Complex Systems Summer School 2000

A Simple, 5x8 Pattern Recognition Neural Network

For my second Java based exploration, I decided to write a very simple neural network. Luckily, I happened across a good book for this (and other) complex systems programming projects: The Nonlinear Workbook by W. Steeb.

The particular net is a fairly simple Hopefiled network, a one layer system designed to store patterns and retrieve one of them given a new pattern that is similar to one of the stored ones.

Java Conversion

The initial program in Steeb was in C++, and deliberately used no graphics or special class capabilities of C++ in order to keep the function of the neural network clear. The conversion to Java was relatively simple. I included two classes beyond the required "main" class: Vec (for the neurons, both initial patterns and test input patterns) and Net which encapsulates the weights array and performs the tests against new patterns.

Here is the source for Hopfield.java

Building the Network

The initial network given in the book consists of only three 5x8 text characters patterns. The test program uses these to build the networks W[][] array.

pats[0]; Energy=1516    pats[1]; Energy=1520    pats[2]; Energy=1484
    *                       ***                    *  * 
   **                      *   *                   *  * 
    *                          *                   *  * 
    *                        **                    *****
    *                       *                         * 
    *                      *                          * 
    *                      *****                      *

Network built, W[40][40]= 0-1-3 1 1 1-1-1 3-1 3 1-1 3-1 3 3-1 1 3 1-1-1 3 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 -1 0 1 1 1 1-1-1-1 3-1 1-1-1 3-1-1-1 1-1 1 3-1-1 1 3 1-1-1 1 3 3 1 1 3 1 1 1 1 1 -3 1 0-1-1-1 1 1-3 1-3-1 1-3 1-3-3 1-1-3-1 1 1-3-1 1-1 1-3-1 1 1 3-1 1-1-1-1-1-1 1 1-1 0-1 3-3-3 1 1 1-1-3 1 1 1 1 1 3 1-1 1-3 1-1 1-1-3 1-1 1 1-1 3 1-1-1-1-1-1 1 1-1-1 0-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 3 1 1-1-1 1 3 3 3 3 3 1 1-1 3-1 0-3-3 1 1 1-1-3 1 1 1 1 1 3 1-1 1-3 1-1 1-1-3 1-1 1 1-1 3 1-1-1-1-1-1 -1-1 1-3 1-3 0 3-1-1-1 1 3-1-1-1-1-1-3-1 1-1 3-1 1-1 1 3-1 1-1-1 1-3-1 1 1 1 1 1 -1-1 1-3 1-3 3 0-1-1-1 1 3-1-1-1-1-1-3-1 1-1 3-1 1-1 1 3-1 1-1-1 1-3-1 1 1 1 1 1 3-1-3 1 1 1-1-1 0-1 3 1-1 3-1 3 3-1 1 3 1-1-1 3 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 -1 3 1 1 1 1-1-1-1 0-1 1-1-1 3-1-1-1 1-1 1 3-1-1 1 3 1-1-1 1 3 3 1 1 3 1 1 1 1 1 3-1-3 1 1 1-1-1 3-1 0 1-1 3-1 3 3-1 1 3 1-1-1 3 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 1 1-1-1 3-1 1 1 1 1 1 0 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 3 1 1-1-1 1 3 3 3 3 3 -1-1 1-3 1-3 3 3-1-1-1 1 0-1-1-1-1-1-3-1 1-1 3-1 1-1 1 3-1 1-1-1 1-3-1 1 1 1 1 1 3-1-3 1 1 1-1-1 3-1 3 1-1 0-1 3 3-1 1 3 1-1-1 3 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 -1 3 1 1 1 1-1-1-1 3-1 1-1-1 0-1-1-1 1-1 1 3-1-1 1 3 1-1-1 1 3 3 1 1 3 1 1 1 1 1 3-1-3 1 1 1-1-1 3-1 3 1-1 3-1 0 3-1 1 3 1-1-1 3 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 3-1-3 1 1 1-1-1 3-1 3 1-1 3-1 3 0-1 1 3 1-1-1 3 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 -1-1 1 1-3 1-1-1-1-1-1-3-1-1-1-1-1 0 1-1-3-1-1-1-3-1-3-1-1-3-1-1 1 1-1-3-3-3-3-3 1 1-1 3-1 3-3-3 1 1 1-1-3 1 1 1 1 1 0 1-1 1-3 1-1 1-1-3 1-1 1 1-1 3 1-1-1-1-1-1 3-1-3 1 1 1-1-1 3-1 3 1-1 3-1 3 3-1 1 0 1-1-1 3 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 0 1 1 1 3 1 3 1 1 3 1 1-1-1 1 3 3 3 3 3 -1 3 1 1 1 1-1-1-1 3-1 1-1-1 3-1-1-1 1-1 1 0-1-1 1 3 1-1-1 1 3 3 1 1 3 1 1 1 1 1 -1-1 1-3 1-3 3 3-1-1-1 1 3-1-1-1-1-1-3-1 1-1 0-1 1-1 1 3-1 1-1-1 1-3-1 1 1 1 1 1 3-1-3 1 1 1-1-1 3-1 3 1-1 3-1 3 3-1 1 3 1-1-1 0 1-1 1-1 3 1-1-1-3 1-1 1 1 1 1 1 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 0 1 3 1 1 3 1 1-1-1 1 3 3 3 3 3 -1 3 1 1 1 1-1-1-1 3-1 1-1-1 3-1-1-1 1-1 1 3-1-1 1 0 1-1-1 1 3 3 1 1 3 1 1 1 1 1 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 0 1 1 3 1 1-1-1 1 3 3 3 3 3 -1-1 1-3 1-3 3 3-1-1-1 1 3-1-1-1-1-1-3-1 1-1 3-1 1-1 1 0-1 1-1-1 1-3-1 1 1 1 1 1 3-1-3 1 1 1-1-1 3-1 3 1-1 3-1 3 3-1 1 3 1-1-1 3 1-1 1-1 0 1-1-1-3 1-1 1 1 1 1 1 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 0 1 1-1-1 1 3 3 3 3 3 -1 3 1 1 1 1-1-1-1 3-1 1-1-1 3-1-1-1 1-1 1 3-1-1 1 3 1-1-1 1 0 3 1 1 3 1 1 1 1 1 -1 3 1 1 1 1-1-1-1 3-1 1-1-1 3-1-1-1 1-1 1 3-1-1 1 3 1-1-1 1 3 0 1 1 3 1 1 1 1 1 -3 1 3-1-1-1 1 1-3 1-3-1 1-3 1-3-3 1-1-3-1 1 1-3-1 1-1 1-3-1 1 1 0-1 1-1-1-1-1-1 1 1-1 3-1 3-3-3 1 1 1-1-3 1 1 1 1 1 3 1-1 1-3 1-1 1-1-3 1-1 1 1-1 0 1-1-1-1-1-1 -1 3 1 1 1 1-1-1-1 3-1 1-1-1 3-1-1-1 1-1 1 3-1-1 1 3 1-1-1 1 3 3 1 1 0 1 1 1 1 1 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 3 1 1-1-1 1 0 3 3 3 3 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 3 1 1-1-1 1 3 0 3 3 3 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 3 1 1-1-1 1 3 3 0 3 3 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 3 1 1-1-1 1 3 3 3 0 3 1 1-1-1 3-1 1 1 1 1 1 3 1 1 1 1 1-3-1 1 3 1 1 1 3 1 3 1 1 3 1 1-1-1 1 3 3 3 3 0

Running The Network

After building the network, it was run on the obvious: some of the input patterns with a few simple "distortions" to see
how well the network functioned. Not unexpectedly, it worked quite well, resolving with only two steps in general.

Processing in[0]
in[0] Step: 0 Energy: 384
 **  
  *  
  *  
   **
  *  
   * 
  *  
*  * 

in[0] Step: 1 Energy: 1516
  *  
 **  
  *  
  *  
  *  
  *  
  *  
     

Processing in[1]
in[1] Step: 0 Energy: 572
 *** 
    *
    *
***  
 *   
*   *
**** 
   * 

in[1] Step: 1 Energy: 1520
 *** 
*   *
    *
  ** 
 *   
*    
*****
     

Processing in[2]
in[2] Step: 0 Energy: 560
** * 
 * * 
*  * 
 ****
   * 
*  **
   * 
*    

in[2] Step: 1 Energy: 1484
*  * 
*  * 
*  * 
*****
   * 
   * 
   *

Well, naturally, the book didn't really challenge the system, so it was time to ad-lib a bit by giving it a few odd inputs.

The first was a set of stripes. Two worked normally, returning one of the stored patterns. One, however, caused the system to fail nicely, returning an "inverted" (white on black) figure four.

Processing in[3]
in[3] Step: 0 Energy: -4
*****
     
*****
     
*****
     
*****
     

in[3] Step: 1 Energy: 1520
 *** 
*   *
    *
  ** 
 *   
*    
*****
     

Processing in[4]
in[4] Step: 0 Energy: 44
* * *
* * *
* * *
* * *
* * *
* * *
* * *
* * *

in[4] Step: 1 Energy: 1484
 ** *
 ** *
 ** *
     
*** *
*** *
*** *
*****

Processing in[5]
in[5] Step: 0 Energy: 44
 * * 
 * * 
 * * 
 * * 
 * * 
 * * 
 * * 
 * * 

in[5] Step: 1 Energy: 1484
*  * 
*  * 
*  * 
*****
   * 
   * 
   * 
     

Processing in[6]
in[6] Step: 0 Energy: -52
 * * 
* * *
 * * 
* * *
 * * 
* * *
 * * 
* * *

in[6] Step: 1 Energy: 940
**** 
*  **
*   *
 *** 
 *   
*  * 
*****
     

in[6] Step: 2 Energy: 1520
 *** 
*   *
    *
  ** 
 *   
*    
*****

Conclusion

After the initial build and tests, I rebuilt the network using more input patterns including more of the text characters. I also tested it with more demanding inputs. Not surprisingly, it started to fail fairly seriously.

The exercise certainly is an interesting one and shows the potential of this style of computing. More sophisticated network certainly would be required for any interesting problem.

As an aside: I spent a few hours browsing around the web and found several very sophisticated implementations of multiple layer networks and other specialized networks for such as the traveling salesman problem (3D, no less) and similar problems.

My next trial will likely be from the "Machine Learning" book of Tom Mitchell, again using Java rather than C or C++.