The BrainWave neural network simulator is an excellent online resource for students interested in an in-depth understanding of neural network architecture. It was written by Simon Dennis and Devin McAuley, with contributions from Raymond Smith, Janet Wiles and Rachael Gibson as a tool to help undergraduate students taking a neural networking course at the University of Queensland, Australia. This allowed them to produce an excellent, complex system that is nonetheless accessible to undergraduate students without much background in neural networks.
Our tutorial is designed to introduce readers to the BrainWave program in a step-by-step fashion by exploring two of the sample networks. To run BrainWave you will need a Java enabled browser (Netscape 3.0 or higher, or Internet Explorer 3.0 or higher, for example). It is recommended that you keep this tutorial available to you while working with the program, either in another window, or in print form (to open the BrainWave page in a new window, without closing this one, click here).Getting Started
The page that loads contains some information about the Brainwave simulations, as well as the program itself. If you do not see the program upon first loading the page, don't worry, it can take a few minutes to load. In some browsers (including Netscape) the page may even appear to stop loading (your browser may report that it is finished, or the animation indicating that loading is in progress may stop) but even in these cases, the load is still in progress. When the load does finish, you will see a window in the middle of the page with a rather complicated looking tangle of squares and lines connecting them, this is the neural net. Before we can understand this network, we need to discuss the idea of distributed representation.Distributed Representation (and Content Addressability)
One dominant metaphor for human memory is that of a library. That is, memories are stored in some vast database and recall is a process of searching the database for a specific item. Likewise, Symbolic AI stores human knowledge as a list of statements and if-then rules. Knowledge retrieval involves searching entries until the correct one is found. In order to accomodate for the possiblity of large databases, just as a library must, knowledge has to be indexed or sorted (much like the phone book is organised alphabetically so that you need not read 2 million names before finding the one you want). In contrast, connectionism uses a distributed representation scheme, with content-addressable retrieval. In a distributed representation, the same set of neurons can be used to represent multiple concepts or facts. For example, if we have three neurons, A, B, and C, one set of activity levels (e.g. A=0.5, B=1.0, C=-0.176) can represent one idea, while the very same neurons, with different activation levels (e.g. A=0.0, B=-0.8, C=0.45) can represent something totally different. Thus, it is no longer necessary to dedicate a group of neurons for each distinct idea or fact in the database (as in the metaphor of distinct books in a library). Instead, if the connection strengths between neurons are set up properly, many ideas and facts can be represented as distributed activation patterns across the same set neurons. A distributed representation possesses another interesting property referred to as content-addressability. If part of the activation pattern is specified, it is often possible to retrieve the pattern as a whole. In fact, for every concept or fact, numerous partial activation patterns exist which will allow retrieval of this knowledge item. Consequently, there is no need to sort the stored knowledge according to any key or index (as would be required by a library style organisation).Jets and Sharks
and Sharks network is an attempt by James McClelland and David Rumelhart to
show that distributed representation (that is, storing many memories as pattern
of activation over the same set of neurons) and content-addressability (that
is, accessing a given memory from any of its component parts) are not just theoretically
possible in networks, but can actually be built. The jets and sharks network
stores information about two rival gangs, the jets and the sharks. Unlike the
back-Propagation style networks discussed so far in the Neural Nets tutorial,
Jets and Sharks is an Interactive Activation and Competition (IAC) net. IAC
nets do not have discrete input and output nodes, and information does not flow
in one particular direction. Instead, the nodes are labeled with particular
attributes (e.g. Marital Status: Single/Married/Divorced) and interconnected
in such a way that activation flows between certain nodes, and competes with
(or inhibits) others. To understand how the net works, here is the information
we have about the Jets and Sharks (taken from the BrainWave information page):
Let's say we are trying to solve a burglary. Police know the man was in his 20's, and wearing a Jets jacket, but that's all. Now if our memory was stored in library format, sorted say, by name, then we would be lost, our only hope would be to examine every memory until we found what we wanted. If our memory system is content addressable though, any part of the memory (e.g. his age, and gang affiliation) should be enough to activate the other parts. To represent what we know in the simulator, click on the box marked "Burglar". The burglar box should fill itself in red, this is the program's way of indicating its activation, the more activation, the larger the red box. Now do the same with the Jets box, click on it to activate it fully. Now to watch the activation spread we have to cycle the network. A cycle is a step in the activation-spreading process. In each cycle, each node's activation is broadcast to all nodes connected to it, and each node's activation is recalculated based on the inputs it receives. In the bottom left corner of the program screen, there is a button labeled cycle. By default, this performs 30 cycles which is fine for our purposes, click the cycle button and watch what happens. Several nodes we have not touched have become activated! What's more, the nodes that are activated correspond to all of Lance's attributes, his age, marital status, education, etc. are all activated and none of the other attributes are activated at all. Even better - Lance is the only person on the list that fits the description of the suspect. This pattern of activation is what's meant when we talk about distributed representation - these nodes, being activated in the particular way they are represent our memory of Lance. A different activation would represent someone else, or maybe no one at all. To see the effect again, click on "Zero Units". This resets every node to an activation level of 0, returning the net to its initial state. This time, activate the name Sam. Then cycle it twice, that is, click cycle, allow the 30 cycles to complete, then click cycle again. Over the 60 cycles, we see activation spreading to only those nodes connected to Sam - the network knows Sam's education level, marital status and age. Feel free to experiment more with the program and see what emerges, remember that some patterns may take more than one click on "cycle" to appear.
What you may notice is that certain nodes never get co-activated. For instance, no activation pattern will result in both Married and Single becoming strongly activated. In fact, if you intentionally activate Married and Single, they cancel each other out until one or the other wins (try it). That seems clear enough to us, no person can be both Married and Single, but how does the network know? Remember that blue lines represent inhibition, and remember that this network is called an Interactive Activation and Competition network. Those blue lines are there because attributes from the same class compete with one another - they cannot co-exist. In this sense the net is very much like a human as well - if I asked you to name one of your friends or family that was both married and single, you too would have great difficulty.
The BrainWave homepage also contains some thorough documentation of its own written at a more technical level. Students interested in learning more about neural nets are encouraged to read the discussion and self-study topics provided at the bottom of the Brainwave page. These exercises require that you register with the site (click on "Individual Register" from the left hand menu bar), but registration is free and the provided material is of very high quality.