Avalanche Pattern
Recognition in Neural Networks
Richard
Keene
Phone: 801-961-3668
2009 Tommy Moe Place, Park City UTAH 84098
rkeene@xmission.com
(2014 now rmkeene@gmail.com)
Keene.Dick@amstr.com
http://www.xmission.com/~rkeene
Abstract
The results of
research into avalanche pattern recognition, neural fatigue, and
successive pattern sequencing are presented. Given an array of
neurons that are interconnected such that each neuron is trying to
predict the next state of all the other neurons, and stimulate the
other neurons to the predicted state, successive patterns can be
taught to the system. Then a partial stimulation of the pattern
will cause an avalanche effect into the full pattern. If the
neurons have a fatigue limit, the current pattern can be made to
fade, and an avalanche into another pattern may occur.
Motivation for Research - The Subsumptive Regular Architecture
The major building block for this theory is the cortex layer that avalanches or cascades into a pattern match. This paper is the result of test programs that implemented various algorithms for avalanche pattern recognition.
1.The Avalanche Process
The usual use of neural networks is to arrive at a degree of confidence that the inputs are of a given pattern. The classic example of differentiating between Little Red Riding Hood and the Wolf uses cape color, ear size, tooth size, and eye size to arrive at a confidence level of which choice it is. This is done by presenting input pattern (can be partial) and, in one cycle, getting an output that is the degree of confidence.If we now take such a neural network and feed the output back around into the inputs such that a match of Wolf will increase the inputs [ears, teeth, eyes] and decrease the input [red cape], and a match of Red Riding Hood will do the opposite, then successive cycles should avalanche into the original teaching pattern.
The pattern may also be partial. If only the ears input is slightly turned on, then the feedback should cause an avalanche into the full Wolf pattern.
Once on the state the system is now ‘locked up’ in that state. In a more general and complex system, if one desires to have the neural system progress to successive states (a requirement for the SRS) then there must be a method for getting out of the state. This is where neural fatigue comes into play. After a neuron has been firing for a certain time it becomes fatigued and turns off. This allows an new avalanche into another state.
To get time sequenced states there must be some embodiment in the system of the current state and the past state.
2.First Experiment - Three Pairs of Neurons
The first experiment used three neurons that were stimulated in one of three patterns. The patterns were 100, 010, 001. All neurons had teachable connections to all other neurons and there selves. The idea being that a neuron would feed-back on it’s self and progress to a full-on state. This caused balance problems where all neurons would progress to a mid level state to where the self-stimulus was balanced by the inhibition from the other two neurons.The next generation of the program uses three pairs of neurons called A, a, B, b, C, c where all neurons have teachable connections to all other neurons but not to self. This works very well. The system is first taught with the three patterns [110000, 001100, 000011] until the patterns are well established. Then the system is set to random neuron activity levels and cycled. Which ever neuron pair has the highest activity will dominate the others and the system will usually avalanche into the corresponding pattern. Occasionally the system will get into an intermediate and balanced state where no pattern dominates.
2.1The Teachable Connections
The weighted and teachable connections were tested in two different styles. There is a connectivity factor used in both styles that determines how strongly all neurons are connected.2.1.1Level Seeking Connections
The first style is not the normal neural network style connection where the activity level (charge) of the neuron is added to the target neuron. Instead the connection has a expected value and a strength (confidence level). This allows the connection to force the target toward a given state.For example if a connection has a expected value of 0.1 and a strength of .99 and the current neuron charge is 0.9 then the target neuron will be pushed toward 0.1 by .99 * .9 * connection_factor plus any external stimulus.
This approach results in the fastest avalanches. If the expected use of the system is to be taught a series of patterns, and then freeze the connection factors (turn learning off), and then match patterns, this is the best connection method. If learning is constant and always turned on, then this approach ‘softens’ the learned patterns fairly rapidly, due to the connections trying to learn the in-between patterns during an avalanche event.
2.1.2Classical Connections
The second style is a more classical learning connection where the charge of the neuron times the connection weight is added to the target neuron. Neurons are randomly excitory or inhibitory when first created. This approach results in any given pattern having some excitory and some inhibitory neurons as members. The neurons can then force the entire neuron array into the pattern by exciting the ON neurons of the pattern and inhibiting the OFF neurons of the pattern.Avalanches works well with this approach if there are a statistically significant number of neurons in all patterns. It also preserves learning during avalanches. In fact avalanches tend to reinforce the already-learned patterns. With this approach connection weights tend to be either almost 0 or almost 1. Intermediate weights represent learning-in-progress. Such a system should represent intermediate input and output levels with multiple neurons.
2.2Conservation of Charge
If a given network has a total energy which is the sum of all the charges of the neurons, and there is no external stimulus, then total charge should increase on successive cycles. Fast increases represent fast avalanches. Too fast of an increase tends to dominate teaching input so much that the learned patterns tend to be fuzzy.With the level seeking connections, if charge is not conserved then the system rapidly becomes inactive. If charge is increasing then the system rapidly runs away until all charges are at maximum. The actual algorithm used clips the neuron charge to a range between 0 and 1 inclusive after each cycle. This limits run-away and limits the total system energy. The charge of a neuron ‘leaks’ toward 0 on each cycle by about 1%. This allows for 99% increase on total system charge each cycle. The balance between the clipping drawing off charge and the inter-neuron stimulus generating charge allows the avalanching state to dominate the system energy pattern. This makes the system continue changing states while not overloading. In fact, the actual leak rates and inter-neuron stimulus factors (called the connectivity factor in the code) can vary considerably and the system balance is not effected.
The classical connections also must conserve charge. The neurons have a relaxation constant such that a charged neuron, with no stimulus will decay to 0 charge. The balance between the relaxation constant, the connectivity factor, and the ratio of inhibitory to excitory neurons determines the avalanche rate, and whether the system will even avalanche. The levels where found by trial and error.
The connectivity factor very strongly effects the avalanche rate by being an overall throttle on system energy. To low of a connectivity factor results in slow avalanches and more often the system can not reach any particular state. Too high of a factor causes very fast avalanches but then the external stimulus is dominated by the avalanches. Still the working range is fairly broad.
2.3Neuron Fatigue
Once the system is avalanched into a given state, it is stuck there. (Much like an electronic flip-flop.) After some number of cycles the fatigue level of the neuron reaches a certain level and the neuron state changes to fatigued. The neuron then stops stimulating its targets for some number of cycles while it recovers from the fatigue. The neuron charge leaks toward zero. While this is happening, other neurons can begin to avalanche to some other state.The number of cycles it takes for a neuron to fatigue must be more than the avalanche time or the full state will not usually be reached. If it is much more than the avalanche time then the system simply reaches successive states more slowly and may miss time-dependent factors. The fatigue recovery time must be somewhat longer than the avalanche time.
A problem encountered in the test program was where one neuron in a group matching a pattern would start fatigue recovery before another neuron reached a saturated avalanched state. This caused the system to oscillate on a single state. The solution is to have a delay of a few cycles between setting the fatigued state and beginning recovery to allow all the neurons in a pattern to become fatigued.
See the source code
for the actual implementation of the algorithms.
2.4Test Results
The test program showed that avalanching does occur most of the time. A typical avalanche time was 15 to 19 cycles. The fatigue time was set to 25 cycles, the recovery delay was 5 cycles, with a recovery of about 20 cycles. (Implemented as when the sum of Charge over time reached 25 with a slight relaxation factor.)If an avalanche did not occur then the fatigue algorithm would eventually upset the balance and the system would avalanche to some other state.
The system would continually avalanche to various states. The test program had no time-dependent components.
3.Test Program - 5x5 Grid of Neurons with Graphic Output
The third test program added in a graphical view of the neuron activity level and made the grid of neurons any size. The tests were done on a 5 by 5 grid of neurons that are all connected to each other, but not self connected.Pattern sets that are non-overlapping result in connection expected values that are exact and strong. This results in very clean avalanches.
Patterns that share common neurons, such as a rotating bar on the center ( patterns that look like the ‘twirly wait bar’ which is a sequence that looks like | / - \ and the center pixel is shared by all patterns, see appendix B) avalanche very well. All patterns have the same total energy level.
Patterns that share some neurons but have different total number of neurons in the pattern generally avalanche into the patterns with the most active neurons. (Highest total energy). One can lightly (50%) set the low energy patterns and they will still avalanche.
3.1Other Algorithms Tested
Several other algorithms were tested.Test were done on the firing algorithm. For the level seeking style connections: The first algorithm used added some percent of the difference between the expected target charge and the actual target charge such that the target charge would avalanche asymptotically toward the expected value. This resulted in slow avalanches. The algorithm was changed to do a linear seek toward the expected value. This resulted in very fast avalanches. Also with constant learning turned on, linear seeking resulted in less forgetting.
The learning of the confidence value (strength) for a level seeking style connection has been done in several ways, none of which are really satisfactory. This needs work. One interesting algorithm was to lower the learning rate for connections with a high confidence. This attempted to compensate for the forgetting that happens when learning is always turned on. It actually had the exact same effect as a lower learning rate would have.
A test was done where instead of weighted connections the connections kept a running sum-of-charge and sum-of-charge-squared and then calculated the weighted mean and standard deviation. This became the expected value and the deviation was used for the confidence level. This did not work well for connections to bi-modal neurons.
4.The Program Code
The test programs are written in Java. They are available at http://www.xmission.com/~rkeene.5.Conclusions and Future Research
This series of tests has shown that interconnected arrays of neurons with fatigue algorithms can avalanche into successive states of learns patterns.There is still the difficulty of arriving at some algorithm for turning learning on and off.
The next step is to add groups (also called maps) of neurons that act together to represent some external state, and let the groups interact with each other. If some of the maps represent past states, while others represent current states, the system should be able to repeat patterns series in the order they were presented.
In large system all neurons would not be connected to all other neurons. Instead each neuron would connect to N other randomly selected neurons where N is the fanout.
6.References
- Keene, Richard 1995, A New Model for the Cognitive Process
- Artificial Cognition, International IEEE Symposium on
Intelligence in Neural and Biological Systems, IEEE Press
- Haykins, Simon 1993.
Neural Networks, A Comprehensive Foundation, IEEE Press
Appendix A - Output From the Three-Pair Test
Here is the three pair system after it has learned the three
patterns, has been set to random charge values and is about to
avalanche. Neurons 2 and 3 will dominate because that pair has the
highest total energy.
Neuron 0: Id A: C
0.111791: Fanout 5: Fatigue 2.25323: State 0
Target 0(a): Expected 0.98659: Strength 1
Target 1(B): Expected 0.00476323: Strength 1
Target 2(b): Expected 0.00665825: Strength 1
Target 3(C): Expected 0.0172547: Strength 1
Target 4(c): Expected 0.0178598: Strength 1
Neuron 1:
Id a: C 0.0739657: Fanout 5: Fatigue 1.73814: State 0
Target 0(A): Expected 0.987056: Strength 1
Target 1(B): Expected 0.0126316: Strength 1
Target 2(b): Expected 0.014937: Strength 1
Target 3(C): Expected 0.00880566: Strength 1
Target 4(c): Expected 0.00888966: Strength 1
Neuron 2:
Id B: C 0.133838: Fanout 5: Fatigue 3.72987: State 0
Target 0(A): Expected 0.00983726: Strength 1
Target 1(a): Expected 0.0158047: Strength 1
Target 2(b): Expected 0.982846: Strength 1
Target 3(C): Expected 0.0147772: Strength 1
Target 4(c): Expected 0.0139572: Strength 1
Neuron 3:
Id b: C 0.218574: Fanout 5: Fatigue 3.18379: State 0
Target 0(A): Expected 0.0257993: Strength 1
Target 1(a): Expected 0.0133968: Strength 1
Target 2(B): Expected 0.983183: Strength 1
Target 3(C): Expected 0.0133469: Strength 1
Target 4(c): Expected 0.0111659: Strength 1
Neuron 4:
Id C: C 0.0244048: Fanout 5: Fatigue 4.25388: State 0
Target 0(A): Expected 0.0160625: Strength 1
Target 1(a): Expected 0.00663962: Strength 1
Target 2(B): Expected 0.0146463: Strength 1
Target 3(b): Expected 0.0163282: Strength 1
Target 4(c): Expected 0.987261: Strength 1
Neuron 5:
Id c: C 0.0540738: Fanout 5: Fatigue 2.00599: State 0
Target 0(A): Expected 0.00808988: Strength 1
Target 1(a): Expected 0.0166216: Strength 1
Target 2(B): Expected 0.00707348: Strength 1
Target 3(b): Expected 0.00900241: Strength 1
Target 4(C): Expected 0.989449: Strength 1
Here is the same
system part way into the avalanche.
Neuron 0: Id A: C
0.0185807: Fanout 5: Fatigue 1.84688: State 0
Target 0(a): Expected 0.98659: Strength 1
Target 1(B): Expected 0.00476323: Strength 1
Target 2(b): Expected 0.00665825: Strength 1
Target 3(C): Expected 0.0172547: Strength 1
Target 4(c): Expected 0.0178598: Strength 1
Neuron 1:
Id a: C 0.0147144: Fanout 5: Fatigue 1.23754: State 0
Target 0(A): Expected 0.987056: Strength 1
Target 1(B): Expected 0.0126316: Strength 1
Target 2(b): Expected 0.014937: Strength 1
Target 3(C): Expected 0.00880566: Strength 1
Target 4(c): Expected 0.00888966: Strength 1
Neuron 2:
Id B: C 0.684502: Fanout 5: Fatigue 7.66481: State 0
Target 0(A): Expected 0.0103409: Strength 1
Target 1(a): Expected 0.0161878: Strength 1
Target 2(b): Expected 0.974916: Strength 0.99733
Target 3(C): Expected 0.0144067: Strength 1
Target 4(c): Expected 0.0143724: Strength 1
Neuron 3:
Id b: C 0.693296: Fanout 5: Fatigue 7.48861: State 0
Target 0(A): Expected 0.0249297: Strength 1
Target 1(a): Expected 0.0152: Strength 1
Target 2(B): Expected 0.946959: Strength 0.985975
Target 3(C): Expected 0.0153209: Strength 1
Target 4(c): Expected 0.0130931: Strength 1
Neuron 4:
Id C: C 0.0180507: Fanout 5: Fatigue 3.79868: State 0
Target 0(A): Expected 0.0160625: Strength 1
Target 1(a): Expected 0.00663962: Strength 1
Target 2(B): Expected 0.0146463: Strength 1
Target 3(b): Expected 0.0163282: Strength 1
Target 4(c): Expected 0.987261: Strength 1
Neuron 5:
Id c: C 0.0161788: Fanout 5: Fatigue 1.52298: State 0
Target 0(A): Expected 0.00808988: Strength 1
Target 1(a): Expected 0.0166216: Strength 1
Target 2(B): Expected 0.00707348: Strength 1
Target 3(b): Expected 0.00900241: Strength 1
Target 4(C): Expected 0.989449: Strength 1
Here is the same
system now fully in the pattern state. Fatigue is building up.
Neuron 0: Id A: C
0.0185807: Fanout 5: Fatigue 1.65732: State 0
Target 0(a): Expected 0.98659: Strength 1
Target 1(B): Expected 0.00476323: Strength 1
Target 2(b): Expected 0.00665825: Strength 1
Target 3(C): Expected 0.0172547: Strength 1
Target 4(c): Expected 0.0178598: Strength 1
Neuron 1:
Id a: C 0.0147144: Fanout 5: Fatigue 1.02516: State 0
Target 0(A): Expected 0.987056: Strength 1
Target 1(B): Expected 0.0126316: Strength 1
Target 2(b): Expected 0.014937: Strength 1
Target 3(C): Expected 0.00880566: Strength 1
Target 4(c): Expected 0.00888966: Strength 1
Neuron 2:
Id B: C 0.93139: Fanout 5: Fatigue 12.7788: State 0
Target 0(A): Expected 0.0103409: Strength 1
Target 1(a): Expected 0.0161878: Strength 1
Target 2(b): Expected 0.974916: Strength 0.99733
Target 3(C): Expected 0.0144067: Strength 1
Target 4(c): Expected 0.0143724: Strength 1
Neuron 3:
Id b: C 0.959067: Fanout 5: Fatigue 12.7305: State 0
Target 0(A): Expected 0.0249297: Strength 1
Target 1(a): Expected 0.0152: Strength 1
Target 2(B): Expected 0.946959: Strength 0.985975
Target 3(C): Expected 0.0153209: Strength 1
Target 4(c): Expected 0.0130931: Strength 1
Neuron 4:
Id C: C 0.0180507: Fanout 5: Fatigue 3.60477: State 0
Target 0(A): Expected 0.0160625: Strength 1
Target 1(a): Expected 0.00663962: Strength 1
Target 2(B): Expected 0.0146463: Strength 1
Target 3(b): Expected 0.0163282: Strength 1
Target 4(c): Expected 0.987261: Strength 1
Neuron 5:
Id c: C 0.0161788: Fanout 5: Fatigue 1.31921: State 0
Target 0(A): Expected 0.00808988: Strength 1
Target 1(a): Expected 0.0166216: Strength 1
Target 2(B): Expected 0.00707348: Strength 1
Target 3(b): Expected 0.00900241: Strength 1
Target 4(C): Expected 0.989449: Strength 1
Appendix B - The 5x5 System With Rotating Bar
Pattern
These three pictures
are a random charges, than a cascade into a pattern, then the full
pattern with fatigue just beginning to set in.
This is after an avalanche to another pattern.
No comments:
Post a Comment