HTM Networks: Beyond Pyramids
Greg Kochaniak, July 1, 2008
Contact: see Hyperionics Contacts page
I’m trying to go beyond a simple pyramidal structure of an HTM network and create something more complex, branching into different parts at some level. Eventually we will all need something like that for better accuracy of networks, and better performance on parallel computing platforms. Let me explain this graphically.
Here is a cross-section of Numenta’s 4-level Pictures network along X or Y axis:
Simple pyramid
Level
4: 48
categories
51054
coincidences
Accuracy 99.6% clean
73.3%
distorted
Level 3:
About 10,000
coinc/node
Level 2:
1566
coinc/node
Level 1:
173
coinc/node
As you can see above, the numbers of coincidences on each level grow in geometric progression, and so do the numbers of groups (not shown). The new algorithms released with NuPIC 1.6 help a little bit to limit that growth (as did the “prune” parameters in Zeta1 algorithms), but at a cost of accuracy of solution, and ultimately we cannot help the bigger complexity of the models that the higher level nodes build. Let’s say I’m satisfied with the performance and the numbers of coincidences and groups the first two levels provide, but would like to split my network above level 2 into something like this:
Branch A:
Branch B:
Level 4A: 24 categories Level 4B:
24 categories
27818 coincidences 22725
coincidences
Accuracy:
100% clean, Accuracy:
100% clean
79.1%
distorted 77.8%
distorted
Level 3A: Level 3B:
About
6200 coinc/node About
5200 coinc/node
Level 2:
1566 coinc/node
Level 1:
173
coinc/node
Of course, ultimately I would like to have any number of such branches, not only two of them, A and B as shown above. I see this split of nodes on levels 3 and 4 above as an analogy to the brain recruiting (or growing in the neurogenesis process) additional neurons to handle an increased amount of data.
I trained the nodes in Level 1 and Level 2 on all 48 categories of Pictures “clean” set, but here I stopped and trained separately the nodes in levels 3 and 4 of Branch A (blue) on the first 24 categories of the “clean” set, and the nodes in Branch B (red) on the second half of the “clean” set, categories 24 and above. It’s easy to guess that each of these branches tested against its own half of the training and test (distorted) set, achieved better accuracy than the original “pyramidal” network. Incidentally, they had slightly better accuracy than the simple “pyramidal” networks trained from the bottom to top on a half of the training set only: for the 1st 24 categories I got 77.9% accuracy for simple pyramidal network, and 79.1% for my “split” network, for the 2nd half it was 76.9% vs. 77.8%.
So far it looked promising, and the lower numbers of coincidences and groups in each branch should be a performance bonus as well, when executed on a parallel computing platform. But how would I know to which branch the processing should be directed, or from which branch should I read the final category for classifying unknown data?
One idea was to look at the level of “confidence” that the top level classifiers have in their recognition – I peaked at it quickly and decided that this would not tell me much and cause as many or more errors than the simple pyramid. The next idea I have is to look at the level of confidence of the spatial poolers in level 3 nodes. After all, if the final category should be in branch A (lower 24 Pictures categories), level 3 nodes should see something familiar there and be slightly more “certain” of their recognition or match to stored patterns, than nodes in branch B... Maybe I could take a vote of level 3 nodes? Or the normalized sum of their “confidence” in their top categories, and use that to direct the final processing to either branch A or B?
Any other ideas on this? Or on the more generic idea of “recruiting additional nodes” for processing larger amounts of data? If the above idea or its modification work, it would be a good way to extend the network capabilities to additional categories later as well. Say I want to add 20 categories more to the set, and have 68 of them instead of 48. I could keep my branches A and B already trained and intact, and connect a branch C to level 2 nodes, train it only on the new 20 categories...
Greg