[nupic-discuss] Tests for spatial pooling

Jeff Hawkins jhawkins at numenta.org
Thu Feb 6 16:34:09 EST 2014


Dean.

Your comment made me smile!  This is the first time anyone has said I work
at the 10,000 foot level of neuroscience!  My problem is always the
opposite.

 

The theory and models we work on are by most standards very biological.  For
example, I know of no other cortical theories/models that incorporate any of
the following, active dendrites, differences between proximal and distal
dendrites, synapse growth and decay, potential synapses, dendrite growth,
depolarization as mode of prediction, mini-columns, multiple types of
inhibition and their corresponding inhibitory neurons, etc.  The new
temporal pooling mechanism we are working on requires metabotropic receptors
in the locations they are, and are not, found.  Again, I don't know of any
theories that have been reduced to practice that incorporate any, let alone
all of these concepts.  The machine learning community in general is
ignorant of any of these kinds of details.

 

One of the things I struggle with is how to describe the algorithms without
the neuroscience.  Most people are unfamiliar with the biological terms and
concepts.  But I am very happy to discuss them on this forum.

 

Jeff

 

 

From: nupic [mailto:nupic-bounces at lists.numenta.org] On Behalf Of Dean Horak
Sent: Thursday, February 06, 2014 11:31 AM
To: NuPIC general mailing list.
Subject: Re: [nupic-discuss] Tests for spatial pooling

 

Great thanks. I'll do just that.

 

Dean

 

On Thu, Feb 6, 2014 at 1:28 PM, Matthew Taylor <matt at numenta.org> wrote:

Dean,

 

This mailing list is entirely appropriate for those types of discussions. If
you have a specific topic or question, please feel free to start a new
conversation. 




---------

Matt Taylor

OS Community Flag-Bearer

Numenta

 

On Thu, Feb 6, 2014 at 11:25 AM, Dean Horak <dean.horak at gmail.com> wrote:

This may be a little off topic for this thread, but is there a proper place
for discussions regarding neuroscience and how the CLA does or does not
approach problems the way biology does? I bring this up here because while
Jeff's comments are indeed correct, they are at a 10,000 ft level. It seems
to me there are many insights that might well prove useful within the CLA
that can be derived from discussions of neuroscience at a much deeper level.

 

Thanks,

Dean

 

On Thu, Feb 6, 2014 at 8:30 AM, Christian Cleber Masdeval Braz
<christiancleber at gmail.com> wrote:


 Kevin, i think i understood better the relation between inhibition and
topology. If topology is taking into account, so the position of the input
bits matter. In this case, the sub set of bits each column will "look" are
not random but organized somehow. For an image, the first column will be
connected to the first n bits (or some 2D portion in the beginning of the
image), the second will be connected to the adjacent bits and so on. In this
case, local activity and the job of inhibition makes sense to guarantee
sparsity and distribution.  

 On the other hand, if no topology exist, there is no difference in use
inhibition or not. Will be ok in choosing the columns with the highest
overlap values because each of them will be looking an aleatory sub set of
bits spread all over the input. So, even if happens what you described, the
resulting columns will have a good distribution.



Regards,

Christian

 

 

2014-02-06 Kevin Martin <youcancallmekevin at gmail.com>:

 

Thanks everyone,

 

The term topology is very clear to me now. But in Fergal's explanation, he
had said that 


"In the absence of topology, it's more efficient to just pick the highest
activation potentials globally"

But this means a distributed representation is not guaranteed right? Suppose
we had 1000 columns and 200 input bits and desired sparsity is 1% (10
columns). Each column is potentially connected to say 75% of the input bits.
Let the inhibition radius be 20 and desiredlocalactivity 5. Even though it
is very unlikely, let us assume that 10 columns with the highest overlap
values occur within a particular inhibition radius.

That is, if the highest 11 overlap values are 125,124,122,....115 (for the
sake of simplicity) in the entire layer, and the columns that have these
values are ordered like this :

x x 125 120 x 119 115 x 118 123  Y x  x 121 122 x 116 x 117 x 124

Where x and Y are unknown overlaps. In this case, the inhibition radius of Y
contains all the top 11 overlaps. Without inhibition, the columns with
overlaps 125,124,123,122,121...116 will be the winners to give us the 1%
sparsity. They all occur together. But with inhibition only 5 columns will
be winners here and the other 5 will come from a different part of the
layer. So even though just getting the columns with the highest overlaps is
way more easier and efficient, will it not result in a higher chance of
getting a representation that is not very distributed?

Thanks,

Kevin Martin Jose

 

On Thu, Feb 6, 2014 at 6:33 AM, Fergal Byrne <fergalbyrnedublin at gmail.com>
wrote:

Thanks Jeff, 

 

Perhaps the difficulty arises from the disparity between cortex (which just
is as it is) and a realisation in software. Neurons can only operate
locally, and their computation depends on the topology of the axons passing
by them.

 

In NuPIC, we can design both the topology of the neurons' inputs and also
the inhibition strategy.

 

I think Kevin is asking about the semantics of topology (and
sparseness/distribution) in that sense.

 

Regards,

 

Fergal Byrne

-
Sent from Mailbox <https://www.dropbox.com/mailbox>  for iPhone

 

On Thu, Feb 6, 2014 at 12:52 AM, Jeff Hawkins <jhawkins at numenta.org> wrote:

I hope to clarify what topology is.  I agree with some of what Fergal wrote
but disagree with other parts, so maybe we are thinking different things.

 

In a brain, the cortex is a sheet of tissue. Everywhere you look, it is
about 2mm thick and consists of 4 or 5 layers of cells (depends on how you
count).  The total area of human neocortex is about 1000 cm2. Cortical
regions are 2D areas.  Some of the regions, such as the primary visual or
touch regions are large.  The retina sends about a million axons to the
primarily visual cortex, or V1.  Each axon represents a small part of the
visual space, sort of like a pixel in an image.  When this bundle fibers
reaches V1 the 2D topology of the axons is maintained.  It is as if a 1M
pixel image were projected on to the area of V1.  (The "image" is inverted,
split, and highly distorted, but in all cases any two adjacent areas of V1
receive input from adjacent areas of the retina.)  Scientists would say V1
has a "retinotopic" orientation.

 

Now comes the interesting part.  The cells in V1 learn common patterns
coming from the retina.  The way the SP works is each bit in the SP tries to
learn the largest spatial pattern it can.  And each bit competes with the
other bits.   The number of patterns coming from the retina is so large that
the SP bits are forced to learn only small spatial patterns that are
localized on the retina.  So cells on the right side of V1 learn small
patterns on the left side of the retina (remember the image is inverted).
In biological V1 all the learning rules are local.  Inhibition between
columns is restricted in radius.  The axons that a cell can possibly connect
to are restricted to a radius.

 

When we say a region has topology it means that inhibition between columns
is limited to a local radius and not the entire region.  It means that the
set of potential connections that a cell or column can make are also within
a radius and not all possible input bits or all possible cells in the
region.  This is the way real neocortex works.  When we were applying the
CLA to vision-like problems we implemented it with topology.  It worked as
expected but it is more complicated to test.  The current implementation of
the CLA has no topology.  This means that each column competes with all the
other columns and the set of potential connections between cells is a subset
of all the cells in the region.  There is no ordering of the columns,
nothing is "adjacent" to anything else.  For applications that are small no
topology works great.  If the application had a large 2D sensory array then
topology would likely be required.

 

I hope that helped.

Jeff

 

From: nupic [mailto:nupic-bounces at lists.numenta.org] On Behalf Of Fergal
Byrne
Sent: Wednesday, February 05, 2014 5:54 AM
To: NuPIC general mailing list.
Subject: Re: [nupic-discuss] Tests for spatial pooling

 

HI Kevin,

 

>> The following ideas are not things I think relate to "topology".

 

Topology is the jargon we use to describe the semantic meaning (or lack of
it) of a column (or input bit) position, and for connections between them. 

For example, a typical encoding of category data randomly picks 21 bits on
out of 128. Each bit position has no real connection with the input category
it encodes. This representation has no topology at all.

Conversely, the sliding window of on-bits in a scalar encoding has
one-dimensional topology. In this case, bit 0 might mean the input is < 10,
while bit 23 might mean an input between 11 and 33.  

 

>> These ideas are things I think relate to "topology".

 

An example of a 2d topology in an SDR is a retina-like sensor. In this case
the light levels or presence of particular features in a certain spatial
position would be represented by bits in the corresponding position.

Topology in the SP relates to how each column position is connected with its
input bits. An SP which is learning to recognise features in an SDR coming
in from a retina would have columns "looking at" a set of input bits
directly "below it" in the retina. The V1 primary visual cortex has exactly
this property.

If you have topology, you'll usually use local inhibition to pick winning
columns, where local is defined according to the topology (1-d, 2-d
distance). In the absence of topology, it's more efficient to just pick the
highest activation potentials globally. 

 

Regards

 

Fergal Byrne

 

 

 

On Wed, Feb 5, 2014 at 11:25 AM, Kevin Martin <youcancallmekevin at gmail.com>
wrote:

Hi all,

Thanks for the help. But I'm not sure if I understand what you mean by
topology. As far as I understand, the spatial pooler has 2 functions :
Return a set of columns that are 1)sparse and 2)distributed.

Sparsity can be maintained by making sure that the pooler returns only n% of
columns and this criterion can be made into a test. And distribution can be
tested by computing the average distance between winning columns. What is
topology? Is it another way of referring to a distributed set of columns? I
might need some help with local learning too. I'm afraid I have never heard
the term before (its in the white paper? A quick search on the pdf did not
see any hits for "local learning")

Thanks,

Kevin Martin

 

On Wed, Feb 5, 2014 at 7:37 AM, Scott Purdy <scott at numenta.org> wrote:

I would go over the properties of the SP and create tests accordingly. The
output, for instance, should be tolerant to spatial noise. So show an
instance a set of very different spatial patterns over and over a number of
times. Then show one of the patterns with a single bit flipped and make sure
the output is very close (possibly identical if columns are saturated) to
the same.

There are a bunch of similar things you could check.

Ian had a hackathon demo showing the connected bits for each column. He was
able to validate different aspects of the SP visually by using different
numbers of columns and % overlap.

You could do something similar to make sure you see the right behavior but I
don't know if you could make an automated test out of it.

http://numenta.org/blog/2013/11/06/2013-fall-hackathon-outcome.html#sp_viewe
r

On Feb 4, 2014 11:23 AM, "Jeff Hawkins" <jhawkins at numenta.org> wrote:

Kevin,

Adding on to what Fergal said..

 

Here is what we did.  We started with a fixed number of columns and no
topology.  In this situation we picked the n% most innervated columns as the
output of the SP.  Thus we were guaranteed to get the correct sparsity of
active columns.  With this setup we did a lot of other tests to verify that
the SP was forming good representations.

 

The tricky part is if you want to do this with topology and local learning
rules.   Is that your concern?

 

With topology we didn't try to guarantee the entire layer of cells will have
exactly n% active columns.  Exact numbers are not essential.  We tried
several different local learning rules, all of them produced about the right
level of sparsity.  We than ran the other tests to make sure that local
learning rules didn't cause the SP to form poor representations.  You won't
get exactly the same results with topology and without topology but my
recollection was we were able to come up with a set of local inhibition
rules that produced good results.

Jeff

 

From: nupic [mailto:nupic-bounces at lists.numenta.org] On Behalf Of Fergal
Byrne
Sent: Tuesday, February 04, 2014 10:43 AM
To: NuPIC general mailing list.
Subject: Re: [nupic-discuss] Tests for spatial pooling

 

Hi Kevin,

 

If you're using no topology (and no local inhibition) then the fraction of
active columns (2% in NuPIC) is the only important factor. With topology,
you should choose winners relative to their neighbours. In either case, the
first test is to count the fraction of active columns. In the second case,
measure the average distance between active columns.

 

Regards,

 

Fergal Byrne

 

On Tue, Feb 4, 2014 at 6:05 PM, Kevin Martin <youcancallmekevin at gmail.com>
wrote:

Hi,

I'm writing my own version of the CLA. I have not reached anywhere
significant and I have been playing with a very small number of columns
until now. I am starting by implementing a spatial pooler and spatial
pooling requires a large number of columns. So far it has been easy to
manually check if the synapses are distributed throughout the input, if the
winning columns are really sparse etc. But it is going to be impossible
when, say,I use 1000 columns. Have somebody used a test to check if the
spatial pooler outputs a sparse distribution? Any suggestions on how to
write a test?

Thanks,

Kevin Martin Jose


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org





 

-- 


Fergal Byrne, Brenter IT

 

http://inbits.com - Better Living through Thoughtful Technology

 

e:fergalbyrnedublin at gmail.com t:+353 83 4214179 <tel:%2B353%2083%204214179> 

Formerly of Adnet editor at adnet.ie http://www.adnet.ie <http://www.adnet.ie/>



_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org

 


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org





 

-- 


Fergal Byrne, Brenter IT

 

http://inbits.com - Better Living through Thoughtful Technology

 

e:fergalbyrnedublin at gmail.com t:+353 83 4214179 <tel:%2B353%2083%204214179> 

Formerly of Adnet editor at adnet.ie http://www.adnet.ie <http://www.adnet.ie/>


 


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org

 


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org

 


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org

 


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org

 


_______________________________________________
nupic mailing list
nupic at lists.numenta.org
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.numenta.org/pipermail/nupic_lists.numenta.org/attachments/20140206/ec857bb4/attachment-0001.html>


More information about the nupic mailing list