Testing of information condensation in a model reverberating spiking neural network

0.24 MB·English

by  Alexander K. Vidybida

1
1
0 Testing of information condensation in a model
2
reverberating spiking neural network1
n
a
J

0
1
A. K. Vidybida

]
Department of Synergetics, Bogolyubov Institute for Theoretical Physics
C
Metrologichna str., 14-B, 03680 Kyiv, Ukraine
N
E-mail: [e-mail]
.
o
i
b
-
Abstract
q
[ Information about external world is delivered to thebrain in theform of structured in time spike trains.

2 During further processing in higher areas, information is subjected to a certain condensation process, which
v resultsinformationofabstractconceptualimagesofexternalworld,apparently,representedascertainuniform
0 spiking activity partially independent on the input spike trains details. Possible physical mechanism of
9 condensation at the level of individual neuron was discussed recently. In a reverberating spiking neural
4 network, due to this mechanism the dynamics should settle down to the same uniform/periodic activity in
0 response to a set of various inputs. Since thesame periodic activity may correspond to different input spike
.
2 trains, we interpret this as possible candidate for information condensation mechanism in a network. Our
1 purpose is to test this possibility in a network model consisting of five fully connected neurons, particularly,
0 theinfluenceofgeometricsizeofthenetwork,onitsabilitytocondenseinformation. Dynamicsof20spiking
1 neuralnetworksofdifferentgeometricsizesaremodelledbymeansofcomputersimulation. Eachnetworkwas
: propelled into reverberating dynamics by applying various initial input spike trains. We run the dynamics
v
i until it becomes periodic. The Shannon’s formula is used to calculate the amount of information in any
X input spike train and in any periodic state found. As a result, we obtain explicit estimate of the degree of
r information condensation inthenetworks,andconcludethatitdependsstronglyonthenet’sgeometricsize.
a
1 Introduction
bestsituationiswhenthebrainreceivesmaximum
information from sensory systems. But compare
thiswith [10]. Anotherparadigm, which is aswell
Theabilityofabiological object toobtainenough
mature, [23], concentrates on self-organization of
information about external world is essential for
spiketrainswhenprimarysensoryactivityspreads
the object’s survival. The information is deliv-
to higher brain areas. Self-organization is ac-
ered to the central nervous system through vari-
companied with information loss, [15], Indeed, a
ous sensory pathways in the form of spike trains.
kind of standartization of activity evoked by var-
The pathways’ information throughput has been
ious primary sensory inputs is obsereved exper-
studyed for a long time in theoretical [22] and ex-
imentally in higher brain areas during olfactory
perimental [25] research. The paradigm of those
[6, 38] and auditory [4] perception. In visual sys-
researchisconsistentwiththeassumptionthatthe
1Accepted intheInternational JournalofNeuralSystems,http://www.worldscinet.com/ijns/ijns.shtml
2 A. Vidybida
tem, simple examples are the transformation of ingdynamicsisallowed togofreely untilit settles
scene representation to viewpoint-invariant, [7] or down to a periodic one. This allows to figure out
retinotopic-invariant [32] coordinates. Here, spike sets of input stimuli bringing about the same pe-
trains in the optic nerve must depend on certain riodic dynamics. The numberof different periodic
information (retinal position, viewpoint), which dynamicsandthenumberofinputstimuliinaset,
is removed in higher areas of conceptual repre- whichcorrespondstoadefiniteperiodicdynamics,
sentation. In the context of cognitive physiol- characterize thenet’s ability tocondense informa-
ogy,theprocessofreductionofinformation aimed tion. We found that this ability depends strongly
at conceptual representation/recognition of exter- on thenet’s size.
nal objects is known as information condensation,
[21]. Usually pattern recognition phenomenon,
2 Methods
which is closely related to information condensa-
tion, is considered in parallel with training, see
[12,17,24,28,30,31,39],learning, [1,11,13,21], 2.1 The Binding Neuron Model
or other plasticity, [16], in the corresponding net-
work. Inabiological network,thelearningmecha- The understanding of mechanisms of higher brain
nisminvolvesbiosynthesis[27]andisthereforevery functions expects a continuous reduction from
slow process, which requires seconds or minutes, higher activities to lower ones, eventually, to ac-
[3]. Atthesametime,recognition ofobjects invi- tivities in individual neurons, expressed in terms
sual scene can be accomplished within 150 ms, or of membrane potentials and ionic currents. While
faster, citeThorpe1,Thorpe. During this short pe- thisapproachiscorrectscientifically anddesirable
riod of time, the network has constant structure, for applications, the complete range of the reduc-
but it is the spiking activity which evolves within tion is unavailable to a single researcher/engineer
it from information-rich at the sensory periphery due to human brain limited capacity. In this con-
into information-poor, representing concrete enti- nection, it would be helpful to abstract from the
ties/concepts at higher brain areas. rulesbywhichaneuronchangesitsmembranepo-
tentials to rules by which the input impulse sig-
We now put a question: What could be the
nals are processed into its output impulses. The
physicalmechanismofinformationcondensationin
coincidence detector, and temporal integrator are
aspikingneuralsystem? Onepossiblemechanism,
the examples of such an abstraction, see discus-
[37] which operates at the level of single neuron,
sion by K¨onig et al., [20]. One more abstraction,
was discussed, see Fig. 1, 2 and n. 2.1.12.1.1, be-
thebindingneuron(BN)model,isproposedassig-
low. In a reverberating spiking network, due to
nal processing unit, [34] which can operate as ei-
this mechanism the spiking dynamics should set-
ther coincidence detector, or temporal integrator,
tle down to the same definite periodic activity in
depending on quantitative characteristics of stim-
responsetoanystimulusfromadefinitesetofvar-
ulation applied. This conforms with behavior of
iousinputs,andtoanotherperiodicactivityinre-
real neurons, see, e.g. work by Rudolph & Des-
sponse to members of another set of inputs. If so,
texhe, [26]. The BN model describes functioning
then the definite periodic dynamical state can be
of a neuron in terms of discret events, which are
considered as an abstract representation of a fea-
input and output impulses, and degree of tempo-
ture,whichallstimulifromthedefinitesethavein
ral coherence between the input events, see Fig.
common. And this is just what is expected from
1. Mathematically, this can be realized as follows.
thecondensation of information.
Each inputimpulseis stored in theBN for a fixed
Our purpose in this work is to study how the time, τ. The τ is similar to the tolerance inter-
ability of a simple spiking neural net to condense val discussed by MacKay, [23]. All input lines are
information in the above described sense depends excitatoryanddeliveridenticalimpulses. Theneu-
on its physical parameter — the net’s geomet- ronfiresanoutputimpulseifthenumberofstored
ric size. For this purpose we simulate dynamics impulses, Σ, is equal or higher than the threshold
of a net composed of five binding neurons placed value,N0. Inthismodel,inhibitionisexpressedin
equidistantly on a circle. The circle’s radius, R, decreased τ value. Itis clear, that BNis triggered
characterizes the net’s geometric size. The net is when a bunch of input impulses is received in a
fully connected, and propagation velocity is taken narrow temporal interval. In this case, the bunch
the same for all connections and all values of R. could be considered as compound event, and the
Thus,thevariationsinRareexpressedexclusively outputimpulse—asanabstractrepresentationof
in the variations in the interneuronal propagation this compound event. One could treat this mech-
times. Initially, the net is stimulated by a spike anism as binding of individual input events into
train of inputimpulses triggering each of fiveneu- a single output event, provided the input events
ronsattimes{t0,t1,...,t4}. Afterwards,thespik- are coherent in time. Such interpretation is sug-
Testing of Information Condensation 3
elementary
-
event
Binding of the
elementary - elementary elementary event
event events based for secondary neurons (cid:26)(cid:26)>
. - (cid:26) -
... on their (representstheboundevent) HHHj
.. temporal
coherence
elementary
-
event
{
inhibition controls binding
Figure 1: Signal processing in the binding neuron model.[35, 36]
gested by binding of features/events in largescale thatthetriggering condition issatisfied. Inaneu-
neuronal circuits, [5, 8, 9, 36]. ron, which needs more than one input impulse to
It would be interesting to characterize the BN fire,variationsoftemporalpositionofimpulses,re-
input-outputrelationsintheformoftransferfunc- ceived just before the triggering one, do not influ-
tion, which allows exact calculation of output in ence the moment of emitting the output impulse,
terms of input. In our case, input is the se- provided those variations are in resonable limits
quence of discrete arriving moments of standard and arrival moment of the triggering impulse re-
impulses: Tin = {l1,l2,l3,l4,...}. The output is mains the same.
the sequence of discrete firing moments of BN: τ
Tout={f1,f2,...}.ItisclearthatTout⊂Tin.The
transferfunction inourcasecould bethefunction
σ(l), l ∈ Tin, which equals 1 if l is the firing mo- t t t t t
25 24 23 22 21
ment, l ∈ Tout, and 0 otherwise. For BN with
threshold N0 the required function can be con-
structedasfollows. ItisclearthatthefirstN0−1
inputimpulsesareunabletotriggerneuron,there-
t t t t t
fore σ(l1) = 0,...,σ(lN0−1) = 0. The next input 15 14 13 12 11
is able to trigger if and only if all N0 inputs are
coherent in time:
Figure 2: Example of two different inputs into
σ(lN0)=1 if and only if lN0−l1 ≤τ. a single neuron, which produce identical out-
puts.
In order to determine σ(lN0+k), k ≥ 1, one must Thus, different input spike trains can produce ex-
takeintoacountallpreviousinputmoments,there-
actly the same output. This looks like if some de-
foreweusenotation σTin instead ofσ. Thevalues tailes of the input stimulus, which is composed of
of σTin(lN0+k) can bedetermined recursively: several impulses, were reduced/condensed in the
output. In the BN, the triggering condition is
σTin(lN0+k)=1 if and only if lN0+k−lk+1≤τ and
that the number of impulses in the BN’s inter-
σTin(li)=0 for all i∈{k+1,...,N0+k−1}. nal memory equals to N0. Consider BN with
TmhoedeflufnocrtaiornbitσrTairny tdhersecsrhiboelds vcaolmueplNet0el≥y 2th.e BN sNp0ike=tr4a,inwshwiicthhiinspsuttimimulpautelsdeswairtrhivatwltoimdeisffeSr1en=t
{t11,t12,t13,t14,t15}, t11 < t12 < t13 < t14 < t15,
2.1.1 Information Condensation in a and S2 = {t21,t22,t23,t24,t25}, t21 < t22 < t23 <
t24 <t25. Let the arrival moments satisfy the fol-
Single Neuron
lowing conditions:
It is worth noticing, that any firing (triggering)
t14−t11 <τ, t25−t22 <τ,
moment of a spiking neuron is determined by the
moment of last input impulse, which just ensures t24−t21 >τ, t14 =t25 =to.
4 A. Vidybida
Inthiscase, bothS1 andS2,iffed totheBN,will 2.3 Data Acquisition Algorithm
produceexactly thesame output,namely,thesin-
2.3.1 Set of Stimuli
gle impulse at moment to. This is illustrated in
Fig. 2.
The net was entrained to reverberating dynamics
by applying initial input spike train of five im-
2.2 The Network Model pulses,onetriggeringimpulseperneuron,attimes
(in dt units) {t0 = 1,t1,t2,t3,t4}. The triggering
As a reverberating spiking neural net we take the
moment of neuron # 0 is taken 1 for all stimuli
net of fiveneurons placed equidistantly at a circle
in order to exclude rotational symmetry between
ofradiusR,seeFig,3. Eachneuronhasthreshold
thestimuliapplied. Otherfourtriggeringmoments
N0 = 4, and internal memory, τ = 10 ms. The runindependentlythroughtheset{1,2,...,tmax},
net is fully connected. The connection lines are
where tmax is choosen proportional to R for each
characterized with length and propagation veloc-
net size. In choosing tmax, we follow two different
ity, v, which is the same in all lines. For R fixed,
paradigms. In the first paradigm of short stim-
there are two types of connection line, the short
uli we restrict the overall duration of the stim-
one, with propagation delay d, and the long one
ulus train with the value tmax = d. Thus, the
with propagation delay D. Each neuron has addi- set of stimuli has d4 different stimuli. If ti ≤ d,
tionally the external stimulus input line, which is
i = 0,...,4, then any neuron in the net never
used to start the net dynamics. Single impulse in
obtains impulse from other neurons before it ob-
thestimulusinputlinedeliverstoitstargetneuron
tains its external inputstimulation. In the second
just threshold excitation. This causesfiringat the
paradigmofextendedstimuli,werestricttheover-
moment of the stimulus impulse arrival. For nu-
all duration of the stimulus train with the value
mericalsimulationsweuse20networksofdifferent
tmax =M, whichis aboutthreetimeslongerthan
sizes,seeTable1. Thepropagationvelocityinany
d for each network (see Table 1). Here, all stim-
interconnection line is taken v=0.1 m/s.
uli{t0 =1,t1,t2,t3,t4},whichwerepresentedtoa
network,coverthesetofM4differenttrains,which
2.2.1 Numerical Simulation
equals from 625 different stimuli for net #1 to
100000000differentstimulifornet#20(seeTable
As programming language we use Python under
1). The stimuli were sampled in accordance with
Linux operating system. The dynamics was mod-
standardalgorithm of4-digit counter. Namely,we
elled by advancing time with step dt = 200 µs.
started from stimulus {1,1,1,1,1}, the next stim-
The delay values d and D, when measured in the
dt units, were rounded to the nearest from below ulus is obtained by advancing t1 by 1, and so on.
Thestimulusnextto{1,M,1,1,1}is{1,1,2,1,1},
integers, see Table 1. As a result, the simulating
the onenext to {1,M,M,1,1} is {1,1,1,2,1} and
programoperatesinwholenumberswithnoround-
soon,untilstimulus{1,M,M,M,M}ispresented.
ing errors involved.
Intheextendedparadigm,thelateexternalin-
Each single tick of the program, the network’s
tick,advancestimebydt,andconsistsofthreepar- put impulse can enter corresponding neuron after
itreceivedimpulsesfromneuronsalreadytriggered
tial ticks, which are performed in the given order.
by early external input impulses.
Namely,(i) inputtick,whichadvancestimeinthe
input lines, (ii) axonal tick, which advances time The second paradigme is in concordance with
intheinternalconnectionlines,(iii)neuronaltick, visual information processing, [2] where activity
which advancestime in theneurons. from higherbrainareas, whichwasinvokeddueto
This mannerof updatingstates can betreated visual stimulation at earlier time, is retroinjected
as synchronous in a sense that each component of to areas V1 and V2 in the primary visual cortex,
the network has the same physical time when the where it interacts with activity invoked by visual
network’s tick is complete. On the other hand, input at later time during perception.
viewed as interneuronal communication process,
the dynamics should be treated as asynchronous 2.3.2 Figuring out Periodic States
dueto nonzero propagation delays.
During the step (ii), a neuron can get impulse After the last input impulse from the train {t0 =
intoitsinternalmemory. Ifaneuronappearsinthe 1,t1,t2,t3,t4} reachs its target neuron, the pro-
state “Fire” as a result of the network tick, then gram begins appending at each time step the in-
the output impulse it produces can appear in the stantaneous state of thenet toa Python list. The
connectionlinesonlyduringthenextnetworktick. instantaneousstateconsistsofstatesofall20con-
This introduces effective delay of one dt between nectionlinesandall5neurons(seeFig. 4). Before
delivering the triggering impulse to a neuron and appending, the program checks if the current in-
emitting output impulse by that neuron. stantaneous state was already included in the list.
Testing of Information Condensation 5
1
t
n
1
t d
2
n
2
D
t = 1
0
n
0
n
3
t3
n
4
t
4
Figure 3: The network, used for simulations. Here {t0,t1,t2,t3,t4} — is the input spike train, d, D — are the
propagation delays in the connection lines. Any line can be either empty, or propagating one impulse.
net # 1 2 3 4 5 6 7 8 9 10
R, mm 0.029 0.057 0.086 0.114 0.143 0.171 0.200 0.229 0.257 0.286
d, dt 1 3 5 6 8 10 11 13 15 16
D, dt 2 5 8 10 13 16 19 21 24 27
M, dt 5 10 15 20 25 30 35 40 45 50
net # 11 12 13 14 15 16 17 18 19 20
R, mm 0.314 0.343 0.371 0.400 0.429 0.457 0.486 0.514 0.543 0.571
d, dt 18 20 21 23 25 26 28 30 31 33
D, dt 29 32 35 38 40 43 46 48 51 54
M, dt 55 60 65 70 75 80 85 90 95 100
Table 1: Dimensions of networks used for simulations; dt=200 µs.
net # 1 2 3 4 5 6 7 8 9 10
31 61 91 111 141 171 201 221 251 281
104 154 184 234 284 324 364 414 454
123 187 227 287 347 407 447 507 567
246 296 376 456 526
net # 11 12 13 14 15 16 17 18 19 20
301 331 361 391 411 441 471 491 521 551
494 544 584 634
Table 2: Distinct periods in dt units of found periodic states in the short stimuli paradigm. Superscript denotes
the number of different periodic states with this period.
6 A. Vidybida
Ifitwas,thentheperiodicdynamicalstateisfound andk3 >ki, i=0,1,2leadstocontradiction, and
with its complete cyclic trajectory covered by in- so on.
stantaneous states between the inclusion and the This number of triggering is either 1 or 2 for
last record in the list, inclusively. Measures are trajectories found, see examples in Fig. 6. Some
takeninordernottocountthesamecyclictrajec- netshaveonlyoneperiodicstate,whichcoresponds
tory,whichwasentrainedatitsdifferentpoints,as tosynchronousfiringofall5neuronsandsymmet-
different periodic states. rical states of connection lines at any moment of
time. This is thecase for nets number1 and from
15 to 20 for short stimuli, and for nets number19
The data for each net were stored in two
and 20 for extended stimuli.
MySQL tables. Table STATES included the se-
rial number of any periodic state found, one ele-
mentfromthecorrespondingcyclictrajectory,and 3.2 Condensation of Information
period of the state (see Fig. 4). Single record
Inordertoestimatethedegreeofinformationcon-
in the INPUTS table included the input stimulus
densationinthecourseoftransformationofanex-
{t0 = 1,t1,t2,t3,t4}, the serial number of the pe-
ternalspiketrainintoacertainperiodicstateofthe
riodic stateit leadsto(thisnumberis0for fading
net, one needs to calculate information amount,
dynamics), and relaxation time, namely, thetime,
which is delivered by specifying a spike train, and
which isspentbetween thelast externalinputim-
which is delivered by specifying the state, it leads
pulse is delivered and the net’s entrance moment
to.
into theperiodic regime.
3 2 0,1 4
state #319
0 50
3 Results
2,3 0,4 1
state #107
0 50
3.1 Characterization of Periodic
0,3 1,2 4 0,3 1,2 4
States Found state #199
0 66
After initial stimulation, networks from #1 to #7
entrain to periodic activity after any stimulation, Figure 6: Examples of periodic states found
and networks #8 to #20 either entrain to peri- for net number 9 in the extended stimuli
odic dynamics, or stops from any activity after paradigm. Spikes indicate the firing moments,
some time. This takes place for both short and
labelsneareachspikegivenumbersofneurons,
extended stimuli.The number of different periodic
firing at this moment. The two upper trains
states found for each network is shown in Fig. 5.
show states with period 10 ms, the lowerone -
Here, the maximal number of periodic states ob-
with period 13.2 ms.
tained withshort stimuliis18, andthisnumberis
achievedfornetnumbersfrom3to7. Exactvalues ThiscanbedonebywellknownShannon’sformula
of periods, and numberof different periodic states [29]
withthisperiodisshowninTable2forshortstim- H =−Xpilog2pi, (1)
uli. We omit similar table for extended stimuli. i
The maximal number of different periodic states where pi is the probability to obtain case num-
obtained with extended stimuli is 485, which is ber i from a set of cases. At the input end we
achieved in thenet number9. The maximal num- have the set of d4, or M4 different external input
ber of different periodic states with the same pe- spike trains. In our statement of the problem, it
riodis294forperiod50·dtinnet#9. Itshouldbe is natural to consider all external input trains as
mentioned that two periodic states, which can be equallyprobable. Ifso,theninformation delivered
turned into eachother by suitable renumeration of by specifying certain train is
neurons, were considered as different.
Itisevidentthatinthenetworkoffiveneurons Hs=4log2d, (2)
withthreshold4,eachneuronistriggeredthesame
for theshort stimuli paradigm, and
number of times during period. Indeed, expect
thatneuronn4 firesk4 times,andanyoftheother He=4log2M, (3)
fourfiresless duringperiod: k4 >ki, i=0,1,2,3.
In order to be triggered k4 times, n4 must obtain for theextended stimuli paradigm.
not less than 4k4 input impulses during period. While estimating information, delivered by
Butitcanobtainonlyk0+k1+k2+k3,whichisless specifying certain periodic state, one should take
than required. Similarly, situation when k4 = k3 intoaccountthatprobabilitiesofdifferentperiodic
Testing of Information Condensation 7
mysql> select * from STATES_5_9 where num=269;
+------+------------------------------------------------------------------------------
----------------------------------------------+--------+
| num | state
| period |
+------+------------------------------------------------------------------------------
----------------------------------------------+--------+
| 269 | 0 1 1 0 8 8 17 17 24 15 15 24 8 8 0 0 8 17 17 8
0 False False
0 False False
0 False False
0 False False
0 False True 49 1
| 82 |
+------+------------------------------------------------------------------------------
----------------------------------------------+--------+
1 row in set (0.02 sec)
mysql>
Figure 4: Example of single record in the MySQL table STATES. The first field (num) is numerical, and gives
the serial number of periodic state found. The second field (state) is a string, which describes instantaneous
state from the periodic state found (a point from the cyclic trajectory,which represents the whole trajectory,or
periodic state). The first 20 numbers in the string describe states of all connectionlines: ’0’ means that the line
is empty, positive number specifies after how many ticks the propagating impulse will rich the targeted neuron.
The next five chunks confined between the ”next line” symbols describe states of neurons. The first number in
eachchunk isthe ”kick”—the totalnumber ofimpulsesobtainedbyneuronafterthe axonaltickwascomplete.
During the neuronal tick, corresponding to that axonal tick, the ”kick” is utilized and set to zero. The next
boolean in the chunk indicates if the neuron is in the ”Fire” state. The next booleanindicates if the neuron has
any impulses inits internalmemory. If it has,thennext couplesofnumbers (up to three couples)describe those
impulses. In this example, neuron #4 has 1 impulse with time to leave 49·dt. The third field (period) specifies
period (in dt units) of this periodic state.
20 500
450
odic states 15 odic states 334050000
peri 10 peri 250
of of 200
umber 5 umber 110500
n n
50
0 0
2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 14 16 18 20
net # net #
Figure 5: Number of different periodic states found for each net with short (left panel), and extended (right
panel) stimuli.
8 A. Vidybida
statesarenotthesame. Inordertocalculateprob-
ability pn of a periodic state Cn, we calculate the 8
nnuammebleyr, Tofn,inapnudtdsipviikdeettrhaiisnsnulemabdeinrgbytoththeetoCtnal, dic states 67 e
afomroTunntinoFfidg.iff7e)rent input stimuli (see histogram of perio 45
on 3 s
Tn mati 2
pn= d4, (4) nfor 1
i
0
2 4 6 8 10 12 14 16 18 20
for theshort stimuli paradigm, and
net #
Tn
pn= M4, (5) Figure 8: Dependence of information amount,
which is ascribed to periodic state, on the net
for the extended stimuli paradigm. Then we use size. Curve’e’correspondstoextendedstimuli
Eq. (1) with probabilities of individual periodic
paradigm, ’s’ — to the short stimuli one.
states found in accordance with (4), (5) to cal-
culate information which should be ascribed to informationofaperiodicstate,calculatedinaccor-
any periodic state. In this calculations, we treat dancewithEq. (1)withprobabilitiesfounddueto
Eqs. (4), (5), varies between 6.93 and 7.33 bits
uniformly with others the external input stimuli,
which lead to fading dynamics. Correspondingly, for extended stimuli, and between 3.17 and 3.46
thestatewithnoactivityistreateduniformlywith for short stimuli. In the plateau, the degree of in-
formation condensation calculated as input infor-
periodic states. This is in the contrast with data
presented in Fig. 5, and Table 2, where the state mation divided by the periodic state information,
with no activity is excluded. varies between 9.02 and 12.27 for extended stim-
uli, and between 11.2 and 19.31 for short stimuli.
Out of a plateau range, for greater net sizes the
amount of information in a periodic state drops
sharply due to simplification of the set of peri-
90
odic states. Namely, the total number of periodic
ne bin 7800 states decreases to one, with the second one with
o
domains in 456000 nuthoneeavdceetnigvlryieteyb,oeatfnwidneeftonhrmethpaitrsioobtnwabociolsnittdyaetinesss.datisiTotrnhibiasustleehadigdvhseratyos
ber of 2300 41000forextendedstimuli,and690forshortones.
m
u 10
n
0
0 10000 20000 30000 40000 4 Discussion
size of conceptual domain
Any reverberating spiking neural net can repre-
Figure 7: Histogram of conceptual domain sent complicated dynamical behavior. If the net’s
sizes for net #9 in extended stimuli paradigm. instantaneous states can be described with whole
Thebinsizeis518. 23domainswithsizesfrom numbers, then the net will inevitably either en-
50764 to 193732are not presented. train to periodic dynamics, or stop its activity at
all. Inthisstudy,itisappearedthataverysimple
As it could be expected, the information net of Fig. 3 can be engaged into a considerably
amount in an external input spike train increases large set of different periodic activities. It is not
as logarithm of the net size, in accordance with clear whichpartofall possiblein thisnetperiodic
Eqs. (2), (3), varying from 25 to 81 bits for short states was discovered in our simulation. As it fol-
paradigm, and from 37 to 106 bits for extended lowsfromcomparisonbetweenshortandextended
paradigm. Information, which could be ascribed stimuliparadigm, thenumberofdifferentperiodic
to a periodic state, depends on the net size in a states found increases with increasing range of in-
more complicated manner, see Fig. 8. A remark- put stimuli. Certainly, thisincrease must saturate
able feature is a kind of plateau between net #3 somewhere. Thisisbecauseanytwodifferentperi-
and #9 for both short, and extended stimulation odicregimesarerepresentedbytheircyclictrajec-
paradigme. Inthe plateau, the tories,whichhasnocommonpoints(instantaneous
states). On the other hand, the total number of
instantaneousstatesthenetworkcanhaveisfinite
Testing of Information Condensation 9
duetofinitenessofthesetofstatesofeachelement extended stimuli paradigm. left, cross-section
thenetwork is composed of. is made with plane (t3 = 23,t4 = 23), right,
The number of periodic states found in a net (t3 = 21,t4 = 23). Origin for (t1,t2) is in
depends on the net’s geometric size. Variations the upper left corner. Both t1 and t2 run
in the net’s size display themselves exclusively in through the set {1,2,...,45} of values. Num-
variations of interneuronal propagation delays d
bersinpolygonsindicateserialnumbersofcor-
and D. On the other hand, the duration of neu-
responding periodic states.
ronalinternalmemory,τ,isthesamefornetofany
size. Thus, it is namely the relationships between It would be interesting to have a look on
thetimesanimpulsespendsfortravellingbetween the topology of the conceptual domains in the 4-
neurons, and time it is allowed to spend in a neu- dimensional space of all stimuli. For this pur-
ronwaitingforadditionalimpulses,whichcontrols pose we figured out 2-dimensional cross-section
possible numberof periodic states. of the input stimuli (see Fig. 9). In the cross-
Itisworthnoticingthateachnethasonecom- section, a typical conceptual domain is composed
pletely synchronized periodic state. The com- of several coherent clusters disconnected with ea-
chother. The histogram of sizes of conceptual do-
pletely synchronozed state is stable and achieved
during finite time. This is in the contrast to the mains found is given in Fig. 7.
caseofpulse-coupledoscillators withdelayedexci- Decompositionofthewholesetofinputstimuli
into a number of conceptual domains represented
tatory coupling, (see, e.g. Ref. [40]).
bycorresponding periodicdynamicsresembles ap-
In the four-dimensional set of stimuli we used,
proachinanalysesofmultivariatedatasets,seere-
the neighbouring stimuli differ from eachother by
view in [14]. The difference is that here the 4-
one dt in one of four dimensions. This can be
dimensional dataset (aconceptual domain) is rep-
treatedasanalogousrepresentationofsomereality.
resentedbyunidimensionalcyclictrajectory,which
The set of periodic states should be considered as
corresponds to the domain, and the trajectory is
a set of discrete entities due to qualitative differ-
composed of points/vectors, which have other di-
ence between any two states. This conforms with
mension than the dataset vectors (see Fig. 4).
aparadigm discussed in cognitivephysiology, [21].
Nevertheless,havingthenetwork,thewholecyclic
The process of transformation of initial analogous
trajectorycanbereproducedstartingfromitsany
inputsintoadiscretsetofperiodicstatesimpliesa
single point. Thus, here the datasets are reduced
loss of information and can be treated as conden-
down to individual points. This is in concordance
sation of information.
with theinformation condensation idea.
If we take a set of input stimuli, any of which
Whydowestickourselveswithnamelythepe-
leadstothesameperiodicstate,thenthatperiodic
riodic dynamics? The answer is related to the
state can be considered as an abstract/conceptual
memory/learning problem, even if we do not con-
representation of a feature, which all stimuli from
sider any plasticity in this study. It is known [3]
thesethaveincommon,andthecorrespondingset
that modification of synaptic strength may hap-
could be named as “conceptual domain”. What
penduetorepetitivedelivery ofimpulses tothose
kindof featureorconceptdoes theconceptualdo-
synapses. Periodic dynamical states are just well
mainrepresent? Ifournetwastrainedtorecognize
suited for such repetitive delivery. All other dy-
acertainrealfeature,thenitwouldbethatfeature.
namical behaviors are of transient type, and have
Inthecontextofthisstudy,thecommonfeatureis
less chances to cause plastic changes in biological
thatallstimulifromtheconceptualdomainengage
network. Ontheotherhand,successfulperception
namely this net into namely this periodic dynam-
expectsabilitytoreportaboutwhatwasperceived,
ics.
which is impossible without memory.
11111111112222222222333333333344444412345678901234567890123456789012345678901234535182322495567323298059101112138142145511661718192024122221328077312121449425810224622452261127232853422921370871130321350322332163477235083612222222222222222237666666222666666767898765410145678902383940224317745203247304422233472074550839 1111111111222222222233333333334444441234567890123456789012345678901234567890123451312332200390145522333099000675021207389101112138142152516561791819203333222292455551565216420290924223212228707053112216211394944257842220146728245112622327532811142723298532137002731160327233834222222222222212222366622266666667666756541014567892898701376813871138862139907149032324371672247725050247342444022232342307475589093 annarotooanffoemntutapsisUivlroomoooeisdngnnnsuecyfilsolhlbyi,atrtalnt.te[oneiu1eooTxegt8nnwxceh]afs.itaoittiesnraaerrR.kltndnyooedra,uRrrpaleyrwamerqs,enaaetuaslipabimetcrlpubniyeacurtitssnaooliiinlnlsaat,logoyocbogwhtofihithcnuahdoaaictscibvrlrshaetre4nwaecar%aegihsenttaetqawehrslrouddeoaifitsarsrchdkseitbtentasehiithrno2aneaiislu5cb.oqstlmipgiuuntcieobdaeceruleieaidnysr--l
Finally, what happens if we use another neu-
ronalmodelinthenetwork? Ouropinionisthatre-
Figure 9: 2-dimensional cross-sectionof the 4- sults will be qualitatively similar. Using the bind-
dimensional space of inputs for network #9 in ing neuron here is natural, since it represents in
10 A. Vidybida
References
refined form what a spiking neuron does with sig-
nalsitreceives. Additionally,theBN modeleasily
allows to develop a program operating in whole [1] Acharya, R., Chua, E.C.P., Chua, K.C.,
numbers. This excludes possible dynamical arte- Min, L.C., and Tamura, T. (2010), Anal-
facts dueto roundingerrors. ysis and Automatic Identification of Sleep
Infuturework,it would beinteresting tocom- Stages using Higher OrderSpectra, Interna-
pare results, if another spiking neuron model is tional Journal of Neural Systems, 20:6.
used in the network, to study the topology of [2] J. Bullier, “Integrated model of visual pro-
conceptual domains and how the topology could cessing,”BrainRes.Rev.,36,96–107(2001).
change if a plasticity is introduced in the network
[3] D. V. Buonomano and M. M. Merzenich,
model.
“Cortical plasticity: from synapses to
maps,” Annu. Rev. Neurosci., 21, 149–186
5 Conclusions (1998).
[4] P. Cariani, “Temporal codes, timing nets,
A network composed of spiking neurons is able to and music perception,” J. New Music Res.,
condense information due to the fact that differ- 30, 107–135 (2001).
ent initial stimuli could lead the network to the
[5] A.R.Damasio,“Thebrainbindsentitiesand
same periodic dynamics. This happens by means
eventsbymultiregional activation from con-
ofinitial/basiccondensationofinformationinspik-
vergence zones,” Neural Comput., 1, 123–
ing neurons, as it is described in n. 2.1.12.1.1,
132 (1989).
above. The network’s geometric size, which deter-
[6] P. Duchamp-Viret and A. Duchamp, “Odor
mines the interneuronal transmission delays, has
processing in the frog olfactory system,”
considerable influence on the net’s ability to con-
Prog. Neurobiol., 53, 561–602 (1997).
dense information, mainly due to influence on the
numberofdifferentperiodicstatesthenetworkcan [7] R. Durbin and G. Mitchison, “A dimension
have, see Fig. 5. The latter has influence on the reduction framework for understanding cor-
amount of information, which should be ascribed tical maps,” Nature, 343, 644–647 (1990).
to a single periodic state, see Fig. 8. As a re-
[8] R. Eckhorn, R. Bauer, W. Jordan, M.
sult,thedegreeofinformation condensationvaries
Brosch,W.Kruse,M.MunkandH.J.Reit-
between 9 and 41000, see n. 3.23.2 for details.
boeck, “Coherent oscillations: a mechanism
The networks considered here are too primi-
for feature linking in the visual cortex?,”
tivetohavereliablebiologicalimplications. Atthe Biol. Cybern., 60, 121–130 (1988).
same time, numerical parameters, see Table 1 and
[9] A. K. Engel, P. K¨onig, A. K. Kreiter, C.
n. 2.22.2,suchasnetworksizesandspikepropaga-
M. Gray and W. Singer, “Temporal coding
tionvelocity,aretakencorrespondingtobiological
by coherent oscillations as a potential so-
data. The threshold value 4 does not contradict
lution to the binding problem: physiolog-
to biological reality, as experimentally registered
ical evidence,” In Schuster, H.G., Singer,
thresholds are between 1 and 300. The BN inter-
W. (ed), Nonlinear Dynamics and Neuronal
nal memory duration, τ, is commensurable with
Networks, 3–25. VCH Weinheim (1991).
halfdecay time of the excitatory postsynaptic po-
tentials (EPSP). Thus, in the framework of this [10] J.Feldman,“Ecological expectedutilityand
extremelysimplemodel,onecouldexpectthatthe themythicalneuralcode,” Cognitive Neuro-
abilityofbiologicalneuralnetworktocondensein- dynamics, 4, 25–35 (2010).
formation should depend on its geometric size, or
[11] Ghosh-Dastidar,S.andAdeli,H.(2007),Im-
on the relationships between interneuronal trans-
proved Spiking Neural Networks for EEG
mission delays and theEPSP halfdecay time.
Classification and Epilepsy and Seizure De-
tection, Integrated Computer-Aided Engi-
AcknowledgmentsThis work was supported neering, Vol. 14, No.3, pp.187-212.
by the Program of basic research of the National
[12] S. Ghosh-Dastidar and H. Adeli, “Spiking
Academy of Science of Ukraine.
neuralnetworks,”Intern. J. Neural Sys. 19,
Content of this work was partially published
295–308 (2009).
in an abstract form in the abstract book of
the 2nd International Biophysics Congress and [13] S. Ghosh-Dastidar and H. Adeli, A New
BiotechnologyatGAP&21thNationalBiophysics Supervised Learning Algorithm for Multiple
Congress, (5-9 Oct. 2009) Diyarbakır, Turkey, SpikingNeuralNetworkswithApplicationin
http://www.ibc2009.org/ EpilepsyandSeizureDetection,Neural Net-
works 22, 1419–1431 (2009).