Neural Networks 32 (2012) 26–34

Contents lists available at SciVerse ScienceDirect

Neural Networks

journal homepage: www.elsevier.com/locate/neunet

2012 Special Issue

Synchrony: A spiking-based mechanism for processing sensory stimuliCornelius Glackin a,∗, Liam Maguire b, Liam McDaid b, John Wade b

a Adaptive Systems Research Group, School of Computer Science, University of Hertfordshire, College Lane, Hatfield, Hertfordshire, AL10 9AB, United Kingdomb Intelligent Systems Research Centre, University of Ulster, Magee Campus, Derry BT48 7JL, Northern Ireland, United Kingdom

a r t i c l e i n f o

Keywords:Spiking neural networksSynchronyLateral inhibitionConnectivity regimesMutual information

a b s t r a c t

Synchronous behaviour of neurons is both beneficial and detrimental to the neural code. On the oneextreme, synchronous firing activity is well known to be a symptom of epileptic seizures, whilst on theother synchrony provides a mechanism for coordinating brain activity. This paper briefly reviews somecurrent thinking with regard to synchrony, and outlines some experiments with LIF neurons that harnessnear-synchronous states for processing biologically-realistic sensory stimuli. Inspired by the topology ofneurons in the cochlear nucleus, laterally connected leaky integrate and fire neurons, operating in near-synchronous states, are investigated for their ability to reduce noisy spikes and increase spectral contrastof auditory stimuli. Two connectivity parameters, referred to as connection length and neighbourhoodradius, are introduced to configure lateral inhibitory connectivity to generate this neural behaviour.Information-theoretic principles are then employed to quantify the information retained by the coding,and then this is compared to the information retained by the various output topologies.

© 2012 Elsevier Ltd. All rights reserved.

1. Introduction

One of the main aims of neural network research is tounderstand how neurons respond selectively to certain stimuli,and bind these responses together for joint processing. Synchronyof neural responses, which is characterised by phase-locking ofspikes, has long been thought to be the principle means by whichvariable binding is implemented (von der Malsburg & Schneider,1986). The so-called binding-by-synchrony hypothesis suggeststhat neural circuits integrate distributed circuits into coherentrepresentational patterns. Physical evidence for the hypothesiswas found soon after (Gray, König, Engel, & Singer, 1989), yet itis only relatively recently that networks of spiking neuron modelshave been experimented with, to study how synchrony works(Börgers & Kopell, 2003; Dhamala, Jirsa, & Ding, 2004; Lubenov &Siapas, 2008; Van Vreeswijk, Abbott, & Ermentrout, 1994). Froman experimental viewpoint it is difficult to see how synchronyaids computation. Synchronous states generated in experimentsthus far typically show entire populations of neurons firing inphase. If all neurons are firing at the same time, how is synchronypromoting selectivity to certain stimuli?

There is still much to understand with regard to synchrony.Nevertheless, there have been some important recent discoveriesmade. Arguably, it is now widely accepted that it is inhibition notexcitation that brings about synchrony, specifically when the rise

∗ Corresponding author. Tel.: +44 0 1707284490.E-mail address: c.glackin2@herts.ac.uk (C. Glackin).

0893-6080/$ – see front matter© 2012 Elsevier Ltd. All rights reserved.doi:10.1016/j.neunet.2012.02.020

time of a synapse is longer than the duration of an action potential(Van Vreeswijk et al., 1994). It has also been discovered thatsynchrony arises even with sparse connectivity (Börgers & Kopell,2003), it can be influenced with modification of inhibitory timeconstants, and the interplay between these latencies and synapticcoupling strength (Dhamala et al., 2004). Most implementationsof synchrony with spiking neurons advocate an all-things-being-equal approach, keeping parameters such as coupling strengthsbetween excitatory and inhibitory cells uniform. Recently, anovel approach advocated modifying the level of coupling withunsupervised learning (Lubenov & Siapas, 2008). Hence, it wasreported that a more useful state exists between randomnessand full epileptic synchrony that promotes self-organisationof spontaneously active neuronal circuits (Lubenov & Siapas,2008). It was found that spike timing-dependent plasticity (STDP)decouples neurons (breaks synchrony) by modifying synapses in anon-uniformway. Interestingly, anti-STDP (STDP learningwindowinverted) brings the neuronal circuit back to a synchronous state.Yet it is still difficult to see how this near-synchronous transitorystate aids computation.

Since inhibition is crucial for bringing about synchronousstates, it is logical to study the biological evidence for inhibitoryconnectivity, to see if some clues as to how synchrony aidscomputation can be discerned. In studies of vision, specificallythe retina (Ratliff, Hartline, & Miller, 1963), lateral inhibitoryconnectivity is thought to produce edge/peak enhancement. Ananalogous process in audition is performed in the anteroventralcochlear nucleus by T-Stellate cells, which extract the profile of thesound spectrum (Shamma, 2001). Yet both studies are unclear as

C. Glackin et al. / Neural Networks 32 (2012) 26–34 27

to what specific connectivity regimes produce this computationalcapability. Therefore this paper advocates the use of specificregimes for lateral inhibitory connectivity. Biologically plausibleinput data in the form of spoken isolated digits processed usingLyon’s Passive Ear cochlear model (Lyon, 1982) will be used in thepresented experiments. Two connectivity parameters will be usedto specify the connectivity regime and demonstrate how specificlateral connectivity canproduce edge/peak enhancement, promotefeature selection, and reduce noise. The paper builds on previouslypublished work (Glackin, Maguire, McDaid, & Wade, 2011) in thatit uses information theory to quantify the effects of the varioustopologies presented.

Within the artificial neural network literature, ideas aboutinformation preservation between layers of neurons are wellestablished, arguably since the formulation of Linsker’s InfomaxPrinciple (Linsker, 1988). This principle circumvents to classicalcredit assignment problem of determining which particularsynapses within a multiple layer network should be rewardedfor correct final output. Instead the infomax principle advocatesa local rule of maximum information preservation in each layer,from which emerges a sequence of feature analysing functions.The ethos behind the infomax principle with regard to preservingthe information between the various layers of a network in orderto extract features from perceptual data is also relevant for thework presented here. This is achieved by calculating the mutualinformation between the stimuli and the multiple layers of thevarious lateral inhibitory topologies presented.

The calculation of mutual information between stimuli andspikes is in general problematic, particularly in the biologicalsetting. For this reason, neuroscientists when trying to determinethe information transmission between stimulus and neuralresponse typically use mutual information estimators (for adetailed review of the different estimators see Paninski, 2003)to cope with the variability of neural response. However, insome scenarios, mutual information calculated directly can beaccurate if neural responses are limited to a small set and stablerecordings can be made (de Ruyter van Steveninck, Lewen, Strong,Koberle, & Bialek, 1997). The stimulus encoding and spiking neuronmodels employed in this work are deterministic, the data isalso tonotopically arranged by frequency, hence a naive mutualinformation calculation can be made to assess the informationpreservationwithin the tonotopic frequency bands of the data. Theaim is to evaluate the mutual information retained within thesefrequency bands and transmitted throughout the various neuronsin each layer (which are also tonotopically arranged).

Section 2 outlines the pre-processing required to obtainbiologically realistic speech signals. Section 3 demonstrates howthe analogue speech signals are converted to digital spike trains.Section 4 provides background on the information theoreticconcepts used to analyse the encoding and the various topologiespresented. Section 5 outlines how lateral inhibitory networkscan be designed to promote feature selection, produce edge/peakenhancement and reduce noise. Finally, discussion and conclusionsfollows this in Section 6.

2. Speech pre-processing

The speech samples used are isolated digits from the TI46database (Liberman, 1993). A sample of digits spoken by 5 femalespeakers, with 10 utterances for each digit, and 10 total digits(0–9) making 500 speech samples, is selected for this work. Pre-processing of speech information in order to enhance speechfeatures and any resulting recognition is an important undertaking.With regards to the way in which the pre-processing is conducted,there are two main approaches; computational and biologically-inspired respectively. It has long been recognised that sounds are

Time [ms]

Cha

nnel

#

Cochleagram

100 200 300

10

20

30

40

50

60

70

Fig. 1. Cochleagram generated using Slaney’s Auditory Toolbox (Slaney, 1998) forsample: s1-u1-d1.

best described in the frequency domain. Thus the computationalapproach begins by performing spectral analysis to decomposethe speech sample into its frequency components. Typically, theshort-time Fourier Transform is preferred over the standard fastFourier Transform as it is more suited to time-varying signals.The frequency components are usually distributed along the Melscale, which is linear for low frequencies and logarithmic for highfrequencies, corresponding to the physical properties of the humanear. Thus the human auditory critical bands are approximatedusing triangular filters, distributed in a combination of linear andlog positions (there are several methods), and a discrete cosinetransform is used to decorrelate the features and produce Mel-Frequency Cepstrum Coefficients (MFCC) (Pols, 1966).

The biologically-inspired approach typically involves the useof a cochlear model to extract biologically realistic frequencyinformation from the speech sample. Both methodologies canresult in the generation of a frequency-based representation ofthe sound. For the computational approach this representationis referred to as a spectrogram, and for the biologically-inspiredapproach a cochleagram. In this work the biological approachis preferred since the increased contrast of the cochleagramover the spectrogram facilitates the encoding procedure (Glackinet al., 2011). Fig. 1 shows a cochleagram generated using Lyon’sPassive Ear (LPE) cochlea model (Lyon, 1982) for speaker 1,utterance 1, digit 1 (s1-u1-d1) from the TI46 speech corpus(Liberman, 1993). This particular sample was chosen arbitrarilyand will be used throughout this paper to illustrate the featureextraction and information theoretic analysis. The cochlea modelis employed to approximate the frequency composition alongthe basilar membrane. The LPE cochlea model contains a notchfilter bank which models the sensitivity of the cochlea to certainfrequencies. The cochleagram shown in Fig. 1 was generated usingSlaney’s Auditory toolbox (Slaney, 1998) with a sample rate of12.5 kHz.

The typical MFCC methodology suffers from the limitation thatthe temporal nature of the speech sample is treated in an arbitraryway with the features of the sound signal effectively averagedwithin each frame. Such treatment results in a fixed number offeatures that are ordered at arbitrary time intervals rather thanattempting to preserve the temporal information. This treatmentexplainswhy theMFCC technique has been researched extensivelyin an attempt to improve its performance, particularly with noisydata.

28 C. Glackin et al. / Neural Networks 32 (2012) 26–34

100 200 300

10

20

30

40

50

60

70

Time [ms]

Inpu

t Cha

nnel

#

BSA Spike Input

Fig. 2. BSA (Schrauwen & Van Campenhout, 2003) speech encoded spike input forsample: s1-u1-d1.

3. Spike encoding

The next step is to obtain biologically realistic spike trains,and this requires a conversion from the continuous data in thecochleagram into discrete spike trains. There are various waysin which this can be performed (Schrauwen & Van Campenhout,2003). The most efficient method considers spike trains as digitalsignals, since it is thought that it is only the timing of the indi-vidual neurons that biological neurons use to communicate withone another. Therefore, the generation of spike trains from thecochleagram may be considered as an analogue to digital conver-sion. There are algorithms in the literature that have been devel-oped to convert continuous data into discrete spike timing, theseare referred to as spiker algorithms (de Garis, Nawa, Hough, & Ko-rkin, 1999; Schrauwen & Van Campenhout, 2003). The two mainmethodologies are referred to as HSA (Hough Spiker Algorithmde Garis et al., 1999) and BSA (Ben’s Spiker Algorithm Schrauwen& Van Campenhout, 2003) respectively. Both algorithms utilisea convolution/deconvolution filter that was optimised for encod-ing/decoding (de Garis et al., 1999) using a genetic algorithm tominimise the error in the encoding and decoding process. For filterspecifications, impulse responses, frequency spectra and pseudocode refer to Schrauwen and Van Campenhout (2003). Fig. 2 showsthe resulting spike trains generated using BSA.

Fig. 2 illustrates how the 78 frequency channels generatedusing the convolution filter and BSA are converted into spiketrains. It can be seen from the diagram that individual spiketrains begin firing, end firing, and fire maximally at specific times.The literature describes these features of sound signals as onsets,offsets and peak firing rates respectively (Wills, 2001). Onsets canbe extracted (Glackin,Maguire, &McDaid, 2010; Smith, 2004) fromspike trains using spiking neurons with depressing synapses thatare configured to consume their synaptic resources when firingin response to the first presynaptic spike in a spike train andconsuming all their synaptic resources. Offset detection is a littlemore complicated but can be performed using three neurons, twoinput neurons connecting to an output neuron. One input neuronis inhibitory and the other input neuron is excitatory with a smalldelay, for further details see Glackin et al. (2010).

4. Analysing spiking topologies with information theory

The various raster plot presented throughout this work give aclear pictorial representation of the accuracy of the encoding of

the input spike trains (when compared to the cochleagram), andalso of the relationship between the spike inputs and the variousspike outputs. Whilst this pictorial representation is informative,this work extends on the previous work (Glackin et al., 2011)by attempting to quantify the information each of these figurescontains. Information theory provides a vehicle for measuring theinformation at each stage of the spiking network topologies fromthe encoding to the output. In particular it would be beneficial tocharacterise themutual information between the cochleagram andthe spike input encoding, and the cochleagram and the variousspike outputs. The mutual information I between two randomvariables X and Y is give by (Cover & Thomas, 1991):

I(X; Y ) =

x∈X

y∈Y

p(x, y) logp(x, y)p(x)p(y)

(1)

where p(x, y) is the joint distribution, p(x) and p(y) are themarginals. By calculating themutual informationbetween cochlea-gram and input, and cochleagram and output, the amount of infor-mation retained by the encoding and the amount of informationtransmitted by the lateral inhibitory networks can be quantified.

4.1. Mutual information between cochleagram and spike encoding

Armedwith themutual information equation, it is nowpossibleto determine in information theoretic terms the accuracy of thespike encoding performed by BSA (Schrauwen & Van Campenhout,2003), this will serve as a baseline for mutual informationcalculations between the cochleagram and the various outputlayers of the network. In order to calculate the mutual informationI in Eq. (1), it is first necessary to calculate the joint distributionp(x, y). Here the random variables X and Y are taken to representthe continuous valued data in the cochleagram and the discretebinary data in the spike encoded input respectively. But herein liesthe first difficulty, that of calculating the joint distribution betweencontinuous and discrete random variables.

The approach advocated in this work is to apply the well-known technique of binning to discretise the continuous-valuedcochleagram data X . Binning the data can be problematic, if doneincorrectly it can introduce bias into the mutual informationcalculation that follows Paninski (2003). The idea is to bin thecontinuous data in a way that maximises the mutual informationcalculation. It can be shown that maximising mutual informationis equivalent to maximising the entropy of the bin probabilitydistribution (Cover & Thomas, 1991). It is well known, that theentropy of a particular random variable is maximised when therandom variable in question is uniformly distributed (Cover &Thomas, 1991). In this way, by ensuring that the number of datapoints within each bin (for the continuous cochleagram data) areequal. This can be achieved by ordering each continuous value inthe distribution of X in order of size and then determining therange of the first 1/K of the total values (where K is the numberof bins), and using the range values as limits for the values thatare to be placed in the bin. The range of the next 1/K valuesdefine the second bin, and this continues until the limits for all thearbitrarily chosen number of bins are defined. In this way each binencompasses the same number of continuous values (maximisingthe entropy of the bin distribution), a technique known as adaptivebinning (Olsson, Nehaniv, & Polani, 2005) or sampling equalisation(Nadal & Parga, 1999).

The binning of the spike data is less problematic than themaximum entropy binning of the cochleagram data as the spikedata is discrete. All that needs to be determined is the binaryword length (spike train segment) to be used in conjunction withthe continuous cochleagram samples when calculating the jointdistribution between the cochleagramdata and the spike encoding.

C. Glackin et al. / Neural Networks 32 (2012) 26–34 29

Fig. 3. Mutual information for various time shifts for sample: s1-u1-d1.

The simplest spike binning is to consider word lengths of one biti.e. two bins, namely 0 or 1, however this size of bin for the spikedata results in low estimations of mutual information. Larger wordlengths can be used but for very large word lengths, there is arisk for a given amount of data that the joint distribution will betoo sparsely populated. It was determined that the optimumwordlength for the given amount of data in the speech sample was 3bits (resulting in 23 bins). Even with such a small word length,care needs to taken in the construction of the joint distributionto determine where continuous data at time t in the cochleagrammaps to in the spike encoding. Since the spike encoding employs aconvex convolution function which is applied over a time intervalof 20ms, it is necessary to consider thatmutual informationmaybemaximised at any point up to 20 ms before the generation of eachspike in the encoding. By implementing a time-shift parameter anditerating the construction of the joint and marginal distributionsand hence the calculation of the mutual information using Eq. (1),it can be determined how long it takes information from thecochleagram to manifest itself in the spike encoding, as isillustrated with Fig. 3.

Fig. 3 shows the mutual information plotted against a range ofnegative to positive time shifts. As can be seen from the figure, themutual information clearly peaks for a time-shift of−11ms (11msbefore the generation of the spike at time t). This coincideswith thepeak of the convex convolution function employed by the spikeralgorithm (Schrauwen & Van Campenhout, 2003). BSA encodinguses this convolution function (which was optimised using agenetic algorithm de Garis et al., 1999), to smoothly quantise thecontinuous data to discrete spiking output. The maximum mutualinformation for the s1-u1-d1 sample is 2.22 bits, meaning thateach time step of the continuous cochleagram input, 2.22 bitsof information is transmitted to the encoded spike output. Themutual information will be calculated throughout the followingsections of the paper in order to quantify the effects of the variouslateral topologies presented. The next section provides backgroundinformation about the construction of these topologies.

5. Lateral inhibitory networks

All of the spiking neurons used throughout this paper are of theLeaky Integrate-and-Fire (LIF) variety (Gerstner & Kistler, 2002).The standard form of the LIF neuron is of the form:

τmdVdt

= −V (t) + RI(t) (2)

where V (t) is the membrane potential and τm is the membranetime constant. R is the membrane resistance and I(t) is an injected

Fig. 4. A simple three neuron SNN (top left) used to test the ability of laterallyconnected neurons to produce coincidental firing and synchrony (Abbott, 2001).The SNN receives two input spike trains, one with a firing rate reducing from 28.5to 21.5 Hz, the other with a constant firing rate of 25 Hz (top right). The bottom twosubplots show the actual input and output spike rasters of inputs 1 and 2 (i/p 1 andi/p 2) and resultant spike output for neurons 1, 2, and 3, respectively.

current. This simple neuron model has been altered slightly inthis work so that it uses spike input, where current injection isdetermined by the synaptic weight if and only if a pre-synapticspike arrives. The resulting neuron model is computationallyinexpensive and enables timely simulation of the medium sizednetworks with extensive lateral connectivity that follows.

One way to discover when neurons are firing at the same rateis by synchrony (Hopfield & Brody, 2001). This can be investigatedwith a simple experiment originally proposed by Abbott (2001).Fig. 4 shows the interaction between two LIF neurons with lateralinhibitory connections, connecting to an excitatory output neuron.Each neuron in the input layer receives as input the inhibitoryoutput of the other. Each of these neurons has its own excitatoryinput, one receives a fixed firing rate of 25 Hz, the other a firingrate linearly changing from 28.5 to 21.5 Hz.

As can be seen from Fig. 4, when the two input neurons arefiring at different frequencies, the input neurons take turns atsuppressing the output of one another, depending on spike timing.The output neuron, neuron 3, fires maximally (coincidently)when the two input neurons are firing at the same frequency(Abbott, 2001). Abbott’s experiment (Abbott, 2001) illustrates howfrequency information can be extracted from a spike train. Itdemonstrates how to findwhen a spike train is firing at a particularfrequency. This experiment is particularly useful from a featureextraction point of view as it can be used to determine when spiketrains in the spike array of Fig. 2 are firing at similar frequencies.In fact there is no limit to the number of spike trains that can becompared in this way, coincidence detection can be performedfor any number of spike trains as long as they have ‘sufficientconnectivity’ (Börgers & Kopell, 2003).

30 C. Glackin et al. / Neural Networks 32 (2012) 26–34

Fig. 5. Neurons connected laterally as determined by a connection lengthparameter.

5.1. Connection length

The output from the LPE and subsequent BSA encoding of spiketrains is tonotopically arranged. Therefore, it does not necessarilymake sense to associate every input neuron and hence soundfrequency with every other, as this disregards this tonotopicarrangement. It seems more likely that the lateral connectivity ofthe input layer can be described in terms of a connection lengthparameter. A particular connection length of c would mean thateach input layer neuron is laterally connected to c neurons eitherside of it. Fig. 5 illustrates this idea, the black lines of various stylesrepresent connection lengths between 1 and 3 for an example layerof laterally connected neurons.

In this way, a layer of N neurons can have a maximumconnection length of cmax = N − 1. In the case of the spike trainsin Fig. 4, a layer containing N = 78 neurons similar to those inFig. 2 was implemented with an initial connection length of 0 (i.e.no lateral connectivity). Successive simulations were conductedwith the connection length increased by 1 up to the maximumconnectivity of cmax = 77. Fig. 6 presents a selection of the spikeoutputs from these experiments.

As can be seen from Fig. 6, synchrony appears with much lessthan full connectivity, in fact it can be seen to begin to formwith neighbourhoods laterally connected to tonotopically adjacentneurons with a small connection length (c = 15). Thus, even with

sparse lateral inhibition, the layer of neurons in Fig. 5 is shownto be capable of attaining a synchronous state in accordance withBörgers and Kopell (2003).

5.2. Multiple layers of synchrony

The layer of neurons with full lateral connectivity producedsynchronised spike activity from tonotopically arranged soundinformation. An obvious line of enquiry is to investigate whathappens when the synchronous spike activity is passed throughmultiple layers of fully connected lateral inhibitory neurons. Fig. 7shows the input and output of 3 layers of such connectivity.

Fig. 7 demonstrates the effects layers of synchrony have onspike input. The output from the first layer is as expected fromFig. 6. However, it can be seen that iterating the lateral processingover several layers serves to successively remove more spikesat each layer. The spikes that remain tend to aggregate intosynchronised bursts across layers. Contrastingly, areas of spikeswhich are not densely packed are removed. The removal ofnoisy spikes is a feature of this connectivity that is potentiallyuseful, particularly if it can be implemented without the periodicremoval of all spikes at regular time intervals in the output, asthis represents a potentially crucial loss of information. Thereforethe focus of the remainder of the paper will be to investigatehow edges can be enhanced, and how noisy spikes can beremoved, but without the periodic removal of spikes associatedwith such epileptic synchrony. Fig. 8 shows themutual informationcalculated for the same multiple time shifts and spike wordlengths as was used for the encoding. There has been some lossof information compared with the mutual information for thespike encoding (compare to Fig. 3), this is to be expected, howeverlater the mutual information across all layers will be discussedto determine how well this topology preserves information insuccessive layers.

The mutual information for layer 1 peaks at 0.87 bits, arounda time shift of 0 ms. This is precisely why multiple time shiftsare examined, for layer 1 there is a 10 ms delay in inhibitory

Connection Length = 0 Connection Length = 15

Connection Length = 77Connection Length = 30

Fig. 6. Effects of increasing connection length on synchrony for sample: s1-u1-d1. A sample of results are shown for connection lengths of 0, 15, 30, and 77.

C. Glackin et al. / Neural Networks 32 (2012) 26–34 31

Fig. 7. Input spike trains for sample: s1-u1-d1 (top left) routed through three successive layers (L1–L3) of neurons laterally connected with inhibition.

Time Shift [ms]

MI [

bits

]

0.4

0.5

0.6

0.7

0.8

0.9

1

–50 0 50

Fig. 8. Mutual information between cochleagram and layer 1 synchronous outputwith multiple time shifts for sample: s1-u1-d1.

synapses, hence the mutual information peaks 10 ms after themutual information in the encoding.

5.3. Neighbourhood radius

The concept of a neighbourhood in the connectivity of neuronswas introduced by Kohonen with the ‘winner-take-all’ competi-tive learning algorithm. Essentially, ‘winning’ neurons had theirweights increased along with neurons topologically close to them,i.e. in the same neighbourhood. The idea of defining neighbour-hoods can also be introduced in the case of spiking neural net-works. In this paper, as already discussed, the connection lengthparameter defines how tonotopically far away a neuron in thesame layer can be connected using lateral inhibition. Another pa-rameter referred to as neighbourhood radius can also be introduced,that describes how neurons that are tonotopically close to one an-other are not connected laterally. Fig. 9 illustrates this idea.

In general three distinct feature extraction properties have beenobserved with different connectivity parameters for randomly

Fig. 9. Introducing the neighbourhood radius parameter to layers of lateralconnectivity specified by the connection length parameter.

selected samples from the TI46 dataset, these are synchrony, edgeenhancement, and noise reduction. Suppose onewishes to identifythe most important frequency channels for a particular sample,and how the importance of these channels varies over time. Interms of the spike encoded speech sample this means the aim isto find the highest frequency spike trains at each time intervalthroughout the sample. This can be achieved for a maximumconnection length and a relatively smaller neighbourhood size, asillustrated by Fig. 10.

5.4. Edge enhancement and noise reduction

In order to make meaningful the tonotopic arrangement ofthe spike encoded sound channels, it follows that neurons thatare tonotopically distant from one another should have littleconnectivity. In terms of the connectivity parameters alreadydefined, a less than maximum connection length has interestingeffects on small neighbourhoods as is illustrated by Fig. 11.

As can be seen from the figure the small neighbourhoodradius, and slightly larger connection length do not result in asynchronous state. Instead, the connectivity has served to removenoisy spikes and extract a clearer representation of the soundsignal by sharpening the main contours of the spike distribution.This kind of noise reduction and edge enhancement capability hasbeen observed in retina and cochlear nucleus cells, likely with

32 C. Glackin et al. / Neural Networks 32 (2012) 26–34

Fig. 10. Maximum lateral inhibitory connectivity (connection length = 77) with a neighbourhood radius of 30 for sample: s1-u1-d1. Partial synchrony has removed someof the spikes from the resultant rasters.

Fig. 11. Neighbourhood radius = 5, Connection length = 20 for sample: s1-u1-d1. Single and low frequency ‘noisy’ spikes are removed with each successive layer of thisconnectivity regime.

similar regimes of connectivity. Fig. 12 shows the subsequentmutual information over time calculation, aswith the synchronoustopology the mutual information peaks around t = 0. The amountofmutual information is higher than for the synchronous networks(peaking at 0.93 bits).

The increase in information transmission is small in this firstlayer but subsequent layers need to be examined to see if thedifferences in information preservation are more pronounced.Thus the mutual information between the cochleagram and theconsecutive layers of synchrony (max connection length), and the

mutual information between the cochleagram and the successiveneighbourhood layers (noise reduction topology with connectionlength = 20; neighbourhood radius = 15) are compared. Fig. 13shows the change in mutual information for the two topologiesthrough successive layers, note that layer 0 refers to the mutualinformation between the cochleagram and the encoded spikes.

As can be seen from Fig. 13, both the topologies lose someinformation from the encoding to the output of the first layer.However, what is interesting is what occurs in the subsequentlayers, the noise reduction topology is more adept at preserving

C. Glackin et al. / Neural Networks 32 (2012) 26–34 33

Time Shift [ms]

MI [

bits

]

1

0.4

0.5

0.6

0.7

0.8

0.9

–50 0 50

Fig. 12. Mutual information between cochleagram and layer 1 output of topologywith neighbourhood radius = 5, connection length = 20 with multiple time shiftsfor sample: s1-u1-d1.

Layer #

MI [

bins

]

Synchronous Layer

Noise Reduction Layer

0.5

1

1.5

2

0 1 2 3

Fig. 13. Mutual information between cochleagram and synchronous layers, andcochleagram and noise reduction layers for sample: s1-u1-d1.

the information between layers, whilst the synchronous topologyprogressively loses information.

6. Discussion and conclusions

The work presented demonstrates that mutual informationcan be calculated directly when the encoding and neuron mod-els employed are deterministic, as they are in the artificial settingof this work. In a biological setting where one wishes to calcu-late mutual information for spike trains in vivo, a mutual infor-mation estimator will often be required, as there is much moreuncertainty associated with biological spike transmission. A po-tential criticism for using the naive approach to the calculationof the mutual information arises with the calculation of mutualinformation between the cochleagram and the outputs spikerasters of the various laterally connected inhibitory layers. Sincethere is substantial connectivity between layers, in theory mutualinformation can be calculated across all frequency channels simul-taneously. Unfortunately, calculating mutual information across78 dimensions is computationally an intractable proposition at thistime, particularly because of the binary word lengths and resultingamounts of data such a calculation would require. Furthermore, inthis work it was desirable to determine the relationship between

the input channels and the corresponding output channels, forother applications involving laterally connected neurons this maynot be of interest. Future work will investigate how the methodol-ogy presented here can be extended to thismulti-information case.For the time being the authors are satisfied that the clear tonotopicorganisation between the layers is sufficient to capture the mutualinformation if only in relative terms (relative between the encod-ing and the various output layers).

In contrast to work in reservoir computing where a typicallyrandom connectivity ethos is preferred, this paper representsongoing work on attempting to quantify what the effects ofparticular lateral connectivity regimes are on neural coding andinformationpreservation. In thisway, two connectivity parameterswere defined that modify the connectivity in a conceptuallymeaningful way. The connection length parameter dictates howfar away (tonotopically) lateral neurons are connected to oneanother. Conversely, the neighbourhood radius parameter dictateshow tonotopically distant neurons can be that are not connectedlaterally. The interplay between the two parameters is the mainfocus of this paper. Initially experiments consider modifying theconnection length parameter alone, in topologies where spikingneurons are sparsely laterally connected to each other withinhibitory synapses, and investigate to what extent synchronyis affected by increasing the degree of lateral connectivity(increasing connection length). In agreement with other researchit is discovered that synchrony results from sparse connectivity(Börgers & Kopell, 2003). However, as connectivity increasessynchrony becomes more defined. Similarly multiple layers ofsynchrony further reduces spike output and produces morepronounced synchronous states.

Most synchrony experiments advocate an all-things-being-equal approach where synaptic weights for example are keptthe same in order to facilitate synchrony (and also because ofa lack of a rationale for varying weights). In experiments whereweights are modified (Lubenov & Siapas, 2008) activity is shownto unsynchronise. Similarly, adding a neighbourhood radiusparameter which dictates a local lack of connectivity for neuronsresults in spikes that are not synchronised. However, with largeconnection lengths, neighbourhoods of neurons compete withone another, with ‘winning’ neighbourhoods suppressing allothers. Interestingly, when the neighbourhood radius is smalland the connection length parameter is only slightly larger, theconnectivity serves to reduce noisy spikes whilst preserving edgesin the spike rasters. Such connectivity experiments could shedsome light on similar capabilities of retina cells and cells in thecochlear nucleus, which are known to extensively employ lateralinhibition. In summary, this work illustrates some interestingfindings with experiments with various forms of spiking neuronconnectivity, and in particular seeks to unravel some of thefunctionality of inhibitory neurons. Spiking neural networktopologies that implement this kind of connectivity could havesome significantly improved capabilities for pattern recognitionproblems.

The work presented augments previously published work(Glackin et al., 2011) by introducing information theoretic analy-sis to quantify the mutual information between the cochleagramstimulus, spiker encoding algorithm, synchronous, and noise re-duction topologies. The analysis shows the effectiveness of theencoding and uses this as a relative baseline for comparing theinformation lost by the synchronous layer (maximum connectionlength) to the information lost by the combination of neighbour-hood radius and connection length (noise reduction) topology. Theneural processing in all of the networks presented show a largereduction in mutual information once the encoded spikes are pro-cessed by the various lateral inhibitory networks. This is to be ex-pected as it is the networks task to transform the input encodingto somethingmore useful. In the case of the synchronous networksthis reduction in mutual information is more aggressive, verified

34 C. Glackin et al. / Neural Networks 32 (2012) 26–34

by the accompanying raster plots which show periodic removal ofspikes. Contrastingly, the noise reduction networks show a preser-vation of mutual information through the various layers, again ascan be seen by the accompanying rasters that show that althoughnoisy spikes have been removed, the remaining spikes still quali-tatively represent the stimulus accurately. Additionally, the totalmaximum information for the synchronous and noise reductiontopologies demonstrate that where the former topologies progres-sively lose information about the stimulus, the latter retain it. Thisclearly points to the detrimental effects of such epileptic synchronyin information terms (albeit with a naive mutual information ap-proach). Therefore clearly demonstrating that near-synchronousstates (such as those employed by the noise reduction networks)are far more useful for neural processing than full synchronousstates.

Acknowledgements

The authors wish to thank Dr. Daniel Polani, Christoph Salge,Malte Harder, and all the members of the Sepia group of theAdaptive Systems Research Group, School of Computer Science,University of Hertfordshire, for their many useful commentsand insight into the information theoretic concepts discussedthroughout the paper.

This research is supported under the Centre of Excellence inIntelligent Systems (CoEIS) project, funded by the Northern IrelandIntegrated Development Fund and InvestNI.

References

Abbott, L. F. (2001). The timing game. Nature Neuroscience, 4(2), 115–116.Börgers, C., & Kopell, N. (2003). Synchronization in networks of excitatory and

inhibitory neurons with sparse, random connectivity. Neural Computation,15(3), 509–538.

Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. John Wiley &Sons.

de Garis, H., Nawa, N. E., Hough, M., & Korkin, M. (1999). Evolving an optimalde/convolution function for the neural net modules of ATR’s artificial brainproject. In Proc. international joint conference on neural networks. Vol. 1(pp. 438–443).

de Ruyter van Steveninck, R. R., Lewen, G. D., Strong, S. P., Koberle, R., & Bialek, W.(1997). Reproducibility and variability in neural spike trains. Science, 275(5307),1805–1808.

Dhamala, M., Jirsa, V., & Ding, M. (2004). Enhancement of neural synchrony by timedelay. Physical Review Letters, 92(7), 074104-1–074104-6.

Gerstner, W., & Kistler, W. M. (2002). Spiking neuron models: single neurons,populations, plasticity. Cambridge University Press.

Glackin, C., Maguire, L., & McDaid, L. (2010). Feature extraction from spectro-temporal signals using dynamic synapses, recurrency, and lateral inhibition. InProc. IEEE world congress computational intelligence. (pp. 1–6).

Glackin, C., Maguire, L., McDaid, L., & Wade, J. (2011). Lateral inhibitory networks:synchrony, edge enhancement, and noise reduction. In Proc. IEEE internationaljoint conference on neural networks.

Gray, C. M., König, P., Engel, A. K., & Singer, W. (1989). Oscillatory responses incat visual cortex exhibit inter-columnar synchronization which reflects globalstimulus properties. Nature, 338(6213), 334–337.

Hopfield, J. J., & Brody, C. D. (2001). What is a moment? transient synchrony as acollective mechanism for spatiotemporal integration. Proceeding of the NationalAcademy of Sciences, 98(3), 1282–1287.

Liberman, M. (1993). TI46 speech corpus. Linguistic Data Consortium [Online].Available: http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC93S9.

Linsker, R. (1988). Self-organisation in a perceptual network. IEEE Computer , 21,105–117.

Lubenov, E. V., & Siapas, A. G. (2008). Decoupling through synchrony in neuronalcircuits with propagation delays. Neuron, 58(1), 118–131.

Lyon, R. G. (1982). A computational model of filtering, detection, and compressionin the cochlea. In Proc. IEEE international conference on ICASSP’82. Vol. 7.

Nadal, J. P., & Parga, N. (1999). Sensory coding: information maximization andredundancy reduction. Neural Information Processing , 7(1A-2), 164–171.

Olsson, L., Nehaniv, C. L., & Polani, D. (2005). Sensor adaptation and developmentin robots by entropy maximization of sensory data. In Proc. computationalintelligence in robotics and automation (pp. 587–592).

Paninski, L. (2003). Estimation of entropy and mutual information. NeuralComputation, 15(6), 1191–1253.

Pols, L. C. W. (1966). Spectral analysis and identification of Dutch vowelsin monosyllabic words. Ph.D. dissertation. Free University. Amsterdam, TheNetherlands.

Ratliff, F., Hartline, H. K., & Miller, W. H. (1963). Spatial and temporal aspects ofretinal inhibitory interaction. Journal of the Optical Society of America, 53(1),110–120.

Schrauwen, B., & Van Campenhout, J. (2003). BSA, a fast and accurate spike trainencoding scheme. In Proc. international joint conference on neural networks. Vol.4 (pp. 2825–2830).

Shamma, S. (2001). On the role of space and time in auditory processing. Trends inCognitive Sciences, 5(8), 340–348.

Slaney, M. (1998). Auditory toolbox: a MATLAB toolbox for auditory modelingwork. Interval Research Corporation [Online]. Available: http://web.interval.com/papers/1998-010.

Smith, L. S. (2004). Robust sound onset detection using leaky integrate-and-fireneurons with depressing synapses. IEEE Transaction on Neural Networks, 15(5),1125–1134.

Van Vreeswijk, C., Abbott, L. F., & Ermentrout, G. B. (1994). When inhibition notexcitation synchronizes neural firing. Journal of Computational Neuroscience,1(4), 313–321.

von der Malsburg, C., & Schneider, W. (1986). A neural cocktail-party processor.Biological Cybernetics, 54, 29–40.

Wills, S. A. (2001). Recognising speech with biologically-plausible proces-sors. Hamilton Prize [Online]. Available: http://www.inference.phy.cam.ac.uk/saw27/hamilton.pdf.