Implementing Fuzzy Reasoning on a Spiking

Neural Network

Cornelius Glackin, Liam McDaid, Liam Maguire, andHeather Sayers

University of Ulster,Faculty of Engineering,

School of Computing & Intelligent Systems,Magee Campus, Londonderry, BT48 7JL, Northern Ireland

{glackin-c1,lj.mcdaid,lp.maguire,hm.sayers}@ulster.ac.uk

Abstract. This paper presents a supervised training algorithm that im-plements fuzzy reasoning on a spiking neural network. Neuron selectivityis facilitated using receptive fields that enable individual neurons to beresponsive to certain spike train frequencies. The receptive fields behavein a similar manner as fuzzy membership functions. The network is su-pervised but learning only occurs locally as in the biological case. Theconnectivity of the hidden and output layers is representative of a fuzzyrule base. The advantages and disadvantages of the network topology forthe IRIS classification task are demonstrated and directions of currentand future work are discussed.

Key words: Spiking Neuron Model, Dynamic Synapse, Supervised Learn-ing, Receptive Field, Fuzzy Reasoning

1 Introduction

The history of neural network research is characterised by a progressively greateremphasis paid to biological plausibility. The evolution of neuron modelling withregard to the complexity of computational units can be classified into three dis-tinct generations [1]. The third generation of neuron modelling (spiking neurons)is based on the realisation that the precise mechanism by which biological neu-rons encode and process information is poorly understood. The spatio-temporaldistribution of spikes in biological neurons holds the key to understanding thebrains neural code.

There exists a multitude of spiking neuron models that can be employed inspiking neural networks (SNNs). The extensive amount and variety of neuronmodels exist in acknowledgement of the fact that there is a trade-off between theindividual complexity of spiking neurons and the number of neurons that canbe modelled in a neural network. In addition to the variety of neuron models,biological neurons can have two different roles to play in the flow of informationwithin neural circuits. These two roles are excitatory and inhibitory respectively.Excitatory neurons are responsible for relaying information whereas inhibitory

2 Cornelius Glackin, Liam McDaid, Liam Maguire, and Heather Sayers

neurons locally regulate the activity of excitatory neurons. Ongoing physiologicalexperiments continue to illuminate the underlying processes responsible for thecomplex dynamics of biological neurons. Therefore it is imperative to determinewhich biological features improve computational capability whilst enabling anefficient description of neuron dynamics. Ultimately neuro-computing seeks toimplement learning in a human fashion. In any kind of algorithm where humanexpertise is implicit, fuzzy IF-THEN rules provide a language for describing thisexpertise [2]. In this paper, the rationale for the distribution of biologically-inspired computational elements is prescribed by the implementation of fuzzyIF-THEN rules.

In Section 2, unsupervised and supervised learning methods, dynamic synapsesand receptive fields are reviewed. Section 3 includes a brief discussion of howfuzzy reasoning can provide a basis for structuring the network topology andintroduces a generic network topology outlining the specific models and algo-rithms used to implement fuzzy reasoning. Experimental results and remarksfor the complex nonlinear Iris classification problem are given in Section 4, andconclusions and future research directions are presented in Section 5.

2 Review

The modelling of the neurons synapse is an essential aspect for an accuraterepresentation of real neurons, and one of the key mechanisms to reproducing theplethora of neuro-computational artefacts in SNNs. From a biologically plausiblepoint-of-view synaptic modification in spiking neurons should be based on thetemporal relationship between pre and post-synaptic neurons, in accordance withHebbian principles . In fact, Hebbian learning and its ability to induce long-term potentiation (LTP) or depression (LTD) provide the basis for most formsof learning in SNNs. Hebbian learning gains great computational power from thefact that it is a local mechanism for synaptic modification but also suffers fromglobal stability problems as a consequence [3].

2.1 Unsupervised and Supervised Learning

There are several learning algorithms that can be used to evoke LTP or LTDof synaptic weights. Spike-timing dependent plasticity (STDP) is arguably themost biologically plausible means of inducing LTP and LTD learning since it isa temporal interpretation of Hebbs well-known first generation learning rule. Interms of the temporal coding of spikes it is the order of individual pre and post-synaptic spikes that determines whether the weight is increased or decreasedusing STDP. BCM and STDP are of course unsupervised learning algorithms,and as such they do not obviously lend themselves to applications requiring aspecific goal definition, since this requires supervised learning.

Table 1 presents a review of supervised learning algorithms and their encod-ing schemes. Time-to-first-spike encoding schemes [4] [5] [6] [7] have the disad-vantage that they cannot be used to learn sequences involving multiple spikes.

Implementing Fuzzy Reasoning on a Spiking Neural Network 3

Additionally, the evolutionary strategy [5] and the linear algebra approach areonly suitable for offline learning [8]. With the statistical approach [9] the re-ported findings have been limited to networks consisting of two neurons so it isdifficult to know how robust the technique will be for larger networks. SupervisedHebbian Learning [7] is arguably the most biologically plausible of the learningalgorithms. However, it suffers from the limitation that even after the goal firingpattern has been achieved the algorithm will continue to change the weights.Of all the supervised learning algorithms presented in Table 1, the ReSuMe [10]approach is perhaps the most efficient. With this approach synaptic weights canbe altered as training progresses in an accurate and stable manner.

Table 1. Comparison of Supervised Learning Approaches [10]

Approach Coding Scheme References

SpikeProp Time-to-first-spike [4]Statistical approach Precise spike timing [9]Linear algebra formalisms Precise spike timing [8]Evolutionary Strategy Time-to-first-spike [5]Synfire Chains Relative spike time [6]Supervised Hebbian Learning Time-to-first-spike [7]Remote Supervision Precise spike timing [10]

2.2 Dynamic Synapses & Receptive Fields

Synaptic efficacy changes on very short-time scales as well as over the longertimescale of training. The rate at which synaptic efficacy changes, is determinedby the supply of synaptic resources such as neuro-transmitter and the num-ber of receptor sites. Dynamic synapse models are typically either deterministicor probabilistic. However, it is important that the modelled magnitude of thepost-synaptic response (PSR) changes in response to pre-synaptic activity [11].Further, biological neurons have synapses that can either facilitate or depressthe synaptic transmission of spikes [11].

Modelling the dynamics of limited synaptic resources makes neurons selectiveto particular spike frequencies. The filtering effects of dynamic synapses occurbecause there is a frequency of pre-synaptic spike trains that optimise the post-synaptic output [12]. A likely explanation for this specificity of frequency is thatfor certain presynaptic spike train frequencies the synapse will not run out ofresources whereas for another it probably will. Between these two pre-synapticspike frequencies there will be an optimum state where the post-synaptic spikefrequency is maximised. This means that certain neurons and synapses can po-tentially be targeted by specific frequencies of pre-synaptic spike trains. Thisphenomenon has been described as ’preferential addressing’ [12]. Constructinga network of neurons using synapses that operate at different frequency bandsis desirable from the perspective of promoting neuron selectivity and richness of

4 Cornelius Glackin, Liam McDaid, Liam Maguire, and Heather Sayers

information flow. However, it is particularly difficult to tune dynamic synapsemodels to operate at specific frequency bands by changing the various modelparameters. One way to guarantee that synapses are responsive to certain fre-quencies is with the use of receptive fields (RF).

As far back as 1953, experiments with retinal ganglion cells in the frog showedthat the cells response to a spot of light grew as the spot grew until somethreshold had been reached [13]. The part of the visual world that can influencethe firing of a neuron is referred to as the RF of the neuron [13]. In Barlow’swork [13], it was demonstrated that a spot of light within the centre of theRF produces excitation of the neuron whereas when the spot of light is largerthan the RF or outside the RF inhibition occurs. The implications for SNNs arethat RFs can be used in conjunction with neuron models to promote featureselectivity and hence enhance the ’richness’ of information flow.

3 Fuzzy SNN Topology

Fig. 1. Generic FSNN Topology

Biological neuron dynamics are determined by the relationships betweenspike trains, synaptic resources, post-synaptic currents and membrane poten-tials. Neuron selectivity can be further strengthened using RFs. The dilemma isin the way in which all these various elements can be combined in a logical wayresulting in SNNs that provide insight into the biological neurons code, and areuseful from an engineering perspective. Biological neurons obviously implementa form of human reasoning. Human reasoning is fuzzy in nature and involvesa much higher level of knowledge representation [2]. Fuzzy rules are typicallydefined in terms of linguistic hedges, e.g. low, high, excessive, reduced, etc. Tak-ing a cue from fuzzy reasoning, the aim of this paper is to demonstrate how

Implementing Fuzzy Reasoning on a Spiking Neural Network 5

the components necessary to define a fuzzy rule in turn dictate the distributionof the various biologically plausible computational elements in an SNN. FuzzyIF-THEN rules are of the form:

IF (x1isA1)AND...AND(xN isAN)THEN(yisZ) (1)

where x1 to xN represent the network inputs, A1 to AN represent hidden layerRFs and y is the network output. Figure 1 presents the generic Fuzzy SNN(FSNN) topology for a typical classification problem. Each layer uses variouscomputational elements to manage the information flow and implement fuzzyreasoning.

3.1 Input Layer

The function of the input neurons is to simply encode feature data into anappropriate frequency range. Spike trains are then generated from the data usinga linear encoding scheme. The encoding scheme takes the frequency data pointsand converts them into an inter-spike interval (ISI) which was then used to createlinear input spike trains.

3.2 Hidden Layer

All of the synapses in the FSNN are dynamic. The dynamic synapse model usedin this research has kinetic equations that describe inactive, active and recoveredstates, see [11] for further information. Each pre-synaptic spike arriving at aparticular time activates a fraction of synaptic resources, which then quicklyinactivate with a time constant of a few milliseconds and then recover slowly.The post-synaptic current is taken to be proportional to the fraction of resourcesin the active state. The post-synaptic membrane potential is calculated usinga leaky-integrate and fire (LIF) passive membrane function. The hidden layerneurons in the FSNN are connected to the input layer using facilitating synapses.Careful choice of parameters in the synapse model determine the type of synapse,again for further details refer to [11]. Figure 2. A shows an example of the Post-synaptic Response (PSR) of a facilitating synapse in response to a spike trainof 20 Hertz, whereas Figure 2. B shows the PSR of a depressing synapse tothe same spike train. Gaussian RFs are placed at every synapse between the

0 500 10000

5

10

15

Time [ms]

V

A. Facilitating

0 500 10000

5

10

15

Time [ms]

V

B. Depressing

Fig. 2. A. Facilitating and B. Depressing Membrane Potentials

input and the hidden neurons. The frequency dependent RFs determine where

6 Cornelius Glackin, Liam McDaid, Liam Maguire, and Heather Sayers

an input frequency fi is in relation to the central operating frequency of theRF FO. The weight is then scaled by an amount ki when calculating the PSR.The response of the synapse to an input frequency is determined by the RF.This whole process relates to the IF (xi is Ai) part of the fuzzy rule, where xi

is the input and Ai represents the RF. In this way the RF makes the synapsefrequency selective in such a way that the numerous parameters in the dynamicsynapse model do not have to be carefully tuned.

The function of each hidden layer neuron is to impose the remaining partof the antecedent fuzzy IF-THEN rule, namely the conjunctive AND. Simplysumming the post-synaptic potentials is tantamount to performing a disjunctiveOR. The collective aim of the RFs connecting to each hidden layer neuron isto only allow spikes to filter through to the output layer when all of the inputfrequencies presented at each synapse are within the Gaussian RFs. This canbe ensured by making the RF excitatory within the RF and inhibitory outside.The excitatory (positive) part of the Gaussian RF scales the weight in the range[0,1]. The inhibitory (negative) part of the RF scales the weight in the range [0,(1 − m)], where m is the number of input neurons. If even one input frequencylies outside the RF, the resultant inhibitory post-synaptic potential will havesufficient magnitude to negate the post-synaptic potentials from all the othersynapses connecting to the same hidden layer neuron. Figure 3 illustrates thisprocess.

Fig. 3. The conjunctive AND part of the fuzzy rule using excitatory/inhibitory RFs

3.3 Output Layer

Depressing synapses model the fact that biological neurons often only respond tothe first few spikes in a spike train before running out of synaptic resources. Thistype of synapse is illustrated by Figure 2. B. The action potential with thesetypes of synapses is only significantly high in magnitude for a very short interval.In this way, this type of synapse can be described as being a coincidence detector.The aim of the depressing synapses connecting the hidden layer to the outputlayer is to produce spikes in the output in a regimented stable manner. It is thenthe task of the ReSuMe supervised learning algorithm [10] to associate the hidden

Implementing Fuzzy Reasoning on a Spiking Neural Network 7

layer neurons to the output layer neurons. Thus performing the fuzzy inferencingbetween the hidden layer (antecedents), and the output layer (consequents).

With ReSuMe weights are changed whenever there is a correlation (or lack ofcorrelation) between the spike frequency coming from the hidden layer neurons(analogous to the ’rule firing strength’) and the target teacher signal at a partic-ular class output neuron. Refer to [10] for a detailed explanation of the ReSuMealgorithm. In summary, the presented FSNN topology provides a rationale forthe use of RFs, excitatory and inhibitory neurons, as well as facilitating anddepressing synapses. The next section of the paper describes how such a net-work may be used to solve the complex non-linear Iris benchmark classificationproblem [14].

4 Iris Classification Results

The Iris classification problem [14] is well-known in the field of pattern recogni-tion. The data-set contains 3 classes of 50 types of Iris plant. The 3 species ofplant are Iris Setosa, Iris Versicolour, and Iris Virginica. In the interests of claritythe three classes shall be referred to as class 1, class 2, and class 3 respectively.Class 1 is linearly separable from classes 2 and 3. Classes 2 and 3 are not linearseparable and make Iris a complex non-linear classification problem.

The Iris data was sorted into two sets, one of 90 samples and the second of60, to be used for training and testing respectively. Each set contained an equalnumber of instances of each class, 30 of each in the training set and 20 of eachin the testing. The Iris training data was normalized and then linearly scaledinto the frequency range [10, 40] (Hertz). The scaled data was then convertedinto ISIs and the linear input spikes were generated. A sample length of 1 secondwas used to encode each data point, which adequately satisfies Nyquist samplingcriteria. For best results, introducing padding (samples with no spikes) ensuresthat the order of the data does not affect the filtering by the hidden layer.

4.1 Positioning Receptive Fields

The task of determining the number, position and spread of RFs is an importantstep in tuning a fuzzy system. This is because the maximum number of possiblerules governing the fuzzy inferencing process is determined by R = IM where R

is the number of rules, M is the number of membership functions and I is thenumber of inputs. For the Iris classification task there are 4 inputs (4 features),therefore the number of rules (number of hidden layer neurons) is given byR = 4M . The number of possible rules grows exponentially as the number ofmembership functions increases, this phenomenon is known as ’rule explosion’.There are many methodologies for optimising RF placement [15]. For this datasetFuzzy C-Means (FCM) clustering was used to calculate the optimum positionsfor RF placement.

Fuzzy clustering is distinct from hard clustering algorithms such as K-meansin that a data point may belong to more than one cluster at the same time. FCM

8 Cornelius Glackin, Liam McDaid, Liam Maguire, and Heather Sayers

clustering was first developed by Dunn in 1973 [16]. FCM is better at avoidinglocal minima than K-means but it is still susceptible to the problem in somecases.

The Iris training set was clustered with FCM in a five-dimensional sense (4features and the class data). Careful selection of appropriate cluster widths canensure, where possible, that hidden layer clusters are associated with a singleclass. Matlab’s FCM algorithm was used to perform the clustering. The FCMprogram returns the cluster positionss and a fuzzy membership matrix for allthe data samples in the training set. By setting appropriate thresholds on thefuzzy memberships to each cluster, it is possible to determine which data samplesshould be within each cluster and which should be excluded. Once the thresh-olding has determined which data samples should belong to each cluster, thevariances for each feature can be calculated from the features of each includeddata sample. These feature variances are then used to specify the widths of eachGaussian RF. Figure 5 shows the output of the hidden layer in response to thetraining data. As can be seen from Figure 6 there are 3, 3 and 4 clusters (and

Fig. 4. Hidden Layer Output

hence hidden neurons) associated with classes 1, 2 and 3 respectively. The origi-nal ordering of the training data is used for clarity, with each successive 60 secondinterval corresponding to classes 1, 2 and 3. The thresholding technique couldnot exclusively associate hidden neurons 5 and 6 to class 2. Similarly, hiddenneurons 9 and 10 could not be exclusively associated with class 3 without thehidden layer excluding some points all together. In all there were 9 data points(10% of the training set) that were not uniquely associated with one particularclass. This means that 10% of the data in the training set is unresolved in termsof the FSNN associating the data correctly with the class data. It is the task ofthe ReSuMe supervised training regime to resolve this error.

Implementing Fuzzy Reasoning on a Spiking Neural Network 9

4.2 ReSuMe Training

The first step in implementing the ReSuMe training was to specify the super-visory spike trains to be used. The ReSuMe algorithm was ’forgiving’ in thisrespect producing good convergence for a wide range of supervisory frequen-cies. Therefore supervisory spike trains of a 50 Hertz frequency were deliveredto the appropriate class output neuron whenever the data sample belonged tothat class. The supervisory signal for the other class neurons for the given datasample was of zero frequency. The learning windows were equal and opposite inmagnitude producing a maximum possible weight update of 0.05.

Each training data sample is deemed correctly classified when it producesthe maximum number of output spikes at the correct output class neuron. Thetraining error smoothly converges to the minimum training error in as few as6 epochs. The training resulted in 2/90 misclassified data samples (97.78% ac-curacy), improving the FCM clustering accuracy by 7.33% . Once training wascompleted, all weights were then fixed and the unseen testing data (60 sam-ples) were presented to the network. During the testing phase 3/60 data pointswere misclassified (95% accuracy), showing good generalisation. Table 2 com-pares the training results for FSNN against some other well-known supervisedlearning algorithms for the Iris classification task. As can be seen from the table,

Table 2. Comparison of FSNN to other Supervised Learning Algorithms

Algorithm Training Testing

SpikeProp 97.4% 96.1%Matlab BP 98.2% 95.5%Matlab LM 99.0% 95.7%Weight Limit Learning 100% 96.6%FSNN 97.8% 95.0%

FSNN compares well with the other algorithms. It may be possible to improveon the accuracy of the FSNN algorithm in any number of ways. More hiddenlayer neurons and different techniques are among the multitude of modificationsto the FSNN that could be attempted to improve the accuracy. However, for apreliminary attempt the results are encouraging.

5 Conclusions

This paper presents an FSNN topology which was demonstrated to producecomparable results to existing ANN and SNN techniques for the Iris classificationproblem. The FSNN topology presented provides a rationale for the assembly ofbiological components such as excitatory and inhibitory neurons, facilitating anddepressing synapses, and receptive fields. In particular, the paper demonstrateshow receptive fields may be configured in terms of excitation and inhibition toimplement the conjunctive AND of the antecedent part of a fuzzy rule. Currently,

10 Cornelius Glackin, Liam McDaid, Liam Maguire, and Heather Sayers

the work relies on the use of FCM clustering as a strategy for combating therule-explosion phenomenon of fuzzy logic systems. Ideally, it would be preferablefor the FSNN to determine the number, placement and spread of fuzzy clusterswithout relying on an external statistical or clustering technique. For this reason,future work in recognition of the growing interest in biological receptive fields,will involve the development of dynamic receptive fields. The ultimate aim is todevelop a biologically plausible FSNN that tunes itself using strictly biologicalprinciples, and in that regard this work represents a significant first step.

References

1. Maass, W.: Networks of Spiking Neurons: The Third Generation of Neural NetworkModels. In: Electronic Colloquium on Computational Complexity (ECCC), vol. 3,no. 31, (1996)

2. Zadeh, L.A.: Fuzzy sets. Information and Control, vol. 8 no. 3, pp. 338.353 (1965)3. Abbott, L. F., Nelson, S. B.: Synaptic plasticity: taming the beast. Nature Neuro-

science, vol. 2, pp. 1178-1183 (2000)4. Bohte, S. M., Kok, J. N., La Poutre, H.: Error-backpropagation in Temporally En-

coded Networks of Spiking Neurons. Neurocomputing, vol. 48, pp. 17-37 (2002)5. Belatreche, A., Maguire, L. P., McGinnity, T. M., Wu, Q. X.: A Method for Su-

pervised Training of Spiking Neural Networks. In: IEEE Cybernetics Intelligence .Challenges and Advances (CICA), pp. 39-44 (2003)

6. Sougne, J. P.: A learning algorithm for synfire chains. Connectionist Models ofLearning, Development and Evolution, pp. 23-32 (2001)

7. Ruf, B., Schmitt, M.: Learning temporally encoded patterns in networks of spikingneurons. Neural Processing Letters, vol. 5, no. 1. pp. 9-18 (1997)

8. Carnell, A., Richardson, D.: Linear algebra for time series of spikes. In: 13th Euro-pean Symposium on Artificial Neural Networks (ESANN), (2005)

9. Pfister, J. P., Barber, D., Gerstner, W.: Optimal Hebbian Learning: A ProbabilisticPoint of View. In: ICANN/ICONIP Lecture Notes in Computer Science, vol. 2714,pp. 92-98 (2003)

10. Kasinski, A., Ponulak, F.: Comparison of Supervised Learning Meth-ods for Spike Time Coding in Spiking Neural Networks. Available:http://matwbn.icm.edu.pl/ksiazki/amc/amc16/amc1617.pdf (2005)

11. Tsodyks, M., Pawelzik, K., Markram, H.: Neural Networks with Dynamic Synapses.Neural Computation, vol. 10, no. 4, pp. 821-835 (1998)

12. Natschlager, T., Maass, W., Zador, A.: Efficient temporal processing with biologi-cally realistic dynamic synapses. Network: Computation in Neural Systems, vol. 12,pp. 75-87 (2001)

13. Barlow, H. B.: Summation and inhibition in the frogfs retina. J. Physiol., vol. 119,pp. 69-88 (1953)

14. Fischer, R. A.: The Use of Multiple Measurements in Taxonomic Problems. Annalsof Eugenics, vol. 7, pp. 179-188 (1936)

15. Abdelbar, A. M., Hassan, D. O., Tagliarini, G. A., Narayan, S.: Receptive FieldOptimisation for Ensemble Encoding. Neural. Comput. & Applic., vol. 15, no. 1,pp. 1-8 (2006)

16. Dunn, J.: A fuzzy relative of the ISODATA process and its use in detecting compactwellseparated clusters. Journal of Cybernetics, vol. 3, pp. 32-57 (1973)