D.-S. Huang et al. (Eds.): ICIC 2009, LNAI 5755, pp. 21–29, 2009. © Springer-Verlag Berlin Heidelberg 2009

Spiking Neural Network Performs Discrete Cosine Transform for Visual Images

Qingxiang Wu, T.M. McGinnity, Liam Maguire, Arfan Ghani, and Joan Condell

Intelligent Systems Research Centre, University of Ulster at Magee Campus Derry, BT48 7JL, Northern Ireland, UK

{q.wu,tm.mcginnity,lp.maguire,a.ghani,j.condell}@ulster.ac.uk

Abstract. The human visual system demonstrates powerful image processing functionalities. Inspired by the principles from neuroscience, a spiking neural network is proposed to perform the discrete cosine transform for visual im-ages. The structure and the properties of the network are detailed in this paper. Simulation results show that the network is able to perform the discrete cosine transform for visual images. Based on this mechanism, the key features can be extracted in ON/OFF neuron arrays. These key features can be used to recon-struct the visual images. The network can be used to explain how the spiking neuron-based system can perform key feature extraction. The differences be-tween the discrete cosine transform and the spiking neural network transform are discussed.

Keywords: spiking neural networks; visual system; discrete cosine transform; visual image.

1 Introduction

The human visual system demonstrates powerful image processing functionalities. The retina contains complex circuits of neurons that extract salient information from visual inputs. Signals from photoreceptors are processed by retinal interneurons, inte-grated by retinal ganglion cells and sent to the brain by axons of retinal ganglion cells. Different cells respond to different visual features, such as light intensity, colour or moving objects [1–5]. Mammalian retinas contain approximately 55 distinct cell types, each with a different function [1]. The exact neuronal circuits for extraction of key features and identification of various visual images still need to be determined. Similarly, there is a need to simulate the neuronal circuits in electronic circuits as a precursor to their application in artificial intelligent systems. These are still open questions. The Discrete Cosine Transform (DCT) [6] is an efficient approach for key feature extraction in the image processing domain. Based on the spiking neuron model [7-12], a neuronal circuit is proposed to perform discrete cosine transform and explain how a spiking neural network can extract key features of a visual image. The neuronal circuit has been simulated using Matlab and the results show that the key features of a visual image can be extracted by the neural network and the visual image can be reconstructed using the key features.

22 Q. Wu et al.

The remainder of this paper is organized as follows. In Section 2, the architecture of the neural network is proposed to extract the key features and the network is shown to be able to reconstruct the visual image. The network model is based on simplified conductance-based integrate-and-fire neurons. The behaviors of the neural network are governed by a set of equations discussed in Section 3. Simulation results are pre-sented in Section 4. The network performance is discussed in Section 5.

2 Spiking Neural Network Model for Discrete Cosine Transform

The human visual system performs feature extraction very efficiently. Neuroscientists have found that there are various receptive fields from simple cells in the striate cor-tex to those of the retina and lateral geniculate nucleus [13]. The different pathways, which are composed of different types of neurons, play different roles in the visual system. ON/OFF pathways [3] were found in the visual system. Inspired by the mechanism of ON/OFF pathways, a spiking neural network model is proposed to perform the discrete cosine transform as shown in Fig. 1.

OFF neuron array

N

M

W N(m n) OFF(p q)

ON neuron array W OFF(p,q) RN(m,n)W ON (p,q) RN(m,n)

N

M

PP W N(m,n) ON (p,q)

Input neuron array

Reconstruction neuron array

Excitatory synapse

Inhibitory synapse

Fig. 1. Spiking neural network for feature extraction and image reconstruction

Spiking Neural Network Performs Discrete Cosine Transform for Visual Images 23

Suppose that each neuron in the input neuron array generates spikes induced by a synapse current of photonic receptor according to the corresponding pixel brightness in a visual image. The dimension of the input neuron array is M×N. A neuron in the array is labeled with N(m, n), where m=0,…,M-1 and n=0,…,N-1. Each pixel of the image corresponds to a receptor. The intermediate layer is composed of two neuron arrays; one is the ON neuron array and other is the OFF neuron array. The ON/OFF neuron arrays have the same dimension P×Q, where P≤M and Q≤N. Neurons in the ON/OFF neuron arrays are labeled with ON(p, q) and OFF(p, q), where p=0,…,P-1 and q=0,…,Q-1. Each neuron in the ON/OFF arrays receives spike trains from all neurons in the input array through excitatory synapses with specific strength distribu-tions ( , ) ( , )N m n ON p qW → for ON neurons and ( , ) ( , )N m n OFF p qW → for the OFF neurons.

Based on the principle of the Discrete Cosine Transform (DCT)[6], synapse strength distribution can be set as follows.

( , ) ON( , )

(2 1) (2 1) (2 1) (2 1)cos cos , cos cos 0

2 2 2 2(2 1) (2 1)

0, cos cos 02 2

p q

N m n p q

m p n q m p n qif

M N M NWm p n q

ifM N

π π π πα α

π π→

+ + + +⎧ >⎪⎪= ⎨ + +⎪ ≤⎪⎩

(1)

( , ) OFF( , )

(2 1) (2 1) (2 1) (2 1)cos cos , cos cos 0

2 2 2 2(2 1) (2 1)

0, cos cos 02 2

p q

N m n p q

m p n q m p n qif

M N M NWm p n q

ifM N

π π π πα α

π π→

+ + + +⎧− <⎪⎪= ⎨ + +⎪ ≥⎪⎩

(2)

where 0 1, and 0 1p M q N≤ ≤ − ≤ ≤ − ,

1 / , 0

2 / ,1 1p

M p

M p Mα

⎧ =⎪= ⎨≤ ≤ −⎪⎩

,1 / , 0

2 / ,1 1q

N q

N q Mα

⎧ =⎪= ⎨≤ ≤ −⎪⎩

(3)

If less neurons are used in the ON/OFF neuron arrays, i.e. P<<M and Q<<N, only key features remain. Using these key features the visual image can still be recon-structed. The reconstruction neuron layer is used to demonstrate reconstruction of the visual image using these key features. Suppose that all neurons in the ON array con-nect to a neuron in the reconstruction neuron array with excitatory synapses with specific strength distribution ( , ) ( , )ON m n RN p qW → and all neurons in the OFF array con-

nect to the same neuron with inhibitory synapses with specific strength distribution

( , ) ( , )OFF m n RN p qW → . Fig.1 demonstrates a neuron in the reconstruction array receiving

spikes through excitatory synapse from an ON neuron and through inhibitory synapse from an OFF neuron. The strength distribution is set as follows.

( , ) ( , ) ( , ) ( , )ON p q RN m n N m n ON p qW W→ →= (4)

( , ) ( , ) ( , ) ( , )OFF p q RN m n N m n OFF p qW W→ →= (5)

If the key features appear in the ON/OFF neuron arrays, the image will be recon-structed in the reconstruction neuron array. A colour image can be dealt with similar networks, in three chanels corresponding to R, G, B colours.

24 Q. Wu et al.

3 Spiking Neuron Model and Simulation Algorithms

Considering computational cost, a simplified conductance-based integrate-and-fire neuron model is used to simulate the network. Let Gm,n(t) represent the gray scale of an image pixel at a point (m, n) at time t. Suppose that each photonic receptor trans-fers the image pixel brightness to a synapse current Im,n(t). The synapse current can be represented as follows:

,, ,

( ) 1( ) ( ).m n

m n x y

dI tI t G t

dtα

τ= − + (6)

where α is a constant for transformation from pixel value to current, and τ is a time constant for decay of the synapse current. According to the integrate-and-fire neuron model [6-11], the potential of a neuron N(m, n) is governed by the following equation:

,, , 0

( )( ( )) ( )m n

l l m n m n

dv tc g E v t I t I

dt= − + + , (7)

where lg is the membrane conductance, El is the reverse potential, , ( )m nv t is the

membrane potential of the neuron N(m, n), c represents a capacitance of the mem-brane, and I0 is simulated by an average current produced by background noise. If the membrane potential passes threshold vth, then the neuron generates a spike. Let Sm,n(t) represent the spike train generated by the neuron such as that:

,

1 ( , ) .( )

0 ( , ) .m n

if neuron m n fires at timetS t

if neuron m n does not fire at timet

⎧= ⎨⎩

(8)

Through equations (6)-(8), gray scale Gm,n(t) has been transferred to spike trains. These spike trains are transferred to the ON/OFF neuron arrays in the intermediate layer through excitatory synapses with strength distributions ( , ) ( , )N m n ON p qW → and

( , ) ( , )N m n OFF p qW → . As each neuron in the ON/OFF arrays receives all spikes from the

top neuron array, the total synapse currents ( , ) ( )ON p qI t and ( , ) ( )OFF p qI t , which are

caused by the spikes, can be represented as follows.

1 1( , )

( , ) ( , ) ( , ) ,0 0

( ) 1( ) ( )

M NON p q

ON p q N m n ON p q m nm n

dI tI t W S t

dtβ

τ

− −

→= =

= − + ∑∑ , (9)

1 1

( , )( , ) ( , ) ( , ) ,

0 0

( ) 1( ) ( )

M NOFF p q

OFF p q N m n OFF p q m nm n

dI tI t W S t

dtβ

τ

− −

→= =

= − + ∑∑ , (10)

where β is a constant. The neuron potential ( , ) ( )ON p qv t in the ON array is governed by

the following equation:

( , )( , ) ( , ) 0

( )( ( )) ( )ON p q

l l ON p q ON p q

dv tc g E v t I t I

dt= − + + . (11)

Spiking Neural Network Performs Discrete Cosine Transform for Visual Images 25

Potential ( , ) ( )OFF p qv t in the OFF array is governed by following equation.

( , )( , ) ( , ) 0

( )( ( )) ( )OFF p q

l l OFF p q OFF p q

dv tc g E v t I t I

dt= − + + . (12)

Let SON(p,q)(t) represent a spike train which is generated by the ON neurons and SOFF(p,q)(t) represent a spike train which is generated by the OFF neurons such that:

( , )

1 ( , ) .( )

0 ( , ) .ON p q

if ON neuron p q fires at timetS t

if ON neuron p q does not fire at timet

⎧= ⎨⎩

(13)

( , )

1 ( , ) .( )

0 ( , ) .OFF p q

if OFF neuron p q fires at timetS t

if OFF neuron p q does not fire at timet

⎧= ⎨⎩

(14)

For the reconstruction neuron array (bottom layer), each neuron RN(m,n) receives spikes from all neurons in the ON array through excitatory synapses and receives spikes from all neurons in the OFF array through inhibitory synapses. The synapse current to neuron RN(m, n) is as follows:

11( , )

( , ) ( , ) ( , ) ( , )0 0

11

( , ) ( , ) ( , )0 0

( ) 1( ) ( )

( ),

QPRN m n

RN m n ON p q RN m n ON p qp q

QP

OFF p q RN m n OFF p qp q

dI tI t W S t

dt

W S t

γτ

γ

−−

→= =

−−

→= =

= − +

−

∑∑

∑∑

(15)

where γ is a constant. The membrane potential of the neuron RN(m,n) is governed by the following equation:

( , )( , ) ( , ) 0

( )( ( )) ( )RN m n

l l RN m n RN m n

dv tc g E v t I t I

dt= − + + . (16)

Let SRN(m,n)(t) represent spike train generated by the neuron RN(m,n) in the bottom layer. The firing rate for the neuron RN(m,n) is calculated by the following expression:

( , ) ( , )1

( ) ( )t T

RN m n RN m nt

r t S tT

+= ∑ . (17)

A firing rate map can be obtained by plotting rRN(m,n)(t). Let rmax represent the maxi-mum firing rate in the neuron array. The reconstructed image Rm,n(t) with 255 gray scale levels is obtained using the following equation:

, ( , )max

255( ) ( )m n RN m nR t r t

r= . (18)

4 Simulation Results

This network is simulated in Matlab using the Euler method with a time step of 0.1 ms. Corresponding to biological neurons [9], the following parameters for the

26 Q. Wu et al.

(a) (b)

Fig. 2. Images to test network: (a)Gaussian grayscale distribution; (b)Lena image

network were used in the experiments. vth = -60 mv. El= -70 mv. gl =1.0 μs/mm2. c=8 nF/mm2. τ = 16ms. T=400ms. These parameters can be adjusted to get a good quality output image. α=0.02. β=0.13. γ=5.1. I0=7µA. These parameters can be determined by trial/error method according to single neuron behaviours.

In order to demonstrate the behavior of the network, two images (as shown in Fig.2) are presented to the network. Fig.2(a) is a black point with a Gaussian gray scale distribution in a 16×16 area (shown with increased pixel size for clarity). Fig. 2(b) is the Lena image which is widely used as a benchmark image in the image processing domain. When the Gaussian black point presented to the network for a period 400ms, the firing rate maps of top-right 8×8 neurons in the ON/OFF neuron arrays are shown in Fig.3(a) and Fig.3(b). If these firing patterns go to the bottom neuron array through excitatory and inhibitory synapses shown in Fig.1, a firing rate map of the reconstruction array is obtained in Fig.3(c). Transformed the firing rate map to a gray scale image by Equation (18), the result is obtained in Fig.3(d). This means that the key feature of the Gaussian black point in dimension 16×16 can be preserved by the firing patterns of the ON/OFF neuron arrays in dimension 8×8. If the dimension of the ON/OFF array is the same as the N neuron array (top layer), almost the same black point can be obtained in the construction image shown in Fig.3(e). If less neurons are used to preserve the pattern in the ON/OFF arrays (e. g. only 4×4 neurons in the ON/OFF arrays are used), a black point will be obtained in the recon-struction image Fig.3(f). It is more vague than Fig.3(d), however the key features still remain. This example shows that the key features can be preserved by a small number of top-right neurons in the ON/OFF arrays.

In order to demonstrate this behaviour further the network was used to transfer the Lena image (see Fig. 2(b)). In our experiments, the Lena image (512×512) has been divided into 32×32 blocks. Each block contains 16×16 pixels. Each block is trans-ferred to ON/OFF firing patterns with only 4×4 neurons, and then the firing pattern is transferred to the reconstructed image blocks. Reconstructed image blocks are put together to form a fully reconstructed image as shown in Fig. 4(a). This image is comparable with the DCT reconstructed image in Fig. 4(b) with a 4×4 mask filter. Comparing Fig. 4(a) to the original Lena image in Fig. 2(b) the deference errors can be computed as shown in Fig. 4(c) which is also comparable with the DCT errors shown in Fig. 4(d).

Spiking Neural Network Performs Discrete Cosine Transform for Visual Images 27

(e)(d) (f)

(a)ONneuronarray (b)OFFneuron array (c)Firing rate of reconstruction

Fig. 3. Firing rate in neuron arrays (a-c) and reconstructed images (d-f)

(c) (d)

(a) (b)

Fig. 4. Comparison between SNN network and DCT results

28 Q. Wu et al.

5 Discussion

Using ON/OFF neuron pathways, a spiking neural network is proposed to extract features from a visual image. An integrate-and-fire spiking neuron model and a syn-apse current mechanism are used to construct and simulate the network. The experi-mental results show that the network can perform the image transformation similar to the discrete cosine transform and can be used to extract key features in visual images. In the simulations the network is assumed to be implemented in a fully parallel net-work like biological system. The results can be achieved by presenting test images to the network for 400ms. If the hardware implementation of the network can reach the speed as same as biological neuron network, the key feature extraction can be com-pleted in 400ms, and this network can be applied to spiking neuron-based artificial intelligent systems to extract key features from visual images. However, there are still some issues for discussion. (1) It is very efficient for the brain to extract key features from a visual image. This network can be used to explain how the spiking neuron-based network is capable of extracting key features in the manner of the proposed network. Further research is required to establish the actual mechanisms employed by the visual cortex. (2) In this network, the information is transferred based on spike trains. Using such a biological information transfer manner, the network is more ro-bust to noise. For example, the average noise current has been considered in the simu-lation. However it is very difficult to exactly perform the inverse transform as in mathematical discrete cosine transforms. (3) In this paper, the synapse strength dis-tributions are set based on principles taken from discrete cosine transform theory. If it is considered to use the spike timing dependent plasticity of synapses, it is possible to train the network to obtain the synapse strength using sample visual images. This is a topic for further study.

References

1. Masland, R.H.: The Fundamental Plan of the Retina. Nature Neurosci. 4, 877–886 (2001) 2. Wassle, H.: Parallel Processing in the Mammalian Retina. Nature Rev. Neurosci. 5,

747–757 (2004) 3. Nelson, R., Kolb, H.: On and Off Pathways in the Vertebrate Retina and Visual System.

In: Chalupa, L.M., Werner, J.S. (eds.) The Visual Neurosciences, pp. 260–278. MIT Press, Cambridge (2003)

4. Demb, J.B.: Cellular Mechanisms for Direction Selectivity in the Retina. Neuron 55, 179–186 (2007)

5. Taylor, W.R., Vaney, D.I.: New Directions in Retinal Research. Trends Neurosci. 26, 379–385 (2003)

6. Ahmed, N., Natarajan, T., Rao, K.R.: Discrete Cosine Transform. IEEE Trans. Computers, 90–93 (1974)

7. Koch, C.: Biophysics of Computation: Information Processing in Single Neurons. Oxford University Press, Oxford (1999)

8. Dayan, P., Abbott, L.F.: Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems. The MIT Press, Cambridge (2001)

9. Gerstner, W., Kistler, W.: Spiking Neuron Models: Single Neurons, Populations, Plastic-ity. Cambridge University Press, Cambridge (2002)

Spiking Neural Network Performs Discrete Cosine Transform for Visual Images 29

10. Müller, E.: Simulation of High-Conductance States in Cortical Neural Networks. Masters thesis, University of Heidelberg, HD-KIP-03-22 (2003)

11. Wu, Q.X., McGinnity, T.M., Maguire, L.P., Glackin, B., Belatreche, A.: Learning Mecha-nism in Networks of Spiking Neurons. In: Studies in Computational Intelligence, vol. 35, pp. 171–197. Springer, Heidelberg (2006)

12. Wu, Q., McGinnity, T.M., Maguire, L.P., Belatreche, A., Glackin, B.: Adaptive co-ordinate transformation based on a spike timing-dependent plasticity learning paradigm. In: Wang, L., Chen, K., S. Ong, Y. (eds.) ICNC 2005. LNCS, vol. 3610, pp. 420–428. Springer, Heidelberg (2005)

13. Kandel, E.R., Shwartz, J.H.: Principles of Neural Science. Edward Amold Publishers Ltd., London (1981)