Pattern Recognition for Industrial Security using theFuzzy Sugeno Integral and Modular Neural Networks

Patricia Melin, Alejandra Mancilla, Miguel Lopez, Daniel Solano, Miguel Soto,Oscar Castillo

Department of Computer ScienceTijuana Institute of Technology

Tijuana, Mexico, pmelin@tectijuana.mx

Abstract. We describe in this paper the evolution of modular neural networksusing hierarchical genetic algorithms for pattern recognition. Modular NeuralNetworks (MNN) have shown significant learning improvement over singleNeural Networks (NN). For this reason, the use of MNN for pattern recognitionis well justified. However, network topology design of MNN is at least an orderof magnitude more difficult than for classical NNs. We describe in this paperthe use of a Hierarchical Genetic Algorithm (HGA) for optimizing the topologyof each of the neural network modules of the MNN. The HGA is clearly neededdue to the fact that topology optimization requires that we are able to manageboth the layer and node information for each of the MNN modules. Simulationresults prove the feasibility and advantages of the proposed approach.

Keywords: Evolution, Neural Networks, Pattern Recognition, Biometrics

1 Introduction

Today, a variety of methods and techniques are available to determine uniqueidentity, the most common being fingerprint, voice, face, and iris recognition. Ofthese, fingerprint and iris offer a very high level of certainty as to a person's identity,while the others are less exact. A large number of other techniques are currently beingexamined for suitability as identity determinants. These include (but are not limitedto) retina, gait (walking style), typing style, body odour, signature, hand geometry,and DNA. Some wildly esoteric methods are also under development, such as earstructure, thermal imaging of the face and other parts of the body, subcutaneous veinpatterns, blood chemistry, anti-body signatures, and heart rhythm, to name a few.

The four primary methods of biometric authentication in widespread use today areface, voice, fingerprint, and iris recognition. All of these are supported in ourapproach, some more abundantly than others. Generally, face and voice areconsidered to be a lower level of security than fingerprint and iris, but on the otherhand, they have a lower cost of entry. We describe briefly in this section some ofthese biometric methods.

2 Patricia Melin, Alejandra Mancilla, Miguel Lopez, Daniel Solano, Miguel Soto, OscarCastillo

Face Recognition. Facial recognition has advanced considerably in the last 10 to 15years. Early systems, based entirely on simple geometry of key facial referencepoints, have given way to more advanced mathematically-based analyses such asLocal Feature Analysis and Eigenface evaluation. These have been extended thoughthe addition of "learning" systems, particularly neural networks.

Face recognition systems are particularly susceptible to changes in lightingsystems. For example, strong illumination from the side will present a vastly differentimage to a camera than neutral, evenly-positioned fluorescent lighting. Beyond this,however, these systems are relatively immune to changes such as weight gain,spectacles, beards and moustaches, and so on. Most manufacturers of face recognitionsystems claim false accept and false reject rates of 1% or better.Voice Recognition. Software systems are rapidly becoming adept at recognising andconverting free-flowing speech to its written form. The underlying difficulty in doingthis is to flatten out any differences between speakers and understand everyoneuniversally. Alternatively, when the goal is to specifically identify one person in alarge group by their voice alone, these very same differences need to be identified andenhanced.

As a means of authentication, voice recognition usually takes the form of speakinga previously-enrolled phrase into a computer microphone and allowing the computerto analyse and compare the two sound samples. Methods of performing this analysisvary widely between vendors. None is willing to offer more than cursory descriptionsof their algorithms--principally because, apart from LAN authentication, the largestmarket for speaker authentication is in verification of persons over the telephone.Fingerprint Recognition. The process of authenticating people based on theirfingerprints can be divided into three distinct tasks. First, you must collect an imageof a fingerprint; second, you must determine the key elements of the fingerprint forconfirmation of identity; and third, the set of identified features must be comparedwith a previously-enrolled set for authentication. The system should never expect tosee a complete 1:1 match between these two sets of data. In general, you could expectto couple any collection device with any algorithm, although in practice most vendorsoffer proprietary, linked solutions.

A number of fingerprint image collection techniques have been developed. Theearliest method developed was optical: using a camera-like device to collect a high-resolution image of a fingerprint. Later developments turned to silicon-based sensorsto collect an impression by a number of methods, including surface capacitance,thermal imaging, pseudo-optical on silicon, and electronic field imaging.

As discussed, a variety of fingerprint detection and analysis methods exist, eachwith their own strengths and weaknesses. Consequently, researchers vary widely ontheir claimed (and achieved) false accept and false reject rates. The poorest systemsoffer a false accept rate of around 1:1,000, while the best are approaching1:1,000,000. False reject rates for the same vendors are around 1:100 to 1:1000.Iris Recognition. Iris recognition is based entirely on a concept originated by Drs.Leonard Flom and Aran Safir, and a software process developed by Dr. JohnDaugman, all of Cambridge University, England. US Patent 5,291,560 issued in thename of Daugman has been assigned to Iridian Corp., one of the world's principalcompanies of iris-based systems. Extensive research has determined that the human

Pattern Recognition for Industrial Security using the Fuzzy Sugeno Integral and ModularNeural Networks 3

iris is essentially unchanged in structure and appearance from the eighth month ofgestation until a few minutes after death. Although a neonatal eye can be darklycoloured until a few months after birth, this darkness is not an influence in theinfrared wavelengths normally used to collect an iris image.

In the history of iris recognition, there has never been a false acceptance. In fact,the equal error rate is 1:1,200,000, with a typical false accept rate of 1:100,000,000and false reject rate of 1:200,000. Note that these are theoretical values based onstrong analysis of limited data (only 5 to 10 million iris scans have ever beenperformed); they also do not take into account the perceived level of difficulty inusing the system. Overall, iris scanning is the system to use if you are concernedabout strongly authenticating users.

2 Proposed Approach for Recognition

Our proposed approach for human recognition consists in integrating the informationof the three main biometric parts of the person: the voice, the face, and the fingerprint.Basically, we have an independent system for recognizing a person from each of itsbiometric information (voice, face, and fingerprint), and at the end we have anintegration unit to make a final decision based on the results from each of themodules. In Figure 1 we show the general architecture of our approach in which it isclearly seen that we have one module for voice, one module for face recognition, andone module for fingerprint recognition. At the top, we have the decision unitintegrating the results from the three modules.

Fig. 1. Architecture of the proposed approach.

4 Patricia Melin, Alejandra Mancilla, Miguel Lopez, Daniel Solano, Miguel Soto, OscarCastillo

3 Modular Neural Networks

This section describes a particular class of "modular neural networks", which have ahierarchical organization comprising multiple neural networks; the architecturebasically consists of two principal components: local experts and an integration unit,as illustrated in Figure 2. In general, the basic concept resides in the idea thatcombined (or averaged) estimators may be able to exceed the limitation of a singleestimator. The idea also shares conceptual links with the "divide and conquer"methodology. Divide and conquer algorithms attack a complex problem by dividing itinto simpler problems whose solutions can be combined to yield a solution to thecomplex problem. When using a modular network, a given task is split up amongseveral local experts NNs. The average load on each NN is reduced in comparisonwith a single NN that must learn the entire original task, and thus the combined modelmay be able to surpass the limitation of a single NN. The outputs of a certain numberof local experts (Oi) are mediated by an integration unit. The integrating unit puts theoutputs together using estimated combination weights (gi). The overall output Y is

Yi = Σgi OI (1)

Nowlan, Jacobs, Hinton, and Jordan [7] described modular networks from acompetitive mixture perspective. That is, in the gating network, they used the"softmax" function, which was introduced by McCullagh and Nelder [9]. Moreprecisely, the gating network uses a softmax activation gi of ith output unit given by

Gi = exp (kui)/ Σj exp (kuj) (2)

Fig. 2. Architecture of a modular neural network.

Where ui is the weighted sum of the inputs flowing to the ith output neuron of thegating network. Use of the softmax activation function in modular networks providesa sort of "competitive" mixing perspective because the ith local expert's output Oi witha minor activation ui does not have a great impact on the overall output Yi.

Module1

Module2

Module

GatingNetwork

∑y=f(x)

…

1y

2y

Ny

1g

2g

Ng

Pattern Recognition for Industrial Security using the Fuzzy Sugeno Integral and ModularNeural Networks 5

4 Genetic Algorithm for Neural Network Optimization

The bottleneck problem for NN application lies within the optimization proceduresthat are used to obtain an optimal NN topology. Hence, the formulation of theHierarchical Genetic Algorithm (HGA) is applied for this purpose [10]. The HGAdiffers from the standard GA with a hierarchy structure in that each chromosomeconsists of multilevel genes. Each chromosome consists of two types of genes, i.e.control genes and connection genes. The control genes in the form of bits, are thegenes for layers and neurons for activation. The connection genes, a real valuerepresentation, are the genes for connection weightings and neuron bias.

With such a specific treatment, a structural chromosome incorporates both activeand inactive genes. It should be noted that the inactive genes remain in thechromosome structure and can be carried forward for further generations. Such aninherent genetic variation in the chromosome avoids any trapping at local optima,which has the potential to cause premature convergence. Thus it maintains a balancebetween exploiting its accumulated knowledge and exploring the new areas of thesearch space. This structure also allows larger genetic variations in chromosomewhile maintaining high viability by permitting multiple simultaneous genetic changes.As a result, a single change in high level genes will cause multiple changes (activationor deactivation in the whole level) in lower level genes. In the case of the traditionalGA, this is only possible when a sequence of many random changes takes place.Hence the computational power is greatly improved.

5 The Problem of Speech Recognition

Speaker recognition, which can be classified into identification and verification, is theprocess of automatically recognizing who is speaking on the basis of individualinformation included in speech waves. This technique makes it possible to use thespeaker's voice to verify their identity and control access to services such as voicedialing, banking by telephone, telephone shopping, database access services,information services, voice mail, security control for confidential information areas,and remote access to computers.

Speaker identification is the process of determining which registered speakerprovides a given utterance. Speaker verification, on the other hand, is the process ofaccepting or rejecting the identity claim of a speaker. Most applications in which avoice is used as the key to confirm the identity of a speaker are classified as speakerverification. Speaker recognition methods can also be divided into text-dependent andtext-independent methods. The former require the speaker to say key words orsentences having the same text for both training and recognition trials, whereas thelatter do not rely on a specific text being spoken. Both text-dependent andindependent methods share a problem however. These systems can be easily deceivedbecause someone who plays back the recorded voice of a registered speaker sayingthe key words or sentences can be accepted as the registered speaker. To cope withthis problem, there are methods in which a small set of words, such as digits, are used

6

as key words and each user is prompted to utter a given sequence of key words that israndomly chosen every time the system is used. Yet even this method is notcompletely reliable, since it can be deceived with advanced electronic recordingequipment that can reproduce key words in a requested order. Therefore, a text-prompted speaker recognition method has recently been proposed by [11, 12].

6 Simulation Results for Voice Recognition

We describe below some simulation results of our approach for speaker recognition.First, in Figure 3 we have the signal of the word "example" in Spanish with noise.Next, in Figure 4 we have the identification of the word "example" without noise. Wealso show in Figure 5 the word "layer" in Spanish with noise. In Figure 6, we showthe identification of the correct word "layer" without noise.

0 1000 2000 3000 4000 5000 6000 7000 8000 9000-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Input signal with noise for recognition of the word "Example" in Spanish

Fig. 3. Input signal of the word "example" in Spanish with noise

0 1000 2000 3000 4000 5000 6000 7000 8000 9000-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Pattern recognition for the word "Example" in Spanish

Fig. 4. Indentification of the word "example".

Pattern Recognition for Industrial Security using the Fuzzy Sugeno Integral and ModularNeural Networks 7

0 1000 2000 3000 4000 5000 6000 7000 8000 9000-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Input signal with noise for recognition of the word "Layer" in Spanish

Fig. 5. Input signal of the word "layer" in Spanish with noise added

0 1000 2000 3000 4000 5000 6000 7000 8000 9000-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Pattern recognition for the word "Layer" in Spanish

Fig. 6. Identification of the word "layer".

The idea behind this data partition is that learning will be easier in each of themodules, i.e. a simple NN can learn more easily the behavior of the function in one ofthe regions. We used three-layer feed-forward NNs for each of the modules with theLevenberg-Marquardt training algorithm.

The fitness function used in this work combines the information the error objectiveand also the information about the number of nodes as a second objective. This isshown in the following equation.

10*2*)1(*

1)(

+

=ObjVObjVRanking

zfβα

(3)

The first objective is basically the average sum of squared of errors as calculatedby the predicted outputs of the MNN compared with real values of the function. Thisis given by the following equation.

8

)(11

1 ii

N

iyY

Nf −= ∑

=

(4)

The parameters of the genetic algorithm are as follows:Type of crossover operator: Two-point crossoverCrossover rate: 0.8Type of mutation operator: Binary mutationMutation rate: 0.05Population size per generation: 10Total number of generations: 100We show in Figure 7 the topology of the final evolved modules of the neural

network for the problem of function identification. As we can appreciate, from thisFigure, module 2 is the smallest one and module 3 is the largest one. The result ofMNN evolution is a particular architecture with different size of the modules (neuralnetworks).

Fig. 7. Topology of the final evolved MNN for pattern recognition.

The MNN architecture shown in Figure 7 is the best one for the specific problemof system identification. It is worthwhile noting that this network topology is difficultto design manually, for this reason the HGA approach is a good choice for neuralnetwork design and optimization.

7 Integration of Results for Person Recognition

We also have developed previously methods for face and fingerprint recognition withmodular neural networks [15] and now we need to integrate the results of the threebiometric aspects of a person, to make a decision on the identity of a person. Weshow in Figure 8 the architecture of the fuzzy system used for this decision process.

Module 1

Module 2

Module 3

Pattern Recognition for Industrial Security using the Fuzzy Sugeno Integral and ModularNeural Networks 9

Fig. 8. Architecture of the fuzzy system for person recognition

We show in Table 1 the complete set of fuzzy rules used in the fuzzy system forperson recognition.

Table 1. Fuzzy rule base for person recognition

Rules Face Fingerprint Voice Decision1 A A A A2 A A B A3 A A C A4 A B A A5 A B B B6 A B C B7 A C A A8 A C B C9 A C C C10 B A A A11 B A B B12 B A C A13 B B A B14 B B B B15 B B C B16 B C A C17 B C B B18 B C C C19 C A A A20 C A B A21 C A C C22 C B A B23 C B B B24 C B C C25 C C A C26 C C B C27 C C C C

10

Experiments with this fuzzy system give 100% accuracy 27 persons with a degreeof noise up to 80% (gaussian noise).

8 Conclusions

We described in this paper our hierarchical genetic algorithm approach for modularneural network topology design and optimization. The proposed approach wasillustrated with a specific problem of pattern recognition, which is voice recognition.The best MNN is obtained by evolving the modules (single NNs) according to theerror of identification and also the complexity of the modules. The results for theproblem of voice recognition are very good and show the feasibility of the HGAapproach for MNN topology optimization. A fuzzy system was used to integrate theresults of voice, face and fingerprints, to make a decision on the identity of a person.

References

[1] G. Barna and K. Kaski, “Choosing optimal network structure”, Proceedings of theInternational Neural Network Conference (INNC90), pp. 890-893, 1990.[2] O. Castillo and P. Melin,"Hybrid Intelligent Systems for Time Series Prediction usingNeural Networks, Fuzzy Logic and Fractal Theory", IEEE Transactions on Neural Networks,Vol. 13, no. 6, pp. 1395-1408, 2002.[3] D. J. Chalmers, "The Evolution of Learning: An Experiment in Genetic Connectionism",Proceedings of the 1990 Connectionist Models Summer School, Morgan Kauffman, 1990.[4] J. Feldman, “Neural representation of conceptual knowledge”, in Neural connections,mental computation (Nadel and et al., eds.), MIT Press, 1989.[5] F. Fogelman-Soulie, “Multi-modular neural network-hybrid architectures: a review”,Proceedings of 1993 International Joint Conference on Neural Networks, 1993.[6] B. Happel and J. Murre, “Design and evolution of modular neural network architectures”,Neural Networks, vol. 7, pp. 985-1004, 1994.[7] R. A. Jacobs, M. I. Jordan, S. J. Nowlan and G. E. Hinton, "Adaptive Mixtures of LocalExperts", Neural Computation, vol. 3, pp. 79-87, 1991.[8] R. Jenkins and B. Yuhas, “A simplified neural network solution through problemdecomposition: The case of the truck backer-upper”, IEEE Transactions on Neural Networks,vol. 4, no. 4, pp. 718-722, 1993.[9] M. I. Jordan and R. A. Jacobs, "Hierarchical Mixtures of Experts and the EM Algorithm",Neural Computation, vol. 6, pp. 181-214, 1994.[10] K. F. Man, K. S. Tang and S. Kwong, "Genetic Algorithms: Concepts and Design",Springer-Verlag, 1999.[11] M. Mitchell, "An Introduction to Genetic Algorithms", MIT Press, 1996.[12] C. Monrocq, “A probabilistic approach which provides and adaptive neural networkarchitecture for discrimination”, Proceedings of the International Conference on ArtificialNeural Networks, vol. 372, pp. 252-256, 1993.[13] D. J. Montana and L. D. Davis, "Training Feedforward Networks using GeneticAlgorithms", Proceedings of the International Joint Conference on Artificial Intelligence,Morgan Kauffman, 1989.[14] H. Simon, The sciences of the artificial, MIT Press, 1981.[15] P. Melin and O. Castillo, “Hybrid Intelligent Systems for Pattern Recognition”, 2005.