Application of the Genetic Algorithm Joint with the Powell Method to Nonlinear Least-Squares Fitting...

Application of the Genetic Algorithm Joint with the Powell Method to NonlinearLeast-Squares Fitting of Powder EPR Spectra

Tomasz Spałek,† Piotr Pietrzyk,† and Zbigniew Sojka*,†,‡

Faculty of Chemistry, Jagiellonian University, Ingardena 3, 30-060 Cracow, Poland, and Regional Laboratoryof Physicochemical Analyses and Structural Research, Ingardena 3, 30-060 Cracow, Poland

Received April 22, 2004

The application of the stochastic genetic algorithm (GA) in conjunction with the deterministic Powell searchto analysis of the multicomponent powder EPR spectra based on computer simulation is described. Thisapproach allows for automated extraction of the magnetic parameters and relative abundances of thecomponent signals, from the nonlinear least-squares fitting of experimental spectra, with minimum outsideintervention. The efficiency and robustness of GA alone and its hybrid variant with the Powell method wasdemonstrated using complex simulated and real EPR data sets. The unique capacity of the genetic algorithmfor locating global minima, subsequently refined by the Powell method, allowed for successful fitting of thespectra. The influence of the population size, mutation, and crossover rates on the performance of GA wasalso investigated.

1. INTRODUCTION

Electron paramagnetic resonance (EPR) also known aselectron spin resonance (ESR) spectroscopy is a powerfultool for characterizing chemical species that contain unpairedelectrons such as radicals, transition metal ions, or defectcenters. The extreme sensitivity and information content ofthe spectra are probably the most acknowledged advantagesthat contribute to large applications of EPR spectroscopy inchemistry, solid-state physics, and molecular biology. Withthe advent of the spin Hamiltonian concept1 in the begin-nings, the primary goal in much of EPR practice was a directmeasurement of the spin Hamiltonian parameters from thepositions of spectral lines in the magnetic field. Over thepast two decades the focus has shifted toward computersimulation, the only method suitable for accurately extractingthe EPR parameters, especially in the case of randomlyoriented powders, glasses, or frozen solutions.2 Becauseusually the number of parameters to be optimized is large,single simulations are of rather limited practical value, andautomated fitting has become an indispensable componentin any proficient analysis of the powder EPR spectrum.2,3

The complexity of EPR powder spectra imposes severalrestrictions on apposite optimization methods for searchingthe spin Hamiltonian parameter spaceHp, which shouldpossess the following characteristics: (i) do not require orminimize any derivative information, (ii ) be resistant to localminima, (iii ) be effective in very nonlinear problems, and(iV) be sparing in use of target function evaluations and notaffected adversely when it is large.4 These requirementsdisfavor nonlinear least squares approaches that requirederivative information, despite their advantages discussedelsewhere,5 approving direct search methods,3,4 downhill

simplex algorithms,6 or global techniques such as simulatedannealing7 or genetic algorithms,8,9 to tackle difficult opti-mization problems with many local optima.

The genetic algorithm is an optimization method employ-ing a probabilistic, nonlocal search heuristics that simulatesnatural evolution. The major strength of GA is the ability toexplore and exploit a large parameter space with no initialguesses and derivative information, without trapping in localminima. However, GA exhibits a distinct drawback, the trueminium, i.e., the best solution, is rarely reached albeit it maybe closely approached.8,9 Therefore, regardless of the qualityof the results obtained with the genetic algorithm, traditionalminimum search techniques such as Gauss-Newton10 andHooke and Jeeves11 may be used as complementary secondstep measures to refine the ultimate results. Despite the widerange of applications of GA to many chemical problems,12,13

optimization of the powder EPR spectra remains still anunexplored field of its potential advantage. To our knowl-edge, except for our preliminary work on static randomlyoriented spectra,14 only a paper by Filipic and Strancar15

reported on the application of GA to analysis of the EPRspectra of motionally restricted fluid solutions.

2. THEORETICAL OUTLINE

2.1. Simulation of Powder EPR spectra.Static EPRspectra of randomly oriented isolated paramagnets with S) 1/2 and I > 0 are described by the spin Hamiltonian,typically written in the form

where the subsequent terms represent electronic Zeeman andhyperfine interactions, respectively. In the case of randomlyoriented systems usually the nuclear Zeeman,gNâNIB , andin many situations also the quadruple,I ‚Q‚I , terms (for Ig1) are neglected as they usually do not contribute seeminglyto the final shape of most of the powder EPR spectra.3

* Corresponding author phone:+48 12 663 22 95; fax:+48 12 63405 15; e-mail: [email protected].

† Faculty of Chemistry.‡ Regional Laboratory of Physicochemical Analyses and Structural

Research.

H ) âeS‚g‚B + Σ S‚Ai‚I i (1)

18 J. Chem. Inf. Model.2005,45, 18-29

10.1021/ci049863s CCC: $30.25 © 2005 American Chemical SocietyPublished on Web 11/24/2004

The parameters of the spin Hamiltonian to be determinedexperimentally, usually referred to asg andA tensors etc.,fully describe all EPR lines and their relative intensities andconvey meaningful molecular information about the inves-tigated system. The powder spectrum can be considered asa superposition of single-crystal EPR signals computed fora large number of orientations (θ, φ) of the external magneticfield B over the unit sphere weighted in proportion ofsinθdθdφ . The resultant pattern can be expressed asfollows16

whereP(i,θ,φ,ν) is the transition probability,f(B - B0[ν],σΒ) is the line shape function (usually assuming a Gaussianor Lorentzian form), andC is a constant that incorporatesall instrumental parameters. The most common line widthmodel is based on the classic angular variation formula

whereσq(B), q ) x, y, z, are the input line width values.Because the line positions generally are not strictly linear

in any of the magnetic parameters, and the latter occurintermingled in the energy expressions, the analysis of EPRspectra appears to generally be nontrivial. The problemsproliferate when several paramagnetic species occur simul-taneously, giving rise to multicomponent EPR spectra

In such cases partially resolved and overlapped signals, oftenencountered in practice,17 may severely impede detection ofthe individual EPR features, making determination of thespin Hamiltonian parameters difficult. As a result, computersimulation combined with fitting appears to be practicallythe only way of reliable analysis of the experimental powderEPR spectra. A large number of suitable programs of varioussophistication levels have been developed in past decadesfor this purpose.2,3,5,18

2.2. Optimization Procedure. Mathematically the ex-perimental and calculated EPR spectra constitute two setsof discrete data pointsY ) (Y1.....Yn) and y ) (y1.....yn),respectively. IfP represents all input parameters needed tofully describe the spectrum, theny ) f(P). The simulationproblem consists of finding suchP to makey ) f(P) as closeas possible toY.4 The quality of the fit gauged by an errorfunctionE(y,Y,w) is usually characterized by the sum of theweighted square deviations between the calculated andexperimental spectra, normalized by the noise variance andthe number of data points.19 Some other criterions have alsobeen reported.3 Thus, an automated determination of the EPRparameters is a multivariate optimization problem of findingthe minimum of

wherewi andsi are the peak weighting and spectra scalingfactors, respectively, andN is the number of data points. In

the case of noiseless synthetic spectra by takingwi ) 1 theE(y,Y,w) function is equivalent to the root-mean-square error(RMS), whereas for noisy spectra withwi ) σi

2 it correspondsto ø2.

The success of the traditional local refinement methodsstrongly depends on the starting point10,11 and is rapidlyrestrained with increasing the number of the parameters. Theycan also be easily trapped in local valleys and thereby oftenfail to locate the global minimum. Thus, a promising solutionof such problems can be provided by hybridization of directsearch methods with global genetic algorithms.

2.3. Adapting GA to Automated Analysis of EPRSpectra.Many different variants of evolutionary algorithmshave been developed.20-22 Although the underlying principlesare relatively simple, those algorithms have proven them-selves as robust and powerful search tools, owing to theirremarkable flexibility and adaptability to a given task.Hereafter a brief description focuses on how to tailor agenetic algorithm to automate the process of determiningthe spin Hamiltonian parameters from powder EPR spectravia computer simulation.

For optimization of multicomponent EPR spectra we haveimplemented into a simulation program EPRsim32 (videinfra) an evolution algorithm described elsewhere.23 Thestructure of the algorithm is outlined in a flowchart shownin Figure 1, while the notation used is explained in Table 1.To define properly the GA machinery in a particularapplication, one should provide the sizem of the initialpopulationΩt(m) of individuals Gi, i ) 1,...,m, the codingmethod, the fitness indexF(Gi) of each individual, and the

Y(B,ν) ) C∫θ)0

π/2∫æ)0

2π ∑i

P(i,θ,æ,ν)f(B -

Bo[ν],σB)dcosθdæ (2)

σB ) (σx2(B)cos2φsin2θ + σy

2(B) sin2φsin2θ +

σz2(B) cos2θ)1/2 (3)

Y(B,ν) ) a1Y1(B, ν) + a2Y2(B, ν) +...acYc(B, ν) (4)

E(f(P),Y,w) ) 1/NΣwi(Yi - siyi)2 (5)

Figure 1. Flowchart of the hybrid optimization method forautomatic analysis of powder EPR spectra based on the combinationof a genetic algorithm search with Powell refinement. The doublearrow S indicates a communication with the EPR module.

GENETIC ALGORITHM TO OPTIMIZATION OF EPR SPECTRA J. Chem. Inf. Model., Vol. 45, No. 1, 200519

formulation of genetic operators. It invokes four dataprocessing steps: representation, selection, recombination(crossover), and mutation.

In the adopted representation, a full set of the current inputvariablesP to be optimized was encoded into an individualGi ) X1, X2,..., Xc, composed ofc chromosomes Xjcorresponding to the particular componentsYj(B, V) of theEPR spectrum. The polychromosomic individuals werepreferred over the usual monochromosomic species, sincethey reflect better the structure of the optimization subject(multicomponent spectrum). Each chromosome is a stringof the form Xj ) x1

j, x2j,..., xn

j, wherexnj stands for the

genes, the number of which is equal to the number ofoptimizable parametersn of the component signalj. In theapplied representation the genes are coded as 32-bit floatingpoint numbers from the [-1, 1] range. The adjustablesimulation parameterspij , i.e., the spin Hamiltonian param-eters, line widthsσq(B), and the relative contributionsa ofthe component signalsYj(B, V) of the resultant spectrum arerelated to the corresponding genesxi

j in the following way:pij ) xi

j∆pijo + pij

o, wherepijo corresponds to their initial

values and∆pijo to the assumed uncertainty intervals

(∆po ) [pmin, pmax]). The overall structure of such representa-tion is shown below in more detail.

One of the advantages of floating point representationconsists of retaining the proximity between two points inboth spaces: the representation and the problem one.23

Despite the lack of well established theoretical justificationfor using real-coded genes, the floating point representationappeares to be faster and easier to implement and provideshigher precision than its binary analogue, particularly in largedomains where the binary strings would be prohibitivelylong.24

The chromosome Xj containing the information about theall adjustable parameters of the component signalj, can beregarded as the basal one for a descendant chromosome X′j:[k×1], k < c, which collectsk simulation parametersP′j thatare currently optimized. The Xj and X′j strings are mutuallyrelated through user defined selection vectorT[c×1], whichallocates the parameters for the adjustment in a givensimulation run, following the scheme

Such operation allows for reduction of the number ofvariables, by fixing those parameters that are supposed tobe optimal or currently kept constant for certain reasons(latent variables). By analogy to the gene-parameter relation-ship, the information contained in the chromosome Xj canbe translated into the parameter space using the relation

wherePjo ) p1j

o, p2jo, p3j

o .....pnjo and∆Pj

o ) ∆p1jo, ∆p2j

o,∆p3j

o ..... ∆pnjo are the strings of the starting parameters

and their uncertainties of the component signalj, respectively.A basal set of the simulation parameters for each com-

ponent signalj is obtained by combining the stringP′j of theadjustable parametersg andA tensors,σq(B), aj, .... witha stringQ′j of the associated parametric variablesS, I, θ, φ,h, ..., containing complementary information about the

Table 1. Notation Used in the Flowchart of GA-Based Optimization of EPR Spectra

notation definition

Y, y experimental and calculated EPR spectraP ) pi full set simulation parameterspi

E(y,Y,w) error (objective) functionΩt(m) tth population of the sizem of individuals Gi

Gi ) X1, X2,.., Xc an individuali composed ofc chromosomes Xj corresponding tocomponent EPR signals contributing to total spectrum

Xj[n × 1] ) [x1, x2, ...,xn]xi ε [-1, 1]

a string of chromosome withn genes corresponding ton parameters to be optimized

F(Gi) fitness index of anith individualGwin the fittest individual in current populationtGopt the fittest individual of ultimate GA-Powell optimizationG′i an individuali complemented by its fitness indexF(Gi)P°, ∆P° strings of starting parameters and their uncertaintiescINF(Gi) intimate neighborhood impact factor evaluated for an individual Gi

INFthrs threshold value of INFtmax maximum number of generations

Scheme 1.The Relationship between the Genotype and PhenotypeSpacesa

aThe polychromosomic genotype of an individual Gi is mapped intothe corresponding parameter spaceP ) Pj, and nextP is used forsimulation of the EPR spectrumYj(B, V) ) F(P).

Scheme 2.Selection VectorT[c×1] Relatingk Currently OptimizedParametersP′ andc Adjustable ParametersP

Pj ) Xj(∆Pjo) + Pj

o (6)

20 J. Chem. Inf. Model., Vol. 45, No. 1, 2005 SPAŁEK ET AL.

electronic and nuclear spin values,θ and φ angles, thenumber simulation pointsh, etc.

In the case of nuclei of various isotopic composition likemolybdenum, thePj andQj strings are associated with thefundamental isotopomer of the componentj, i.e., that withthe largest number of the magnetic parameters. They haveto be complemented by relatedPj(i) and Qj(i) stringsaccounting for all descendant isotopomersi, through ap-propriate modification of the reliant parameters (e.g.,Apq(i)) gn(i)/gnApq wheregn are the tabulated nuclearg factors)and suitable amendment of the parametric variablesQj. Forinstance, in the case of O-17 enriched superoxide radicalwith nonequivalent nuclei there are four isotopomers17O17O,17O16O, 16O17O, 16O16O, and the17O17O can be taken as afundamental one. The relative abundances of the descendantisotopomers can either be calculated a priori,25 from theknown isotopic compositions or taken as another adjustableparameter inserted into thePj string. The complete setPj ofthe parameters required for simulation of a given componentsignal j is then defined by an ensemble ofPj, Pj(i) andQj,Qj(i) strings of all isotopomers. Obviously, only the param-eters of the fundamental isotopomers (Pj) have to beoptimized, all dependent parametersPj(i) of the remainingisotopomers, needed for simulation, are derived from thePj

values. Finally, for a multicomponent EPR spectrum the fullsetP of the simulation parameters can be created by simpleaggregation of the stringsPj, of all partaking components,P ) Pj, j ) 1, c.

In such an approach the ensemble of the basal chromo-somes X constituting an individual Gi can be identified witha genotype corresponding to the all simulation parametersGi f P. The associated phenotype is then constituted bythe associated simulated EPR spectrum calculated with thoseparameters (the genotype). Both spaces are linked togetherby translation function (the simulation) converting thegenotype space into the phenotype space (Y ) f(P(Gi)),which is in turn compared with the environment (theexperimental spectrumy). The error functionE(Y,y,w) is usedas a fitness criterion to be minimized (Scheme 1).

The flowchart shown in Figure 1 explains the interplaybetween the GA optimization and the EPR simulationprogram. At the beginning an initial populationΩ1(m) ofthe individuals Gi (potential solutions) has to be created. Eachgene of an individual Gi encodes a preliminary value of theassociated adjustable parameter, selected at random from theassumed uncertainty range. The individuals are next evalu-ated by calling the simulation program EPR and thecalculated fitness indexesF(Gi) are associated with the Gi

strings (Gi f G′i). The fitness index composed of twonumbers, a root-mean-square error (orø2) and the value ofan intimate neighborhood impact factorF(Gi) ) E(Y,y,w),INF(Y,y,ε), was used for scoring the individuals. The INF-(Y,y,ε) gauges the fraction of the simulated data points whichare located within a user defined envelope of the experi-mental spectrum. The widths of this envelope are determinedby anε value, which usually is dictated by the noise level(vide infra).14 The intimate neighborhood impact factorprovides a concise and stabile measure of the globaloptimization successfulness. The value of INF(Y,y,ε) is usedas a criterion for switching from GA to Powell search,whereasE(Y,y,w) is an actual subject of the optimizationprocess. Determining the fitness of an individual is compu-

tationally the most demanding step of the GA optimizationsince it requires numerical simulation of the EPR spectrum.Actually the genetic algorithm makes use of a modifiedfitness functionE′(Y(Gi),y,w) ) E(Y(Go),y,w) - E(Y(Gi),y,w),where E(Y(Go),y,w) is the fitness of the worst adaptedindividual, i.e., that of the highestE(Y,y,w) value in the initialpopulation.

Based on theE′(Y,y,w) values the G′i individuals areranked according to their merit, and the fittest individual (thewinner) is duplicated and saved as Gwin. If at least 95% ofthe calculated spectral points (an arbitrary threshold valueof the INF criterion) appear within the envelope of theintimate neighborhood, i.e., INF(Gwin) > INFthrs, the globaloptimization is considered to be accomplished successfully,and the set of the parameters connected with Gwin is used asa starting point for direct search refinement. Thus, at thisstage the GA optimization is tantamount with the MonteCarlo search for an advantageous starting point, with thenumber of trials equal to the population size.

In the case when the Gwin individual fails the INF test(INF(Gwin) < INFthrs), the population is subjected to thereproduction process using the genetic operations in thesequence indicated in Figure 1. The selection operationchooses the best individuals of the current population forreproduction in the next generation. A fitness proportionateselection (roulette wheel)26 along with an elitist successionwas applied to produce the offspring. In this approach thereproduction probabilityWt(Gi) of an individual Gi isproportional to itsE′(Y(Gi),y,w) value, normalized over thewhole population:

To ensure that good solutions are not lost regularly, a fractionr of the top individuals from the parent population is placedin the next generation (from whichr worst individuals hasbeen removed) without any changes. The actual fraction ofretention defines the selection pressure. When size of theelite is large from one generation to another, the algorithmcan exhibit narrow population diversity. On the other hand,a small fraction might not retain enough of the desirablecharacteristics to overcome convergence problems.

The descendent individuals are produced through theoperations of crossover and mutation. A uniform crossoveroperator was used for this purpose. Its action consists ofarithmetic averaging of the parental chromosomes X1 andX2 (appropriate for the floating-point representation) and canbe described in the following way

whereg is a random number with uniform distribution inthe [0, 1] interval. Such choice has a big advantage, becausenew chromosome values will be automatically inside theirdefinition ranges.

In the last step the chromosomes undergo the mutationoperation. This operation is helpful in preventing prematureconvergence and ensuring the diversity of the individuals.A probability variable controls the mutation process and

Wt(Gi) )E′(Y(Gi),y,w)

∑i)1m E′(Y(Gi),y,w)

(7)

X1new ) X1

old + g(X2old - X1

old)

X2new ) X2

old + g(X1old - X2

old) (8)


usually has a small value inversely proportional to thenumber of genes.27 The floating point representation suggeststhe usage of a nonuniform mutation described by thefollowing fluctuation function

whereτ stands for the evolution time,r is a random numberfrom the [0, 1] range,z limits the maximum variation of thegiven gene, whereasb describes the system nonuniformity.23

The resulting population is sorted by fitness, and the topmindividuals are retained for the next generationΩt+1(m). Theiterative cycles (generations) are repeated until the optimiza-tion or stopping criterium (t e tmax) is fulfilled.

3. EPR SIMULATION PROGRAM

Simulated test spectra were obtained using an EPRsim32program written in Microsoft visual C++ 6.0 using the MFClibrary. The program has building blocks based on theapproach described above. The program runs under MicrosoftWindows with full 32 coding. It has a user-friendly interfacebased on mouse-driven access to pull-down menus anddialogue boxes. The simulated spectrum can be saved as anASCII file, and hardcopy is available using any output devicesupported by Microsoft Windows. The interactive graphicsfor monitoring of the simulation progress, visualization ofdata in conjunction with the experimental spectrum, versatiletools for peak picking and line shape analysis, interactivestick diagram generation for provisional assessment ofg, andhyperfine splittings are available. A short calculation timealong with the possibility of manual stepping of variablesprovides users with a better understanding of spectralresponse to changes of the parameters, in nearly-real time.

The program allows for analysis of the static randomlyoriented continuous wave multicomponent EPR spectra(including isotopomers) with S) 1/2 and coincident principalaxes. The second-order perturbation formula was used tocalculate resonance fields along with the powder averagedtransition probabilities⟨g1

2/g⟩ for evaluation of the lineintensities.3 The line shape can be either Lorentzian, Gauss-ian, or mixed, and the orientation dependent line width canbe used. Because of extensive calling of the simulationmodule, evaluation of the GA performance and spectraoptimization are quite time-consuming. Thus, using of theclosed form perturbation formula instead of demandingenergy matrix diagonalization helped us to speed up thisprocess essentially.

4. DATA SETS

The performance of the GA-based optimization algorithmwas tested using synthetic and real data sets of increasingcomplexity, corresponding to typical spectroscopic propertiesof transition metal ions exchanged into ZSM-5 zeolites, wehave encountered in our studies. A rather straightforwardtest spectrumT1 with 17 adjustable parameters was com-posed of three superimposed signals (S1-1, S1-2, S1-3) thatrefer to various types of copper intrazeolite complexes (theaxial signalS1-1, with g| > g⊥ andA| > A⊥, corresponds tocopper in a square pyramid) and the signalS1-2, with g⊥ >g| andA| > A⊥, to a trigonal bipyramid coordination, whereasthe isotropic signalS1-3 to a hexaaqua complex averaged

by tumbling. These are three generic species of divalentcopper entrapped in zeolites, depending on the dehydrationstate. A test spectrumT2 having 26 adjustable parameterswas composed of three orthorhombic (S2-1, S2-2, S2-3) andone axial signal (S2-4), all without the hyperfine structure,and theg tensor values that are characteristic of mono-, di-,tri-, and tetracarbonyl complexes of nickel in ZSM-5. A testspectrum T3 with 21 parameters to be optimized wasproduced by three largely overlapped axial signals (S3-1, S3-2, S3-3) with pronounced hyperfine structure due to specia-tion of low spin tricarbonyl cobalt intrazeolite adducts. TheT1, T2, andT3 spectra along with the component signals areshown in Figure 2a, b, c, and the corresponding parametersare collected in Table 2.

The experimental EPR spectrumT4 of (CO)nNi9/ZSM-5(Figure 2d) was recorded on a Bruker ELEXYS spectrometer,working at X-band with a rectangular TE102 cavity and 100kHz modulation. Since any distortion of the experimentalspectra will reduce the fidelity of the simulation, our analysisis restricted to spectra for which the modulation distortionsare small, the saturation can be neglected, and the slowpassage condition is fulfilled.

5. OPTIMIZATION OF THE GA PARAMETERS

The operational efficiency of a genetic algorithm isintimately controlled by the values of such parameters asthe population size, the crossover and mutation probabilities,the decay factor, and the percentage of elitist retention, whichcontrol various aspects of the algorithm in a given imple-mentation. There is no sound theoretical basis for a prioriassignment of their values, albeit some general hints areavailable.8,22 Thus, to determine such setting for our applica-tion, parametric sensitivity tests were performed. Becauseof the stochastic nature of GA, sensible conclusions can onlybe based on the average performance over several runs.Therefore, to obtain reliable results and minimize theinfluence of the initial population 10-20 independent trialswith a randomly varying starting population were performedin all testing experiments.

5.1. Establishing of the Crossover and Mutation Rates.The sensitivity of the GA-based fitting of EPR spectra tothe probabilities of crossover and mutation was examinedusing theT1, T2, and T3 data set. Optimization of thoseparameters is an important issue since the relative prob-abilities of the crossover and mutation determine the balancebetween the exploitation and exploration moves that arecrucial for performance of the algorithm. The probability ofthe crossover (pcrs) was paced in the range [0, 1] with a 0.1step, whereas the mutation probability (pmut) was paced inthe range [0, 0.1] with a 0.01 step. As a result a grid of 121data points was created, and the GA optimization, with apopulation of the sizem ) 100, was successively iteratedover the generations for each pair of the values. The resultsof the parametric sensitivity study, averaged over threeindependent runs, are summarized in the form of operatorperformance maps shown in Figure 3, where log of theconvergence efficiency, gauged by the number of simulationsrequired to achieve the INF threshold of 95%, is drawn as afunction of the crossover and mutation probabilities. Fromthe inspection of Figure 3 it is clear that an increase of thecrossover rate tends to enhance the performance of GA for

∆(τ,z) ) z(1 - r(1-τ/tmax)b) (9)


the T1 and T2 targets. The optimal values are found in arather broad range of the values (from 0.5 to 0.9). In thecase of theT3 test spectrum, however, the optimal area israther narrow and is shifted towardpcrs ∼ 0.6. The crossoverprobabilities larger than 0.7-0.8 or smaller than 0.4 distinctlydeteriorate the performance. The optimal ranges for themutation probabilities are more confined in comparison tothe crossover ones. Large (pmut > 0.03-0.02) and very small(pmut < 0.005) rates become detrimental quite rapidly. Forall test spectra the advantageous values appear in the vicinityof 0.01. Comparison of the operator performance maps for

all investigated targets shows that the area of the optimalrates distinctly shrinks on passing fromT1 to T2 and thenT3. This indicates that targets with rather diverse components,like that of T1, imply a more open space of the probabilityrates than those composed of similar objects (T3).

The results of the sensitivity studies for the crossover andmutation operators revealed a rather strong dependence ofthe GA performance on the assumed probability values andon the nature of the investigated target. The mutation ratesaround 0.01 and the crossover rates in the range 0.5-0.7appeared to be optimal settings for further investigations.

Figure 2. Synthetic EPR test spectra along with the component signals (a)T1, (b) T2, (c) T3, and (d) experimental spectrumT4 of nickelcarbonyl complexes in ZSM-5.

Table 2. EPR Parameters of theT1, T2, andT3 Test Spectra Used for Evaluation of Genetic Algorithm Performance

spectrum g-tensor A-tensor/G line widths/G

T1

S1-1 g| ) 2.37( 0.126,g⊥ ) 2.06( 0.126 A| ) 121( 25 δ| ) 45 ( 9, δ⊥ ) 23 ( 5A⊥ ) 20 ( 8

S1-2 g| ) 1.97( 0.126,g⊥ ) 2.32( 0.126 A| ) 131( 26 δ| ) 40 ( 8, δ⊥ ) 40 ( 8A⊥ ) 45 ( 9

S1-3 gav) 2.18( 0.126 Aav ) 160( 8 δ ) 180( 36T2

S2-1 gz ) 2.02( 0.1,gx ) 2.392( 0.1 δz ) 30 ( 6, δx ) 35 ( 9gy ) 2.35( 0.1 δy ) 32 ( 7

S2-2 gz ) 2.191( 0.1,gx ) 2.086( 0.1, δz ) 30 ( 6, δx ) 18 ( 4gy ) 2.066( 0.1 δy ) 18 ( 4

S2-3 gz ) 2.005( 0.1,gx ) 2.2( 0.1 δz ) 12 ( 2.5,δx ) 30 ( 6gy ) 2.162( 0.1 δy ) 30 ( 6

S2-4 g| ) 2.009( 0.1,g⊥ ) 2.13( 0.1 δ| ) 12 ( 2.5,δ⊥ ) 12 ( 2.5T3

S3-1 g| ) 2.011( 0.1,g⊥ ) 2.22( 0.1 A| ) 79 ( 12 δ| ) 18.4( 3, δ⊥ ) 27 ( 4A⊥ ) 38 ( 6

S3-2 g| ) 2.0275( 0.1,g⊥ ) 2.19( 0.1 A| ) 67 ( 10 δ| ) 15 ( 2.25,δ⊥ ) 20 ( 3A⊥ ) 38 ( 6

S3-3 g| ) 2.018( 0.1,g⊥ ) 2.18( 0.1 A| ) 71 ( 11 δ| ) 10 ( 1.5,δ⊥ ) 15 ( 2.25A⊥ ) 38 ( 6


5.2. Optimization of the Population Size.In general thisparameter should be determined according to the complexityof the problem, and usually it is necessary to try out a rangeof the m values. We have investigated this issue in moredetail on the same test data sets as above. The size of thepopulation was systematically varied from 30 to 1000, whileall other parameters were kept constant atpcrs ) 0.5 andpmut ) 0.01. Generally the simulation success averaged over10 runs and measured by the number of simulations requiredto pass the criterion of INF> 95% exhibited a moderatedependence on the population size with rather broad minimaappearing atm ) 250, 150, and 200 for theT1, T2, andT3

targets, respectively. However, for small sizes of the popula-tion, when the gene pool was probably insufficient, the resultswere quite scattered indicating strong dependence of theperformance on the starting point. This was especiallyapparent in the case of theT1 andT2 test spectra composedof diverse EPR signals. In contrast, for theT3 spectrum beinga superposition of the signals similar in shape, the scatteringof the results was more uniform and the optimum in thepopulation size better pronounced. Eventually, the selectionof the initial population size appeared less crucial than the

selection of the crossover and mutation rates, and for furtherinvestigations the values within the range of 150-250 wereused.

6. OPTIMIZATION OF EPR SPECTRA

6.1. Simulated Data. The best way to evaluate theefficiency of an optimization routine is to fit a spectrum witha priori known parameters (a simulated spectrum). Only insuch cases a complete agreement between the calculated andfitted spectra can be obtained, identifying the global mini-mum univocally. In this study we compared the performanceof the Powell, GA, and the hybrid GA-Powell optimizationmethods, by following the evolution of convergence for thenoiselessT i, i ) 1, 2, 3 test spectra, and for the same spectracontaminated with a white noise of SNR) 35 dB. Theselatter spectra are labeled hereafter as n-T i. The signal-to-noise ratio was defined as follows

where amax and σ are the maximum values of the signalamplitude and the variance of the added noise, respectively.27

For the convergence tests an uncertainty limit of(20% wasimposed on all adjustable parameters, whereas the startingpoint was selected at random. Twenty trials were effectedfor the Powell method, and 10 independent initializationsof the genetic algorithm were used to equilibrate the results.The average RMS value corresponding to the INF of 95%was determined empirically for all test spectra, by fittingthe plot RMS vs INF to a logarithmic curve (RMS) aln-[INF] + b). The values of the GA parameters used in thenumerical experiments are collected in Table 3.

The performance profiles for the noiselessT1, T2, andT3

spectra are shown in Figure 4. For these data exact solutions(RMS f 0) are in principle possible. From comparison ofparts a and b of Figure 4 it is clear that for theT1 target thePowell method markedly outperformed GA, out of 20 trials12 reached the INF level in less than 3000 simulations.Among them, in 3 cases a perfect fit (global minimum) wasachieved, whereas the remaining 9 runs were caught in somelocal minima yet below the INF threshold. However, in 8cases a complete failure was observed, demonstrating anobvious sensitivity of the direct search on the starting point.The analogous progress of the convergence for the GAoptimization was distinctly slower but clearly more profi-cient. All 10 runs converged below the INF limit, albeit ina few cases it required more than 7000 simulations. Thedecrease in RMS was approximately exponential in nature,suggesting that the most rapid improvements occurred earlyin the evolutionary process. The obtained final GA fits werealways worse than those observed in the most successful runsof the Powell algorithm.

Figure 3. Operator performance maps for parametric sensitivityof the GA convergence as a function of the mutation and crossoverprobabilities for theT1, T2, andT3 targets. The color coding displaysthe convergence efficiency, gauged by the number of simulationsrequired to achieve the INF threshold of 95% in the log scale,averaged over three independent runs. The shading has colorresulting from bilinear interpolation of the colors at its four vertices.

Table 3. Values of the Genetic Algorithm Parameters Used inNumerical Experiments with theT1, T2, andT3 Test Spectra

GA parameter T1 T2 T3

population size 200 150 250maximum number of generations 750 1000 600crossover probability 0.9 0.6 0.6crossover type uniform uniform uniformmutation probability 0.02 0.01 0.01

SNR) 10logamax2/σ2 (10)


For the more demandingT2 spectrum the situation changeddramatically (Figure 4c,d). While the slow decent of RMSin the genetic algorithm optimization demonstrated clearlythat the overall quality of the fit was steadily improved asthe population evolved, in the case of the Powell methodonly for one fortuitous run a satisfactory suboptimal solutionwas achieved. All other attempts became essentially stackedin various shallow minima situated distinctly above theassumed INF level, already at quite early stages of theoptimization process. Further prolongation of the optimiza-tion course was completely inefficient (Figure 4c) in contrastto GA that showed a moderate but persistent progress,satisfying the INF criterion in all runs (Figure 4d). Thissuperlative behavior of GA was further magnified on passingto the most congested spectrumT3. While the Powell methodcompletely failed in this case, with the genetic algorithm

the INF limit was achieved in most attempts. However, thefinal fits were clearly not optimal, as it can be judged fromthe terminal RMS values (Figure 5e,f). Thus, the testexperiments have revealed an expected robustness of theglobal GA search and its inherent difficulty in fine refine-ment. For improvement of the optimization efficacy thegenetic algorithm search was next hybridized with the Powellmethod.

The performance of the hybrid GA-Powell algorithm isshown in Figure 4b,d,f. After running the genetic algorithmoptimization, once the convergence approached the INF limit,the current values of the adjustable parameters were takenas the starting point for the subsequent Powell refinement,following the flowchart shown in Figure 1. Spectacularconvergence, leading to a practically perfect fit observed forthe T1 and T3 test spectra, revealed that GA can very

Figure 4. Performance profiles for the Powell (a, c, e) and the GA (green lines) combined with the Powell (red lines) optimization of thenoiselessT1, T2, andT3 test spectra (b, d, f). The horizontal dashed line indicates an average RMS value corresponding to INF of 95% (seethe text for more details).


effectively locate promising regions in the search space thatare close to the global minimum. However, in the case oftheT2 spectrum the performance of the hybrid approach wasclearly deteriorated as the global minimum was located onlytwice. Most of the subsequent Powell refinements becameconfined in various suboptimal regions of theHp space.However, the use of synthetic test spectra allows verifyingthe quality of the result by comparing the optimizedparameters with their known input values used to generatetheT2 spectrum (Table 4). The final adjusted values are givenas the average over 10 runs of the hybrid algorithm togetherwith their standard deviations. From the comparison of theadjusted parameters and the corresponding input values wecan conclude that despite the suboptimal solution most ofthe optimized parameters agree well with the original values.

The difficulty in reaching the global minimum for theT2

spectrum was apparently caused by the relatively large errorsin determination of theσz line widths for the S2-1 and S2-4signals. Indeed, bothgz-components being very weak arelargely suppressed by the dominantgz band of the S2-2 signal(Figure 2b). As a result, contributing only slightly to the RMSvalue, they cannot be precisely determined.

The progress of convergence for the test spectra contami-nated with the noise is shown in Figure 5. For the n-T1

spectrum, the Powell method with only 8 out of 20 successfultrials significantly diminished its convergence efficiency incomparison to the noiselessT1 spectra (Figure 5a). Obviouslylocation of the global minimum in the presence of the noiseis a more difficult task. The performance of GA seems tobe not much affected by the spectral noise as for most of

Figure 5. Performance profiles for the Powell (a, c, e) and the GA (green lines) combined with the Powell (red lines) optimization of thenoisy n-T1, n-T2, and n-T3 test spectra (b, d, f). The horizontal dashed line indicates an average RMS corresponding to INF of 95% (seethe text for more details).


the runs the INF limit was achieved, within the number ofthe simulations comparable to that ofT1. In the case of n-T2

and n-T3 spectra the evolution of the fitness was quite similar(Figure 5b-f). The Powell optimization after some rapidprogress at the very beginning became quickly saturated,distinctly above the INF level of 95%. Although, for the n-T3

target an accidental success could be noted, it was mostprobably owing to a very fortuitous choice of the startingpoint. Again GA clearly surpassed the performance of thePowell method and in tandem with the Powell refinementoptimal solutions (ø2 close to 1) were always reached (Figure5d,f). The comparison between the original and the fittedtest spectra n-Ti is shown in Figure 6.

The outcomes of the hybrid optimization technique, shownin plots b, d, and f in Figures 4 and 5, exhibit high diversityregarding the number of simulations needed to accomplishthe task. There are several possible reasons, acting inconjunction, that may lead to such a result. They include (i)complications of the error function arising from the firstderivative nature of EPR spectra, (ii ) complex structure ofthe search space, (iii ) stochastic nature of GA, and (iV) scarcecoherence between RMS and INF. Although precise evalu-ation of their particular relevance would require separatedetailed investigations, being now in progress, a provisionalphenomenological explanation of such behavior seemsplausible, based on the inspection of Figures 4 and 5. The

transition from GA to Powell optimization is controlled bythe critical value of INF) 95%. It takes place within theregion where RMS function exhibits a distinct plateauindicating saturation of the GA search efficiency. Since thereis no strict functional dependence between RMS and INF,the threshold value of INF in subsequent runs may beachieved at distinctly different RMS values. The very flatshape of the RMS function in the mature stages of the globalsearch implies that even small differences in the actual RMSvalues may easily translate into quite large changes in thenumber of the necessary simulations, while reaching the INFlimit. In other words, the pronounced diversity of the resultsreflects an apparent unsteadiness of the system in selectingthe moment of passing from global to local search, whichbecause of the stochastic nature of GA and flattened RMSbecomes widely scattered. Statistical evaluation of theoptimization results forTi and n-Ti test spectra is shown inTable 5.

6.2. Real Spectra.To test the GA-Powell hybrid approachon experimental data, fitting of the complex EPR spectrumT4 due to nickel carbonyl complexes was chosen as a casestudy (Figure 7a). Optimization of real spectra is moredemanding not only because of the ubiquitous noise but alsodue to ambiguities of the baseline correction and other signaldistortions, which are not accounted for by the simulationalgorithm. Nonetheless, the result of a fully automated

Table 4. Comparison of the Parameters Optimized with the Hybrid GA-Powell Method with the Known Input Values of theT2 SpectrumAveraged over 10 Runs

center parameter input values averaged optimized values for n-T2 standard deviation

S2-1 abundance 0.100 0.099 0.002gz, gx, gy 2.020, 2.392, 2.350 2.016, 2.390, 2.348 0.003, 0.001, 0.001δz, δx, δy/G 30.0, 35.0, 32.0 29.7, 37.9, 30.3 2.2, 0.3, 1.1

S2-2 abundance 0.500 0.499 0.01gz, gx, gy 2.191, 2.086, 2.066 2.196, 2.080, 2.072 0.006, 0.010, 0.010δz, δx, δy/G 30.0, 18.0, 18.0 28.9, 17.9, 18.0 1.7, 0.04, 0.04

S2-3 abundance 0.250 0.256 0.007gz, gx, gy 2.005, 2.200, 2.162 2.0050, 2.198, 2.1620 7× 10-5, 0.003, 2× 10-4

δz, δx, δy/G 12.0, 30.0, 30.0 11.3, 29.9, 30.5 0.1, 0.5, 0.4S2-4 abundance 0.150 0.144 0.004

g|, g⊥ 2.009, 2.130 2.022, 2.1300 0.01, 10-5

δ|, δ⊥/G 12.0, 12.0 13.1, 11.9 0.9, 0.1

Table 5. Summary of the Performance of the Hybrid GA-Powell Method onTi and n-Ti Test Spectra

T1 T2 T3 n-T1 n-T2 n-T3

number of runs 10 10 10 10 10 10best convergence speeda 1996 19580 35586 4719 13800 24490worst convergence speed 12981 48957 158765 16812 66158 146247average convergence speed 6174 33494 117433 9167 34436 52056

a Expressed as the number of the simulations.

Figure 6. Comparison of the original and fitted n-Ti spectra: (a) n-T1, (b) n-T2, and (c) n-T3. Solid line indicates original spectrum anddotted line fitted spectrum.


optimization using the hybrid approach, shown in Figure 7a,clearly demonstrates the power of the method, despite largeuncertainties in the values of the starting parameters. Theevolution of the convergence, for 10 independent runs (Figure7b), gave practically the same results, but the number of therequired simulations was clearly different, ranging from 2000to 9000 in the extreme cases. This was caused by the differentspeed of GA, while moving toward the optimal solutiondepending on the starting point, since the subsequent Powellrefinement was always quite rapid. The choice of the input

parameters, however, was not crucial for achieving of thefinal success, influencing only the time of the optimizationprocess. The results of the hybrid optimization of thespectrumT4 resolved into three componentsS4-1, S4-2, andS4-3 are collected in Table 6.

7. CONCLUSIONS

A simulation program has been developed to automate theentire process of determining the magnetic parameters andanalysis of the powder EPR spectra. Genetic algorithmcombined with Powell local optimization search provides arobust method for automated extraction of the parametersfrom experimental EPR spectra with minimum outsideintervention. Owing to the combination of the exploitationand exploration processes, the genetic algorithm efficientlylocates the most promising regions of the search spaceproviding advantageous starting point for subsequent Powellrefinement. The efficiency of the method was demonstratedusing synthetic and real multicomponent test spectra ofvarious complexities. The ranges of the optimized parametersdo not need to be narrow, so knowledge of the accuratestarting values is not necessary. The results reported hereaugur well for future successful applications of the hybridapproach to determine the parameters of complex powderEPR spectra.

ACKNOWLEDGMENT

This work was supported by the Committee for ScientificResearch of Poland, KBN, as the Research Project number3 T09A 147 26.

REFERENCES AND NOTES

(1) Abragam, A.; Bleaney, B.Electron Paramagnetic Resonance ofTransition Ions; Clarendon Press: Oxford, 1970.

(2) Pilbrow, J. R. EPR spectroscopy of poorly characterized systems: ahistorical and current view.Appl. Magn. Res. 1994, 6, 161-181.

(3) Mubbs, F. E.; Collison, D.Electron Paramagnetic Resonance of dTransition Metal Compounds; Elsevier: Amsterdam, 1992.

(4) Eldershaw, C.Report on Determination of Spin Hamiltonian Param-eters from Electron Paramagnetic Resonance Spectra Using NumericalNonlinear Optimization Methods; The University of Queensland,Australia, 1996.

(5) Misra, S. K. A Rigorous Evaluation of Spin-Hamiltonian Parametersand Line width from a Polycrystalline EPR Spectrum.J. Magn. Reson.1999, 140, 179-188.

(6) Fajer, P. G.; Bennett, R. L. H.; Polnaszek, C. F.; Fajer, E. A.; Thomas,D. D. General Method for Multiparameter Fitting of High-ResolutionEPR Spectra Using a Simplex Algorithm.J. Magn. Reson. 1990, 88,111-125.

(7) Kirkpatrick, S.; Gelatt, C. D.; Vecchi, M. P. Optimization by simulatedannealing.Science1983, 220, 671-680.

(8) Goldberg, D. E.Genetic Algorithms in Search, Optimization, andMachine Learning; Addison-Wesley: New York, 1989.

(9) Davis, L.Handbook of Genetic Algorithms; Van Norstrand Reinhold:New York, 1991.

(10) Bernhard, W. A.; Fouse, G. W. Simulation of Powder ESR Spectraof Organic Free Radicals.J. Magn. Reson. 1989, 82, 156-161.

(11) Griffin, M.; Muys, A.; Noble, C.; Wang, D.; Eldershaw, C.; Gates,K. E.; Burrage, K.; Hanson, G. R. XSOPHE, a Computer SimulationSoftware Suite for the Analysis of the Electron Paramagnetic Spectra.Mol. Phys. Rep. 1999, 26, 60-84.

(12) Lucasius, C. B.; Kateman, G. Understanding and Using GeneticAlgorithms, Part 1. Concepts, Properties and Context.Chemom. Intell.Lab. 1993, 19, 1-33.

(13) Hibbert, D. B. Genetic Algorithms in Chemistry.Chemom. Intell. Lab.1993, 19, 277-293.

(14) Adamski, A.; Spałek, T.; Sojka, Z. Application of EPR Spectroscopyfor Eludidation of Vanadium Speciation in VOx/ZrO2 Catalysts Subjectto Redox Treatment.Res. Chem. Intermediat.2003, 29, 793-804.

Figure 7. The best fit of the EPR spectrumT4 due to nickelcarbonyl in ZSM-5, obtained with the hybrid GA-Powell optimiza-tion together with its INF envelope and the component signals (a)and progress of the fit with the number of simulations (b). Thehorizontal dashed line indicates an average RMS corresponding toINF of 95% (see the text for more details).

Table 6. Parameters of the ExperimentalT4 Spectrum Due toNickel Carbonyl Complexes in ZSM-5 Zeolite, Determined Usingthe Hybrid GA-Powell Approach

center parameter averaged optimized values standard deviation

S4-1 abundance 0.264 0.007gz, gx, gy 2.081, 2.240, 2.090 0.004, 0.002, 0.002δz, δx, δy/G 49.6, 26.2, 37.2 6.3, 1.8, 0.5

S4-2 abundance 1.047 0.020gz, gx, gy 2.0249, 2.2169, 2.1716 10-5, 7 × 10-5, 10-4

δz, δx, δy/G 9.8, 24.2, 28.1 0.8, 1.6, 1.7S4-3 abundance 2.544 0.084

g|, g⊥ 2.0224, 2.1392 6× 10-5, 10-5

δ|, δ⊥/G 3.91, 9.80 0.09, 0.05


(15) Filipic, B.; Strancar, J. Tuning EPR Spectral Parameters with a GeneticAlgorithm. Applied Soft Computing2001, 1, 83-90.

(16) Pilbrow, J. R.Transition Ion Electron Paramagnetic Resonance;Clarendon Press: 1990.

(17) Sojka, Z.; Che, M. EPR Techniques Applied to the Study of Chemistryand Catalysis on Oxide Surfaces.Appl. Magn. Reson.2001, 20, 433-456.

(18) Weil, J. A. The Simulation of EPR Spectra: A Mini-Review.Mol.Phys. Rep.1999, 26, 11-24, and references therein.

(19) Kriste B. Methods for Automated Analysis and Simulation of ElectronParamagnetic Resonance Spectra.Anal. Chim. Acta1992, 265, 191-200.

(20) Fogel, L.; Owens, A.; Walsh, M.Artificial Intelligence throughSimulated EVolution; John Wiley: 1966.

(21) (a) Rechenberg, I.EVolutionsstrategie: Optimierung TechnischerSysteme nach Prinzipien der Biologischen EVolution; Frommann-Holzboorg: Stuttgart, 1973. (b) Schwefel, H.-P.Numerical Optimiza-tion of Computer Models; John Wiley and Sons: 1981.

(22) Holland, J.Adaptation in Natural and Artificial Systems; Universityof Michigan Press: 1975.

(23) Michalewicz, Z.Genetic Algorithms+ Data Structures) EVolutionPrograms; Springer: Berlin, 1992.

(24) Davis, L.Handbook of Genetic Algorithm; Van Nostrand Reinhold:New York, 1991.

(25) Sojka, Z.; Stopa, G. Analysis of the Isotropic EPR Spectrum ofK3[Cr(CN)5NO]. An Inorganic Chemistry Laboratory Experiment.J.Chem. Educ.1993, 70, 675-678.

(26) Jong, A. K. D.An Analysis of the BehaVior of a Class of GeneticAdaptiVe Systems, Ph.D. Thesis, University of Michigan, Ann Arbor,1975.

(27) Choy, W. Y., Sanctuary, B. C. Using Genetic Algorithms with a PrioriKnowledge for Quantitative NMR Signal Analysis.J. Chem. Inf.Comput. Sci.1998, 38, 685-690.

CI049863S


Application of the Genetic Algorithm Joint with the Powell Method to Nonlinear Least-Squares Fitting...

Documents

Transcript of Application of the Genetic Algorithm Joint with the Powell Method to Nonlinear Least-Squares Fitting...