Applied Soft Computing 3 (2003) 97–122

Efficient Differential Evolution algorithms formultimodal optimal control problems

I.L. Lopez Cruz∗, L.G. Van Willigenburg, G. Van StratenSystems and Control Group, Wageningen University, Mansholtlaan 10,6708 PA, Wageningen, Netherlands

Received 29 April 2002; received in revised form 24 August 2002; accepted 16 December 2002

Abstract

Many methods for solving optimal control problems, whether direct or indirect, rely upon gradient information and thereforemay converge to a local optimum. Global optimisation methods like Evolutionary algorithms, overcome this problem. In thiswork it is investigated how well novel and easy to understand Evolutionary algorithms, referred to as Differential Evolution(DE) algorithms, and claimed to be very efficient when they are applied to solve static optimisation problems, perform onsolving multimodal optimal control problems. The results show that within the class of evolutionary methods, DifferentialEvolution algorithms are very robust, effective and highly efficient in solving the studied class of optimal control problems.Thus, they are able of mitigating the drawback of long computation times commonly associated with Evolutionary algorithms.Furthermore, in locating the global optimum these Evolutionary algorithms present some advantages over the Iterative DynamicProgramming (IDP) algorithm, which is an alternative global optimisation approach for solving optimal control problems. Atpresent little knowledge is available to the selection of the algorithm parameters in the DE algorithm when they are appliedto solve optimal control problems. Our study provides guidelines for this selection. In contrast to the IDP algorithm the DEalgorithms have only a few algorithm parameters that are easily determined such that multimodal optimal control problemsare solved effectively and efficiently.© 2003 Elsevier B.V. All rights reserved.

Keywords: Optimal control; Evolutionary algorithms; Differential Evolution algorithms; First-order gradient algorithm; Iterative DynamicProgramming

1. Introduction

Indirect numerical methods for optimal controlbased on Pontryagin’s minimum principle (PMP) usegradient information and local search methods. There-fore, if the optimal control problem is multimodal,convergence to a local optimum is likely. Determin-istic direct methods for optimal control parameterisethe controls and also use gradient information and

∗ Corresponding author. Tel.:+52-595-952-1551;fax: +52-595-952-1551.E-mail address: ilopez@taurus1.chapingo.mx (I.L. Lopez Cruz).

local search methods to solve the resulting non-linearprogramming (NLP) problem. Consequently, theymay also converge to a local solution. The simplestway to increase the chances of finding the global solu-tion by these approaches is by repeating them severaltimes with different control initialisations. Doing so,there still are optimal control problems that require avery close guess to the global optimum. To locate theglobal optimum or a sufficiently close approximation,global optimal control approaches are needed. An ap-proximate global solution may be used to initialise adirect or indirect local optimisation method to obtainthe global solution accurately.

1568-4946/$ – see front matter © 2003 Elsevier B.V. All rights reserved.doi:10.1016/S1568-4946(03)00007-3

98 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

Global optimisation methods or multimodal opti-misation algorithms can roughly be divided into twogeneral groups: deterministic[1–6] and stochastic[2,7–13]. Simulated annealing is another well-knownglobal stochastic optimization approach[14,15].Although Evolutionary algorithms (Genetic algo-rithms [16–18], Evolution Strategies[19,20], Evo-lutionary Programming[21]) can be considered asheuristic–stochastic global search methods, they havereceived limited attention among the global optimi-sation research community[22]. One possible expla-nation is the extended believe that all Evolutionaryalgorithms are not efficient enough to solve contin-uous global optimisation problems[23]. However,recently, new algorithms inspired by evolution havebeen developed that are rather different from a simpleGenetic algorithm[24,25].

Moreover, though global optimisation methodsgenerically are deterministic and stochastic, one canfind that actually four approaches have been at-tempted in the solution of multimodal optimal controlproblems, namely, Deterministic algorithms, Stochas-tic algorithms, Iterative Dynamic Programming, andEvolutionary algorithms.

Rosen and Luus[26] proposed a method based online search techniques to determine starting points fortheir NLP solver. Their approach fails when the prob-lem is highly multimodal. Strekalovsky and Vasiliev[27] presented a search method for convex objectivesbut convergence of their approach is guaranteed foronly a limited class of problems. Esposito and Floudas[28] have proposed a Deterministic algorithm to ad-dress the non-linear optimal control problem to globaloptimality. The approach is based on a branch andbound algorithm (�BB). By using some benchmarkproblems from non-linear optimal control a theoreti-cal guarantee of attaining the global optimum of themultimodal optimal control is offered as long as rig-orous values of the parameters needed or rigorousbounds on the parameters are obtained. However, theissue of global optimality remains open. Papamichailand Adjiman[29] have proposed a new Deterministicglobal optimization algorithm for dynamic optimiza-tion problems, based on BB techniques for NLP prob-lems. However, their algorithm has not been evaluatedwith highly multimodal optimal control problems asthe authors acknowledged. One advantage of Deter-ministic algorithms is that convergence proofs can be

presented. The same does not happen in case of prob-abilistic algorithms. One disadvantage is that it de-mands certain structure of the NLP problem in orderto guarantee the global optimum is achieved.

Ali et al. [30] applied and evaluated the perfor-mance of several Controlled Random Search[8–10]and modified controlled random search algorithms[31] on the solution of a low multimodal optimalcontrol problem. This class of algorithms is based onthe use of a population of search points. Within thisclass, Banga and Seider[32] proposed the IntegratedControlled Random Search for Dynamic SystemsEngineering. They applied their algorithm to somedynamic optimization problems of bioprocesses andbioreactors[33]. Carrasco and Banga[34] proposedanother Stochastic algorithm for global optimizationnamed Adaptive Randomly Directed Search for Dy-namic Systems. Both algorithms use only one searchpoint and utilize normal and uniform random distribu-tions, respectively, in order to generate new solutions.The first algorithm did not achieve the global solutionof a highly multimodal optimal control problem, soa combination of Stochastic algorithms and a deter-ministic approach has been proposed recently[35].

Luus and co-workers have attempted to solvemultimodal optimal control problems by DynamicProgramming[36]. Luus and Galli[37] showed themultimodality of a CSTR dynamic optimization byusing dynamic programming. Luus and Bojkov[38]applied dynamic programming iteratively to solvethe bifunctional catalyst problem which is a highlymultimodal optimal control problem[39]. Bojkovand Luus [40] found advantages in using randomvalues for control in their Iterative Dynamic Program-ming approach. Several classes of optimal controlproblems have been solved by using IDP. Singularoptimal control problems[41], problems with finalstate constraints[42], problems with state inequalityconstraints[42], time optimal control problems[43]and problems from chemical engineering like the op-timal control of fed-batch bioreactors. A summary ofthe IDP approach to dynamic optimization problemsin presented in Luus[44]. Recently, Mekarapirukand Luus[45] applied both randomly and determin-istically selected candidates for control in the IDPalgorithm. They found some advantages in using thishybrid approach when solving some highly non-linearand multimodal chemical engineering optimal control

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 99

problems. Two types of deterministic control candi-dates (shifting and smoothing candidates) are chosenon the basis of the control policy obtained in the pre-vious iteration. In spite of its success IDP has manyalgorithm parameters that have to be tuned.

Although several researchers have applied Evo-lutionary algorithms to optimal control problems inthe past, to our best knowledge neither there areprevious studies on the performance of Evolutionaryalgorithms nor on the use of Differential Evolutionalgorithms to solve multimodal optimal control prob-lems. Michalewicz et al.[46] applied floating-pointGenetic algorithms to solve discrete time optimalcontrol problems. Sewald and Kumar[47] appliedcanonical Genetic algorithms to solve optimal controlproblems with linear controls. Yamashita and Shima[48] used classical Genetic algorithms to solve freefinal time optimal control problems with terminalconstraints. Smith[49] proposed an evolution pro-gram for continuous optimal control problems. Dakevand co-workers[50,51] used algorithms based on theBreeder Genetic algorithm to solved optimal controlproblems. Pohlheim and Hei�ner [52,53] appliedGenetic algorithms and also Evolution Strategies tosolve dynamic optimization problems of greenhouseclimate. Sim et al.[54] used a combination of Geneticalgorithms and a shooting method to solve continuousoptimal control problems. Roubos et al.[55] appliedGenetic algorithms with floating-point chromosomesto solve continuous time optimal control problemsof bioreactors. Hashem et al.[56] applied a modifiedEvolution Strategy on the solution of discrete timeoptimal control problems. Recently, Wang and Chiou[57] applied a Differential Evolution algorithm to theoptimal control and location time problems of differ-ential algebraic equations. Lee et al.[59] used a mod-ified Differential Evolution algorithm to the dynamicoptimization of continuous polymer reactor. Chiouet al.[58] proposed two new operators to improve thespeed of convergence and to avoid convergence to lo-cal minima of Differential Evolution algorithms andthey solved some static and dynamic optimization offed-batch fermentation. An extended review on theapplications of evolutionary algorithms for optimalcontrol and additional information on DifferentialEvolution algorithms is given in reference[60].

The present paper, studies Evolutionary algorithms(EAs) which are used to solve two optimal control

problems that are known to have several local min-ima. Firstly, a First-order gradient algorithm fromclassical optimal control theory is used to solve bothproblems. The objective is to illustrate some limita-tions of this approach in solving multimodal optimalcontrol problems. Next, the performance of four Evo-lutionary algorithms is compared with that obtainedby Iterative Dynamic Programming. It is well knownthat many Evolutionary algorithms (i.e. Genetic algo-rithms) tend to be inefficient computationally whenthey are applied to continuous parameter optimisationproblems. Since the computation time is often criticalin solving optimal control problems, the design ofmore efficient evolutionary algorithms is an importantchallenge. In this work it is investigated how wellnovel and easy to understand evolutionary algorithms,referred to as Differential Evolution[24,61–64] al-gorithms, and claimed to be very efficient when theyare applied to solve static optimisation problems,perform on solving multimodal optimal control prob-lems. Additionally, almost no knowledge is availableon how to choose the algorithm parameters that steerthe optimisation process, when Differential Evolu-tion algorithms are applied to solve multimodal dy-namic optimisation problems. Hence, in this work,it is investigated how the DE algorithm parameters‘population size’, ‘crossover constant’ and ‘differ-ential variation coefficient’ act upon its efficiencyand effectiveness in solving the selected benchmarkproblems.

The paper is organised as follows: inSection 2,a general description of the class of optimal controlproblems we are interested in is given. InSection 3, ageneral description of an Evolutionary algorithm, andthe specific characteristics of both a real-valued Ge-netic algorithm with sub-populations and the Differen-tial Evolution algorithm are provided. InSection 4, abrief description of a first-order gradient algorithm forthe solution of optimal control problems is given andalso the main properties of the Iterative Dynamic Pro-gramming algorithm are described.Section 5presentsresults obtained when the studied evolutionary algo-rithms were applied to two benchmark optimal con-trol problems belonging to the class of interest. Theseresults are then compared to those obtained with theindirect and gradient method and the direct IterativeDynamic Programming algorithm. A general discus-sion is presented at the end of the paper.

100 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

2. The class of optimal control problems

Consider the class of optimal control problemswhere the system is described by the non-linear time-varying dynamic equation:

x = f(x(t), u(t), t), (1)

wherex(t) ∈ Rn is the state andu(t) ∈ Rm is thecontrol. The control inputs are constrained,

ai(t) ≤ ui(t) ≤ βi(t), i = 1,2, . . . , m, (2)

whereαi(t) andβi(t) are known time functions. Fur-thermore,

x(0) = x0, (3)

is the known initial condition. Then, the optimal con-trol problem is to find the inputu∗(t), t ∈ [t0, tf ] thatdrives the plant along the trajectoryx∗(t), t ∈ [t0, tf ]such that the cost function

J(u(t)) = φ(x(tf ), tf )+∫ tf

0L(x(t), u(t), t)dt (4)

is minimised where the final timetf is fixed[65]. Thereare two general approaches to solve these problemsnumerically: indirect and direct methods[66]. The firstgroup is based on the solution of a calculus of vari-ations problem through the use of the Pontryagin’sminimum principle (PMP)[67]. In a direct approach,on the other hand, the optimal control problem (1)–(4)is approximated by a finite dimensional optimisationproblem, which can be cast in a non-linear program-ming (NLP) form and solved accordingly[68,69]. Thisis achieved through control parameterisation. In ourcase the controlu(t) is assumed to be piecewise con-stant

u(tk) = uk, t ∈ [tk, tk+1], k = 0,1, . . . , N − 1,

t0 = 0, tN = tf . (5)

This is a realistic assumption in the case of digi-tal control. As a resultN × m parameters determinethe control over [0,tf ]. The NLP problem is to findthe stacked control vectoru ∈ Rm×N defined byu = [uT

0 , uT1 , . . . , u

TN−1] = [u1, . . . , um×N ], where

ui, i = 1,2, . . . , m×N are scalars.

3. Two classes of Evolutionary algorithms:Breeder Genetic algorithms and DifferentialEvolution

Following Bäck [20] a generic description ofan Evolutionary algorithm is presented inFig. 1.An Evolutionary algorithm is a stochastic searchmethod, which maintains a populationP(g) :={�a1(g), . . . , �aµ(g)} of individuals (chromosomes)�ai ∈ I; i = 1, . . . , µ, at generationg, whereI is aspace of individuals, andµ is the parent populationsize. Each individual represents a potential solution ofthe problem and is implemented as some generic datastructure (strings of bits in Genetic algorithms, realnumbers in Evolution Strategies). By means of themanipulation of a family of solutions, an Evolutionaryalgorithm implements a survival of the fittest strategyin order to try to find the best solution to the problem.

Each individual is evaluated by a fitness functionΦ : I → R, such that a real value is assigned toeach potential solution, which is a measure of howindividuals perform in the problem domain. Next, aniterative process starts in which a set of evolution-ary operators is applied to the population in order togenerate new individuals.

From a set{wΘ1, . . . , wΘz|wΘi : Iλ → Iλ} ∪{wΘ0 : Iµ → Iλ} of probabilistic evolutionarywΘioperators[70] (for instance: crossover, mutation), eachone specified by parameters given in the setsΘi ∈ R,some operators are applied to the population and anew evaluation of its fitness is calculated. The mainevolutionary operators applied to the populationP(g)

Fig. 1. Procedure Evolutionary algorithm.

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 101

are recombination (crossover)rΘr : Iµ → Iλ andmutationmΘm : Iλ → Iλ. A selection operatorsΘs :(Iλ ∪ Iµ+λ)→ Iµ which may modify the number ofindividuals fromλ or λ + µ to µ, is applied as well,whereλ,µ ∈ N, λ the number of offspring. As before,the selection operator may be governed by a set ofparameters defined inΘs. The setQ ⊂ P(g) denotesan additional set of individuals. The functionι : Iµ →{true, false} represents the termination criterion for theevolutionary algorithm. After a number of generations,it is expected that the best individual of the populationrepresents a near-optimum global solution.

3.1. Two Evolutionary algorithms based on theBreeder Genetic algorithm

The Breeder Genetic algorithm (BGA) is one ofthe most efficient Genetic algorithms available inthe domain of continuous parameter optimisation. Inaddition, an extended theory has been proposed thatverifies some practical results[71]. BGA utilises areal number representation for a chromosome. Thatis, µ real vectors of dimensiond make up the pop-ulation P(g). According to the previous notation:�a = �x = (x1, . . . , xd) ∈ Rd . Each potential solu-tion in the evolutionary framework consists of thevector u = [u1, u2, . . . , um×N ] of parameters ob-

Fig. 2. A graphical representation of both operators mutation of the BGA and discrete recombination in the two-dimensional space.

tained from the transformation of the continuousoptimal control problem into a NLP problem. Theonly necessary modification is a rearrangement ofparameters, in order to ensure that consecutive real-izations of one single element of the control vectorappear in adjacent positions in a chromosome. Thatis, a chromosome is implemented specifically as thevector of floating-point numbers:�a = u, so d =m×N.

The genetic operators that have been chosen to makeup the Evolutionary algorithm are (i) crossover by dis-crete recombination (seeFig. 2), (ii) mutation by theoperator of the Breeder Genetic algorithm (seeFig. 2),and (iii) selection by Stochastic Universal Sampling.Also the option of sub-populations, to be describedlater, has been implemented in order to increase thechances to find the global optimum. According toMühlenbein and Schlierkamp-Voosen[71], the dis-crete recombinationrd : I2 → I (crossover operator)is defined as follows: let�a1 = (a1

1, . . . , a1d) and�a2 =

(a21, . . . , a

2d) be two parent chromosomes. Then each

element of the offspring�a3 = (a31, . . . , a

3d) is com-

puted by

a3i =

{a1i if rand( ) < 0.5

a1i otherwise

, i = 1, . . . , d , (6)

102 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

where rand( ) is a uniform random number from [0,1].This operator is appliedµ times by picking up parentsrandomly in order to create an offspring population.

The mutation operator of the Breeder Geneticalgorithm m{pm,rs} is defined as follows: let�a =(a1, . . . , ad) be a parent solution. Then, for a givenprobability of mutationpm ∈ [0,1] and a specified‘shrinking mutation range’rs ∈ [0,1] a gene (vari-able) a′1 is selected and modified to generate a newvariable according to:

a′i ={ai+mr · rangei · δ, if rand() < 0.5

ai −mr · rangei · δ, otherwise, i = 1, . . . , d , (7)

where

rangei = 12rs(βi − αi),

mr = 1, if rand( ) < pm

0, otherwise, δ =

19∑j=0

γj2−j ,

rand( ) is a uniform random number from [0,1],γj ∈[0,1] with probability 0.05,pm = 1/d normally, andrs are algorithm parameters that have to be specified,αi andβi denote the lower and upper boundaries of thevariableαi. With the given settings forδ the mutationoperator is able to locate the optimum up to a precisionof rangei · rs × 2−19.

The selection operators : Iµ+λ → Iµ consists ofa combination of an elitist selection mechanism andthe stochastic universal sampling algorithm. Firstly,the objective functionf(�ai = J(ui), i = 1, . . . , µ)is calculated, which is equal to the cost function ofthe optimal control problem. The costJ(u) is eval-uated through integration of the dynamic equation(Eq. (1)) given parameterised control. Then, the fit-ness functionΦ(�a) is calculated using a linear rankingscheme:

Φ(�ai) = 2 − sp + 2(sp − 1)f ′(�ai)− 1

µ− 1;

i = 1, . . . , µ, (8)

with the selection pressure coefficientsp = 2.0 andf ′(�ai) the index position in the descending orderedpopulation of the objective function value of indi-vidual i. The stochastic universal sampling algorithmpicks the parent chromosomes for the new populationsuch that the probability for�ai being picked equals

ps(�ai) · ps(�ai) are calculated according to:

ps(�ai) = Φ(�ai)sΦ

, sΦ =µ∑j=1

Φ(�aj), (9)

whereΦ(�ai) is the fitness of individual�ai. To im-plement an elitist selection scheme new individualsare generated as a fraction of the population sizeλ = µggap whereggap is termed the generation gap, aparameter determined by the user. Once offsprings are

generated and their fitness functions calculated theyare inserted into the new population. An insertionfunction replaces the worst performing individuals al-lowing the best previous solutions to belong to the newpopulation in order to maintain the size of the originalpopulationµ.

Fig. 3 shows a flow chart of the Breeder Geneticalgorithm in which the main operators are indicated.When the algorithm does not consider several popu-lations migration does not take place.

The sub-population methodology divides the wholepopulation in multiple subpopulations or demes. Theevolutionary operators evolve during a number of gen-erations for each sub-population. From time to timesome individuals migrate from one sub-population toanother. Three parameters have to be specified: themigration rate, the manner of selection of individualsfor migration and the topology over which migrationtakes place. The migration rate is only a scalar num-ber, which specifies the number of individuals to bemigrated. The individuals to be migrated can be se-lected randomly or according to their fitness. There arethree main migration topologies: a ring in which onlyadjacent sub-populations can interchange individuals,a neighbourhood migration, which is an extension ofthe previous one where migration in each adjacentsub-population is allowed. Finally, unrestricted migra-tion topology, in which individuals may migrate fromany sub-population to another. There is some evi-dence showing that sub-populations help evolutionaryalgorithms to locate the global optimum[72]. Thecomputer implementation of these Evolutionary algo-rithms is given in the Genetic algorithm toolbox for usewith MATLAB [73]. The integration of the dynamic

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 103

Fig. 3. Flow chart describing the Breeder Genetic algorithm with subpopulations.

equations was implemented by using a C-MEX fileroutine in order to speed up the simulations.

3.2. Differential Evolution algorithms

Differential Evolution algorithms are evolutionaryalgorithms that have already shown appealing featuresas efficient methods for the optimisation of continuousspace functions. Storn[24] have reported impressiveresults that show DE outperformed other evolution-ary methods and the stochastic differential equations

Fig. 4. A graphical representation of the crossover and mutation operators for the DE/rand/1/bin algorithm in the two-dimensional space.

approach on solving some benchmark continuousparameter optimisation problems. These algorithmsuse a floating-point representation for the solutions inthe population. However, the evolutionary operatorsare completely different from those used in methodsknown as Genetic algorithms, Evolution Strategiesand Evolutionary Programming. In DE, the mutationoperatorm{F } : Iµ → Iµ consists of the generationof µ mutated vectors according to:

�vi = �ar1 + F(�ar2 − �ar3), i = 1,2, . . . , µ, (10)

104 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

where the random indicesr1, r2, r3 ∈ [1,2, . . . , µ]are mutually different and also different from theindex i. F ∈ [0,2] is a real constant parameter thataffects the differential variation between two vectors(seeFig. 4). Greater values ofF and/or the populationsize (µ) tend to increase the global search capabilitiesof the algorithm because more areas of the searchspace are explored.

The crossover operatorr{CR} : I2 → I combinesthe previously mutated vector�vi = [v1i, v2i, . . . , vdi]with a so-called target vector (a parent solution fromthe old population)�ai = [a1i, a2i, . . . adi] to generate aso-called trial vector�a′i = [a′1i, a

′2i, . . . , a

′di] according

to:

a′ji ={vji, if (randb(j) ≤ CR) or j = rnbr(i)

aji, if (randb(j) > CR) and j �= rnbr(i), j = 1,2, . . . , d; i = 1,2, . . . µ , (11)

where randb(j) ∈ [0,1] is thejth evaluation of a uni-form random number generator, rnbr(i) ∈ 1,2 . . . , dis a randomly chosen index. CR∈ [0,1] is thecrossover constant, a parameter that increases thediversity of the individuals in the population. The‘hyper-cube’ boundaries generated by the crossoveroperator represented by the dotted rectangle inFig. 4.�a′i can take one of the three corners except for�ai.Greater values of CR give rise to a child vector (�a′i)more similar to the mutated vector (�vi). Therefore, the

Fig. 5. Flowchart of a Differential Evolution algorithm.

speed of convergence of the algorithm is increased.As can be seen fromEq. (11), each member of thepopulation plays once the role of a target vector.It is important to realise that even when CR= 0,Eq. (11)ensures that parent and child vectors differby at least one gene (variable). The three algorithmparameters that steer the search of the algorithm, arethe population size (µ), the crossover constant (CR)and the differential variation factor (F). They remainconstant during an optimisation.The selection oper-ator s : Iµ → Iµ pares the cost function value ofthe target vector�ai with that of the associated trialvector�a′i, i = 1,2, . . . , µ and the best vector of thesetwo becomes a member of the population for the

next generation. That is,

If Φ(�a′i(g)) < Φ(�ai(g)), then

�ai(g+ 1) := �a′i(g); else

�ai(g+ 1) := �ai(g+ 1) := �ai(g), i = 1, . . . , µ,

(12)

whereg denotes the current generation.Fig. 5presentsa flowchart with the main operators of a DifferentialEvolution algorithm.

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 105

It appears that the differential mutation as designedin DE fulfil three properties that are crucial for an ef-ficient mutation scheme[61]. Firstly, DE guaranteesa distribution with zero mean by randomly samplingdifference vectors, i.e.�xr1 − �xr2 has the same prob-ability of being selected than the opposite�xr2 − �xr1has. Secondly, the distribution of vector differentialsis automatically self-scaling. DE scales mutation stepsizes by scaling their relative magnitude. Thirdly, DEis rotational invariant since the mutation distributiongenerated by difference vectors will always have thesame orientation as the level lines of the objectivefunction. In addition, it seems that DE algorithmshave a property that Price has called a universal globalmutation mechanism or globally correlated mutationmechanism, which seems to be the main propertyresponsible for the appealing performance of DE asglobal optimisers.

Different Differential Evolution (DE) algorithmscan be distinguished by the properties of the muta-tion and crossover operators. In this research two DEalgorithms that are claimed to be the most efficientare tested. First, the Differential Evolution algorithmdenoted by DE/rand/1/bin[24,61–63]that stands for:a Differential Evolution algorithm in which the mu-tant vector is selected randomly from de population,while only one difference vector is considered by themutation operation as described byEq. (10)while thecrossover scheme is due to independent binomial ex-periments. The second DE algorithm studied is shortlydescribed as DE/best/2/bin[24,62,63]. In this casethe mutated vectors instead of (10) are described by

�vi = �abest+ F(�ar1 + �ar2 − �ar3 − �ar4), (13)

where�abest is the current best solution in the popula-tion.

Since originally DE algorithms were designedto solve unconstrained static optimisation problems[24], a modification is required in order to deal withconstraints for the controls (Eq. (2)). A clipping tech-nique has been introduced to guarantee that onlyfeasible trial vectors are generated after the mutationand crossover operators:

a′ji(g) ={βj, if a′ji(g) > βjαj, if a′ji(g) > αj

, j = 1,2, . . . , d, i = 1,2, . . . , µ, (14)

where αj and βj represent the lower and upperboundaries of the control variables, respectively. Aremarkable advantage of Differential Evolution algo-rithms is its simplicity, which means that it is rela-tively easy to understand how they work. Also theyare easy to program. Our computer implementationis based on the MATLAB environment. The core ofthe algorithm is an m-file that calls a Simulink modelprogrammed as a C-MEXs-function, which containsthe dynamic equation of the system and the objectivefunction.

4. The first-order gradient algorithm and theIterative Dynamic Programming algorithm

4.1. The gradient algorithm

The numerical solution of the optimal controlproblem described inSection 2can be accomplishedby means of a first-order gradient algorithm prop-erly modified with a clipping technique to dealwith constraints for the controls. The basis is thealgorithm described by Bryson[67]. However, aline search procedure was introduced in order tocalculate the value of the step size parameter (k),which in Bryson’s algorithm is constant. The gra-dient algorithm is described next and applies to aMayer formulation of the optimal control problem[67].

(i) Guessu(t) atN+1 pointst−t0 = 0, . . . , N ,T,,T = (tf − t0)/N. N is an even number.

(ii) Integrate the state equations forward. Storex(t)at t − t0 = ,T, . . . , N ,T .

(iii) Evaluateφ[x(tf )] andλT(tf ) = (∂φ/∂x)(tf ).(iv) Compute and storeλ(t) and the function

(∂H(x, u, λ, t)/∂u) at t − t0 = ,T, . . . , N ,T ,by integrating backward in timeλ = −λT

(∂f(x(t), u(t), t)/∂x), starting at λ(tf ), where(∂H(x, u, λ, t)/∂u)=λT(t)(∂f(x(t), u(t), t)/∂u).

(v) Apply a line search algorithm to determine thestep size parameter (k).

106 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

(vi) Computeδu(t) and the newu(t) according to:δu(t) = −k(∂HT/∂u)(t), u(t) := u(t)+ δu(t).

(vii) Clip the controls if necessary in accordancewith:

u(t) ={αt, if u(t) > α(t)

βt, if u(t) > β(t).

(viii) If |δuavg| < εg stop. Otherwise go to step (ii),whereεg > 0 is a desired precision andδuavg =√(1/tf )

∫ tf0 δu

T(t)δu(t)dt.

The previous algorithm was implemented in an en-hanced MATLAB-Simulink program with a C-MEXfile s-function so as to speed up the simulation of thedynamic system.

4.2. Iterative Dynamic Programming algorithm

An iterative version of the Dynamic Programmingalgorithm has been proposed by Luus[44] as a highlyreliable method for locating the global optimum inoptimal control problems. A brief description of thisapproach is given next. For more details one is referredto [44].

Step 0. Initialisation: The time interval [0,tf ] is di-vided into N time intervals, each of lengthL. The con-trol is approximated by the piecewise constant controlpolicy u(t) = u(tk) ∈ [tk, tk+1], k = 0, . . . , N − 1,tk+1 − tk = L, t0 = 0, tN = tf . Chooseu0(0), . . . , u0(N − 1) and r0(0), . . . , r0(N − 1)wherer0(k), k = 0,1, . . . , N − 1 specifies the rangeu0(k) ± r0(k) of allowable values of control for thenext iterationu1(k), k = 0, . . . , N − 1 Select thenumber of allowable values for controlR > 1 to betried at each stagek = 0, . . . , N − 1. Choose the re-gion contraction factor 0.5 ≤ γ < 1.0 and the numberof grid points M for the states. Finally, specify thenumber of iterationsI. Set iteration numberi = 1.

Step 1. Use the best control policy from the pre-vious iteration (the guess solution at iteration 1)u∗i−1(0), . . . , u

∗i−1(N − 1), and generateM − 1

other control policies within the regionu∗i−1(k) ±

ri−1(k), k = 0, . . . , N − 1. Integrate the dynamicequation (Eq. (1)) from t = 0 to tf for all M controlpolicies. TheM values ofxm(k), k = 0, . . . , N − 1,

m = 1,2, . . . ,M at the beginning of each time stageare stored.

Step 3.

(a) At stageN, for each of theM stored valuesfor xm(N − 1), integrate the dynamic equation(Eq. (1)) from tf − L to tf , with each of theRallowable values for the control, which are gener-ated by:

ui(N − 1) = u∗i−1(N − 1)+ D · r i−1(N − 1),

(15)

where u∗i−1(N − 1) is the best control value

obtained in the previous iteration andD is adiagonal matrix of different uniform randomnumbers between−1 and 1. To deal with con-straints of the controls whenever an unfeasiblesolution is generated it is set to the violated limit,according to:

ui(N − 1) ={α(t), if ui(N − 1) < α(t)

β(t), if ui(N − 1) > β(t).

(16)

(b) From theR values of the control choose the oneu∗i (N − 1) that gives the best performance index

and store these values.

Step 4.

(a) Step back to stageN − 1 and repeat step 3a wereN is replaced byN − 1.

(b) The integration is continued fromx(N − 1)over the last time intervaltf − L to tf using thestored value foru∗

i (N − 1) corresponding tothe state grid point closest to the value of thecalculated state vector at timetf − L. From theR values of the control selectu∗

i (N − 2) thatgives the best performance over the time interval[N − 2, N].

Step 5. Continue the procedure until stageN = 1 isreached corresponding to the initial timet = 0. Here,there is only one state grid point that corresponds tothe initial conditions (Eq. (3)). Store the trajectoriesu∗i (k), x

∗(k), k = 0,1, . . . , N − 1.

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 107

Step 6. Reduce the region for allowable values of thecontrol.

ri+1(k) = γ · ri(k), k = 0,1, . . . N − 1. (17)

Select the best control obtained from step 5 as themidpoint for the allowable values for control. Seti =i+ 1 and go to step 1.

The previous algorithm is continued for the speci-fied number of iterationsI and after that the results areanalysed. Sometimes, the allowable values for controlsare selected from a uniform distribution (evenly spacedgrid) instead of randomly. Also for some problems itis desirable to use a multi-pass method, which selectsthe value of the region contraction factorγ equal toa fraction of its size at the beginning of the previouspass. This is implemented to prevent a premature col-lapse of the search region[44]. In that case anotherparameter called region restoration factor 0.5 ≤ η ≤1.0 is used. The number of passes (P) must be definedas well.

From the previous description is apparent that theIDP algorithm has numerous algorithm parametersthat can be varied. The region contraction factor (γ),number of allowable values for control (R), numberof grid points (N), initial region size values (r0(k)),and restoration factor (η) in case of multiple passes.Some insight has been obtained about their valuesas one is applying IDP to a particular problem, butin general a parameter tuning approach is required.Luus has reported[44] that with too small values ofthe region contraction factor (γ) premature collapseof the regionr(k), k = 0,1, . . . , N − 1 is very likelyand too large values give rise to a very slow conver-gence rate or no convergence at all. Also it is knownthat small values ofγ work properly with sufficientlylarge values of the allowable values for control (R).Conversely, when small values ofR are used, highvalues ofγ are required to increase the chances offinding the global optimum. The allowable numberfor controls should be chosen as small as possiblein order to reduce the computational load. Regardingthe number of grid points (M) is known that for someproblemsM = 1 works fine, but in other problemsM > 1 may be necessary. Our computer programof the described Iterative Dynamic Programmingalgorithm for the MATLAB-Simulink environmentis an enhanced code, which uses a C-MEX file

s-function to speed up the simulation of the dynamicequations.

5. Benchmark problems

5.1. The optimal control of a non-linear stirredtank reactor

A multimodal optimal control problem has beenused by Luus[44] to evaluate his Iterative DynamicProgramming algorithm. Ali et al.[30] solved thisproblem by stochastic global optimisation algorithms.Also, this problem is a member of the list of bench-mark problems proposed in the Handbook of TestProblems in Local and Global Optimization[74]. Afirst-order irreversible chemical reaction carried outin a continuous stirred tank reactor (CSTR) has beenmodelled by two non-linear differential equationsthat are the result of a heat and mass balance of theprocess.

x1 = −(2 + u)(x1 + 0.25)

+ (x2 + 0.5)exp

(25x1

x1 + 2

), (18)

x2 = 0.5 − x2 − (x2 + 0.5)exp

(25x1

x1 + 2

), (19)

wherex1 represents the deviation from dimensionlesssteady-state temperature andx2 stands for the devi-ation from the dimensionless steady-state concentra-tion. The controlu(t) represents the manipulation ofthe flow-rate of the cooling fluid, which is inserted inthe reactor through a coil. The optimal control prob-lem is to determine the unconstrainedu∗(t) that min-imises the performance index:

J =∫ tf

0(x2

1 + x22 + 0.1u2)dt, (20)

where tf = 0.78. The initial conditions arex(0) =[ 0.09 0.09]T. It can be shown that this problem hastwo solutions. In solving this problem numerically theintegration of the dynamic system was performed withthe ode45 routine available in MATLAB, with the rel-ative tolerance error set to 1×10−8. The initial guessesfor the controls of the different algorithms were se-lected from the interval 0≤ u(t) ≤ 5.0.

108 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

5.1.1. First-order gradient methodA solution of this problem by the gradient method

was obtained keeping the step size parameter (k =0.12) constant. So the line search was not used. Sincethis is an optimal control problem with fixed finaltime and without bounds for the controls, and becausethe partial derivatives can be calculated analytically,Bryson’s Matlab code for Continuous Dynamic Opti-mization[67] without constraints was applicable. Theaccuracy of the criterion of convergence was specifiedas εg = 0.0001. The convergence of the algorithmwas straightforward. The convergence of the first or-der gradient algorithm to the local or global optimumdepends on the initial values for the control. Actually,by using a constant pattern as initial guess,u0(t) =c, 0 ≤ t ≤ tf the gradient algorithm always convergedto the local optimum (J∗ = 0.2444) if u0(t) ≤ 1.8;otherwise it converges to the global optimum (J∗ =0.1330).Fig. 6 shows the two optimal control trajec-tories associated with the two values of the functional(J). It can be seen that the local and global optimalcontrol have completely different shapes.

Fig. 6. Optimal control trajectories of the CSTR problem calculated by a first-order gradient method.

5.1.2. Iterative Dynamic ProgrammingIn order to solve this problem by means of direct

methods, the time interval [0,tf ] was discretized inN = 13 time intervals since it has been reported thata good approximation to the continuous-time optimalcontrol is obtained by doing so[30,44]. A piecewiseconstant approximation for the control was used ateach of those time intervals. In the IDP algorithmthe parameters values suggested by Luus[44] werechosen: the number of state grid pointsM = 1, thenumber of allowable values for controlR = 15. Theregion reduction factor wasγ = 0.80, the number ofiterationsI = 20, the number of passesP = 3 and theregion restoration factorη = 0.5 after each pass. Theallowable values for control were generated randomly(seeSection 4.2). In order to achieve comparable con-ditions among all the direct methods, firstly, the initialcontrol trajectory was chosen constant, i.e.u0(tk) = c,k = 0,1, . . . , N − 1 with c a value randomly takenfrom the control interval 0≤ u(t) ≤ 5. A similarprocedure was followed for the selection of the initialregion valuer0(tk), k = 0,1, . . . , N − 1 which was

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 109

Table 1Optimal control of a multimodal CSTR by Iterative Dynamic Programming

u0(tk) r0(tk) J∗ CPU time (s)a u0(tk), r0(tk) = 4 J∗ CPU time (s)a

3.4990 2.2999 0.1355869 600.79 1.0000 0.1355852 700.502.3744 3.1557 0.1356138 620.46 4.2823 0.1356766 686.300.9718 4.1646 0.1355876 600.16 2.0389 0.1356422 668.741.1568 1.2498 0.2446122 589.43 3.9007 0.1356262 665.343.8732 4.4433 0.1356089 590.77 0.7900 0.1355806 764.054.0504 3.4722 0.1355970 635.90 0.0108 0.1355933 701.570.7557 1.2997 0.2446122 599.58 2.3551 0.1355905 872.464.7105 3.7785 0.1355856 602.31 2.7851 0.1355866 849.924.6422 2.2919 0.1355816 599.64 1.0293 0.1356085 851.204.0494 0.8660 0.1355828 588.51 2.9137 0.1355811 683.33

0.1356070b 744.34b

a Measured on a Pentium III 700 MHZ PC, function evaluations: 2100.b Mean.

selected from the interval 0≤ r0(tk) ≤ 5. Since thecontrol has no constrainsEq. (16)was not used.

Table 1shows the results obtained by the IDP al-gorithm. From the values presented in the first fourcolumns it is evident that IDP may still converge tothe local optimum (J∗ = 0.2444). This occurs whenboth the initial value for the control and the initialregion size are too small. Otherwise IDP convergesto the global optimum (J∗ = 0.1365). However, Luusand Bojkov [38] have reported that when the stategrid is equal toM = 1, it is beneficial to use a greaterregion size. Therefore, by selecting a sufficiently largeinitial region size valuer0(tk) ≥ 4, convergence to theglobal optimum is always obtained regardless of theinitial value for the controlu0(tk). This is shown inTable 1; columns five to seven. Repeated optimisationis necessary since the IDP algorithm (seeSection 4.2)generates randomly the allowable values for control(Eq. (15)).

5.1.3. Evolutionary algorithmsIn order to solve the CSTR optimal control prob-

lem by Evolutionary algorithms, firstly a convergencecriterion for all of them was defined. A measure ofsimilarity of the population seems to be a good crite-rion [19]. This involves a way to measure in absoluteor relative sense how similar solutions in the popu-lation are. Sometimes, researchers use the value toreach (VTR) as a stopping criterion, which evidentlycan be applied only when a solution is already known.Since the states (x), control (u), and alsoJ are dimen-

sionless in this problem it is a good option to select anabsolut convergence criterion. It was defined as fol-lows: letJb be the best objective function value in thepopulationJb = minJ(ui), i = 1, . . . , µ, andJw theworst function valueJw = maxJ(ui), i = 1, . . . , µ,then an absolute convergence criterion can be de-fined byJw − Jb < εc. In the current applicationεcwas selected to beεc = 0.00001, which guaranteesgood accuracy of the solution. Since this optimalcontrol problem is unconstrainedEq. (14) was notused.

Four variants of evolutionary algorithms were im-plemented and evaluated. Two of them are based onthe Breeder Genetic algorithm and two are Differ-ential Evolution algorithms.Tables 2–4present themain results for various values of the population sizeµ(20,15,10). The results reported in the tables areaverages from 10 runs. EA1 means the Breeder Ge-netic algorithm with only one population, EA2 denotesthe Breeder Genetic algorithm with sub-populations,and EA3 denotes the DE algorithm DE/rand/1/bin andEA4 stands for the DE algorithm DE/best/2/bin. Evo-lutionary algorithms are evaluated on the basis of fourcriteria: (i) the number of function evaluations, whereeach evaluation involves the integration of the dynamicequations (Eq. (1)), (ii) the CPU time (measured ona Pentium III personal computer at 700 MHZ), (iii)the performance index value (J∗), and (iv) the conver-gence efficiency (C.E.%) to the global optimum whichis measured by the percentual number of times thatthe algorithm found the global solution.

110 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

Table 2Optimal control of a multimodal CSTR by Evolutionary algorithms (population sizeµ = 20)

Algorithm

EA1 BGA EA2 BGA-subpopulations EA3 DE/rand/1/bin EA4 DE/best/2/bin

Function evaluations 7401.80 7143.4 3494 2270CPU time (s)a 430.02 451.10 242.51 160.44J∗ 0.1358249 0.1355985 0.1355966 0.1355850Variance 9.818× 10−8 2.3192× 10−7 2.3503× 10−10 4.7643× 10−12

CE (%) 100 100 100 100Iterations 410.1 397.3 174.70 113.50Parameters pm = 0.09 pm = 0.09, mr = 0.8 CR = 0.6 CR= 0.5

ggap = 0.9 ggap = 0.9, subpop= 2 F = 0.4 F = 0.4

a Measured on a Pentium III 700 MHZ PC.

Table 3Optimal control of a multimodal CSTR by evolutionary algorithms (population sizeµ = 15)

Algorithm

EA1 BGA EA2 BGA-subpopulations EA3 DE/rand/1/bin EA4 DE/best/2/bin

Function evaluations 6985.60 5272.6 3535.5 1783.5CPU time (s)a 406.85 314.93 244.06 134.26J∗ 0.1362200 0.1362479 0.1355920 0.1355970CE (%) 100 100 100 100Variance 8.8488× 10−7 3.1354× 10−7 4.2769× 10−10 5.5529× 10−10

Iterations 497.9 438.80 235.70 118.90Parameters pm = 0.09 pm = 0.09, mr = 0.8 CR = 0.5 CR= 0.5

ggap = 09 ggap = 0.9, subpop= 2 F = 0.5 F = 0.4

a Measured on a Pentium III 700 MHZ PC.

A parameter tuning approach was applied in orderto determine what combination of algorithm parame-ters gives the best performance index with the leastnumber of function evaluations. Storn[24] have sug-gested values for the population size from the interval5d ≤ µ ≤ 10d for static optimisation problems,whered is the dimension of the problem. Price[61]

Table 4Optimal control of a multimodal CSTR by evolutionary algorithms (population sizeµ = 10)

Algorithm

EA1 BGA EA2 BGA-subpopulations EA3 DE/rand/1/bin EA4 DE/best/2/bin

Function evaluations 2225.8 7925 2907 1719CPU time (s)a 135.57 449.50 200.89 134.80J∗ 0.1449189 0.1365219 0.1356905 0.1356052Variance 9.8458× 10−5 1.4124× 10−6 4.6861× 10−8 1.3848× 10−9

CE (%) 100 100 100 100Iterations 246.20 792 290.70 171.9Parameters pm = 0.09 pm = 0.09, mr = 0.8 CR = 0.5 CR= 0.5

ggap = 0.9 ggap = 0.9, subpop= 2 F = 0.6 F = 0.5

a Measured on a Pentium III 700 MHZ PC.

proposed selecting the population size from the inter-val 2d ≤ µ ≤ 20d. Figs. 7–9show the correspondingoptimal control trajectories obtained by one realiza-tion of the four Evolutionary algorithms, according tothe population size that was applied. FromFigs. 7 and8, it is apparent that all EAs converge to the global so-lution. FromFig. 9, observe the small deviations from

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 111

Fig. 7. Optimal control trajectories calculated by a realization of the four EAs, for the CSTR optimal control problem with populationsize: 20 individuals.

the global optimum of the optimal control trajectoriesobtained by EA1 and EA2 but still they did not con-verged to the local minima.

Since it is apparent that for optimal control prob-lems greater population sizes may increase the com-putation time dramatically, in this work the use ofpopulation sizes around the dimension of the optimi-sation problemd = mN was chosen. After a popula-tion size was fixed other parameters of the algorithmswere tuned in order to obtain the best performanceindex with the least number of function evaluations.The values reported in the tables are obtained in thisway. In Tables 2–4, the variance of the 10 runs isreported as well.

5.2. The bifunctional catalyst blend optimalcontrol problem

A very challenging multimodal optimal controlproblem has been studied by Luus[38,44]. Luus[44]

showed that this problem has many local optima (25).Esposito and Floudas[28] found recently 300 localminima. This problem is also proposed as a bench-mark in the Handbook of Test Problems in Localand Global Optimization[74]. A chemical processconverting methylcyclopentane to benzene in a tubu-lar reactor is modelled by a set of seven differentialequations:

x1 = −k1x1, (21)

x2 = k1x1 − (k2 + k3)x2 + k4x5, (22)

x3 = k2x2, (23)

x4 = −k6x4 + k5x5, (24)

x5 = k3x2 + k6x4 − (k4 + k5 + k8 + k9)x5

+ k7x6 + k10x7, (25)

x6 = k8x5 − k7x6, (26)

112 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

Fig. 8. Optimal control trajectories obtained by a realization of the four EAs for the CSTR optimal control problem with a populationsize of 15 individuals.

x7 = k9x5 − k10x7, (27)

wherexi, i = 1, . . . ,7 are the mole fractions of thechemical species, and the rate constants (ki) are cubicfunctions of the catalyst blendu(t):

ki = ci1 + ci2u+ ci3u2 + ci4u3, i = 1, . . . ,10.

(28)

The values of the coefficientscij are given in[44].The upper and lower bounds on the mass fractionof the hydrogenation catalyst are: 0.6 ≤ u(t) ≤ 0.9,and the initial vector of mole fraction isx[0] =[

1 0 0 0 0 0 0]T. This is a continuous

process operated in steady state, so that ‘time’ inEqs. (21)–(28)is equivalent to travel time and thuslength along the reactor. The optimal control prob-lem is to find the catalyst blend along the length ofthe reactor, which in the control problem formula-tion is considered at times 0≤ t ≤ tf where the

final effective residence timetf = 2000 g h/mol suchthat the concentration in the reactor is maximised:J = x7(tf )× 103.

5.2.1. First-order gradient algorithmIn order to solve this problem by means of a first or-

der gradient algorithm a clipping technique was addedto the basic gradient algorithm so as to deal with con-trol constraints. A line search method as describedbefore was added to adjust the step size parameterkefficiently. The convergence tolerance was set toε =0.000001. Despite both enhancements the classicalmethod failed to locate the global optimum as can beseen inTable 5that shows the results of twenty optimi-sations. The best solutions (emphasized inTable 5) areclearly far from the global solution which equalsJ∗ =10.0942 for a piece-wise constant approximation forthe controls. However, when the gradient method wasstarted using a solution generated with a direct method(for example IDP or any evolutionary algorithm) it

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 113

Fig. 9. Optimal control trajectories calculated by a realization of the four EAs for the CSTR optimal control problem with a populationsize of 10 individuals.

converged quickly to the valueJ∗ = 10.1042. Clearly,due to the presence of many local minima in this prob-lem, a first-order gradient algorithm is easily trappedby one of them. The gradient algorithm is able to con-verge to the global optimum only if the initial control

Table 5Optimal control of bifunctional catalyst blend by a first-order gradient algorithm (N = 10)

Constantu0(t) Randomu0(t)

u0(t) J∗ Iterations CPU time (s) J∗ Iterations CPU time (s)

0.80 9.6419 16 166.66 8.7627 22 2052.400.70 8.1215 20 3689.00 8.0054 37 6811.800.85 9.6419 8 97.86 8.4409 24 5450.100.65 8.1214 23 4593.30 8.1691 46 6076.100.90 9.6419 6 36.03 8.5083 30 6235.300.79 9.7577 38 494.68 8.4300 21 1931.100.60 8.1214 27 6928.60 9.3883 17 1344.300.72 8.1223 49 9072.10 8.2718 22 3860.800.78 9.6574 31 3355.40 9.1816 38 3396.100.82 9.6419 9 131.05 9.0628 89 13156.00

trajectory is in the vicinity of the true solution. There-fore, the use of a global optimisation method such asEvolutionary algorithms to approximate the global so-lution followed by a local optimisation method suchas a first-order gradient algorithm to reach the global

114 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

optimum exactly seems a good approach in solvingmulti-modal optimal control problems.

5.2.2. Iterative Dynamic ProgrammingIn order to solve this problem by means of direct

methods, the time interval was divided inN = 10time subintervals and the control was approximatedby a piecewise constant signal at each time interval.In solving it by Iterative Dynamic Programming, theinitial control trajectoryu0(tk); k = 0, . . . , N−1 waschosen constant with a value randomly chosen fromthe control interval 0.6 ≤ u(t) ≤ 0.9. A similar pro-cedure was followed to select the initial region valuer0(tk), k = 0, . . . , N − 1 which was chosen, from theinterval 0.6 ≤ r0(tk) ≤ 0.9.

The algorithm parameters of the IDP algorithmwere: number of state grid pointsM = 1, number ofallowable values for the controlR = 15. The allow-able values for control were generated randomly. Theparameter region contraction factor wasγ = 0.80,which was selected according to Luus’ suggestions[44]. The maximum number of iterations was (I =30), but the optimisation was stopped when it reachedthe conditionJ∗ − J ≤ 0.00005.Table 6shows themain results, which show indeed the convergence to

Table 7Optimal control of bifunctional catalyst by Iterative Dynamic Programming

r0(tk) = 0.27 r0(tk) = 0.30

u0(tk) FE J∗ Iterations CPU time (s) u0(tk) FE J∗ Iterations CPU time (s)

(a)0.75 2250 10.0395 30 293.78 0.65 2250 10.0528 30 263.180.70 1125 10.0942 15 370.76 0.68 675 10.0942 9 88.810.80 2250 9.94429 30 295.76 0.70 450 10.0942 6 61.440.65 2250 10.0395 30 299.92 0.72 975 10.0942 13 121.720.85 2250 10.0395 30 286.77 0.75 2250 10.0395 30 320.650.68 2250 10.0395 30 297.81 0.78 450 10.0942 6 57.420.72 1050 10.0942 14 145.40 0.80 2250 10.0395 30 306.410.78 2250 10.0395 30 300.68 0.82 2250 10.0395 30 328.190.82 2250 9.8978 30 321.38 0.85 900 10.0942 12 514.370.75 2250 10.0395 30 293.78 0.88 1275 10.0942 17 385.43

r0(tk) 0.40 0.50

(b)Function evaluations 938.70 1177.5Iterations 13.70 15.70CPU time (s)a 132.33 150.64

J∗ 10.0942 10.0942

a Measured on a Pentium III 700 MHZ PC.

Table 6Optimal control of bifunctional catalyst by Iterative Dynamic Pro-gramming

u0(tk) r0(tk) FE J∗ Iterations CPUtime (s)

0.6658 0.7814 1500 10.0942 20 174.360.8727 0.6677 1350 10.0942 18 165.580.6353 0.6330 1125 10.0942 15 126.590.8574 0.8600 600 10.0942 8 72.410.7537 0.7033 1125 10.0942 15 121.190.6842 0.7090 1500 10.0942 20 137.520.6357 0.6099 825 10.0942 11 92.580.8515 0.7711 1425 10.0942 19 152.200.8035 0.8417 1575 10.0942 21 169.650.7915 0.6154 1050 10.0942 14 138.23

1207.5a 10.0942a 16.1a 135.03a

a Mean.

the global optimum all the time and the associatednumber of function evaluations and iterations. It isclear from the table that if the initial region size islarge enough IDP always finds the global optimum.

However, the sensitivity of IDP to the choice of theinitial region valuer0(tk) can be illustrated by choos-ing different values for this parameter.Table 7ashowsthat with a value ofr0(tk) = 0.27 the IDP algorithm

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 115

Table 8Optimal control of the bifunctional catalyst blend problem by EAs (population sizeµ = 25)

Algorithm

EA1 BGA EA2 BGA-subpopulations EA3 DE/rand/1/bin EA4 DE/best/2/bin

Function evaluations 7007.10 4890 3172.5 3607.5CPU time (s)a 1186.60 515.14 549.75 632.84J∗ 10.0942 10.0929 10.0941 10.0941Variances 0.0012 0.0019 4.8889× 10−9 9.8889× 10−9

CE (%) 70 80 100 100Iterations 278.28 202.50 126.90 144.30Parameters ggap = 1 ggap = 1, mr = 0.2 F = 0.9 F = 0.9

pm = 0.18 pm = 0.2, subpop= 4 CR = 0.0 CR= 0.0

a Measured on a Pentium III 700 MHZ PC.

converged to the global optimum only in 20% of thecases. By usingr0(tk) = 0.30 this percentage is in-creased to 60%. With greater values thanr0(tk) ≥0.40, k = 0,1, . . . , N − 1 for the initial region sizethe IDP is capable to always converge to the globaloptimum as it is shown inTable 7b. Table 7bshowsthe average of 10 optimisations with different randomallowable values for control.

5.2.3. Evolutionary algorithmsIn order to solve this problem by the selected Evo-

lutionary algorithms, first a proper and common con-vergence criterion based on a measure of the qualityof the solutions in the population was chosen. In con-trast to example one, where an absolute criterion wasselected, here the following relative convergence cri-terion was applied:

µ

εd(Jw − Jb) ≤

∣∣∣∣∣µ∑i=1

J(ui)

∣∣∣∣∣ , (29)

Table 9Optimal control of the bifunctional catalyst blend problem (population sizeµ = 20)

Algorithm

EA1 BGA EA2 BGA-subpopulations EA3 DE/rand/1/bin EA4 DE/best/2/bin

Function evaluations 11493 18227.5 2496 2736CPU time (s)a 1896.90 3237.20 453.92 537.62J∗ 10.0934 10.0916 10.0941 10.0941Variances 0.0064 3.9754× 10−4 6.2222× 10−9 8.4444× 10−9

CE (%) 60 80 100 100Iterations 572.66 910.12 124.8 136.80Parameters ggap = 1 ggap = 1, mr = 0.2 F = 0.9 F = 0.9

pm = 0.28 pm = 0.45, subpop= 4 CR = 0.0 CR= 0.0

a Measured on a Pentium III 700 MHZ PC.

whereJw andJb are defined as before,J(ui) is the per-formance index, andεd = 0.001 is a constant valueselected according to the desired precision. The ini-tialisation for all the EAs was done randomly from thecontrol input domain 0.6 ≤ u0(tk) ≤ 0.9. As before,a parameter tuning approach was applied and the bestresults obtained with the selected parameter values arereported.Tables 8–10show the averages of 10 runs forthree values of the population size (µ = 15,20,25).Reported values of the performance (J∗) are averagesover successful optimisations.Figs. 10–12show theoptimal control trajectory obtained by one realizationof the evolutionary algorithms.

Looking at Figs. 10 and 11we can see only tinydifferences among the optimal control trajectories forthe four evaluated evolutionary algorithms. This is thetrajectory is the associated with the global optimumvalue of the performance indexJ = 10.0942. In caseof Fig. 12, a local minima is plotted which was ob-tained by EA1. The Differential Evolution algorithmsalways approached the global optimum solution.

116 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

Table 10Optimal control of the bifunctional catalyst blend problem (population sizeµ = 15)

Algorithm

EA1 BGA EA2 BGA-subpopulations EA3 DE/rand/1/bin EA4 DE/best/2/bin

Function evaluations 6552.90 12718 1752 2268CPU time (s)a 578.99 2433.40 341.28 385.44J∗ 10.0937 10.0854 10.0940 10.0939Variances 0.0155 0.0039 2.3330× 10−8 1.0316× 10−6

CE (%) 70 50 100 100Iterations 434.85 793.60 116.80 151.20Parameters ggap = 1 ggap = 1, mr = 0.2 F = 0.9 F = 1.0

pm = 0.29 pm = 0.56, subpop= 4 CR = 0.0 CR= 0.0

a Measured on a Pentium III 700 MHZ PC.

Fig. 10. Optimal control trajectories calculated by a realization of the EAs for the highly multimodal OCP using a population size of 25individuals.

6. Discussion

6.1. Non-linear stirred tank reactor

The first-order gradient method is capable of solv-ing low-multimodal optimal control problems like the

multimodal CSTR analysed only by repeated initiali-sation of the guess control trajectory. A gradient-basedalgorithm requires the partial derivatives (gradients)of the problems and also a initial control trajectoryin the neighbourhood of a local minima or globaloptimum.

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 117

Fig. 11. Optimal control trajectories calculated by a realization of the EAs to the highly multimodal OCP with a population size of 20individuals.

The Iterative Dynamic Programming algorithm dueto its stochastic nature (random values for the con-trol) can potentially solve a multimodal optimal con-trol problem since it is not based on the calculation ofthe gradients. However, this problem has shown thatthis approach is rather sensitive to other parameter ofIDP such as the initial region size value. Similar be-haviour of IDP was reported by Luus and Galli[37]in the past.

Concerning the Evolutionary algorithms several re-marks need to be made. Firstly, the four evaluated al-gorithms converged to the global optimum. Even in thecase that a population size of ten individuals was cho-sen, an acceptable value for the performance index inthe neighbourhood of the global optimum was calcu-lated, in contrast to the expectation that with a smallervalue of the population size the algorithms might con-verge to the local optimum. Secondly, the DifferentialEvolution algorithms turned out to be more efficientthan those based on the Breeder Genetic algorithm tak-

ing into account the accuracy of the solutions. Thirdly,within the Differential Evolution algorithms the onethat solved the problem with the lowest number offunction evaluations was DE/best/2/bin.

In both algorithms EA1 and EA2 the mutation rate(pm) was selected a bit greater than the default valuefrequently chosenpm = 1/(m·N), in order to improvethe probability of the algorithm to converge to theglobal optimum. Clearly, this gives rise to a highernumber of function evaluations. In case of EA2 it wasnot possible to obtain a better solution by increasingthe number of subpopulations by more than two. ForEA2 the migration rate (mr) between populations wasallowed each 20 generations.

As far as parameter tuning of the Differential Evo-lution algorithms is concerned, the heuristic rulesapplied to determine the values of the algorithmparameters ‘amplification variation’ (F) and ‘crossoverconstant’ (CR) were as follows. The initial valueswereF = 0.5, CR= 0.1 according to Storn[24], and

118 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

Fig. 12. Optimal control trajectories calculated by a realization of the Evolutionary algorithms for the highly multimodal optimal controlproblem using a population of 15 individuals.

then depending on whether a good convergence wasobserved the value of the crossover constant was in-creased in order to improve the efficiency of the algo-rithm. Also smaller values forF were tried. It can beseen from the tables that values forF within the range0.4 ≤ F ≤ 0.6 were sufficient to obtain a convergenceefficiency of 100%. Greater values of CR resulted insolutions with worst (less accurate) performance butnot necessarily to a local optimum. Also it was no-ticed that the change of CR from 0.1 to 0.5 resultedin considerable faster convergence. In contrast to thepopulation size values suggested by Storn and Price5(m ·N) ≤ µ ≤ 10(m ·N) relatively small populationsalso allowed to find a good solution. It seems that be-cause the problem is not highly multimodal a relativesmall value of the differential variation parameterFsuffices to explore properly the whole search space.

Compared to results obtained with the IDP algo-rithm the DE/best/2/bin algorithm was able to solve

this problem with a smaller number of function eval-uations (cf.Table 1). This shows that DE algorithmsare actually very efficient evolutionary algorithms. Todetermine the values of the three algorithm param-eters that steer the optimisation only a few experi-ments are required. In the IDP algorithm there aremore algorithm parameters to be tuned than in DE. Incontrast to the IDP algorithm, the algorithm parame-ters that guarantee a convergence efficiency of 100%are easily obtained for the DE algorithms consideredhere.

6.2. The bifunctional catalyst blend problem

This problem showed that a first-order gradientalgorithm can find the global optimum if and onlyif the initial control trajectory is at the vicinity ofthe global optimum. The difficulty is that one cannot know in advance were that vicinity is. Therefore,

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 119

highly multimodal optimal control problems cannotbe solved satisfactory with this approach.

The Iterative Dynamic Programming algorithm per-formed very efficiently in solving the multimodal opti-mal control problem. However, from our experiments,one can see clearly a high sensitivity to the initial valueof the region size. Therefore, in order to solve a newdynamic optimisation problem it is required to attemptseveral values of this parameter of IDP. These previousexperimentation should be included in the evaluationof the efficiency of the algorithm. Shiaw and Hwang[75] have attempted to improve the convergence ofthis approach by using robust sequence random num-ber generators. Iterative Dynamic Programming is in-deed a powerful and efficient method to solve difficulthighly multimodal optimal control problems.

Regarding evolutionary algorithms EA1 and EA2did not reach 100% of convergence efficiency whensmall population sizes were used. However, it wasfound that by increasing adequately the populationsize, EA2 improves remarkably. As a matter of fact,by using a population size ofµ = 60 with foursubpopulations and 15 individuals each one, a gen-eration gapggap = 0.9, a migration ratemr = 0.2,mutation ratepm = 0.1 and a elapsed time of 10generations between migrations, EA2 converged al-ways to a valueJ∗ = 10.0942. The average requirednumber of function evaluations was 7048.6 with avariance of 1× 10−9. EA1 converged only 80% ofthe times to the global optimum with population sizeµ = 40 and 50. These results illustrate the benefitsof using sub-populations. The Differential Evolutionalgorithms always reached a convergence efficiencyof 100%. Moreover, both DE algorithms were con-siderably more efficient than the Breeder Geneticalgorithms.

By comparingTables 8–10it can be seen that bothEA1 and EA2 require a greater value of the muta-tion rate parameter (pm) to obtain reasonable solutionswhen the population size is diminished. This some-times leads to an increased number of function evalu-ations. In contrast, the Differential Evolution requiresless function evaluations, and usually has better con-vergence to the global optimum. As for the number offunction evaluations the best algorithm in this case isEA3. But, EA4 is not significantly less efficient thanEA3, while it performs better when the population sizeis further reduced (Table 10).

As before the DE algorithms were tuned by apply-ing some heuristic rules to determine the differentialvariation parameter (F) and crossover constant (CR)that lead to the global optimum efficiently. For thechosen population sizes, starting with initial valuesF = 0.5, and CR= 0.1 premature convergence (con-vergence to a local solution) was observed. Therefore,the value of the parameterF was increased. It was dis-covered that increasing CR neither improves the speedof convergence nor locating of the global optimum.On the contrary, it was observed that neglecting com-pletely the effect of the crossover operator, by settingCR = 0, the effectiveness increases considerably. Thisvalue (CR= 0) gave the best convergence of the al-gorithms at the expense of more function evaluations.It seems that the large multimodality of this problemdemands an extensive exploration of the search space.It is recommended to select a value close to one for thedifferential variation parameter (F), and a crossoverrate (CR) of zero in highly multimodal problems.

It must be said that in this problem DE algorithmsrequired more function evaluations than the IDP algo-rithm. However, one could argue that this difference isnot significant when taking into account that the IDPalgorithm requires more previous experiments to tuneits critical algorithm parameters than in the straight-forward procedure associated to the DE algorithms.

7. Conclusions

Evolutionary algorithms are robust search methodscapable of locating the global optimum of multimodaloptimal control problems. These algorithms are notsensitive to the initial control trajectory. They can beinitialised randomly. Evolutionary algorithms basedon the Breeder Genetic algorithm are able to solvecomplex multimodal optimal control problems butthey demand a large population size or a high mu-tation rate (probability of mutation). Both propertiesgive rise to an increased number of function evalu-ations (simulations) and hence a long computationtime. The use of sub-populations can improve theirconvergence to the global optimum as problem 2 ofthis research has shown.

This research has shown that within the family ofEvolutionary algorithms, Differential Evolution algo-rithms stand out in terms of efficiency as compared

120 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

to the Breeder Genetic algorithm. In contrast to themajority of Evolutionary algorithms, where manyalgorithm parameters have to be tuned, in DE onlythree algorithm parameter values (the population size,the crossover constant and the differential variationcoefficient) have to be selected. The population sizeplays a crucial role in solving optimal control prob-lems. Selecting a too small population size reducesthe probability of finding the global solution. Increas-ing the population size increases the chances that thealgorithm finds the global optimum but the compu-tation time increases. The two investigated Differen-tial Evolution algorithms solved the two benchmarkmultimodal optimal control problems properly andefficiently. In solving the first problem the efficiencyachieved by DE was clearly comparable to that of the(non-Evolutionary) IDP algorithm. As for the secondproblem the efficiency of DE was slightly inferior tothe one required by the IDP algorithm when the algo-rithm parameters have been tuned. On the other hand,the determination of appropriate values of the algo-rithm parameters for IDP is more difficult and moreinvolved [cf. 30]. In summary, Differential Evolu-tion algorithms are reliable and relatively efficient tosolve multimodal optimal control problems. Clearly,improving the efficiency of the DE algorithms furtherremains an important issue for future research.

The guidelines to select the algorithm parametervalues crossover constant (CR) and amplification ofthe differential variation (F) in the DE algorithms ob-tained from this investigation can be summarized asfollows. Adopt a smaller population size than in staticoptimisation; a population size less than or equal totwo times the dimension of the optimisation prob-lem (µ ≤ 2(N × m)) is desirable for optimal controlproblems. Highly multimodal optimal control prob-lems may require greater values of the amplificationvariation coefficient (F) and a very small or zero valuefor the crossover constant (CR). Low multimodal op-timal control problems may need medium values ofthe mutation parameter (F) and greater or medium val-ues for the crossover constant (CR). Further researchis needed if one is interested in finding more genericrules for parameter tuning.

In order to solve multimodal optimal control prob-lems more efficiently and accurately, an efficient Evo-lutionary algorithm like Differential Evolution maybe used toapproximate the global minimum. Next, a

classical local optimisation algorithm can be appliedto accurately compute the global optimum. The devel-opment of such a combined method is the aim of ourfuture work.

Acknowledgements

The first author wishes to mention the support of theNational Council of Science and Technology (CONA-CyT) of Mexico to this research.

References

[1] R. Horst, H. Tuy, Global Optimization, DeterministicApproaches, Springer-Verlag, Berlin, 1990.

[2] R. Horst, P.M. Pardalos (Eds.), Handbook of GlobalOptimisation, Kluwer Academic Publishers, Dordrecht, 1995.

[3] R. Horst, P.M. Pardalos, N.V. Thoai, Introduction to GlobalOptimization, Kluwer Academic Publishers, Dordrecht, 1995.

[4] C.A. Floudas, Deterministic Global Optimisation: Theory,Algorithms and Applications, Kluwer Academic Publishers,Dordrecht, 2000.

[5] I. Androulakis, C.D. Maranas, C.A. Floudas,αBB: A globaloptimization method for general constrained non-convexproblems, J. Global Optim. 7 (1995) 337–363.

[6] W.R. Esposito, C.A. Floudas, Deterministic globaloptimization in isothermal reactor network synthesis, J. GlobalOptim. 22 (2002) 59–95.

[7] A. Torn, A. Zilinskas, Global Optimization, Springer, Berlin,1989.

[8] W.L. Price, A controlled random search procedure forglobal optimization, in: L.C.W. Dixon, G.P. Zsego (Eds.),Toward Global Optimization 2, North-Holland Publishing,Amsterdam, 1978, pp. 71–84.

[9] W.L. Price, Global optimisation by controlled random search,J. Optim. Theory Applic. 40 (3) (1983) 333–348.

[10] W.L. Price, Global optimization algorithms for CADworkstation, J. Optim. Theory Applic. 55 (1) (1987) 133–146.

[11] A.H.G. Rinnooy Kan, G.T. Timmer, Stochastic globaloptimization methods. I. Clustering methods, Math. Prog-ramming 39 (1987) 27–56.

[12] A.H.G. Rinnooy Kan, G.T. Timmer, Stochastic global optimi-zation methods. II. Multilevel methods, Math. Programming39 (1987) 57–58.

[13] E.M.T. Hendrix, P.M. Ortigosa, I. Garcıa, On success ratesfor controlled random search, J. Global Optim. 21 (3) (2001)239–263.

[14] Z. Zabinsky, Stochastic methods for practical globaloptimization, J. Global Optim. 13 (4) (1998) 433–444.

[15] M. Locatelli, Convergence of simulated annealing algorithmfor continuous global optimization, J. Global Optim. 18 (3)(2000) 219–233.

I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122 121

[16] J. Holland, Adaptation in Natural and Artificial Systems,University of Michigan Press, Ann Arbor, 1975.

[17] D.E. Goldberg, Genetic Algorithms in Search, Optimizationand Machine Learning, Addison-Wesley, Reading, 1989.

[18] Z. Michalewicz, Genetic Algorithms+ Data Structures=Evolution Programs, Springer-Verlag, Berlin, 1992.

[19] H.P. Schwefel, Evolution and Optimum Seeking, Wiley, NewYork, 1995.

[20] T. Bäck, Evolutionary Algorithms in Theory and Practice,Oxford University Press, NY, 1996.

[21] D. Fogel, Evolutionary Computation: Toward a NewPhilosophy in Machine Learning, Los Alamitos, CA, 1995.

[22] M.F. Hussain, K.S. Al-Sultan, A hybrid Genetic algorithm fornon-convex function minimization, J. Global Optim. 11 (3)(1997) 313–324.

[23] R. Salomon, Re-evaluating Genetic algorithm performanceunder coordinate rotation of benchmark functions, a survey ofsome theoretical and practical aspects of Genetic algorithms,BioSystems 39 (1996) 263–278.

[24] R. Storn, Differential Evolution—a simple and efficientheuristic for global optimization over continuous spaces, J.Global Optim. 11 (4) (1997) 341–359.

[25] D. Corne, M. Dorigo, F. Glover (Eds.), New Ideas inOptimization, McGraw-Hill, New York, 1999.

[26] O. Rosen, R. Luus, Global optimisation approach tonon-linear optimal control, J. Optim. Theory Applic. 73 (3),547–562.

[27] A. Strekalovsky, I. Vasiliev, On global search for non-convexoptimal control, in: I.M. Bomze, T. Csendes, R. Horst, P.M.Pardolos (Eds.), Global Optimisation in Engineering Design,Kluwer Academic Publishers, Dordrecht, 1997, pp. 355–386.

[28] W.R. Esposito, Ch.A. Floudas, Deterministic globaloptimisation in non-linear optimal control problems, J. GlobalOptim. 17 (2000) 97–126.

[29] I. Papamichail, C.L. Adjiman, A rigorous global optimizationalgorithm for problems with ordinary differential equations,J. Global Optim. 24 (2002) 1–33.

[30] M.M. Ali, C. Storey, A. Törn, Application of stochastic globaloptimization algorithms to practical problems, J. Optim.Theory Applic. 95 (3) (1997) 545–563.

[31] M.M. Ali, C. Storey, Modified controlled random searchalgorithms, Intern. J. Computer Math. 53, 229–235.

[32] J.R. Banga, W.D. Seider, Global Optimisation of ChemicalProcesses Using Stochastic Algorithms. State-of-the-Artin Global Optimisation: Computational Methods andApplications, Princeton Univeristy, Princeton, NJ, 1995.

[33] J.R. Banga, A.A. Alonso, R.I. Pérez-Martın, R.P. Singh,Optimal control of heat and mass transfer in food andbioproducts processing, Computers Chem. Eng. 18 (1994)S699–S2705.

[34] E.F. Carrasco, J.R. Banga, Dynamic optimization of batchreactors using adaptive stochastic algorithms, Ind. Eng. Chem.Res. 36 (1997) 2252–2261.

[35] E.F. Carrasco, J.R. Banga, A hybrid method for optimalcontrol of chemical processes, in: Proceedings of the UKACCInternational conference on Control 98, University of Wales,Swansea, UK, September 1998.

[36] R. Luus, Optimal control by dynamic programming usingsystematic reduction grid size, Int. J. Control 19 (1990) 995–1013.

[37] R. Luus, M. Galli, Multiplicity of solutions in using dynamicprogramming for optimal control, Hung. J. Ind. Chem. 19(1991) 55–62.

[38] R. Luus, B. Bojkov, Global optimization of the bifunctionalcatalyst problem, Can. J. Chem. Eng. 72 (1994) 160–163.

[39] R. Luus, J. Dittrich, F.J. Keil, Multiplicity of solutions inthe optimization of a bifuctional catalyst blend in a tubularreactor, Can. J. Chem. Eng. 70 (1992) 780–785.

[40] B. Bojkov, R. Luus, Use of random admissible values forcontrol in Iterative Dynamic Programming, Ind. Eng. Chem.Res. 31 (1992) 1308–1314.

[41] R. Luus, On the application of Iterative DynamicProgramming to Singular Optimal Control problems, IEEE,Trans. Automatic Control 37 (11) (1992) 1802–1806.

[42] W. Mekarapiruk, R. Luus, Optimal control of final stateconstrained systems, Can. J. Chem. Eng. 75 (1997) 806–811.

[43] B. Bojkov, R. Luus, Time optimal control of high dimensionalsystems by Iterative Dynamic Programming, Can. J. Chem.Eng. 73 (1995) 380–390.

[44] R. Luus, Iterative Dynamic Programming, Chapman &Hall/CRC Press, Boca Raton, FL, 2000.

[45] W. Mekarapiruk, R. Luus, Optimal control by iterativedynamic programming and with deterministic and randomcandidates for control, Ind. Eng. Chem. Res. 39 (1) (2000)84–91.

[46] Z. Michalewicz, C.Z. Janikov, J.B. Krawczyk, A modifiedGenetic algorithm for optimal control problems, Comput.,Math. Applic. 23 (12) (1992) 83–94.

[47] H. Sewald, R.R. Kumar, Genetic algorithm approach foroptimal control problems with linearity appearing controls, J.Guidance, Control and Dyn. 18 (1) (1995) 177–182.

[48] Y. Yamashita, M. Shima, Numerical computational methodusing Genetic algorithms for the optimal control problem withterminal constraints and free parameters, Non-linear Anal.,Theory, Meth. And Applic. 30 (4) (1997) 2285–2290.

[49] S. Smith, An Evolutionary Program for a class of continuousoptimal control problems, in: Proceedings of the IEEEConference on Evolutionary Computation, vol. 1, IEEE,Piscataway, 1995, pp. 418–422.

[50] N.V. Dakev, A.J. Chipperfield, P.J. Flemming, A generalapproach for solving optimal control problems usingoptimization techniques, in: Proceedings of the 1995 IEEEConference on Systems, Man and Cybernetics, Part 5 (of 5),Vancouver, pp. 4503–4508.

[51] N.V. Dakev, A.J. Chipperfield, J.F. Whidborne, P.J. Flemming,An evolutionary approach for solving optimal controlproblems, in: Proceedings of the 13th Triennial IFAC WorldCongress, San Francisco, USA, 1996, pp. 321–326.

[52] H. Pohlheim, A. Hei�ner, Optimal control of greenhouseclimate using Genetic Algorithms, in: Proceedings ofthe Mendel’96—2nd International Conference on GeneticAlgorithms, Brno Czech Republik, 1996, pp. 112–119.

[53] H. Pohlheim, A. Hei�ner, Optimal control of greenhouseclimate using a short time climate model and evolutionary

122 I.L. Lopez Cruz et al. / Applied Soft Computing 3 (2003) 97–122

algorithms, in: A. Munack, H.J. Tantau (Eds.), PreprintsMathematical and Control Applications in Agriculture andHorticulture, IFAC Workshop, Hannover, Germany, 1997,pp. 113–118.

[54] Y.C. Sim, S.B. Leng, V. Subramanian, A combined Geneticalgorithm-shooting method approach to solving optimalcontrol problems, Int. J. Syst. Sci. 31 (1) (2000) 83–89.

[55] J.A. Roubos, G. Van Straten, A.J.B. Van Boxtel, Anevolutionary strategy for fed-bacth bioreactor optimization:concepts and performance, J. Biotechnol. 67 (1999) 173–187.

[56] M.M.A. Hashem, K. Watanabe, K. Izumi, A new EvolutionStrategy and its application to solving optimal controlproblems, JSME Int. J., Ser. C, 41 (3) (1998) 406–412.

[57] F.S. Wang, J.P. Chiou, Optimal control and optimaltime location problems of differential-algebraic systems bydifferential evolution, Ind. Eng. Chem. Res. 36 (1997) 5348–5357.

[58] J.P. Chiou, F.S. Wang, Hybrid method of evolutionaryalgorithms for static and dynamic optimisation problems withapplications to a fed-batch fermentation process, Comput.Chem. Eng. 23 (1999) 1277–1291.

[59] M.H. Lee, Ch. Han, K.S. Chang, Dynamic optimisationof continuous polymer reactor using a modified differentialevolution algorithm, Ind. Eng. Chem. Res. 38 (1999) 4825–4831.

[60] I.L. Lopez-Cruz, Efficient Evolutionary Algorithms forOptimal Control, Ph.D. Thesis, Wageningen University,Wageningen, 2002.

[61] K.V. Price, An Introduction to Differential Evolution, in:D. Corne, M. Dorigo, F. Glover (Eds.), New Ideas inOptimization, McGraw-Hill, New York, 1999.

[62] R. Storn, On the Usage of Differential Evolution for FunctionOptimisation, NAFIPS 1996, Berkeley, pp. 519–523.

[63] R. Storn, K. Price, Minimizing the real functions ofthe ICEC’96 contest by Differential Evolution, in: IEEEConference on Evolutionary Computation, Nagoya, 1996,pp. 842–844.

[64] R. Storn, Systems design by constraint adaptation anddifferential evolution, IEEE Trans. Evol. Comput. 3 (1) (1999)22–34.

[65] F.L. Lewis, V.L. Syrmos, Optimal Control, Wiley, New York,1995.

[66] O. von Stryk, R. Bulirsch, Direct and indirect methods fortrajectory optimization, Annal. Oper. Res. 37 (1992) 357–373.

[67] A.E. Bryson, Dynamic Optimization, Addison-Wesley, MenloPark, NY, 1999.

[68] S.K. Agrawal, B.C. Fabien, Optimization of DynamicSystems, Solid Mechanics and its applications, KluwerAcademic Publishers, Dordrecht, The Netherlands, 1999.

[69] C.J. Goh, K.L. Teo, Control parametrization: a unifiedapproach to optimal control problems with general constraints,Automatica 24 (1) (1988) 3–18.

[70] G. Rudolph, Convergence Properties of EvolutionaryAlgorithms, Verlag Dr. Kovac, Hamburg, 1997.

[71] H. Mühlenbein, D. Schlierkamp-Voosen, Predictive modelsfor the Breeder Genetic Algorithm. I. Continuous parameteroptimization, Evol. Comput. 1 (1) (1993) 25–49.

[72] E. Cantú-Paz, Designing efficient and accurate parallelGenetic Algorithms, Ph.D. thesis, University of Illinois atUrbana, Champaign, 1999.

[73] A. Chipperfield, P. Flemming, H. Pohlheim, C. Fonseca,Genetic Algorithm Toolbox for use with MATLAB,Department of Automatic Control and Systems Engine-ering, University of Sheffield, User’s Guide, Version 1.2,1994.

[74] Ch.A. Floudas, P.M. Pardalos, C.S. Adjiman, W.R. Esposito,Z.H. Gumus, S.T. Harding, J.L. Klepeis, C.A. Meyer, C.A.Schweiger, Handbook of Test Problems in Local and GlobalOptimization, Kluwer Academic Publishers, Dordrecht, TheNetherlands, 1996.

[75] L.J. Shiaw, Ch. Hwang, Enhancement of the globalconvergence of using iterative dynamic programming to solveoptimal control problems, Ind. Eng. Chem. Res. 38 (6) (1998)2469–2478.