1

Constructive Neural Networks and their application to ship multi-disciplinary design optimization

Eric Besnard, Adeline Schmitz, Hamid Hefazi and Rahul Shinde

Mechanical and Aerospace Engineering California State University, Long Beach

Abstract

This paper presents a Neural Network-based Response Surface Method for reducing the cost of computer intensive optimizations for applications in ship design. In the approach, complex or costly analyses are replaced by a Neural Network which is used to instantaneously estimate the value of the function(s) of interest. The cost of the optimization is shifted to the generation of (smaller) data sets used for training the network. The focus of the paper is on the use and analysis of constructive networks, as opposed to networks of fixed size, for treating problems with a large number of variables, say around 30. The advantages offered by constructive networks are emphasized, leading to the selection and discussion of the Cascade Correlation algorithm. This topology allows for efficient neural network determination when dealing with function representation over large design spaces without requiring prior experience from the user. During training, the network grows until the error on a small set (validation set), different from that used in the training itself (training set), starts increasing. The method is validated for a mathematical function for dimensions ranging from 5 to 30 and the importance of analyzing the error on a set other than the training set is emphasized. The approach is then applied to the automated Computational Fluid Dynamics-based shape optimization of a fast ship configuration known as the twin H-body. The classical approach yields a design improvement of 26% while the NN-based method allows reaching a 34% improvement at one fifth of the cost of the former. Based on the analysis of the results, areas for future improvements and research are outlined. The results demonstrate the potential of the method in saving valuable development cycle time and increasing the performance of future ship designs.

2

Nomenclature BL = buoyant lift, or displacement CAD = computer aided design CFD = computational fluid dynamics Cpmin = minimum pressure coefficient on surface of configuration DL = dynamic lift DOE = design of experiments DV = design variable f = exact function fNN = Neural Network approximation of the function f L/D, LOD= lift-to-drag ratio GS = generalization set HU = hidden unit MDO = multidisciplinary design optimization n = number of inputs/design variables NN = Neural network Np = number of points in set RSM = response surface method Struc = structural constraint TS = training set VS = validation set

Subscripts ( )p = value at a point of the training set

Definitions

1

1 pN

ppp

A AN =

< >= ∑ = average of A over set

( ) ( ) ( )NNE x f x f x= − = error at x

( )1 1max max ( )

p pM p pp N p N

E E E x≤ ≤ ≤ ≤

= = = maximum error over set

22

1

12

pN

ppp

E EN =

= ∑ = modified squared error over set

( )21

1( )1

pN

ppp

Std E E EN =

= − < >− ∑ = standard deviation of error over set

3

Introduction Artificial Neural Networks basically attempt to replicate functions of the human brain by connecting neurons to each other in specific manners. In most cases, the number of neurons and connections has been limited to tens or hundreds of neurons with a similar order of magnitude of connections between them. This contrasts with the human brain which has many orders of magnitude more neurons and also many more connections between them. The work done so far in this field can therefore be categorized as precursory and the potential of this technology has yet to be fully realized. Jain and Deo (2005) present the scope of applications in ocean engineering which can benefit from that technology. They have been used for predicting environmental parameters (wave heights, sea level, wind speeds, tides, etc.), forces and damage on ocean-going structures, ship and barge motions, in various ship design applications and more. In the area of ship design, Neural Networks (NN) has been employed for various purposes. For example, Koushan (2003) presents a hull form optimization employing NNs where 8 hull form parameters are varied in order to minimize resistance. Similarly, the use of NNs in hull shape optimization in Danõşman et al (2002) and Koh et al (2005) allows using a more advanced flow model (panel method instead of thin-ship flow theory) and thus leads to improved shapes. Koh et al (2004), present a similar approach for investigating resistance, maneuverability and seakeeping characteristics of a high speed hull form. In Mesbahi and Bertram (2000), and Bertram and Mesbahi (2004), NN are used to derive functional relations between design parameters for cargo ships as alternative of using charts. In Maisonneuve (2003), several applications examples from the industry are discussed showing the state of the art European research in marine design; integrating real calculations (using CAD model) and artificial calculations (using NNs as response surface methods) to perform single or multi-objective optimizations. In most of the applications, a single hidden layer feed-forward type network and back propagation were used. Also, the number of input nodes used by the different investigators was relatively small, on the order of 1 to 10. Very few applications used a large number of inputs showing the real prowess of the neural networks. One exception is Danõşman et al (2002) which used a single hidden layer NN with 40 input and 4 output parameters. The inputs represent a series of half breadths of the aft end of a catamaran and the output parameters are related to wave resistance, wave elevation and displacement. The network used back-propagation with a training set of 300 and a validation set of 50 points. The number of hidden units used and the errors obtained on the Neural Network are not discussed in the paper, however. As the number of parameters to be varied increases, training the network becomes more and more challenging. A single hidden network is a universal approximator, provided that sufficiently many hidden units are available (Hornik, 1991). To date, no theoretical bound on the number of hidden units required has been identified so that for each new problem, the number of hidden units installed on the Network must be sufficient to represent the complexity of the function to be approximated, i.e. the network size must be determined by trial and error. Also, for large networks, back propagation is known to be very slow. Other network structures and training algorithms should be investigated.

4

This paper presents an alternative NN structure based on a constructive NN topology and a corresponding training algorithm suitable for large number of input/outputs to address the problems where the number of design parameters is fairly large, say up to 30 or even more. First, the use of NNs as advanced Response Surface Methods (RSM) in the design cycle is reviewed and compared with traditional RSM approaches. The proposed constructive network trained by a modified Cascade Correlation algorithm is then presented. Because the results of the optimization greatly depend on the ability of the Neural Network to represent the function(s) of interest, the NN-based RSM approach is first applied to a mathematical function to illustrate its behavior as the design space increases in dimension. Finally, the approach is used in a practical case to an automated multi-disciplinary design/optimization of a lifting hull where 28 parameters are varied in order to obtain an improved design which was built and tested at sea (Fig. 1). This optimization is compared with the classical global optimization approach where the Computational Fluid Dynamics tools are used directly in the design process.

Fig. 1. Blended-wing-body configuration; the NN approach is applied to the design/optimization of the submerged twin H-body configuration

Neural Networks as Response Surface Methods in the design cycle

The design cycle The MDO method described in this paper is applicable to any level of a system development (see, for example, in Blanchard & Fabrycky, 2006), either at the conceptual design level (i.e. of a ship-based transportation system), or at the detailed design level of a component (i.e. of propeller blade shape). The modern approach used in the design of such system (the ship or component inside the ship) usually includes at some level an optimization as shown in Fig. 2. In practical cases, the design tool may either be an optimization or design-of-experiment software, or a set of test cases identified by an experienced designer interested in conducting trade studies. The analyses performed at each subsystem level rely, in general, on a combination of semi-analytical models, advanced numerical methods such as finite element analysis, and use of existing databases. In Fig. 2, each type is allocated to a particular subsystem for illustrative purposes, but this obviously needs not be the case in the most general design problem.

5

Subsystem 1 Semi-analytical

model

Design Tool

(DOE or optimization)

New Design

Subsystem 3 Large database

Objective(s) & Constraints

Subsystem 2Extensive computer

model

Fig. 2. Generic system design loop utilizing a combination of analytical models, computationally extensive models and/or large databases.

Such optimization or trade study usually has the following requirements: 1. It has to be able to handle a large number of design variables (say up to 30 or more) 2. The design space is to be completely explored The latter stems from the fact that one does not just wish to improve an existing design but also discover new interesting designs in the feasible design space. This is particularly important when investigating new types of configurations for which little experience exists. The optimization tool shall, therefore, be global (as opposed to local). This requirement for using a global optimization method puts additional constraints on the design process because global optimization methods require several thousand evaluations before an optimum can be reached. Response surface methods may be used to reduce the computational cost of using such global optimizers.

Response surface methods and neural networks in design/optimization Because of the cost associated with using advanced analysis tools for function evaluation such as computational fluid dynamics and finite element methods, these are usually introduced later in the design process when general layouts have already been determined. Instead, the higher system levels rely on semi-empirical/analytical models for determining subsystem requirements. Also, when they do get used at the lower levels, the cost associated with their use, both in terms of man and computing power required, usually limits the exploration of the design space. RSM can be used to reduce some of these limitations. They basically all seek to reduce the time associated with extensive computations by estimating the functions being evaluated in the loops of Fig. 2 based on some limited information available. The most widely used RSM method in optimization is based on a polynomial representation. This approach is described in many references and many variants exist to improve its performances, depending on the targeted application. It has been used in optimization problems and compared with artificial NN in a variety of fields. For example, Todoroki et al (2004) apply it to composite structures stacking sequence optimization, Gomes and Awruch (2004) to structural reliability analysis, Lee & Hajel (2001) to

6

structural optimization, Bourquin et al (1998) to pharmaceutical modeling, and Dutt et al (2004) to biochemistry optimization. A third approach consists of using a Kriging model. While this approach has been used for multi-disciplinary design/optimization purposes, Simpson et al (1998) shows that as the number of design variables increases, the size of the amount of CPU time required for manipulating the matrices grows exponentially. This latter model is, therefore, not considered suitable here since it does not scale to large dimension design spaces.

Neural Networks in the design cycle As alluded to earlier, Neural Networks may be inserted directly at all levels of the system design process and on a broad basis. Specialists who use advanced computational tools for detailed analyses are often remotely connected to the design loop. The use of NN allows for them to be indirectly integrated very early into the design cycle by generating a computational database representative of the problem at hand over the desired design space. For example, the database might consist of a few hundred CFD analyses performed for a configuration represented by tens of widely varying design parameters. This database is then used to train –hence its name, Training Set– a Neural Network which is then inserted in the design loop (Fig. 3). At this point, the designer (not the specialist) can use the NN and get a solution for a variety of designs in a fraction of a second. For example, if a network has been trained for estimating the ship resistance, the latter can be obtained instantaneously by the designer for any desired point in the design space. Similar uses of Neural Networks can be made when dealing with available large databases. Such databases may be from one or more sources, numerical and/or experimental. In this case, the database can be used directly to train the NN and the latter can also be integrated into the design loop (Fig. 3). The result is a design approach where the function, such as ship resistance, corresponding to a particular set of design variables, either selected by the designer or by the computer (“design tool”) can be obtained instantaneously. Used in such fashion, NN is particularly well suited for problems which exhibit the following characteristics:

• Large number of design variables (20 to 100) • Functions (objective or constraints) exhibit non-linear behaviors • One or more of the analyses are very time consuming, either computationally or

in man power • Large databases are already available • The objective functions are likely exhibit multiple minima/maxima • Little a priori experience exists for the problem at hand

7

Subsystem 1Semi-analytical

model

Design Tool (DOE or

optimization)

New Design

Subsystem 2NN-2

Subsystem 3NN-3

Objective(s) & Constraints

Training set generation for subsystem 2

analysis

Subsystem 2 NN-2

Large database for subsystem 3

analysis

Subsystem 3 NN-3

Fig. 3. System design loop utilizing Neural Networks. The NN’s are generated outside the design loop based on computationally extensive models and/or large databases.

In practical terms, the introduction of NN allows extracting the time-consuming or difficult operations (performing an advanced numerical analysis or extracting information from a large and evolving database) from the design loop while still keeping their influence on the outcome of the design process via the NN. The cost has thus been moved (and possibly reduced in the process) to the training set generation (if it was not already available) and to the training of the network. The result is a NN which can estimate the function or functions over the design space it has been trained on. This ability to quickly evaluate new designs allows in turn for the use of global optimization tools such as Genetic Algorithms instead of having to rely on local optimization methods or exploring a restricted part of the design space.

Advantages offered by constructive artificial neural networks Artificial neural networks can be viewed as large multi-input multi-output nonlinear systems composed of many simple nonlinear processing elements operating in parallel (neurons). The links between neurons are unidirectional and are called connections. Each neuron has multiple inputs and a single output, and its output serves as input to other neurons. Each connection is assigned a weight factor, determining how strong the connection is. The network structure, or topology, is defined by the way neurons are interconnected. Training the network involves estimating the weight of the connections between neurons. Haykin (1998), for example, presents an overview of Neural Networks and their applications. Artificial Neural Networks can be used for various tasks such as pattern recognition, function approximation, control systems, etc. NN’s have been investigated for many applications outside the field of ocean engineering requiring a large number of function analyses, ranging from chemistry (Agatonovic-Kustrin et al 1998, and Takayama 2003) to structural analysis (Deng et al 2005). Research clearly indicates that NN’s compare favorably with classical RSM (Todoroki et al 2004, Bourquin et al 1998, Gomes & Awruch 2004, Dutt et al 2004 and Lee & Hajel 2001), in particular when the function is non-convex over the desired domain and the function may be highly non-linear. In addition, for problems with a large number of inputs (design variables), the size of the dataset required for the classical RSM rapidly grows. In the area of ship design, Mesbahi (2003), among others, shows how linear programming can be replaced by neural networks.

8

As discussed above, for most marine applications reported to date, however, NN employed either a fixed topology and/or back propagation for training. The former implies that one needs to have some information about the function to approximate and the latter leads to increasingly large CPU time requirements for training in the case of a large number of inputs. Methods which use a fixed network topology involve evaluating in advance (before training) the type of network that would best suit the application (how many neurons, how many hidden-layers…) to match the complexity of the NN to that of the function. This point is best illustrated quoting Kwok and Yeung (1997): “Consider a data set generated from a smooth underlying function with additive noise on the outputs. A polynomial with too few coefficients will be unable to capture the underlying function from which the data was generated, while a polynomial with too many coefficients will fit the noise in the data and again result in a poor representation of the underlying function. For an optimal number of coefficients the fitted polynomial will give the best representation of the function and also the best predictions for new data. A similar situation arises in the application of NN, where it is again necessary to match the network complexity to the problem being solved. Algorithms that can find an appropriate network architecture automatically are thus highly desirable.” There are essentially two approaches for training multi-layer feedforward networks for function approximation which can lead to variable networks (Kwok and Yeung 1997). The Pruning Algorithms start with a large network, train the network weights until an acceptable solution is found, and then use a pruning method to remove unnecessary units or weights (units connected with very small weights). On the other hand, Constructive Algorithms start with a minimal network, and then grow additional hidden units as needed. The primary advantage of constructive algorithms versus pruning algorithms is that the NN size is automatically determined. Constructive algorithms are computationally economical compared to pruning algorithms which spend most time training on networks larger than necessary. Also, they are likely to find smaller network solutions, thus requiring less training data for good generalization. They also require a small amount of memory because they usually use a “greedy” approach where only part of the weights is trained at once, whereas the other part is kept constant. Cascade-Correlation, first introduced by Fahlman and Lebiere (1990), is one such supervised learning algorithm for NN. Instead of just adjusting the weights in a network of fixed topology, Cascade-Correlation begins with a minimal network, then automatically trains and adds new hidden units one-by-one in a cascading manner. This architecture has several advantages over other algorithms: it learns very quickly; the network determines its own size and topology; it retains the structure it has built even if the training set changes; and it requires no back-propagation of error signals through the connections of the network. In addition, for a large number of inputs (design variables), the most widely used learning algorithm, back-propagation, is known to be very slow. Cascade-Correlation does not exhibit this limitation (Fahlman & Lebiere 1990). Table 1 summarizes the characteristics of the various approaches which can be used as response surface methods for optimization discussed above. It shows the main advantages of artificial neural networks – and of the cascade correlation algorithm in

9

particular – for the purpose of the general representation of (possibly non-linear) functions of many variables. The Cascade Correlation algorithm was first presented by Fahlman and Lebiere (1990) for pattern classification but shows promising results for function approximation for large design spaces. Cascade correlation adapted to function approximation is presented in the following section, after presenting in more detail how the NN approach is integrated in the design cycle and the benefits it offers. Table 1. RSM approaches for optimization – Important characteristics related to current project objectives

Artificial Neural Networks Polynomial-based Training with back-

propagation Cascade Correlation

algorithm • Mathematically simple • Requires domain

decomposition for non-convex functions

• Not accurate for highly non-linear functions

• Requires large dataset when design space size increases

• Can represent non-linear functions

• Network size fixed in most cases: need to know size a priori or end up with too small/large of a network

• Training with back-propagation is slow when number of design variables increases

• Can represent non-linear functions

• Network grows to optimum size: smallest suitable network

• Training is faster for large number of design variables – no back-propagation required

Constructive Neural Networks based on Cascade Correlation As mentioned above, the constructive Cascade-Correlation algorithm begins with a minimal network consisting of the input and output layers and no hidden unit (neurons), then automatically trains and adds one hidden unit at a time until the error (Ep) between the targets (fp) of the training set and the outputs from the network (fNN,p) reaches a desired minimal value. Thus, it self-determines the number of neurons needed as well as their connectivity or weights. The original algorithm of Fahlman and Lebiere (1990) was geared towards pattern recognition and has been improved to make it a robust and accurate method for function approximation (Schmitz et al 2002). Although the definitions used here assume that the NN has a single output, i.e. that the NN represents a single scalar function such as ship resistance, the process is easily extended to networks with multiple outputs (Schmitz et al 2002). In the applications considered in this paper, multiple functions, such as ship resistance and ship displacement, are each represented by a separate network, i.e. use multiple single-output networks rather than a single multiple-output network. It was just found during the course of the study that it was more advantageous to train multiple single output networks rather than one large multiple output network in terms of error on an unseen dataset (generalization error) and computing time (since multiple “single output” networks could be trained simultaneously faster than one single “multiple output” network. The training algorithm involves the steps described below and leads to a network with the topology of Fig. 4. 1. Start with the required input and output units; both layers are fully connected. The

number of inputs (design variables) and outputs (objective function, constraints) is dictated by the problem.

10

2. Train all connections ending at an output unit with a typical learning algorithm until the squared error, 2E , of the NN no longer decreases.

3. Generate a large pool of candidate units that receive trainable input connections from all of the network’s external inputs and from all pre-existing hidden units. The output of each candidate unit is not yet connected to the active network (output). Each candidate unit has a different set of random initial weights. All receive the same input signals and see the same residual error for each training pattern. Among candidate hidden units in the pool, select the one which has the largest correlation, C, defined by

( ) ( )0, 01

pN

p pp

C z z E E=

= − < > − < >∑ (1)

where z0,p is the output of the candidate hidden unit and Ep is the error at each point of the training set, calculated at Step 2. The weights of the hidden unit selected are then modified by maximizing the correlation C with an ordinary learning algorithm

4. Only the candidate with the best correlation score is installed. The other units in the pool are discarded. The best unit is then connected to the outputs and its input weights are frozen. The candidate unit acts now as an additional input unit. Training of the input-outputs connections by minimizing the squared error E as defined in Step 2 is then repeated. The use of several candidates greatly reduces the chances that a useless unit will be permanently installed because an individual candidate was trapped in a local maximum during training.

5. Repeat until the stopping criterion is verified. Instead of stopping the training based on the error measured on the Training Set (TS), the stopping criterion makes use of a Validation Set (VS) which is much smaller than the Training Set and distributed throughout the design space. Such approach is used because although the error on the TS decreases as hidden units are added (the network grows), the error on points elsewhere in the design space (such as those of the VS) may increase slightly. This phenomenon is known as over-fitting of the network and employing a VS circumvents this issue (Prechelt 1998). This important phenomenon is discussed below in more detail in the section illustrating the approach for a mathematical function.

HU #1 HU #2

HU #h

Output

Inputs

+ Fig. 4. NN Topology with Cascade Correlation Algorithm

11

For the application of the Cascade Correlation algorithm to function approximation, several improvements have been implemented (Schmitz et al, 2002). Inputs and outputs are non-dimensionalized, weights are constrained to a maximum value in order to limit strong non-linearities of the response surface, training of the input-to-hidden-unit weights and hidden-to-output weights is performed with a second order optimization method (Sequential Quadratic Programming) instead of the QUICKRPOP algorithm introduced by the original authors, several variations of the correlation formula are available for training the input-to-hidden-unit depending on the problem. Several stopping criteria are also available to limit over-fitting of the network; they all continue training slightly past the minimum validation error (error measured on the Validation Set) and the resulting network is that with the smallest squared error on the VS whereas the original authors stop when the error on the training set reaches a predetermined value.

NN for RSM: illustration and characterization As mentioned above, many applications have used NN for low search space dimensions and with little investigations of the errors introduced by the NN approach, particularly for larger search space dimensions. In this section, the capabilities of the network are analyzed by considering a mathematical function. The two-dimensional case is addressed first to illustrate the method. Results are then presented and analyzed when the design space dimension is increased from 2 to 30 in order to gain an insight into the behavior of the NN characteristics as the number of design variables increases and characterize the errors, not only on the training set as is customary, but also at points not used in the training.

Test function and illustration in 2 dimensions A mathematical test function used in a previous global optimization study (Besnard et al 1999) is selected because it has many minima and maxima and is easily extended to larger dimension spaces. The function has been modified such that, as the size of the design space, n, increases, the magnitude of the function remains between 0 and 1. It is defined over the s-dimensional compact [ ], nπ π− by

2 23 / 4

1

1( ) sin ( ) ( )4

ni i

i ii

x yf x A Bn =

−⎛ ⎞= + −⎜ ⎟⎝ ⎠

∑ (2)

where the scalars Ai and Bi are defined by

( ) ( ), ,

1

1 sin cos|| || || ||

n

i i j j i j jj

A a x b xn =

= ++

∑a b (3)

and

( ) ( ), ,

1

1 sin cos|| || || ||

n

i i j j i j jj

B a y b yn =

= ++

∑a b (4)

and where the matrices a and b are given by

12

1 ,

0.5 1 1.5 ....(1 ) 2 (2 ) 2 (3 ) 2 ....

( ( 1) ) (1 2 ) 2 (2 2 ) 2 (3 2 ) 2 ....2

: : : :: : : :

i j n

n n nj i n n n n

< <

⎡ ⎤⎢ ⎥+ + +⎢ ⎥+ −⎡ ⎤= = ⎢ ⎥+ + +⎢ ⎥⎣ ⎦ ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

a (5)

and

1 ,

2 1.5 1 ....( 4 ) 2 ( 3 ) 2 ( 2 ) 2 ....

( 1) 5 ( 4 2 ) 2 ( 3 2 ) 2 ( 2 2 ) 2 ....2

: : : :: : : :

i j n

n n nj i n n n n

< <

− − −⎡ ⎤⎢ ⎥− + − + − +⎢ ⎥+ − −⎡ ⎤= = ⎢ ⎥− + − + − +⎢ ⎥⎣ ⎦ ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

b (6)

Fig. 5 shows the function magnitude for the two-dimensional case (n = 2) and Fig. 6 shows the Neural Network obtained with a Training Set of 750 points distributed over the design space using a Latin Hypercube sampling (Stein 1987). The figures show that the network is able to represent the function quite accurately. Fig. 7 shows the error in the NN function value prediction as compared to the exact function value. For a function on the order of 1, the error is generally less than 0.01 with larger errors observed in a few locations, particularly at the boundaries of the domain. The error is largest in these locations because the Latin Hypercube sampling used as training set has only a couple points located on the boundary.

Fig. 5. Exact function representation in the two-dimensional case, n = 2.

13

Fig. 6. Neural Network representation of the function for n = 2.

Fig. 7. NN error

14

Neural Network accuracy in 5 dimensions Characteristics of the network are analyzed here for the case n = 5. Similarly to the n = 2 case, a training set with 1500 points is generated using a Latin Hypercube. A Validation Set of 200 points and a Generalization Set of 5000 points are also generated using a Latin Hypercube. As discussed above, the VS is used to stop the Cascade Correlation training at an “optimum” point. The much larger GS will be used to evaluate the error over the design space at points not used in the training and compare that error with that on the VS. In order to visualize the accuracy of the network, points in the TS and GS are arranged in ascending order in the value of the function f. The corresponding value of the Network, fNN, is shown and compared with f for the 1500 points of the training set in Fig. 8 when the default stopping criteria is used. With this criteria, training stops when the error on the validation set starts increasing as the number of hidden units increases. The corresponding graph for the 5000 points in the GS is shown in Fig. 9. The average error on the training set is approximately the same as that at other points of the design space (GS). One question which arises is whether continuing the training improves the results on the TS and/or GS. Fig. 10 shows the results for the TS when the network was allowed to reach the maximum number of hidden units (70 here). Here, agreement between the network and the exact function value is excellent and similar to that reported by other researchers (Koushan 2003). In that case, however, errors on the GS (Fig. 11) persist, i.e. further training improved the network behavior on the training set, but not on other points of the design space. This observation, illustrated in Fig. 12 where the average errors on the TS and GS are plotted as the network grows, indicates that the true quality of the NN should be measured on points not included in the TS, but in another set, such as the VS used here.

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 251 501 751 1001 1251 1501

Sample Point #

f(x)

NN

EXACT

Fig. 8. Error on the training set with the “early-stopping criterion” (n = 5)

15

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 501 1001 1501 2001 2501 3001 3501 4001 4501 5001

Sample Point #

f(x)

NN

EXACT

Fig. 9. Error on the generalization set with the “early-stopping criterion” (n = 5)

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 251 501 751 1001 1251

Sample Point #

f(x)

NN

EXACT

Fig. 10. Error on the training set when training is not stopped until 70 hidden units (n = 5)

16

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 501 1001 1501 2001 2501 3001 3501 4001 4501

Sample Point #

f(x)

NN

EXACT

Fig. 11. Error on the generalization set when training is not stopped until 70 hidden units (n = 5)

0

0.01

0.02

0.03

0.04

0.05

0.06

1 11 21 31 41 51 61

Number of Hidden Units

<E>

On TS

On GS

Minimum Generalization Error = 0.027for 30 HU's

Fig. 12. Average error on TS and GS as the number of hidden units in the network increases (case of n = 5)

17

Effect of design space size on training set requirements In order to evaluate the variations in NN performance as the design space size increases, the dimension of the search space, n, is varied from 5 to 30 and the training is performed for various sizes of training set. The cases n = 5, 10, 15 and 30 are considered for TS sizes of 500, 750 and 1,500 points. All TS are generated using a Latin Hypercube. For the 30-dimension case, a TS of 5,000 points is also considered. In all cases, the VS is kept small at 200 pts and the GS has 15,000 points (also generated by Latin Hypercube) which is assumed to be sufficiently large to get a measure of the quality of the network over the design space at points not used during the training. For practical applications, a GS would not be used. Instead, it is expected that the small VS would give an indication of the error at points other than those used for training (TS), i.e. the error on the GS and that on the VS should be similar. Fig. 13 through Fig. 16 show the variation in the average error, <E>, measured on the TS, VS and GS for n = 5, 10, 15 and 30, respectively. The data shows that the difference between the average errors computed on the VS and GS is small, validating the approach of using a small VS as stopping criterion during the NN training process. As expected, the average errors on the VS and GS decrease when the size of the TS is increased, i.e. increasing the number of points used to train the network does improve the quality of the network. However, while large reductions in the average error are observed when the size of the TS is increased from its smallest dimensions, further increasing the TS size produces more limited improvements. For n = 5, the GS and VS average error for a 1,500-pt TS is approximately 0.03 on a function which is between 0 and 1. If a TS with half that size is used, the average error increases only slightly to 0.04. In the 30-dimension case, increasing the TS from 1,500 to 5,000 only reduces the error from 0.054 to approximately 0.051. These results demonstrate that the size of the training set does not need to increase exponentially but that an increase in TS size proportional to the dimension of the design space may be adequate for most well-behaved problems. Based on the results presented here and the experience of the authors, the size of an adequate TS can be determined as a function of the dimension of the search space by

, 100p TSN n≈ (7)

Also as expected, except for the 30-dimension case at a TS size of 1500 points, the error on the training set is smaller than that on the VS or GS.

18

Fig. 13. Average error on TS, VS and GS as the TS size is increased for n = 5

Fig. 14. Average error on TS, VS and GS as the TS size is increased for n = 10

19

Fig. 15. Average error on TS, VS and GS as the TS size is increased for n = 15

Fig. 16. Average error on TS, VS and GS as the TS size is increased for n = 30

20

In summary, the results above lead to a few important conclusions. First, the overall accuracy of a NN should not determined by its accuracy on the TS, but on a (smaller) set different from the TS. The VS, used as a training stopping criterion in the present approach provides an adequate measure of that error. Second, Cascade Correlation provides adequate accuracy if errors on the order of 5% of expected function value are sufficient during the design space exploration. The results also suggest that if the NN approach is to be used for design/optimization, it may be beneficial to perform an optimization which explores and provides local optima in multiple regions of the design space. In this case, should the error at the global optimum lead to an unfeasible design, the next best solution –which might be in a different location of the design space– could be considered. Lin and Wu (2002) present an example of such optimization method based on a multi-island genetic algorithm.

Application to fast ship hull shape design The approach is applied here to the shape optimization of a fast ship configuration, known as the twin H-body (Fig. 17), subject to structural and other constraints. In the systems approach outlined above, this application corresponds to a subsystem design addressing the hull shape optimization subject to constraints emanating from the synthesis-level definition. The resulting optimized geometry was integrated into the first US-built “fast ship”, the HDV-100 technology demonstrator (Fig. 18), which is a larger version of that shown in Fig. 1.

Cross-foil

Strut

H-body

2.74 m

> 7.81 m

> 0.2286 m

>1.524 m

Less than 10.3632 m (34 ft)

9.144 m

Fig. 17. Twin H-body configuration

21

Fig. 18. HDV-100 Technology Demonstrator (shown here at low speed)

The optimization is performed using both a classical approach, in which the CFD analysis is integrated directly inside the optimization loop, and that based on neural networks, in which the CFD analyses are used for TS generation, i.e. outside of the optimization loop. This application has been presented and discussed in detail in Schmitz et al (2004) and only a summary is included here. It is based on an automated CAD-based system driven by a commercial optimization software in a way similar to that of Maisonneuve et al (2003), albeit with different tools.

Objectives, constraints and optimization implementation The objective of the optimization is to maximize lift-to-drag ratio subject to the following conditions and constraints:

• Operational speed of 47 knots (i.e. Reynolds number is 211.63×106 based on a reference chord of 8.69 m in 75 F Hawaii waters)

• Cavitation free at 52 knots, i.e. the minimum pressure coefficient on the configuration, Cp,min, must remain greater than that leading to the onset of cavitation, or Cp,min > - 0.269 (Hawaii water at 75 F).

• Total lift equal to 80 LT • Displacement greater that 16 LT • 2.74 m (9 ft) operating draft • Strut centerline to strut centerline distance is fixed at 9.14 m (30 ft) for mating

with upper hull • Overall beam length should not exceed 10.36 m (34 ft) • Minimum strut thickness 0.2286 m (0.75 ft) • Minimum strut chord of 1.524 m (5 ft) • The cross foil should have the structural integrity not to yield using a material of

yield strength of 344.7379 Mpa (50 ksi), assuming a solid section • Displacement pod length should not exceed 7.81 m.

The configuration is represented by a total of 28 design variables which control the size and shape of each component, which, once assembled, describe the entire configuration. Simple geometrical parameters, such as width, length, chord, etc., are used to characterize the elements. Foil cross-sections are defined by their mean camber line and thickness distributions. Camber is parameterized by classical NACA 2-parameter camber and the thickness distribution is represented by a 6-deg. polynomial, with an additional term in x for controlling the leading edge radius. This configuration is automatically generated by the CAD-based solid modeler, Pro-Engineer here, based on the 28 design variables using scripts.

22

In the second step, constraints which may be determined from the newly generated solid model, such as volume, structural constraint, etc., are evaluated. In third step, the structural constraint is evaluated. Then, if no constraint is violated, a suitable mesh is automatically generated using a commercial meshing package, ICEM-CFD. This mesh is then used along with the CFD input data file to execute the CFD code. The CFD tool used here makes use of the high Froude and Reynolds number approximations by employing a viscous-inviscid approach. The inviscid flow is solved by a higher order panel method with the free surface effects modeled by negative images and the viscous flow is solved using an inverse boundary layer approach which can treat large regions of flow separation. Viscous and inviscid methods are coupled using the blowing velocity/displacement concept and leads to accurate pressure, lift and drag predictions at minimal costs for this type of configuration and flow conditions (see, e.g., Hefazi et al 2002). Hence, only a surface mesh is needed. The use of a Reynolds-Averaged Navier-Stokes method would require the use of a volume mesh which could be implemented with the tools used here (ICEMCFD). Because a global optimization method is used (Genetic Algorithms), the objective function and constraints are integrated into a single “constrained objective function” which is to be minimized. The constrained objective function is such that, when at least one constraint is strongly violated, fc is set close to fmax. fmax is typically on the order of 1 and corresponds to a normalized value of the maximum objective function. The normalization value is given by the user and is typically chosen at the higher values of expected f over the search space. For example, with an optimization where L/D is the objective function on the order of 10-15, a normalization value of 10 to 20 can be chosen. Choosing 10 would mean that fc might reach 1.5. The objective is to minimize f subject to the ncon constraints ( ) 0, 1i cong x i n≤ ≤ ≤ . Two positive parameters 1ε and 2ε are now defined. If for all i, 1ig ≤ −ε , then the constrained objective function is f. If there exists an i such that 2ig ≥ ε , then the penalized cost function gets close to fmax depending on how the constraints are violated. In other words,

1ε decides when constraints become active, and 2ε when they become prohibitive. The generalized constraint G is then defined as

1

11( ) ( )

i

igcon

G x g xn ≥−ε

⎛ ⎞= + ε⎜ ⎟⎜ ⎟

⎝ ⎠∑ (8)

and the constrained objective function becomes

( ) ( ) max( ) 1 . ( ) . . 12

y

cef x y f x y f−⎛ ⎞

⎡ ⎤= − ϕ + ϕ −⎜ ⎟⎣ ⎦⎝ ⎠

(9)

where

21

0 if 0( )

if 0x

xx

e x−

≤⎧⎪ϕ = ⎨>⎪⎩

(10)

and

1 2

10 ( )( )

G xy =ε + ε

(11)

23

Classical optimization First, a classical optimization is performed in order to provide a baseline for comparison with the NN approach. The optimization, shown in Fig. 19 is implemented in iSIGHT which iteratively varies the geometric parameters in the solid model input files (shape.in and *.ptr). Then the new geometry is automatically generated in Pro-Engineer, a mesh suitable for the CFD analysis is also generated for that configuration with ICEM-CFD and the CFD analysis is performed to determine the hydrodynamic performance. The structural load is calculated as well in order to determine whether the structural constraints has been violated. The results are then combined into the constrained objective function which is transferred back to the optimizer for the next iteration. This analysis loop takes approximately 10 minutes per configuration.

Update shape in ProEngineer files (runShape)

Compute flow and process data (runDAC)

ProEngineer D.V.: • crosswing.ptr • hbody.ptr • strut.ptr

Shape D.V.: • shape.in

Optimizer or Latin Hypercube (iSIGHT)

New D.V.

Regenerate model (Proe)

Check if regenerated sucessfully

Check if structural constraint is violated

fc=3 No

fc=1 Yes

Yes

No

Check if runDAC ran sucessfully fc=2

No

Yes Calculated value of fc

Fig. 19. Optimization with classical approach. The same loop is used for training/validation set generation but with Latin Hypercube sampling instead of optimization.

The results for a total of 5000 Genetic Algorithm iterations, summarized in Table 2, give a lift-to-drag ratio improvement of 26%.

24

Table 2. Performance of optimized vs. initial configuration using a classical optimization method in which the CFD tool is integrated directly into the design loop (Fig. 2)

Baseline Optimum Objective Displacement 18.4 LT 16.1 LT > 16 LT Total lift 80.2 LT 81.6 LT = 80 LT Cp,min -0.263 -0.250 > - 0.269 L/D 10.23 12.92 Maximum

NN training For this problem to be treated with the NN approach, either a single 28-input/5-output NN or 5 28-input/single output networks are needed. The latter approach is used here for problem definition flexibility as depicted in Fig. 20, one for the objective function and the others for the constraints, lift-to-drag ratio (LOD), minimum pressure (Cpmin), dynamic lift (DL), buoyant lift (BL) and maximum stress value (Struc). All other constraints can be implemented directly into the problem definition.

LOD NN

Optimizer

New D.V.

Calculated value of constrained objective function, fc and LOD,

Cpmin, DL, BL, Struc

CpminNN

Dynamic Lift (DL)

NN

Buoyant Lift (BL)

NN

Structures (Struc)

NN

fc calculator

Fig. 20. Optimization process with NN

Several TS were generated using the same setup as that used in the classical optimization problem, but this time, instead of using an optimizer, a Latin Hypercube sampling was done over the design space (left hand side of Fig. 3 and Fig. 19) and used to train a set of neural networks. The results did not vary much from one to the other and are presented here for a training set size of 1000 data points. Also, in order to evaluate the quality of the training and compare the error on the 300-pt VS with that at points elsewhere in the design space, a GS of 5000 points was used. Table 3 shows the average errors and standard deviations over the TS, VS and GS. Errors and standard deviations are adequate for the problem at hand. For example, an average error of 0.04 is expected for the displacement (BL) which has a value on the order of 18 and the corresponding standard deviation is also 0.04. Also, average errors and standard deviations on the VS and GS are in excellent agreement, thus validating the VS approach despite the very low number of points used for the design space of 28 dimensions.

25

Table 3. Errors and standard deviations on TS, VS and GS for a 1000-pt. TS and a 5000-pt. GS for the 28-design variable Twin H-body optimization

Typical <E>TS <E>VS <E>GS Std(E)TS Std(E)VS Std(E)GS BL (LT) 18 0.0275 0.0381 0.0419 0.0223 0.0314 0.0427 Cpmin -0.269 0.0055 0.0065 0.0068 0.0066 0.0071 0.0080 DL (LT) 60 0.1607 0.1847 0.1855 0.2002 0.1420 0.1874 LOD 12 0.2270 0.2602 0.2646 0.1775 0.2074 0.2054 Struc (MPa) 300 4.05 5.3 5.3 3.7 5.2 5.1

Optimization with NN approach The 5 neural networks generated are then integrated into the optimization loop in as illustrated on the right hand side of Fig. 3. A Genetic Algorithm (GA) optimization was used with an initial population of 50 and 35000 iterations were run based on the constrained objective function. The best 100 runs resulting from the optimization with the five NNs (best fc) were then run using the Pro-Engineer model and the CFD code to compute the objective function and constraints and compare them with those of the NN near the optimum. Also, a few results from the optimization (i.e. as determined by the NN) with a slightly higher LOD than that of the best fc but with constraints closer to their limit (therefore resulting in a lower fc) were chosen and also analyzed with CFD. A summary of the results is shown in Table 4. For each size of training set, the table shows:

• The best fc as determined by the optimizer (i.e. after 35000 GA iterations using the NN)

• The best LOD based on the NN with minimal constraint violations: corresponds to some user-selected results from the optimization showing a slightly better LOD but with constraints close to the acceptable limits but resulting in a higher fc

• The best fc based on CFD results from the 100 best NN points (as determined by the GA)

• The best LOD based on CFD results from the 100 best NN points (as determined by the GA)

Table 4. Results from NN optimization and comparison with CFD

Output from NN Same Design Variable but with CFD

Results BL TL LOD Cpmin BL TL LOD CpMin

Best fc optimized from NN 15.23 80.59 14.39 -0.265 15.24 81.59 13.76 -0.266

Best LOD optimized from NN 14.75 79.94 14.44 -0.267 14.78 81.04 13.85 -0.268

Best fc (using CFD) out of 100 best runs 15.49 80.81 14.30 -0.264 15.51 81.67 13.70 -0.266

Best LOD (using CFD) out of 100 best runs 15.25 80.60 14.38 -0.265 15.25 81.64 13.78 -0.265

In each case, buoyant lift (BL), total lift (TL), lift-to-drag ratio (LOD) and minimum pressure (Cpmin) values computed by the NN and the CFD method are shown. The

26

stress value (Struc) is not shown because it exhibited little variations between points and because the constraint was not violated. Although the optimization uses DL and BL to determine TL, the latter is presented because it corresponds to a primitive twin H-body requirement. The differences between the NN and the CFD are within acceptable limits. The LOD is over-predicted by the NN and the differences between points are about the same. LOD is greatly improved from the baseline value of 10.23.

Comparison between classical and NN-based method Depending on which design is selected, L/D (LOD) ranges from 13.70 to 13.85, which is a definite improvement from the classical method which lead to 12.92. This improvement is due to the ability to increase the exploration of the design space within the time available in a given design project. For the classical optimization, the Genetic Algorithm was allowed to run for 5000 iterations, which corresponds to approximately one month of constant calculations on an Origin 3200 server. Because of the use of the CFD tool inside the design loop, CPU time available limited the design space exploration and did not lead to a true optimum. On the other hand, a much larger number of iterations could be performed with the NN approach leading to a greater L/D improvement. In this case, most of the CPU time is taken by the training set generation, with all other computations (training and GA iterations) representing a small fraction of the total CPU time. With as low as 1000 points generated to approximate the various functions over a design space with 28 design variables, an improvement of about 34% in L/D is achieved with the NN approach compared with the original baseline twin H-body, this at one fifth the cost needed to get a 26% improvement when using the classical approach (with iterations limited because of CPU time constraints). The results also point to a few improvements which would need to be implemented address non-differentiable functions (to improve Cpmin predictions, for example), in the selection of constrained objective function, and to the selection of mathematical optimization method. The latter comment is particularly pertinent for problems in which the optimizer (GA here) would focus its attention in a region of the design space where one (or more) function (objective or constraint) is not approximated as well as might be desired, thus potentially leading to unusable optimization results. One could benefit from having an optimization method with explores several regions of the design space, as illustrated in Lin and Wu (2002). Results do show, however, that the method can provide its user with a valuable tool for improving designs within a limited time frame and possibly at a lower cost than using conventional analysis tools integrated in the optimization loop. In this latter case, similarly to the twin H-body configuration optimization presented here, the cost of the analyses limits the number of iterations which can be performed in a reasonable time leading to sub-optimal solutions. On the other hand, with the NN approach, a large number of designs can be investigated quickly –almost instantaneously– once a training set has been made available and the NN has been trained.

Conclusions The paper presents a Neural Network-based Response Surface Method for reducing the cost of computer intensive optimizations. The advantages offered by using constructive

27

networks as opposed to (a priori) fixed-size networks for such applications are emphasized leading to the selection and discussion of the Cascade Correlation algorithm which allows for efficient Neural Network determination when dealing with function representation over large design spaces. During training, the network grows until the error on a small set (VS) different from that used in the training itself (TS) starts increasing. The method is characterized for a mathematical function for dimensions ranging from 5 to 30 and the importance of analyzing the error on a set other than the training set is emphasized. The approach is then applied to the automated CFD-based shape optimization of a fast ship configuration known as the twin H-body. Unlike in the classical case where the CFD tool resides inside the optimization loop, the CFD analysis is “extracted” from that loop and, instead, used to generate the (smaller) data sets needed for the NN training (TS & VS). The classical approach yields a design improvement of 26% while the NN-based method allows reaching a 34% improvement at one fifth of the cost of the former. Based on the analysis of the results, areas for future improvements and research are outlined. Most notably, results suggest the use of the approach with a multi-island search algorithm where several regions of the design space are explored concurrently. Then, analyses can be performed for a few promising designs in each of these niches. Used in this fashion, the method has the potential for saving valuable development cycle time and increasing the performance of future ship designs.

Acknowledgements This work was conducted in collaboration with Pacific Marine of Hawaii. The authors would like to express appreciation to Mr. Steven Loui and his group at Pacific Marine for their various contributions. The authors also would like to acknowledge Mr. David Egan for providing all of the CAD and grid generation models and interfaces. This material is based upon work supported by the Office of Naval Research, under Cooperative Agreement No. N00014-04-2-0003 with the California State University, Long Beach Foundation Center for the Commercial Deployment of Transportation Technologies (CCDoTT). The authors thank Stan Wheatley, Steven Hinds, and Carrie Scoville from CCDoTT and Dr Paul Rispin, ONR program manager, for their supports.

References Agatonovic-Kustrin, S., Zecevic, M., Zivanovic, L., and Tucker, I.G. (1998), Application of neural

networks for response surface modeling in HPLC optimization, Analytica Chimica Acta, 364, 265-273.

Bertram, V.; Mesbahi, E. (2004), Estimating Resistance and Power for fast Monohulls Employing Artificial Neural Nets, 4th Int. Conf. High-Performance Marine Vehicles (HIPER), Rome.

Besnard, E., Cordier-Lallouet, N., Schmitz, A., Kural, O., and Chen, H.P. (1999), Design/Optimization with Advanced Simulated Annealing, AIAA Paper No. 99-0186, Reno, NV

Blanchard, B., and Fabrycky, W. (2006), Systems Engineering and Analysis, 4th Ed., Prentice Hall. Bourquin, J., Schmidli, H., van Hoogevest, P., and Leuenberger, H. (1998), Advantages of

Artificial Neural Networks (ANNs) as alternative modeling technique for data sets showing non-linear relationships using data from a galenical study on a solid dosage form, European Journal of Pharmaceutical Sciences, 7, 5-16.

28

Danõşman, D.B., Mesbahi, E., Atlar, M., Gören, O. (2002), A new hull form optimization technique for minimum wave resistance, 10th Int. Mar. Assoc. Mediterranean Congress (IMAM), Crete.

Deng, J., Gu, D., Li, X., and Yue, Z.Q. (2005), Structural reliability analysis for implicit performance functions using artificial neural network, Structural Safety, 27, 1, 25-48.

Dutt, J.R., Dutta, P.K., and Banerjee, R. (2004), Optimization of culture parameters for extracellular protease production from a newly isolated Pseudomonas sp. using response surface and artificial neural network models, Process Biochemistry, 39, 2193–2198.

Fahlman, S.E., and Lebiere, C. (1990), The Cascade-Correlation Learning Architecture, Technical Report CMU-CS-90-100, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA.

Gomes, H. M., and Awruch, A. M. (2004), Comparison of response surface and neural network with other methods for structural reliability analysis, Structural Safety, 26, 49-67.

Hefazi H. (2002), CFD Design Tool Development and Validation, CCDoTT FY 00 Task 2.8 report, Center for the Commercial Deployment of Transportation Technologies, Long Beach, CA

Haykin, S. (1998), Neural Networks, A Comprehensive Foundation, Prentice Hall; 2nd edition. Hornik, K. (1991) Approximation capabilities of multi-layer feedforward networks, Neural

Networks, 4, 251-257 Jain, P., and Deo, M. C. (2005), Neural Networks in Ocean Engineering, SAOS 2006 1, 1, 25-35. Koh, L., Altar, M., Wright, P.N.H., Mesbahi, E., and Cooper, D. (2004), An inverse design

procedure for hydrodynamic optimization of high speed hull form using commercial software, RINA Conference on High Speed Craft.

Koh, L., Janson, C-E, Altar, M., Larsson, L., Mesbahi, E., and Abt, C. (2005), Novel design and hydrodynamic optimization of a high speed hull form, 5th International Conference on High Performance Marine Vessels, Shanghai.

Koushan, K. (2003), Automatic Hull Form Optimization Towards Lower Resistance and Wash using Artificial Intelligence, FAST 2003 Conference, Ischia, Italy.

Kwok, T.Y., and Yeung, D.Y. (1997), Constructive Algorithms for Structured Learning in Feedforward Neural Networks for Regression Problems, IEEE Transactions on Neural Networks, 8, 630-645.

Lee, J., and Hajel, P. (2001), Application of classifier systems in improving response surface based approximations for design optimization, Computers and Structures, 79, 333-344.

Lin, C.Y., and Wu, W.H. (2002), Niche Identification Techniques in Multimodal Genetic Search with Sharing Scheme, Advances in Engineering Software, 22, 779-791.

Maisonneuve, J.J. (2003), Chapter 7: Applications Examples from Industry, Optimistic- Optimization in Marine Design, 2nd edition, Birk, L., and Harries, S. (Editors), Mensch & Buch Verlag.

Maisonneuve, J.J., Harries, S., Marzi, J., Raven, H.C., Viviani, U., and Piippo, H. (2003) Towards Optimal Design of Ship Hull Shapes, 8th International Marine Design Conference - IMDC03, Athens.

Mesbahi, E., and Bertram, V. (2000), Empirical Design Formulae Using Artificial Neural Nets. COMPIT'2000, Potsdam.

Mesbahi, E. (2003), Chapter 10: Artificial Neural Networks – Optimization, Optimistic- Optimization in Marine Design, 2nd edition, Birk, L., and Harries, S. (Editors), Mensch & Buch Verlag.

Prechelt, L. (1998), Automatic Early Stopping Using Cross Validation: Quantifying the Criteria, Neural Networks, 11, 4, 761-767.

Schmitz, A., Besnard, E., and Hefazi, H. (2004), Automated Hydrodynamic Shape Optimization using Neural Networks, Paper No. C6 (D19), SNAME Maritime Technology Conference & Expo, Washington D.C.

29

Schmitz, A., Besnard, E., and Vives, E. (2002), Reducing the Cost of Computational Fluid Dynamics Optimization Using Multi Layer Perceptrons, 2002 World Congress on Computational Intelligence, Honolulu, HI.

Simpson, T.W., Mauery, T.M., Korte, J.J., and Mistree, F. (1998), Comparison of Response Surface and Kriging Models for Multidisciplinary Design Optimization, AIAA Paper No. 98-4755, Proceedings, 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, 1, 381-391.

Stein, M.L. (1987), Large Sample Properties of Simulations using Latin Hypercube Sampling, Technometrics, 29, 2, 143-151.

Takayama, K., Fujikawa, M., Obata, Y., and Morishita, M. (2003), Neural network based optimization of drug formulations, Advanced Drug Delivery Reviews, 55, 1217–1231.

Todoroki, A., and Ishikawa, T. (2004), Design of experiments for stacking sequence optimizations with genetic algorithm using response surface approximation, Composite Structures, 64, 349-357.