Comparison of model structural uncertainty using a multi-objective optimisation method

HYDROLOGICAL PROCESSESHydrol. Process. 25, 2642–2653 (2011)Published online 23 February 2011 in Wiley Online Library(wileyonlinelibrary.com) DOI: 10.1002/hyp.8006

Comparison of model structural uncertainty using amulti-objective optimisation method

Giha Lee,1* Yausto Tachikawa2 and Kaoru Takara3

1 Research Associate, Construction & Disaster Research Center, Department of Civil Engineering, Chungnam National University, Daejeon305-764, Korea

2 Associate Professor, Department of Urban and Environmental Engineering, Kyoto University, Kyoto 615-8540, Japan3 Professor, Disaster Prevention Research Institute, Kyoto University, Uji, Kyoto 611-0011, Japan

Abstract:

This study aims to propose a method for effectively recognising and evaluating model structural uncertainty. It began with acomparative assessment of various model structures that have differing features regarding the rainfall-runoff mechanism andDEM spatial resolution. The assessment applied a multi-objective optimisation method (MOSCEM-UA) with two objectivefunctions (simple least-squares and the heteroscedastic maximum likelihood estimator), and focused on five historical floodevents. The study was based on the assumptions that a structurally sound model assures improved prediction results (eitherminimized or maximized model performance measure), allows constant model performance with regard to objective functions(a small Pareto solution set), and yields good applicability of a calibrated parameter set to various events (good parameterstability). The results indicated that KWMSS, a distributed model, was superior to SFM, a simple lumped model, whenestimating a Pareto solution set and assessing parameter stability for the applied events. In addition, three different spatialresolutions (250 m, 500 m, and 1 km) were compared to assess the structural uncertainty due to changes in the topographicalrepresentation in distributed rainfall-runoff modelling. The results indicated that the 250 and 500 m models were Pareto-equivalent, containing similar Pareto fronts, and both produced Pareto results superior to the 1 km model. Both models alsoyielded parameter stability values that were much more superior to the model based on a 1 km DEM. As the topographicrepresentation became more detailed, the model showed a tendency to have less structural uncertainty in terms of guarantyingbetter performance, better parameter stability, and a smaller Pareto solution set. On the other hand, the output of a spatiallydetailed model was likely to be insensitive to the variation of model parameters (i.e. equifinality). Copyright 2011 JohnWiley & Sons, Ltd.

KEY WORDS equifinality; model structural uncertainty; multi-objective optimisation method; parameter stability; Pareto solutionset

Received 22 September 2008; Accepted 21 August 2009

INTRODUCTION

A basic and principal task in hydrological modelling isidentifying a suitable model and its corresponding opti-mal parameters under given conditions, such as the mod-elling purpose, catchment characteristics, and availabledata (Wagener and Gupta, 2005). However, the quan-tifiable and/or unquantifiable uncertainties involved inmodelling procedures make model identification diffi-cult. Uncertainty can propagate into model predictionsand, in turn, produce unreliable prediction results. Refs-gaard and Knudsen (1996) and Uhlenbrook et al. (1999)demonstrated that different conceptualisations of a catch-ment runoff system can produce equally good numericaloutcomes, even if some models are overly simplified oroutside a hydrologist’s experience/expertise (i.e. struc-tural uncertainty). Beven and Binely (1992) also showedthat despite having differing values, a number of param-eter sets were capable of yielding quite similar model

* Correspondence to: Giha Lee, Research Associate, Construction &Disaster Research Center, Department of Civil Engineering, ChungnamNational University, Daejeon 305-764, Korea.E-mail: [email protected]

performance measures from the best-performing param-eter set (i.e. equifinality).

A great deal of research has focused on recognisingthe propagation of various uncertainty components intomodel predictions (e.g. Beven and Binely, 1992; Jakemanand Hornberger, 1993; Kuczera and Mroczkowksi, 1998;Gupta et al., 1998; Kavetski et al., 2003). In particular,recent research has increasingly focused on model struc-tural uncertainty (Gupta et al., 1998; Yapo et al., 1998;Boyle et al., 2000, 2001; Wagener et al., 2001; Vrugtet al., 2003; Lee et al., 2007). This is a fundamentalproblem in hydrological modelling because a model isa simplification of reality, so conceptually, it representsonly some aspects of the actual hydrological system, nomatter how spatiotemporally sophisticated it may be.

Although the structural uncertainty of a model has asignificant effect on prediction uncertainty, the formeris difficult to assess due to problems with quantifica-tion related to model structural error. Gupta et al. (2003)pointed out that a major consequence of model struc-tural inadequacy is the lack of ability to reproduce entire(or global) hydrological behaviour with a single optimalparameter set estimated using traditional single-objective

Copyright 2011 John Wiley & Sons, Ltd.

COMPARATIVE ASSESSMENT OF MODEL STRUCTURAL UNCERTAINTY 2643

optimisation algorithms (e.g. the shuffled complex evolu-tion (SCE-UA) method; Duan et al., 1992, 1993). In otherwords, when a subjective selection of the objective func-tion is used to calibrate structurally imperfect models, thismay result in an overemphasis on specific response modesin estimating hydrographs, such as low flow or high flow(i.e. local hydrological behaviours). Therefore, differingparameter combinations are required to represent partic-ular local behaviours of the actual rainfall-runoff system(Wagener et al., 2004; Lee et al., 2007).

Gupta et al. (1998, 2003) demonstrated the inherentmulti-objective feature of hydrological modelling andthen proposed an alternative solution, ‘Pareto optima’(often called the non-dominated or trade-off solution),to permit several local hydrological behaviours simul-taneously. The Pareto solution, which allows multipleacceptable parameter sets, has a particular property:moving from one solution to another causes improve-ment in one criterion and decline in another. Yapoet al. (1998) and Vrugt et al. (2003) developed effectiveand efficient algorithms: the Multi-Objective Complex(MOCOM-UA) and the Multi-Objective Shuffled Com-plex Evolution Metropolis (MOSCEM-UA), respectively.Both algorithms are advanced versions of the originalSCE-UA algorithm and were developed to solve themulti-objective problem in hydrological modelling.

As reported in previous studies (Gupta et al., 1998,2003; Yapo et al., 1998), a better (or more stable) modelstructure results in both smaller Pareto sets and moreimproved values, with respect to the given objectivefunctions during calibration trials. This means that ahydrological model with less structural uncertainty can beregarded as a model that provides constant and accuratehydrological behaviours, regardless of the objective func-tions. Therefore, a comparison of Pareto optima usingmulti-objective algorithms can be employed as one indi-cator of model structural uncertainty. Furthermore, if nodrastic changes are made to the land use at the studysite, such a stable model should be applicable to varioustypes of events without changing the parameter values(Klemes, 1986). Thus, a structurally better model has

a superior parameter stability that is not dependant onrainfall events. Consequently, model structural uncer-tainty can be evaluated by testing the consistency ofmodel performance with regard to objective functions,by using a multi-objective optimisation method and bytesting parameter stability under various rainfall events(Lee et al., 2007).

The principal goal of this study is to develop a methodfor effectively recognising and assessing model structuraluncertainty. To this end, three experiments were con-ducted. First, a multi-objective optimisation algorithmwas used to calibrate two rainfall-runoff models withdifferent mathematical forms in order to conceptualizethe rainfall-runoff process: a simple lumped model anda distributed kinematic wave model based on a 250mDEM. These models were compared to investigate theconsistency of model performance for two specific objec-tive functions with differing characteristics. Second, theparameter stability was assessed to determine whetherthe calibrated model was applicable for predicting vari-ous flood events. Five historical flood events were usedfor this purpose. Finally, a case study was conductedusing a distributed kinematic rainfall-runoff model basedon three different DEM spatial resolutions (250m, 500m,and 1 km). This case study investigated how model struc-tural uncertainty, due to variability in the spatial scale ofmodel building units, affected both Pareto solution setsand parameter stability.

MATERIALS

Study area and historical data

The study site is the Kamishiiba catchment, anupstream area of the Kamishiiba Dam, which lies withinthe Kyushu region of Japan and has an area of 211 km2

(Figure 1(a)). The topography of this area is hilly, withelevations from 431 to 1720 m, and most of the landis forested. As shown in Figure 1(b)–(d), the drainagenetworks at this study site are represented by three dif-ferent DEM spatial resolutions (from 250 m to 1 km) for

Figure 1. (a) The Kamishiiba catchment and modelled drainage networks of (b) 250 m, (c) 500 m, and (d) 1 km DEMs

Copyright 2011 John Wiley & Sons, Ltd. Hydrol. Process. 25, 2642–2653 (2011)

2644 G. LEE, Y. TACHIKAWA AND K. TAKARA

Table I. Historical flood events

Event ID Flood term Type Total rainfall (mm) Max. runoff (m3/s) Runoff ratio

Event 1 15–19 Sep. 1997 Typhoon no.9 496 1192 0Ð84Event 2 24 June–3 July 1999 Rainy season (frontal) 463 210 0Ð51Event 3 1–7 Aug. 1999 Typhoons nos.7, 8 474 472 0Ð5Event 4 22–27 Sept. 1999 Typhoon no.18 340 644 0Ð75Event 5a 3–9 Sept. 2005 Typhoon no.14 714 1718 0Ð89

a Note: spatial resolution of radar rainfall and temporal resolution of observed discharge are 2Ð5 km and 1 h, respectively.

distributed rainfall-runoff modelling. The observed dis-charge data (Kyushu Electric Co., Inc.) with a 10 mintemporal resolution was converted from the water levelof the dam inflow, and the operational radar rainfall data(Ejiroyama X-band radar) was available for a 128 kmradius. Table I lists the characteristics of the five histori-cal events used in this study.

Rainfall-runoff models

Two different models were developed based onthe Objective-Oriented Hydrological Modeling System(OHyMoS; http://flood.dpri.kyoto-u.ac.jp, Takasao et al.,1996; Ichikawa et al., 2000), as described below. Thesewere used for a comparative assessment of model struc-tural uncertainty.

Storage function method (SFM). The SFM (Kimura,1960) is a simple nonlinear reservoir model that is widelyused in practical engineering work in Japan despite itssimplicity. The SFM can be expressed as:

dS

dtD re�t � Tl� � q, S D kqp �1�

re D{

f ð r, if∑

r � RSA

r, if∑

r > RSA�2�

where S is the water storage, re is the effective rainfallintensity, r is the rainfall intensity, q is the runoff, tis time, k is a storage coefficient, p is a coefficient ofnonlinearity, f is the primary runoff ratio, Tl is lagtime, and RSA is the accumulated saturated rainfall. Fourparameters (k, p, f, and RSA) must be optimized inthe SFM.

Kinematic wave method for subsurface and surfacerunoff (KWMSS). The KWMSS assumes that a permeablesoil layer covers the hillslope, as shown in Figure 2.The soil layer consists of a capillary layer, whichcontains unsaturated flow, and a non-capillary layer,which contains saturated flow. According to this runoffmechanism, overland flow will result if the depth ofwater, h, is higher than the soil depth, D. The stage-discharge relationship can be defined as:

q D{

vcdc�h/dc�ˇ, 0 � h � dc

vcdc C va�h � dc�, dc � h � ds

vcdc C va�h � dc� C ˛�h � ds�m, ds � h

�3�

∂h

∂tC ∂q

∂xD r�t� �4�

Figure 2. Schematic model structure and stage-discharge relationship ofKWMSS

The flow rate of each slope element (poly-lines ofthe drainage networks shown in Figure 1(b)–(d)) iscalculated by the stage-discharge Equation (3), combinedwith the continuity Equation (4), where vc D kci, va Dkai, kc D ka/ˇ, ˛ D p

i/n, i is the slope gradient, kc isthe hydraulic conductivity of the capillary soil layer, ka

is the hydraulic conductivity of the non-capillary soillayer, and n is a roughness coefficient; the water depthcorresponding to the water content is ds and the waterdepth corresponding to the maximum water content inthe capillary pore is dc. For a detailed explanation ofthe model structure, see Tachikawa et al. (2004). Fiveparameters (n, ka, ds, dc, and ˇ) must be optimizedin the KWMSS; these are assumed to be spatiallyhomogeneous. Moreover, the initial state variables ofthe two models were directly estimated by the firststreamflow observations of all events with assuming asteady-state condition.

METHODOLOGY

Two methods were used to evaluate the structural uncer-tainty of the model. The first step was to estimate thePareto solution sets and evaluate the performance for eachmodel structure using MOSCEM-UA with two differingobjective functions: simple least squares (SLS) and theheteroscedastic maximum likelihood estimator (HMLE).The second step was to assess the parameter stabilityby testing the transposability of the optimal parametersets with respect to the two objective functions from oneevent to another. Then, the extent of parameter stabil-ity was quantified using two indices: the peak dischargeratio (PDR) and the Nash-Sutcliffe coefficient (NSC). Anideal model structure is expected to provide balanced andacceptable simulation results for several local behaviours,regardless of the objective functions and flood events. In



addition, the same methodology, described above, wasused to assess the model structural uncertainty due toeither spatial aggregation or disaggregation in distributedrainfall-runoff modelling using the KWMSS.

Model calibration using MOSCEM-UA

Difficulties in automatic calibration can arise fromthe response surfaces of parametric structures: numerouslocal optima, non-smooth response surfaces, and non-convex shapes around global optima. These have moti-vated researchers to develop global search algorithms foruse in hydrological modelling (Sorooshian and Gupta,1995). Evolutionary algorithms are probably the mostcommonly applied global optimisation methods for thecalibration of non-linear hydrologic models. MOSCEM-UA is a Markov-chain Monte Carlo sampler (MCMCS)designed to solve the multi-objective optimisation prob-lem in hydrological models. It combines the characteris-tics of the original SCE-UA with the probabilistic searchof the Metropolis algorithm and the theory of effectivefitness assignment, and it is well suited for estimatingPareto solutions in high-dimensional multi-objective cal-ibration problems. It is also superior to the conventionalMCMCS in its ability to estimate Pareto optimal solutionsbecause it prevents clustering of evolved solutions in thecompromized region as well as premature convergencewhen highly correlated performance criteria are used formodel calibration (Vrugt et al., 2003). Researchers in var-ious fields have proven its efficacy for finding Pareto opti-mal solutions (Vrugt et al., 2003; Johnsen et al., 2005;Schoups et al., 2005).

Objective functions

The goal of computer-based automatic calibration is tofind model parameter values that minimize or maximizethe numerical value of the objective functions. The mostcommonly used objective functions in rainfall-runoffmodelling are variations of the SLS function, defined as:

SLS D 1

N

N∑tD1

�qobst � qt��2 �5�

where qobst is the observed streamflow value at time t,

qt�� is the simulated streamflow value at time t usinga parameter set �, and N is the number of flow valuesavailable. Although the original SLS is not divided by theN, we revised it as expressed in Equation (5) to comparethe relative fitness of all events used here. One of theproperties of this function is that the residuals betweenthe observed and simulated discharge are evenly weightedacross an event, thereby yielding a parameter set thatmatches well around the peak discharge.

The HMLE is the most successful form of the max-imum likelihood criteria, and correctly accounts forthe non-stationary variance in streamflow measurementerrors (Sorooshian and Dracup, 1980). This new mea-sure incorporates weight, providing a more balanced

performance across the entire flow range. It can beexpressed as:

min�,�

HMLE D

1

N

N∑tD1

wtεt2

[N∏

tD1

wt

] 1N

�6�

where εt D qobst � qt�� is the model residual at time

t, wt is the weight assigned to time t computed aswt D f2��1�

t , ft D qtruet is the expected true flow at time

t, and � is the transformation parameter that stabilizes thevariance. Yapo et al. (1998) recommended the use of ft

as observed flow, qobst for more stable estimation. In this

study, we used the revised form of HMLE, suggested byDuan (1991); it is equivalent to Equation (6) but morestable to obtain the value of the HMLE. The modifiedform of HMLE is derived as:

HMLE D

1

N

N∑tD1

wtε2t

exp[2�� 1�ad]�7�

where ad D 1N

N∑tD1

ln�ft�, we substituted � D 0Ð3 and

ft D qobst into Equation (7) and then estimated the

HMLE values during calibration trials.

COMPARISON OF PARETO SOLUTION SETSBETWEEN TWO TESTABLE RAINFALL-RUNOFF

MODELS

Two rainfall-runoff models, SFM and KWMSS with250 m DEM (hereafter, KWMSS 250m) were calibratedusing MOSCEM-UA with SLS and HMLE for five floodevents involving differing features. Figure 3 plots theresults of the Pareto optimal solutions for Events 1,3, and 5. This figure includes two-dimensional (2D)projections of two criteria solution spaces, which arerepresented by 500 Pareto solutions selected from param-eter samples after 3000 iterations of MOSCEM-UA.Here, the grey diamond and black cross marks indi-cate the SFM and KWMSS 250m results, respectively,and the minimum values for each objective functionare shown for SLS (open circle) and HMLE (opensquare). The results clearly indicate that the distributedmodel, KWMSS 250m, performed better than SFM; allof the minimum values of the two objective functionswere much smaller in KWMSS 250m than in SFM.In addition, the variability of trade-offs indicates thatKWMSS 250m was more constant with regard to thetwo specific objective functions than SFM, becauseKWMSS 250m provided smaller Pareto solution sets.The comparison of Pareto solution sets between the twomodels revealed that KWMSS 250m was more preciseand less uncertain in modelling the Kamishiiba catchmentrainfall-runoff.



Figure 3. Comparison of objective function spaces between SFM and KWMSS 250m; (a) Event 1, (b) Event 3, and (c) Event 5

To show how model structural uncertainty propa-gates into model output simulations, hydrographs werereproduced using optimal parameter sets correspond-ing to the minimum model performance measures, asshown in Figure 3. Tables II and III summarize thebest-performing parameter combinations from the twomodels.

Figure 4 shows that for SFM, the simulated hydro-graphs, based on parameter sets calibrated using SLS,matched the measured runoff for all events well, whilethe hydrographs based on parameter sets calibrated usingHMLE tended to underestimate large flood events; seeEvents 1, 4, and 5 in Figure 4(a). However, SFM pro-vided accurate simulation results for small flood events(i.e. Events 2 and 3), regardless of objective function.This suggests that in the simple lumped model, smallquantities of runoff residual due to small-magnitude rain-fall weakened the role of the objective function. Thus,the two objective functions produced similar optimal

parameter sets and eventually resulted in constant modeloutputs. Indeed, the two best-performing parameter setsfrom SFM for Events 2 and 3 were relatively similar,compared to those from the large flood events: Events 1,4, and 5.

In contrast, the objective function appeared to have nosignificant impact on the KWMSS 250m performance, asshown in Figure 4(b); all simulated hydrographs matchedthe measured discharge well. This indicates that the useof KWMSS 250m for distributed rainfall-runoff mod-elling prevented the problem related to the subjectiveselection of the objective function for model calibra-tion, since it produced satisfactory results for the runoffof both rising and falling periods with a single opti-mal parameter set tuned by any given objective function.Another interesting finding is that while no constantparameter set was found for the two objective func-tions in the KWMSS 250m calibrations (Table III), allthe results from these differing values were visually

Table II. Best-performing parameter sets of SFM for each event

SFM

SLS HMLE

1 2 3 4 5 1 2 3 4 5

k [−] 49·91 35·80 49·94 49·94 49·41 49·94 36·65 49·96 49·98 49·32P [−] 0·529 0·999 0·683 0·544 0·527 0·585 0·999 0·715 0·586 0·626f [−] 0·596 0·846 0·760 0·552 0·172 0·622 0·857 0·827 0·509 0·181RSA [mm] 194·8 295·3 1·402 194·7 220·1 170·4 299·0 1·406 161·6 203·1

Eventparameter

Table III. Best-performing parameter sets of KWMSS 250m for each event

KWMSS

SLS HMLE

1 2 3 4 5 1 2 3 4 5

n [m− 1/ 3s] 0·499 0·495 0·499 0·499 0·474 0·499 0·499 0·495 0·499 0·198ka [m/s] 0·019 0·031 0·014 0·026 0·011 0·023 0·042 0·015 0·024 0·012ds [m] 0·621 0·735 0·886 0·669 0·353 0·691 0·724 0·871 0·660 0·452dc [m] 0·467 0·599 0·510 0·599 0·045 0·555 0·599 0·472 0·599 0·067b [− ] 6·172 5·396 19·53 6·219 2·801 7·016 5·839 1·939 6·018 2·316

Eventparameter



Figure 4. Comparison of hydrographs generated by the two optimal parameter sets for all events between (a) SFM, and (b) KWMSS 250m

acceptable. This kind of phenomenon, when many param-eter combinations can yield equally good outputs interms of model performance measures or hydrologicalvariables in spite of their unreliable values, has beencalled ‘equifinality’ (Beven and Binley, 1992; Savenije,2001). This means that some calibrated parameters losephysical meaning with respect to their numerical valuebut are still meaningful in terms of model predic-tions. Therefore, those cannot be rejected recklessly

until additional evidence becomes apparent or constraintsappear.

Figure 5 demonstrates that KWMSS 250m had ahigher potential for equifinality than SFM. This figureillustrates the Pareto-ensemble simulations from the twomodels, which are associated with the parameter samplesof 500 Pareto solutions. SFM produced a wide simula-tion uncertainty boundary; in particular, Events 1, 4, and5, while KWMSS 250m produced a very narrow region



Figure 5. Simulation uncertainty boundary (grey shaded band) associated with the Pareto solution sets for (a) SFM, and (b) KWMSS 250m

of uncertainty. Within the context of equifinality, it isapparent that even though KWMM 250m had a morestable model structure in terms of model performanceand Pareto solution sets, it was likely to suffer from poorparameter identifiability because of the insensitivity ofmodel performance to parameter values. Wagener et al.(2004) also noted that a lack of identifiability obstructsthe applications of complex continuous model structures

for the regionalisation of model parameters in ungaugedbasins.

ASSESSMENT OF PARAMETER STABILITYFOR VARIOUS EVENTS

As discussed above, a less erroneous model structuremay result in a more acceptable performance when its



Figure 6. Results of parameter stability for the five flood events in (a) SFM, and (b) KWMSS 250m. Note that each point indicates the NSC andPDR values and the bottom label indicates the applied optimal parameter set illustrated in Tables II and III while the upper label is the target event

for parameter stability test

calibrated parameter set is applied to various eventscontaining differing rainfall patterns and magnitudes.Therefore, the stability of identified parameter sets forvarious events can also be used as an indicator to assessmodel structural uncertainty. Here, parameter stability isquantified by two indices: the PDR and the NSC statistic.These can be expressed as:

PDR D qpeaksim /qpeak

obs �8�

NSC D 1 �

N∑tD1

�qobst � qt��2

N∑tD1

�qobst � qmean�2

�9�

where qpeaksim is the simulated peak discharge and qpeak

obs isthe observed peak discharge. PDR measures the tendencyof a simulated peak discharge to be larger or smallerthan the observed peak discharge. NSC measures therelative magnitude of residual variance to the varianceof the observed stream flows. The optimal value of bothmeasures is 1.

The optimal parameter sets for each event, summarizedin Tables II and III, were applied in order to reproducehydrographs for the five events. Figure 6 plots themodel performance, illustrating the quantified results ofparameter stability. Using SFM, the optimal parametersets were mostly inapplicable for differing flood events

(they exhibited poor parameter stability, and the NSC andPDR values plotted far from optimal values). However,even though the global performance was poor, the bestperforming SLS-based parameter sets were suitable forcapturing the peak discharges in large flood eventsbecause of the distinct characteristics of SLS (the SLSparameter sets provided closer PDR values to 1 than theHMLE parameter sets; see the top panel in Figure 6(a)).In contrast, the calibrated parameter combinations ofKWMSS 250m were suitable for simulations of differingevents (they exhibited good parameter stability, and theNSC and PDR values were plotted at near optimalvalues). However, the optimal parameter sets identifiedfrom other events were inapplicable to Event 2 in bothmodel applications. SFM and KWMSS 250m producedNSC values of approximately less than 0Ð6 and 0Ð8,respectively (or much worse), and PDR values weregreatly over-estimated. It is likely that the observed datafor Event 2 contained greater errors than the data for theother events, making them incompatible with the modelsthat were produced using informative observed data.

The distributed model, KWMSS 250m, is sure to pro-vide better parameter stability than the lumped model,SFM. Nevertheless, better parameter stability does notguarantee the identification of an identical single optimalparameter set for the flood events used here. Even non-optimal parameter values can lead to equivalent runoffsimulations for different flood events. As well as theresults of Pareto solution estimation in Section 4, this



Figure 7. Pareto solutions of KWMSS based on 250 m, 500 m and 1 km DEMs for all events. Note that panel (b) shows the enlarged objectivespace for 250 and 500 m spatial resolutions of Event 1, as illustrated in panel (a)

result also supports that a considerable number of param-eter combinations can yield acceptable model perfor-mance measures, such as NSC and PDR, in distributedrainfall-runoff modeling using KWMSS 250m.

ASSESSMENT OF STRUCTURAL UNCERTAINTYDUE TO THE SPATIAL RESOLUTION OF DEM

In the above sections, we dealt with the approach on thestructural uncertainty assessment through the comparisonof two different rainfall-runoff models, which have dif-ferent mathematical forms to describe the rainfall-runoffprocesses. The choice of spatial resolution has a consid-erable effect on the streamflow simulation in distributedrainfall-runoff modelling because the selection of eithercoarse or fine DEM resolution will result in significantchanges to the topographic runoff structure: contributingarea size, length of drainage network, slope, etc. Thesechanges to topographical parameters then influence therainfall-runoff modelling for surface or subsurface flows(Vieux, 2004).

In this section, we aim to analyse the extent of theimpact of topographical representation on both structuraluncertainty and its evaluation by applying the sameassessment methodology, introduced before, to threemodels with different spatial resolutions. As shown inFigure 1(b)–(d), 250, 500, and 500 m grid resolutionswere compared to assess how changes in topographicalparameters due to DEM size affected the estimates ofboth Pareto solutions and parameter stability.

Figure 7 illustrates the corresponding 500 Pareto solu-tions for each DEM for five historical flood events, esti-mated by MOSCEM-UA. KWMSS results based on finerDEMs (250 and 500 m) were remarkably improved withrespect to both objective functions (SLS and HMLE),compared with results based on the coarsest DEM (1 km).The results based on 250 and 500 m DEMs were nearlyPareto-equivalent, with similar and narrow Pareto fronts,and both results were Pareto-superior to KWMSS resultsbased on a 1 km DEM. The model based on a 500 mDEM was able to predict results with the same levelof accuracy as the model based on a 250 m DEM, andit had a structural stability very similar to (and some-times better than) results based on the finest resolution.For Event 3, it produced better results for both objec-tive functions. This finding enables modellers to selectan appropriate resolution size to ensure computationalefficiency and acceptable prediction accuracy. In otherwords, it is not necessary for KWMSS users to insiston more spatially detailed topographical data to obtainimproved predictions in distributed rainfall-runoff mod-elling. Therefore, even if a rainfall-runoff model is wellcalibrated by effective tuning programs, the DEM sizeis also a very important factor with regard to improvingmodel performance.

Another interesting finding is that the Pareto fronts ofthe 250 and 500 m DEMs were wider than the 1 km DEMin Event 5, while their optimal objective functions weresmaller than those of the coarsest DEM. It is probablethat the Pareto solutions converged prematurely because



(a) (b) (c)

Figure 8. Comparison of parameter stability according to DEM sizes; results of KWMSS based on (a) 250 m, (b) 500 m, and (c) 1 km DEMs

a 1-h time step in measured data is not sufficient tocalibrate the models based on a spatial resolution of lessthan 1 km. However, this coarse temporal resolution issuitable for the calibration of a distribution model basedon the coarsest spatial resolution of DEM, as shown inFigure 7(f).

The results of the comparison of parameter stability byDEM size indicate that a finer resolution results in betterparameter transposability from one event to another.As illustrated in Figure 8, each optimal parameter setof KWMSS 250m provided average stability values of0Ð9 NSC and 1Ð0 PDR for all applications. In addition,KWMSS based on a 500 m DEM (KWMSS 500m)produced the same level of parameter stability as a250 m DEM (0Ð89 NSC and 1Ð0 PDR). The modelbased on the coarsest resolution (1 km) produced worseparameter stability values (0Ð81 NSC and 0Ð98 PDR).However, all these distributed models had parameterstabilities superior to the conceptual lumped model, SFM,which produced stability values of 0Ð61 NSC and 0Ð84PDR. As illustrated by the comparison results of Paretosolutions between KWMSS 250m and KWMSS 500m,this comparative assessment of parameter stability alsosuggests that making models more complex in terms ofDEM spatial resolution does not necessarily provide moreaccurate and less uncertain model predictions. Indeed,the distributed models based on 250 and 500 m DEMswere nearly equivalent with regard to Pareto solutionsand parameter stability.

From the results regarding the interaction betweentopographical model complexity and structural uncer-tainty, it can be concluded that KWMSS 250m andKWMSS 500m are structurally sound in terms of thephysical reasoning of the catchment topography but thoseare likely to cause poor parameter identifiability (or equi-finality). On the other hand, SFM and KWMSS 1kmare spatially simplified models with a weak physicalreasoning but are relatively free from the equifinalityproblem. Lee et al. (2009) demonstrated that insufficient

topographical information of lumped models requireswell-identified model parameters while abundant topo-graphical information decreases the role of model param-eters such that various parameter combinations can leadto equivalent simulation results.

CONCLUSIONS

This study proposed a method for effectively recognis-ing and evaluating model structural uncertainty. It beganwith a comparative assessment of various model struc-tures that have differing features regarding the rainfall-runoff mechanism and DEM spatial resolution. Theassessment applied a multi-objective optimisation method(MOSCEM-UA) with two objective functions (SLS andthe HMLE), and focused on five historical flood events.The study was based on the assumption that a better(or less uncertain) model structure provides a balancedsimulation result with respect to several local hydro-graphical behaviours, regardless of objective functionor applied events. A structurally better model shouldproduce improved prediction results (either minimizedor maximized model performance measure), a smallerPareto solution set (constant model performance withregard to objective functions), and superior parameterstability (predictability of a calibrated model to variousevents). This study resulted in the following conclusions.

1) The distributed rainfall-runoff model, KWMSS 250m,performed better and produced smaller Pareto solu-tions than the lumped model, SFM.

2) The objective function had no significant effect onthe hydrograph simulations in KWMSS 250m appli-cations for five historical events, while SFM resultsbased on HMLE were underestimated compared toSFM results based on SLS. However, for small floodevents, SFM produced acceptable results, regardlessof the objective function, due to the small residualbetween the simulated and observed hydrographs.



3) Parameter stability was assessed by transferring thecalibrated models based on one event to another,and was quantified using two indices: NSC andPDR. The parameter combinations for each eventin KWMSS 250m indicated a good suitability forrainfall-runoff simulations of differing events, whilethe SFM produced a very poor parameter stability, soits optimal parameter sets for both objective functionswere unable to provide accurate model predictions.

4) The comparison of Pareto solution sets and parame-ter stability revealed that KWMSS 250m was superiorto SFM. However, KWMSS 250m did not producea unique parameter set to handle all of the objec-tive functions and flood events for distributed rainfall-runoff modelling. Instead, it produced many plausibleparameter sets, which yielded good model perfor-mance measures and even produced indistinguishablehydrographs.

5) Three different spatial resolutions (250 m, 500 m, and1 km) were used to assess the model structural uncer-tainty due to changes in topographical representation,according to the DEM size. A comparative assess-ment revealed that the results based on 250 and500 m DEMs were Pareto-equivalent, containing sim-ilar Pareto fronts and that both models were Pareto-superior to the model based on a 1 km DEM and SFM.Moreover, no remarkable differences appeared in theparameter stability between results based on the 250and 500 m DEMs, while spatially non-detailed models,SFM and KWMSS 1km, led to worse parameter sta-bility than both KWMSS 250m and KWMSS 500m.This finding indicates that modellers do not need touse a very fine spatial resolution of DEM in dis-tributed rainfall-runoff modelling using KWMSS whenthe computational efficiency for model simulation andcalibration is considered.

6) From the comparative assessment results, with respectto Pareto solution estimates and parameter stability,the model became structurally sound (less structurallyuncertain) as the topographical representation becamemore detailed, from SFM to KWMSS 250m. However,the increased model complexity accepted a number ofplausible parameter combinations; some parameter setscould lose their physical meaning while those providedacceptable model predictions.

Conclusions 4) and 6) state that due to uncertainparameter identification, multiple potential flow pathwayscan exist in distributed rainfall-runoff modelling usingKWMSS. As a consequence, the presence of alterna-tive solutions make it difficult to identify the optimal(or unique) model parameter set in distributed rainfall-runoff modelling. Therefore, it is essential to filter out thenon-physical or unreliable parameter sets from a numberof plausible parameter combinations by imposing addi-tional constraints on the model, calibrated using onlystreamflow data. Future research will develop an eval-uative criterion based on multiple hydrological variables,

which will be used to reduce the parameter uncertaintyin distributed rainfall-runoff modelling.

REFERENCES

Beven K, Binely AM. 1992. The future of distributed models: modelcalibration and uncertainty prediction, Hydrological Processes 6:279–298.

Boyle DP, Gupta HV, Sorooshian S. 2000. Toward improved calibrationof hydrologic models: Combining the strengths of manual andautomatic methods, Water Resources Research 36(12): 3663–3674.

Boyle DP, Gupta HV, Sorooshian S, Koren V, Zhang Z, Smith M. 2001.Toward improved streamflow forecast: The value of semi-distributedmodeling, Water Resources Research 37(11): 2739–2759.

Duan Q. 1991. A global optimization strategy for efficient and effectivecalibration of hydrologic models, Ph.D. dissertation, University ofArizona, Tucson.

Duan Q, Sorooshian S, Gupta VK. 1992. Effective and efficient globaloptimization for conceptual rainfall-runoff models, Water ResourcesResearch 28(4): 1015–1031.

Duan Q, Gupta VK, Sorooshian S. 1993. Shuffled complex evolutionapproach for effective and efficient global minimization, Journal ofOptimization Theory and Application 76(3): 501–521.

Gupta HV, Sorooshian S, Yapo PO. 1998. Toward improved calibrationof hydrologic models: Multiple and non-commensurable measures ofinformation, Water Resources Research 134(4): 751–763.

Gupta HV, Sorooshian S, Hogue TS, Boyle DP. 2003. Advances inautomatic calibration of watershed models. In Advances in calibrationof watershed models , Duan Q, Sorooshian S, Gupta HV, Rosseau A,Turcotte R (eds), AGU: Washington, DC; pp. 29–47.

Ichikawa Y, Shiiba M, Tachikawa Y, Takara K. 2000. Object-orientedhydrological modeling system, Proc. of 4th International ConferenceHydro- informatics 2000 CD-ROM, Iowa, USA.

Jakeman AJ, Hornberger GM. 1993. How much complexity is warrantedin a rainfall-runoff model, Water Resources Research 29(8):2637–2649.

Johnsen KP, Mengelkamp HG, Huneke S. 2005. Multi-objective calibra-tion of the land surface scheme TERRA/LM using LITFASS-2003 data,Hydrology and Earth System Sciences 9(6): 586–595.

Kavetski D, Franks SW, Kuczera G. 2003. Confronting input uncertaintyin environmental modeling. In Advances in calibration of watershedmodels , Duan Q, Sorooshian S, Gupta HV, Rosseau A, Turcotte R(eds), AGU: Washington, DC; pp. 49–68.

Kimura T. 1960. Research on the estimation of flood runoff: A study ofthe storage function, Journal of Research of PWRI 4: 23–43.

Klemes V. 1986. Operational testing of hydrological simulation models,Hydrological Sciences Journal 31(1): 13–24.

Kuczera G, Mroczkowski M. 1998. Assessment of hydrologic parameteruncertainty and the worth of multi-response data, Water ResourcesResearch 34: 1481–1489.

Lee G, Tachikawa Y, Takara K. 2007. Identification of model structuralstability through comparison of hydrologic models, Annual Journal ofHydraulic Engineering, JSCE , No. (51): 49–54.

Lee G, Tachikawa Y, Takara K. 2009. Interaction between topographicand process parameters due to the spatial resolution of DEMsin distributed rainfall-runoff modeling, Journal of HydrologicEngineering , DOI : org/10Ð1061/(ASCE)HE.1943–5584Ð0000098.

Refsgaard JC, Knudsen J. 1996. Operational validation and intercompari-son of different types of hydrologic models, Water Resources Research32(7): 2189–2202.

Savenije HHG. 2001. Equifinality, a blessing in disguise? HydrologicalProcesses 15: 2835–2838.

Schoups G, Hopmans JW, Young CA, Vrugt JA, Wallender WW. 2005.Multi-criteria optimization of a regional spatially-distributed subsurfacewater flow model, Journal of Hydrology 311: 20–48.

Sorooshian S, Dracup JA. 1980. Stochastic parameter estimationprocedures for hydrologic rainfall-runoff models: Correlated andheteroscedastic error cases, Water Resources Research 16(2): 430–442.

Sorooshian S, Gupta VK. 1995. Model calibration. In Computer modelsof watershed hydrology , Singh VP (ed), Water Resources Publications:Highland Ranch, USA; pp. 23–68.

Tachikawa Y, Nagatani G, Takara K. 2004. Development of stage-discharge relationship equation incorporating saturated–unsaturatedflow mechanism, Annual Journal of Hydraulic Engineering, JSCE , No.(48): 7–12 (in Japanese).



Takasao T, Shiiba M, Ichikawa Y. 1996. A runoff simulation withstructural hydrological modeling system, Journal of Hydroscience andHydraulic Engineering, JSCE , 14(2): 47–55.

Uhlenbrook S, Seibert J, Leibundgut C, Rohde A. 1999. Predictionuncertainty of conceptual rainfall-runoff models caused by problemsin identifying model parameters and structures, Hydrological SciencesBulletin 44(5): 779–797.

Vieux BE. 2004. Distributed Hydrologic Modeling using GIS , KluwerAcademic Publisher: Dordrecht; pp. 289.

Vrugt JA, Gupta HV, Bouten W, Sorooshian S. 2003. Effective and effi-cient algorithm for multi-objective optimization of hydrologic models,Water Resources Research 39(8): 1214, DOI:10Ð1029/2002WR001746.

Wagener T, Boyle DP, Less MJ, Wheater HS, Gupta HV, Sorooshian S.2001. A framework for development and application of hydrologicalmodels, Hydrology and Earth System Sciences 5(1): 13–26.

Wagener T, Wheater TS, Gupta HV. 2004. Rainfall-Runoff Modeling inGauged and Ungauged Catchments , Imperial College Press: London;pp. 306.

Wagener T, Gupta HV. 2005. Model identification for hydrologicalforecasting under uncertainty, Stochastic Environmental Research andRisk Assessment 19: 378–387.

Yapo PO, Gupta HV, Sorooshian S. 1998. Multi-objective globaloptimisation for hydrologic models, Journal of Hydrology 204: 83–97.


Comparison of model structural uncertainty using a multi-objective optimisation method

Documents

Transcript of Comparison of model structural uncertainty using a multi-objective optimisation method