Understanding the sources of uncertainty to reducethe risks of undesirable outcomes in large-scalefreshwater ecosystem restoration projects:An example from the Murray–Darling Basin,Australia

R.H. Bark a,*, L.J.M. Peeters b, R.E. Lester c, C.A. Pollino d, N.D. Crossman e,J.M. Kandulu e

aCSIRO Ecosystem Sciences, 41 Boggo Road, Dutton Park, QLD 4102, AustraliabCSIRO Land & Water, PMB 2, Glen Osmond, SA 5064, AustraliacSchool of Life and Environmental Sciences, Deakin University, P.O. Box 423, Warrnambool, VIC 3280, AustraliadCSIRO Land & Water, GPO Box 1666, Canberra, ACT 2601, AustraliaeCSIRO Ecosystem Sciences, PMB 2, Glen Osmond, SA 5064, Australia

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8

a r t i c l e i n f o

Article history:

Received 2 January 2013

Received in revised form

29 April 2013

Accepted 29 April 2013

Published on line

Keywords:

Ecosystem restoration

Uncertainty

Risk analysis

Murray–Darling

a b s t r a c t

There are a growing number of large-scale freshwater ecological restoration projects world-

wide. Assessments of the benefits and costs of restoration often exclude an analysis of

uncertainty in the modelled outcomes. To address this shortcoming we explicitly model the

uncertainties associated with measures of ecosystem health in the estuary of the Murray–

Darling Basin, Australia and how those measures may change with the implementation of a

Basin-wide Plan to recover water to improve ecosystem health. Specifically, we compare two

metrics – one simple and one more complex – to manage end-of-system flow requirements

for one ecosystem asset in the Basin, the internationally important Coorong saline wet-

lands. Our risk assessment confirms that the ecological conditions in the Coorong are likely

to improve with implementation of the Basin Plan; however, there are risks of a Type III error

(where the correct answer is found for the wrong question) associated with using the simple

metric for adaptive management.

Crown Copyright # 2013 Published by Elsevier Ltd. All rights

reserved.

Available online at www.sciencedirect.com

journal homepage: www.elsevier.com/locate/envsci

1. Introduction

There is an imperative that large-scale ecosystem restoration

projects be based on the best available science because they are

costly and involve potential trade-offs between the environ-

ment and the economy (Doyle and Drew, 2008; Lee, 1993).

However there are many challenges in using best science in

large complex restoration projects. These challenges include

* Corresponding author. Tel.: +61 7 3833 5678.E-mail addresses: Rosalind.bark@csiro.au, rh.bark@gmail.com (R.H

1462-9011/$ – see front matter. Crown Copyright # 2013 Published bhttp://dx.doi.org/10.1016/j.envsci.2013.04.010

not only the integration of peer-reviewed science into decision-

making and making decisions in the face of scientific

uncertainty (Doody et al., 2012) and limited and/or emerging

scientific understanding (Doyle and Drew, 2008; Harris and

Heathwaite, 2012), but also how to provide information to

‘‘decision-makers in ways that foster good decisions that

increase the likelihood of attaining desired outcomes’’ (Pielke,

2007, p. 30). There are multiple sources of uncertainty (see

Ascough et al., 2008 for a review) in achieving outcomes in

. Bark).

y Elsevier Ltd. All rights reserved.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 898

complex coupled natural and human systems (Liu et al., 2007)

but long-term restoration projects provide opportunities for

adaptive management approaches that involve experimenting,

learning and reducing uncertainties (Lee, 1993; Doyle and Drew,

2008).

The goal of this paper is to provide a clearer understanding

of the types of uncertainty and the sources of risk related to

achieving the outcomes (i.e. benefits) of a large ecological

restoration project. Arguably the biggest restoration project in

Australia is the AU $10.7 billion investment by the Australian

Government to restore the health of freshwater ecosystems in

Australia’s largest river basin, the Murray–Darling.1 A Murray–

Darling Basin Plan, outlining how that investment should be

made, including multiple strategies such as water buy-backs

from irrigators and investment in irrigation efficiency, was

adopted by the Federal Water Minister in late 2012. We focus

on the risks inherent in the Murray–Darling Basin Plan,

drawing on material from a recently-completed study that

quantified the ecosystem service benefits of returning water to

the environment under the proposed Basin Plan (CSIRO, 2012).

‘‘Newig et al. (2005) discuss two types of uncertainty that

affect decision making: normative uncertainty – or decision

uncertainty (doubt as to what to do and how to do it) – and

informational uncertainty (limited knowledge or factual

uncertainty)’’ (Harris and Heathwaite, 2012, p. 93). What they

term informational uncertainty can be further classified as

either statistical or systematic (Daneshkhah, 2004). Statistical

uncertainty results from natural variation in a system which,

based on observations, can be quantified through probability

distributions. In turn these can be used to derive confidence

intervals and risk profiles (Hayes, 2011; Kandulu et al., 2012). A

relevant example of statistical uncertainty, in the context of

restoration of ecosystems, is the variability in the hydro-

climatology of the basin (Peel et al., 2004; Kirby et al., 2006).

Hydrological and ecological response models used to project

outcomes from changes in water resource management are

also subject to this type of uncertainty.

In hydrological models used to assess effects of changes in

water resources management, the uncertainty associated with

input data, such as precipitation, and observed output data, such

as streamflow observations, is often classified as statistical

uncertainty while the model structure and parameters are

considered a source of systematic uncertainty. The input and

observed data should however also be considered as systematic

uncertainty as there are seldom sufficient independent data to

characterise the probability distribution of these quantities

(Kavetskietal.,2006;Shaoetal.,2012).Thesystematicuncertainty

from model structure and parameters arises because it is often

not possible, based on the available data, to infer and conceptu-

alise all processes relevant to the model prediction at the given

spatial and temporal scale (Gupta et al., 2012).

More fully, systematic uncertainty results from: (i) a lack

of data or knowledge about a system; (ii) the inherent

1 Comprising AU $3.1 bn under the Restoring the Balance Pro-gramme for water buybacks, AU $5.8 billion under the SustainableRural Water Use and Infrastructure Programme for irrigation effi-ciency projects most of which will be in the Basin, and an addi-tional AU $1.77 bn announced in October 2012 for additionalinfrastructure investments.

complexities of the system and; (iii) interactions with other

systems such as between human and natural systems. The

inherent complexities stem from, among others: stochastic

parameters that are difficult to estimate, non-linearities, non-

stationarities, and unknown conditioning information (Lo and

Mueller, 2010). Lo and Mueller (2010) describe a spectrum of

systematic uncertainty from fully reducible or partially reduc-

ible to irreducible uncertainty. Systematic uncertainty cannot

be quantified, but new knowledge or data can, in some

instances, reduce this type of uncertainty to statistical

uncertainty. In ecohydrological models sources of systemic

uncertainty include non-stationarity of the catchment, tempo-

ral variability of system characteristics, representation of

processes in models and lack of commensurability in observed

and predicted variables because of scale issues or difference in

meaning (Beven and Alcock, 2012; Harris and Heathwaite, 2012).

Systematic uncertainty is prevalent in modelling ecological

response to policy-contingent water resources management

change. This uncertainty derives from incomplete knowledge

of water requirements for ecological systems, the complexi-

ties, feedbacks and nested nature of diverse ecosystems in the

basin, the potential for non-linearities of ecological response

to changed hydrologic conditions, the non-hydrologic system

influences, the time-scale of ecological recovery (Lee, 1993)

and associated difficulty in measuring changes in ecological

condition due to time lags and measurement effort. In

addition there may be instances where the absence of

spatially-explicit, workable ecological response models is a

constraint on any predictive modelling of policy-contingent

change (CSIRO, 2012) or where available models provide little

information on how to simultaneously achieve multiple

ecological objectives. To estimate the value of water resources

management change there is an added level of uncertainty

stemming from the mismatch between the complexity of

ecological systems and the ecosystem service framework to

monetarily value benefits (Norgaard, 2010) as well as from the

monetary valuation methods.

The link between uncertainty and outcomes is risk.

There are many definitions of risk, for instance the ISO

31000 risk management standard (IEC, 2009) describes risk

as the effect of uncertainty on objectives. The complexity of

the nested hierarchy of uncertainties means that, although

decision-makers might benefit from the provision of

information on the uncertainties intrinsic in modelled

assessment of policy-contingent changes (Bateman et al.,

2011) such risk assessments are rarely undertaken. This

means that the decision-maker has little information on

what objectives are likely to be met with a high probability

of confidence and which ones are less certain outcomes. We

argue that it is important for decision-makers to understand

uncertainty typologies and how to approach each in a risk

management framework.

In this paper we consider both sides of the river restoration

problem identified by Harris and Heathwaite (2012) uncertain-

ty on the biophysical side and decision-making uncertainty on

the policy side. We begin with background to the Murray–

Darling Basin Plan (Section 2). Section 3 provides detail on the

method and results for a worked example using a simplistic

metric and a more complex ecological model. The risk analysis

highlights that, for some outcomes, uncertainties are likely

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8 99

reducible (Lo and Mueller, 2010) whereas for others the

complexity of the system and the interconnections with other

systems (Liu et al., 2007) means that outcomes are less certain.

In the discussion section we address whether such ex ante

programme valuations are sufficient to guide decision-making

and how they might be improved. We find that the risk

framework provides a systematic means of prioritising efforts

for reducing uncertainty to increase the likelihood of meeting

key objectives. The final conclusions section puts lessons

learned into a more generalised context.

2. Background: policy-contingent change, theMurray–Darling Basin Plan

In response to widespread long-term decline in riverine,

floodplain, wetland and estuarine ecosystems and worsening

ecological degradation in the Murray–Darling Basin (see Fig. 1)

during the severe millennium drought (1997–2009) the

Australian Government passed the Water Act 2007 (Common-

wealth). An objective of the Water Act was to ‘‘protect, restore

and provide for the ecological values and ecosystem services

of the Murray–Darling Basin’’ (Water Act 2007, 3 Objects (d)(ii)).

The Act mandates that the Murray–Darling Basin Authority

(MDBA), a federal agency, prepare a Basin Plan to recover

ecosystem functions and to balance the multiple needs in the

Basin. Underpinning the Basin Plan (MDBA, 2011a), the MDBA

determined the environmentally sustainable level of take

(ESLT) for key ecosystem assets and functions to guide basin-

scale water resource management (MDBA, 2011b). Comple-

menting the ESLT was the establishment of a Sustainable

Diversion Limit (SDL) that capped the amount of water that

could be diverted for consumptive use. The current SDL from

the Basin Plan is 2750 GL below current diversion levels

meaning that this volume can be returned to the environment.

An additional 450 GL/year will be made available as part of the

Water Recovery Act, which is separate from the SDL and the

Basin Plan.

The recovery of this water for the environment will be

through two mechanisms. The Australian Government has set

aside AU $3.1 billion to permanently purchase water access

licenses from willing sellers for the environment and a further

AU $5.0 billion to upgrade inefficient irrigation infrastructure

in the basin, with conserved water shared among irrigators

and the environment. An additional $1.77 billion was

announced in November 2012 that can be used to purchase

entitlements, invest in conservation or to reduce system

constraints on environmental watering events, e.g. raising

levees to protect towns from flooding. Once the Australian

Government has taken ownership of the water for the

environment, an Environmental Watering Plan (EWP) will

inform how and where that water is delivered to the

environment to meet the objectives of the Basin Plan.

The proposed Basin Plan released in December 2011 lays

out the management objectives and outcomes to be achieved

(MDBA, 2011a, Chapter 5). They include: protecting and

restoring water-dependent ecosystems in the Basin; ensuring

that these ecosystems are resilient to risks and threats; using

water resources to give effect to international agreements

(listed under Water Act 2007, §4 relevant international

agreement); and to optimise economic and social outcomes

and improve the water security of all uses in the Basin. The

proposed Basin Plan also recognises that there are risks to

achieving these objectives including insufficient water quan-

tity for the environment and/or unsuitable water quality

which may result in poor health of the Basin’s water-

dependent ecosystems. To address these risks the MDBA

has five policy levers above and beyond the SDL: the EWP; a

water quality and salinity management plan; water trading

rules; approval authority over water resource planning by

states (Garrick et al., 2012); and the opportunity to invest in

improved measuring, monitoring and research to improve

knowledge of environmental water requirements, impacts of

land use change including to conservation uses (Smith et al.,

2012), and climate change. This knowledge component is a

clear signal that the MDBA will invest resources to reduce

systematic uncertainty.

A big risk in implementation of the Basin Plan is it may not

meet the objectives desired by the various stakeholders

because of statistical and systematic uncertainty. Assessing

the risk of not meeting the objectives in the Basin Plan requires

an understanding of nested statistical and systematic uncer-

tainty in the hydro-climatology of the basin and the

hydrological, ecological response and socioeconomic valua-

tion modelling underpinning the Basin Plan as well as how the

development of water sharing plans and the EWP can address

these uncertainties. The presence of uncertainty at each stage

does not invalidate assessment of policy-contingent change

under the Basin Plan (CSIRO, 2012): rather by unpacking the

type of uncertainty and whether the magnitude is large

enough to change a decision, it is possible to see where for

instance, further investment in data collection or model

development might reduce uncertainty and where sufficient

information, in spite of uncertainty, already exists to enable a

decision to be taken. While, the final EWP that will guide

management and use of recovered environmental water had

not, at the time of writing, been developed by the Australian

Government, a series of discussion papers point to methodo-

logical frameworks to prioritise watering events, measure and

monitor the ecological responses and other outcomes, and

provide greater flexibility to the Australian Government

(DEWHA, 2009; DSEWPaC, 2011; CEW, 2011).

3. Method and results

This section draws on results from the coupled natural and

human system assessment (CSIRO, 2012). CSIRO (2012)

assessed the ecosystem service benefits of setting an SDL

and delivering recovered water to the environment under a

scenario very similar to the proposed Basin Plan (an SDL of

2800 GL/year). The incremental ecological and ecosystem

service benefits of two scenarios – a baseline scenario (a

simulated 114-year run that preserves hydro-climatologically

variability and current water-resource infrastructure) and a

2800 GL scenario (a simulated 114-year run with the policy-

contingent change, i.e. a further 2800 GL/year additional

environmental water) – were modelled and where possible

valued in monetary terms. The study concluded that returning

2800 GL/year to the environment would halt the decline (and

Fig. 1 – The Coorong, Lake Alexandrina and Lake Albert, Murray–Darling Basin, Australia.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8100

in some case improve the condition) of ecosystems on the

lower floodplain and wetlands. Yet it is the detail of the type,

magnitude, location, and uncertainties of achieving these

incremental benefits, that is paramount to any risk manage-

ment process. An assessment of risk may assist decision-

makers’ understanding of the level of confidence in, and the

likelihood of, achieving multiple benefits under the Basin Plan.

Understanding uncertainties prior to the finalisation of the

EWP and eventual implementation of the restoration policy

may also identify ways to manage risk and achieve stated

objectives with more confidence.

To demonstrate how information on uncertainty affects

the achievement of good ecosystem health, we investigate

modelled outcomes for one key environmental asset in the

Basin, the Coorong saline wetlands. The Coorong, Lake

Alexandrina and Lake Albert were listed under the Convention

on Wetlands of International Importance, called the Ramsar

Convention (Ramsar, 1971) in 1985. This complex of freshwa-

ter, estuarine and hypersaline lakes and lagoons provides a

mosaic of habitats for international migratory wading birds

and waterfowl many of them listed for protection under

international treaties. Preserving ecosystem health of Ramsar

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8 101

sites is a stated rationale behind the Water Act 2007 and the

Basin Plan.

The Coorong Ramsar site is vulnerable: during the millenni-

um drought, insufficient freshwater reached the sea and the

Mouth of the Murray silted up, thus closing the natural

connection between freshwater, estuarine and marine ecosys-

tems in the Coorong lagoon system (Webster, 2010). Freshwater

flows ceased completely for a period of more than three years

(March 2007–October 2010) which was unprecedented in

recorded history for the River Murray (Webster, 2010). A

consequence of the lack of freshwater flows was the develop-

ment of extreme hypersaline conditions (i.e. water went from

twice the salinity of seawater to seven times the salinity of

seawater) in the South Lagoon of the Coorong and a loss of

estuarine conditions in the North Lagoon, closer to the Murray

Mouth (Kingsford et al., 2011). The South Australian State

Government initiated an emergency dredging programme in

2002 to re-establish connectivity between the Coorong and the

Southern Ocean to mitigate the extreme hypersaline conditions

(Kingsford et al., 2011). Estuarine conditions could not be

restored without a return of freshwater flows from the River

Murray which later occurred in 2010.

Hashimoto et al. (1982) described three aspects to system

performance that ideally could be used to assess the outcomes

against stated objectives. The three aspects are: (i) how often

the system fails (reliability); (ii) how quickly the system

returns to a satisfactory state once a failure has occurred

(resiliency, Folke et al., 2004), and; (iii) how significant the

likely consequences of failure may be (vulnerability). Our risk

Fig. 2 – Monthly time series of Mouth Opening Index (MOI) for t

yearly average MOI for baseline (c) and 2800 GL (d) scenario. Ho

assessment addresses the first of these tests, reliability,

explicitly and we discuss the other two. Our proxy for

reliability is to consider two metrics, the risk of Murray Mouth

closure and the risk of degraded ecological conditions in the

Coorong. These two examples were chosen because they are

representative of a management problem: to manage against a

simple metric or against a more complex metric of ecosystem

health.

3.1. Risk of Murray Mouth closure

A risk analysis requires the choosing of a threshold based on

an objective. The first example is the risk of Murray Mouth

closure. This is a proxy for sufficient end-of-system water to

support downstream ecosystems and incidentally to avoid

costly mitigation (dredging) expenses (Kingsford et al., 2011;

CSIRO, 2012; Banerjee et al., 2013).

Several indicators for Murray Mouth closure are available

(Webster, 2010). In this study the Mouth Opening Index (MOI)

from Walker and Jessup (1992) is used. This index is an

empirical relationship based on the tidal energy and flow over

the barrages blocking the Murray Mouth. The monthly time

series of MOI simulated for both the baseline scenario and

2800 GL scenario are presented in Fig. 2a and b, respectively.

The indicator for Murray Mouth closure is defined as the

percentage of years for which the minimum MOI in that year is

less than 0.05, with a hydrological year defined from October to

September. Fig. 2c and d shows the minimum MOI per

hydrological year for the baseline and 2800 GL scenario. Under

he baseline (a) and 2800 GL (b) scenario and time series of

rizontal line in all plots indicates MOI threshold of 0.05.

Fig. 3 – Autocorrelation of monthly baseline MOI time series.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8102

the baseline scenario 22 years out of 114 have a yearly

minimum MOI less than 0.05, while this is only the case for 4

years under the 2800 GL scenario.

A traditional uncertainty analysis on the model results is

not possible. Even for the baseline scenario, the hydrodynamic

model evaluates a combination of current management with

an historical climate. Therefore the model results cannot be

directly compared to observations such as river flow as the

observed flow rates correspond to historical management

practices. To gain some insight however in the variability of

the indicators, block bootstrapping is applied to the time series

of the baseline scenario and 2800 GL scenario. Each time series

is resampled 10,000 times in blocks of 1.5 years to account for

temporal correlation in the resampled time series. This

sampling length is based on the autocorrelation structure of

the baseline MOI time series (Fig. 3). Fig. 3 clearly illustrates a

minimum in autocorrelation at 1.5 years, justifying the use of

that length of time in the block bootstrapping procedure.

Fig. 4a and b shows a histogram and empirical cumulative

distribution of the block-bootstrapping results for the number

of years with a minimum MOI less than 0.05 for both scenarios.

It is clear that the 2800 GL scenario results in a considerable

decrease in the number of years with a risk of mouth closure.

This is confirmed in Fig. 4c and d which shows the histogram

and empirical cumulative distribution of the reduction in

years with minimal MOI less than 0.05 between both

scenarios.

As a result of past closures of the Murray Mouth, the South

Australian government invested in dredging of the Murray

Mouth in 1981 and in the period 2002–2010. The total

investment for the last dredging period amounted to AU $32

million (Kingsford et al., 2011). While it is not feasible to derive

a yearly dredging cost from this total amount, it is clear that

the drastic reduction in mouth closure risk under the 2800 GL

scenario will result in considerable savings in dredging costs,

irrespective of the ecological benefits of maintaining a

connection to the ocean. In a more simplistic analysis CSIRO

(2012) estimated these savings as almost AU $18 million.

3.2. Risk of not achieving good habitat condition

The second example assesses the risk of failing to meet good

habitat condition thresholds in the Coorong tidal lagoon

ecosystem; conditions that are highly valued by the Australian

public (Hatton MacDonald et al., 2011). This example uses a

more complex ecological response model; it assesses the

uncertainties in maintaining healthy ecosystem states of the

Coorong lagoons using an ecological response model that is

based on suites of co-occurring biota and the physicochemical

conditions under which they are observed (see Lester and

Fairweather, 2011 for a description of how the ecosystem

states are calculated). ‘Healthy’ ecosystem states are defined

by those that are simulated to have occurred at the time of

Ramsar listing of the Coorong. Current environmental targets

for the region adopted by the South Australian Government

are based on the salinity targets for the region, but were also

assessed for their ability to avoid the occurrence of degraded

ecosystem states in the Coorong (Lester et al., 2011).

The ecosystem state model simulates the condition of 14

sites in the Coorong. Eight different states are simulated,

ranging from estuarine to hypersaline. Based on the condi-

tions simulated for the 14 sites, the overall condition of the

Coorong can be expressed as the number of years in the

simulation period the Coorong has healthy conditions

(estuarine/marine, healthy hypersaline and average hypersa-

line) and the number of years with degraded conditions. For

the baseline scenario and 2800 GL scenario, bootstrapping was

applied to the time series of conditions. Here, 10,000 samples

were generated with block bootstrapping with a sampling

length of 2 years to retain the autocorrelation in the time

Fig. 4 – Histogram (a) and empirical cumulative distribution (b) of number of years with minimum MOI below 0.05 threshold

for the baseline and 2800 GL scenario and histogram (c) and empirical cumulative distribution plot (d) for reduction in years

with minimum MOI less than 0.05 between both scenarios.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8 103

series. For each bootstrap sample, the proportion of years with

healthy (Fig. 5a–c) and degraded (Fig. 5d–f) ecosystem states

were computed.

For the baseline scenario it was very unlikely that healthy

conditions in the Coorong were achieved in more than 95

percent of years, which corresponds to current targets for the

region. For the 2800 scenario in contrast, it was very likely that

conditions were healthy more than 90 percent of the time.

However, even in this scenario, the probability of having

healthy conditions more than 97 percent of the time was very

small.

Degraded ecosystem states are defined as those occurring

when there has been no freshwater flow from the River Murray

to the Coorong for a period of 339 days. Such an occurrence

would have been likely less than 1% of the time in the absence

of water-resources development in the Basin, and the

documented hydrologic and ecological impacts of such dry

conditions were severe (Lester and Fairweather, 2011). Under

these conditions, salinities reached more than 200 g/L in the

South Lagoon, estuarine conditions disappeared completely,

and Coorong biota contracted in range towards the north, or

disappeared from the system (Kingsford et al., 2011). These

conditions were simulated to occur occasionally under the

baseline scenario shown in Fig. 5. Under the baseline scenario,

it would be highly probable to have degraded conditions more

than 2 percent of the time.

In contrast under the 2800 GL scenario it is highly unlikely

to have degraded conditions more than 2 percent of the time

and there is a small but distinct probability of having no

degraded conditions in the Coorong. Eliminating long periods

Fig. 5 – Histograms and empirical cumulative distribution plots of healthy conditions (a and b) and degraded conditions (c

and d) for the baseline and 2800 GL scenarios.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8104

of low or no freshwater flows to the Coorong lagoons is highly

likely to avert the degraded conditions described above and

improve the resilience of Coorong ecosystems, thus effectively

mitigating the risk of ecological degradation and meeting the

provisions of the Water Act 2007. This would similarly avert the

need for intensive management intervention such as that

described by Kingsford et al. (2011).

The policy implications of the risk of not achieving good

habitat condition are more complex than the risk of Murray

Mouth closure reported in the previous section. The analysis

shows that the hydrologic conditions that result in degraded

ecosystem states in the Coorong are reduced under the 2800

GL scenario but that this reduction is small. However even

though the likelihood of degradation is small the ecological

(resilience) and economic benefits are likely high (CSIRO, 2012)

due to the severity of the impact when such conditions do

occur.

4. Discussion

The South Australian Government spent in excess of AU $2

billion mitigating the human (e.g. insufficient water quality for

irrigation) and ecological costs (e.g. capture and maintenance of

threatened fish as a captive population until conditions were

suitable for their return to the system) associated with degraded

Coorong habitat conditions (Kingsford et al., 2011). Conditions

such as these were unprecedented in recorded history for the

Coorong and the long-term ecological recovery of the system

remains uncertain. To date, no comprehensive analysis has

been undertaken to demonstrate the ability of the system to

recover from this disturbance fully (what Hashimoto et al., 1982

describe as resilience) and it is likely that active restoration

interventions (e.g. translocation of aquatic macrophytes) may

be necessary in the region.

Table 1 – Type I and Type II errors.

H0 true H0 false

Reject H0 Type I error

(false positive)

Correct outcome

(true positive)

Fail to reject H0 Correct outcome

(true negative)

Type II error

(false negative)

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8 105

The concepts of reliability, resilience and vulnerability are

implicitly important in a policy context where decision-

makers are concerned to manage the risks on delivering on

beneficial outcomes. To frame this discussion we utilise the

statistical concepts of Type I and Type II (and later Type III)

errors. An example is used to illustrate these errors and the

real life consequences of the different error types. A Type I

error is when a true null hypothesis (H0, a default condition

that is tested) is rejected and a Type II error is when a false null

hypothesis fails to be rejected. Scientists often focus on Type I

errors but the risks of Type II errors are particularly relevant in

a policy context (Lee, 1993). Table 1 identifies four possible

states.

4.1. Suppose, H0 = there is sufficient water set aside underthe Basin Plan to restore water-dependent ecosystems of theMurray–Darling Basin

A result that returns a false positive (Type I error), i.e. it is

assumed there is insufficient environmental water when

there is actually sufficient water, would result in unneces-

sary alarm about the likely outcomes under the Basin Plan

and perhaps increased pressure for resourcing with

associated opportunity costs of alternative government

expenditures. Using the examples above, modelling may

show that there is likely to be insufficient water to keep

the desired level of connectivity via the Murray Mouth and

to support healthy habitat condition, when this is not

actually the case. Managers may then respond by dredging

unnecessarily, or by providing additional environmental

flows to the region, which may divert needed water from

other environmental assets. Alternatively, a false negative

(Type II error), i.e. it is not true that there is sufficient water

and this false assumption fails to be rejected, would result

in the illusion that the Basin Plan will provide conditions

for good ecosystem health while in reality the Basin Plan

will not deliver on this key objective and a policy

opportunity to reduce the risk of ecosystem collapse is

missed.

While statistical theory is well equipped to deal with the

Type I and Type II errors, it is still hard to find a framework in

which to evaluate Type III errors, ‘‘finding the correct answer

for the wrong question’’ (Kimball, 1957). In hydrology and

ecology considerable uncertainty is present in model out-

comes as the models are based on an incomplete and ever-

evolving conceptual understanding of the system dynamics

(Beven and Alcock, 2012; Gupta et al., 2012; Harris and

Heathwaite, 2012). Bredehoeft (2005) provides a number of

examples from hydrogeological modelling practice, where the

predictions of state-of-the-art groundwater models have

proven to be incorrect due to flawed or incomplete conceptual

models. Landis et al. (2013) cite Type III error as an important

factor in the ecological risk assessment in the context of

climate change.

Indicators and thresholds used in management are also the

result of the often-limited understanding of the system and

the risk exists that even if the desired threshold or indicator

value is achieved, the corresponding system state is not. This

would imply that even in the case that the management

objective meets preset environmental water demands, the

delivery of this water might not be sufficient to restore the

environmental health of the system.

We illustrate Type III error using the Murray Mouth opening

index. The river management model is designed to address the

H0 that the proposed management scenario will result in a MOI

equal or greater than 0.05. By adhering to correct statistical

procedures and modelling protocols the risks of Type I and II

errors can be minimised, so that the null hypothesis can either

be correctly rejected or accepted. The H0 however does not

directly address the key objective, namely a healthy ecological

condition for the Coorong. A Type III error would consist of a

management scenario that would maintain a MOI of 0.05 or

more, while failing to achieve healthy habitat conditions. The

Murray Mouth is a very dynamic system that naturally has a

seasonal cycle of openness and can almost close in many

years. This is generally not a problem for the health of Coorong

lagoons if it then opens again with higher seasonal flows in the

next year. However, using a simplistic metric can mask this

variability and so if water managers actively strive to keep the

mouth at a constant depth and level of openness (e.g. by

delivering 2 GL/day everyday) they may inadvertently be

creating ecological degradation in a system that is not adapted

to constant conditions.

The collapse of the northern cod off Newfoundland is partly

attributable to such error (Walters and Maguire, 1996); the

catch per effort was seen as an adequate metric for cod

abundance and fishery management was partly based on this

metric. The catch per effort however proved to be a gross

overestimate of fish abundance. Other examples can be found

in failed attempts of biological pest control in Australia, such

as the introduction of cane toad to control canegrubs

(Robertson et al., 1995) and the introduction of gambusia to

control mosquitoes (Griffin and Knight, 2012). In both

instances the basic premise is correct; the exotic species

predates the pest. The introduction however fails to address

the underlying objective of reducing the pest populations and,

especially in these cases, results in declining health of

ecosystems.

The risk profile of the government and their constituents

may provide guidance on: (1) which type of error is most

egregious; and (2) how to minimise the consequences of

errors. In determining which type of error to minimise, the

electorate may, for example, have strong preferences for

sustainability and for following a precautionary approach

where large systemic risks are present (Ciriacy-Wantrup, 1952;

Bishop, 1978; Harris and Heathwaite, 2012) and, in fact,

Australia has signed onto the notions of the precautionary

principle in environmental management under the National

Strategy for Ecologically Sustainable Development (1992). The

consequences of errors can be minimised through invest-

ments in more effective monitoring of incremental change in

ecosystem responses, integrative measures of river health,

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8106

complexity indicators and indicators of ecological progress for

adaptive management (Harris and Heathwaite, 2012). Other

strategies include setting safe minimum standards (Bateman

et al., 2011), and the use of probabilistic forecasting coupled

with simple economic models (Verkade and Werner, 2011).

Furthermore, the EWP will be implemented separately each

year and, although Pielke reminds us that ‘‘a continuous

decision process is more complicated than a one-time,

discrete choice’’ (2007: 25), this strategy provides opportunities

for adaptive learning (Lee, 1993; Harris and Heathwaite, 2012).

In a multi-dimensional space it can be easy to lose sight of

the fact some things are more important than others to the

application of a restoration strategy, i.e. all risks are not equal

and many are not critical in policy trade-offs. This might

involve prioritising management focus (Harris and

Heathwaite, 2012) for instance on those natural systems that

are historically abundant but are now quite rare, or the degree

and effort (and cost) of restoring highly-disturbed sites

compared to maintaining or improving minimally-disturbed

sites (Palik, 2000). Newig et al. (2005) recommend participatory

approaches as a way to mediate interests and goals to: manage

normative uncertainty, improve acceptance, and improve

implementation outcomes. Participatory approaches that

incorporate local knowledge may also help to make better

informed decisions. Meanwhile, Harris and Heathwaite (2012)

discuss opportunity to integrate land and water management

for ecological outcomes and Doody et al. (2012) recommend

appraisal of the structures and procedures of the implemen-

tation agent to identify likely failings and address these early

in the implementation phase.

5. Conclusions

Is uncertainty analysis useful for decision-makers or does

acknowledging uncertainties undermine the Basin Plan? It is

true that the science community does not know everything,

nor does it need to (post-normal science; Funtowicz and

Ravetz, 1993); and there is nested uncertainty from hydro-

climatology through hydrological models through ecological

response models and the monetary valuation of benefits. As a

policy process is necessarily one of negotiation and dialogues,

science encompasses the legitimacy of multiple perspectives,

commitments and is a process of discourse and demonstra-

tion through evidence (Funtowicz and Ravetz, 1993). Manage-

ment rules can be designed, monitored and amended as more

information on the water requirements and regimes of water-

dependent ecosystems and on the success of delivery options

becomes available. It is through learning and adaptive

processes that we allow a fresh and progressive approach to

risk-based water resource management.

The benefits of a risk-based approach include the possibili-

ty for more robust planning and therefore better-defined

planning objectives that are more likely to be achieved.

Providing decision-makers and the public with easily inter-

pretable information on uncertainty provides an opportunity

for more fruitful discussion of policy objectives and ways to

achieve good outcomes and to manage expectations. Assess-

ment that includes risk can provide information on the risk

profile you are currently operating under and perhaps identify

management options to get you to the risk profile you wish to

have. The risk assessment provides guidance for those things

that can be managed adaptively and those that are likely

dependent on the actual sequencing of wet and dry years (e.g.

the need to piggy-back on flood events to achieve watering of

the high floodplain) or those objectives that are beyond the

scope of water resource managers without collaboration and

integration with other policy instruments, e.g. native fish

recovery and the need for complementary land management

and investment in fish ladders.

In some cases more water, water managed more efficiently,

or more sensitively managed to achieve co-benefits, i.e.

cultural benefits for indigenous communities in the Basin,

will enable better outcomes or outcomes to be achieved with

greater certainty. In this instance, if sufficient water cannot be

delivered because of flood risk to private property then a

creative solution could comprise the purchase of flood

easements or building up levees around regional towns or

irrigation districts. The additional funding announced in

November 2012 provides opportunity for this type of invest-

ment. In yet other cases, outcomes might be better reached

with a paradigm shift in thinking, for instance a focus on

achieving the desired outcomes opens the solution set,

illuminates the nature of the risks to water-dependent

activities, highlights the long-term nature of the recovery

project and the benefits of a more resilient whole-of-system.

This paradigm shift is straightforward to implement in an

uncertainty analysis/risk management framework. Rather

than focusing on the probability of an event happening, the

same simulations can be used to express the most likely

system state given a scenario of boundary conditions, such as

climate and management strategy.

Acknowledgements

The authors would like to thank The Water for a Healthy

Country Flagship, CSIRO. Darran King provided early insight

into uncertainties in the Basin Planning process. Thanks also

to Martin Nolan for creating Fig. 1.

r e f e r e n c e s

Ascough I.I., J.C., Maier, H.R., Ravalico, J.K., Strudley, M.W., 2008.Future research challenges for incorporation of uncertaintyin environmental and ecological decision-making. EcologicalModelling 219 (3–4) 383–399.

Banerjee, O., Bark, R., Connor, J., Crossman, N., 2013. Anecosystem services approach to estimating economic lossesassociated with drought. Ecological Economics 91, 19–27.

Bateman, I.J., Mace, G.M., Fezzi, C., Atkinson, G., Turner, K.,2011. Economic analysis for ecosystem service assessments.Environmental and Resource Economics 48, 177–218.

Beven, K.J., Alcock, R.E., 2012. Modelling everything everywhere:a new approach to decision-making for water managementunder uncertainty. Freshwater Biology 57, 124–132 (BlackwellPublishing Ltd.).

Bishop, R., 1978. Endangered species and uncertainty: theeconomics of a safe minimum standard. American Journal ofAgricultural Economics 60, 10–18.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8 107

Bredehoeft, J., 2005. The conceptualization model problem—surprise. Hydrogeology Journal 13 (1) 37–46.

CEW, 2011. Commonwealth Environmental Water-tradingArrangements. Discussion Paper. CommonwealthEnvironmental Water, 22 (November).

Ciriacy-Wantrup, S.V., 1952. Resource Conservation: Economicsand Policies. University of California Press, Berkeley.

CSIRO, 2012. Assessment of the ecological and economicbenefits of environmental water in the Murray–DarlingBasin. CISRO Water for a Healthy Country National ResearchFlagship, Australia.

Daneshkhah, A.R., 2004. Uncertainty in probabilistic riskassessment: a review. In: BEEP Working Paper: BayesianElicitation Of Experts’ Probabilities. University of Sheffield,Sheffield, UK. , 9 August (accessed 21.05.12)http://www.sheffield.ac.uk/content/1/c6/03/09/33/risk.pdf.

DEWHA, 2009. A Framework for Determining CommonwealthEnvironmental Watering Actions. Department of theEnvironment, Water, Heritage and the Arts, Canberra(December).

Doody, D.G., Foy, R.H., Barry, C.D., 2012. Accounting for the roleof uncertainty in declining water quality in an extensivelyfarmed grassland catchment. Environmental Science andPolicy 24, 15–23.

Doyle, M., Drew, C.A., 2008. In: Doyle, M., Drew, C.A. (Eds.),Large-scale Ecosystem Restoration: Five Case Studies from theUnited States. Island Press, Washington, DC, p. 325.

DSEWPaC, 2011. A Proposed Monitoring, Evaluation andReporting Framework for use of CommonwealthEnvironmental Water in the Murray–Darling Basin:Discussion Paper. Department of Sustainability,Environment, Water, Population and Communities, 22(June).

Folke, C., Carpenter, S., et al., 2004. Regime shifts, resilience,and biodiversity in ecosystem management. Annual Reviewof Ecology, Evolution, and Systematics S35, 557–581.

Funtowicz, S., Ravetz, J.R., 1993. Science for the post-normalage. Futures 25, 735–755.

Garrick, D., Bark, R., Connor, J., Banerjee, O., 2012.Environmental water governance in federal rivers:opportunities and limits of subsidiarity in the River Murrayof Australia. Water Policy 14, 915–936.

Griffin, L.F., Knight, J.M., 2012. A review of the role of fish asbiological control agents of disease vector mosquitoes inmangrove forests: reducing human health risks whilereducing environmental risk. Wetlands Ecology andManagement 20 (3) 243–252.

Gupta, H.V., Clark, M.P., Vrugt, J.A., Abramowitz, G., Ye, M., 2012.Towards a comprehensive assessment of model structuraladequacy. Water Resources Research (AGU) 48, W08301–W08308, http://dx.doi.org/10.1029/2011WR011044.

Harris, G.P., Heathwaite, A.L., 2012. Why is achieving goodecological outcomes in rivers so difficult? FreshwaterBiology 57 (Suppl. 1) 91–107.

Hashimoto, T., Stedinger, J.R., Loucks, D.P., 1982. Reliabilityresiliency and vulnerability criteria for water resourcesystem performance evaluation. Water Resources Research18 (February (1)) 14–20.

Hatton MacDonald, D., Morrison, M.D., Rose, J.M., Boyle, K.J.,2011. Valuing a multistate river: the case of the River Murray.Australian Journal of Agricultural and Resources Economics55 (3) 373–391.

Hayes, K.R., 2011. Uncertainty and uncertainty analysismethods Issues in quantitative and qualitative risk modelingwith application to import risk assessment. ACERA Project(0705) Report Number: EP102467. .

IEC, 2009. IEC/ISO 31010:2009-11, Risk Management—RiskAssessment Techniques. International ElectrotechnicalCommission, International Standards Organisation.

Kandulu, J.M., Bryan, B.A., King, D., Connor, J.D., 2012. Mitigatingeconomic risk from climate variability in rain-fed agriculturethrough enterprise mix diversification. Ecological Economics79, 105–112.

Kavetski, D., Kuczera, G., Franks, S.W., 2006. Bayesian analysisof input uncertainty in hydrological modeling: 1. Theory.Water Resources Research (AGU) 42, W03407.3, http://dx.doi.org/10.1029/2005WR004368.

Kimball, A.W., 1957. Errors of the third kind in statisticalconsulting. Journal of the American Statistical Association,American Statistical Association 52 (278) 133–142.

Kingsford, R.T., Walker, K.F., Lester, R.E., Young, W.J.,Fairweather, P.G., Sammut, J., et al., 2011. A Ramsar wetlandin crisis—the Coorong, Lower Lakes and Murray Mouth,Australia. Marine and Freshwater Research 62 (3)255–265.

Kirby, M., Evans, R., Walker, G., Cresswell, R., Coram, J., Khan, S.,et al., 2006. The Shared Water Resources of the Murray–Darling Basin. , http://www.csiro.au/files/files/p7g9.pdf.

Landis, W.G., Durda, J.L., Brooks, M.L., Chapman, P.M., Menzie,C.A., Stahl Ralph, G.J., Stauber, J.L., 2013. Ecological riskassessment in the context of global climate change.Environmental Toxicology and Chemistry 32, 79–92.

Lee, K.N., 1993. Compass and Gyroscope: Integrating Scienceand Politics for the Environment, vol. xii. Island Press,Washington, DC, ISBN: 1559631988pp. 243.

Lester, R.E., Fairweather, P.G., 2011. Ecosystem states: creating adata-derived, ecosystem-scale ecological response modelthat is explicit in space and time. Ecological Modelling 222,2690–2703.

Lester, R.E., Webster, I.T., Fairweather, P.G., Young, W.J., 2011.Linking water-resource models to ecosystem-responsemodels to guide water-resource planning: an example fromthe Murray–Darling Basin, Australia. Marine and FreshwaterResearch 62, 279–289.

Liu, J., Dietz, T., Carpenter, S.R., Alberti, M., Folke, C., Moran, E.,et al., 2007. Complexity of coupled human and naturalsystems. Science 317 (5844) 1513–1516.

Lo, A.W., Mueller, M.T., 2010. WARNING: physics envy may behazardous to your wealth. , 20MarcharXiv:1003.2688v3[q*fin.RM].

MDBA, 2011a. Proposed Basin Plan. MDBA publication no: 192/11. , http://download.mdba.gov.au/proposed/proposed_basin_plan.pdf.

MDBA, 2011b. The Proposed ‘Environmentally Sustainable Levelof Take’ for Surface Water of the Murray–Darling Basin:Methods and Outcomes. Murray–Darling Basin Authority,Canberra. , http:\\www.download.mdba.gov.au/proposed/ESLT_MDBA_report.pdf.

Newig, J., Pahl-Wostl, C., Sigel, K., 2005. The role of publicparticipation in managing uncertainty in theimplementation of the Water Framework Directive.European Environment 15, 333–343.

Norgaard, R.B., 2010. Ecosystem services: from eye-openingmetaphor to complexity blinder. Ecological Economics 69 (6), http://dx.doi.org/10.1016/j.ecolecon.2009.11.009.

Palik, B.J., 2000. Using landscape hierarchies to guide restorationof disturbed ecosystems. Ecological Applications 10, 189–202.

Peel, M.C., McMahon, T.A., Finlayson, B.L., 2004. Continentaldifferences in the variability of annual runoff—update andreassessment. Journal of Hydrology 295, 185–197.

Pielke Jr., R.A., 2007. The Honest Broker: Making Sense ofScience in Policy and Politics. Cambridge University Press,New York.

Ramsar Convention, 1971. The Convention on Wetlands ofInternational Importance, Especially as Waterfowl Habitat,in 3 December 1982 as Amended by the Paris Protocol, andRegina Amendments, 28 May 1987. , Ramsar: UN TreatySeries No. 14583.

e n v i r o n m e n t a l s c i e n c e & p o l i c y 3 3 ( 2 0 1 3 ) 9 7 – 1 0 8108

Robertson, L.N., Allsopp, P.G., Chandler, K.J., Mullins, R.T., 1995.Integrated management of Canegrubs in Australia—currentsituation and future-research directions. Australian Journalof Agricultural Research 46 (1) 1–16.

Shao, Q., Lerat, J., Brink, H., Tomkins, K., Yang, A., Peeters, L.,Li, M., Zhang, L., Podger, G., Renzullo, L.J., 2012. Gaugebased precipitation estimation and associated model andproduct uncertainties. Journal of Hydrology, http://dx.doi.org/10.1016/j.jhydrol.2012.04.009 pp. 444–445,100–1120.

Smith, L., Inman, A., Cherrington, R., 2012. The potential of landconservation agreements for protection of water resources.Environmental Science and Policy 24, 92–100.

Verkade, J.S., Werner, M.G.F., 2011. Estimating the benefits ofsingle value and portability forecasting for flood warning.Hydrological and Earth System Science 15, 3751–3765.

Walker, D.J., Jessup, A., 1992. Analysis of the dynamic aspects ofthe River Murray Mouth, South Australia. Journal of CoastalResearch 8 (1) 71–76.

Walters, C., Maguire, J.-J., 1996. Lessons for stock assessmentfrom the northern cod collapse. Reviews in Fish Biology andFisheries 6 (2) 125–137.

Webster, I.T., 2010. The hydrodynamics and salinity regime of acoastal lagoon—the Coorong, Australia—seasonal to multi-decadal timescales. Estuarine, Coastal and Shelf Science 90,264–274.