Management Effectiveness: Assessing Management of Protected Areas

Management Effectiveness: Assessing Management of

Protected Areas?

MARC HOCKINGS*, SUE STOLTON** & NIGEL DUDLEY**

*University of Queensland, Brisbane, Australia and IUCN World Commission on Protected

Areas

**Equilibrium Consultants, Bristol, UK

ABSTRACT To maximise the potential of protected areas, we need to understand thestrengths and weaknesses in their management and the threats and stresses that they face.There is increasing pressure on governments and other bodies responsible for protectedareas to monitor their effectiveness. The reasons for assessing management effectivenessinclude the desire by managers to adapt and improve their management strategies,improve planning and priority setting and the increasing demands for reporting andaccountability being placed on managers, both nationally and internationally. Despitethese differing purposes for assessment, some common themes and information needs canbe identi®ed, allowing assessment systems to meet multiple uses. Protected-areamanagement evaluation has a relatively short history. Over the past 20 years a numberof systems have been proposed but few have been adopted by management agencies. Inresponse to a recognition of the need for a globally applicable approach to this issue, theIUCN World Commission on Protected Areas developed a framework for assessingmanagement effectiveness of both protected areas and protected area systems. Thisframework was launched at the World Conservation Congress in Jordan in 2000. Theframework provides guidance to managers to develop locally relevant assessment systemswhile helping to harmonise assessment approaches around the world. The framework isstrongly linked to the protected area management process and is adaptable to differenttypes and circumstances of protected areas around the world. Examples from FraserIsland in Australia and the Congo Basin illustrate the use of the framework.

Introduction

Since the ®rst protected areas were established, managers and others involved inthe conservation movement have sought to achieve effective management of theseareas. However management effectiveness as a prominent issue in protected-areamanagement is a relatively new phenomenon. It was not raised by speakers at the1972 Second World Conference on National Parks, although in the same year, aConservation Foundation Task Force on the National Parks System did raise theissue of stresses to park systems and the capacity of the Parks Service in America

Correspondence address: The School of Natural and Rural Systems Management, Universityof Queensland, Brisbane, Queensland, Australia. Email [email protected]

1523±908X Print/1522±7200 Online/04/040157-18 ã 2004 Taylor & Francis Ltd.

DOI: 10.1080/1523908042000320731

Journal of Environmental Policy & PlanningVol. 6, No. 2, June 2004, 157±174

to deal with these (Strong et al., 1972). They recommended adoption of an annualsystem of park environmental reports to assure a continued monitoring of eachpark's internal and external environmental factors and to identify mattersrequiring remedial action. They further recommended that these reports shouldbe compiled into a system-wide environmental report.

A decade later, the third World Congress on National Parks (the BaliCongress) contained one paper on monitoring (Croze, 1984) in the section of theCongress dealing with New Directions in Protected Area Management, but this dealtwith monitoring to provide baseline and trend data on ecological systems. Theimportance of such information for effective management was noted but the useof this information to assess and adapt management was not addressed. In theCongress workshop on Managing Protected Areas, papers by Deshler (1982) andThorsell (1982) addressed the issue of management effectiveness explicitly.Deshler's paper consisted of a discussion of what constitutes effective manage-ment, but did not develop this into an assessment system. Thorsell's paperadvocated the use of an evaluation checklist or questionnaire. The development of`tools and guidelines' to `evaluate the ecological and managerial quality ofexisting protected areas' was identi®ed as one of the actions in the Bali Action Planthat was adopted at the end of the Congress (Miller, 1984). The outcomes from theworkshop session were later compiled into a book, Managing Protected Areas in theTropics (MacKinnon and MacKinnon, 1986a), that contained a chapter onEvaluating the Effectiveness of Management.

Following the Bali Congress the issue of management effectiveness ofprotected areas began to appear in international literature and particularly withinthe work and deliberations of Commission on National Parks and Protected Areas(CNPPA)1 (e.g. MacKinnon and MacKinnon, 1986b; MacKinnon and MacKinnon,1986c; IUCN Commission on National Parks and Protected Areas, 1988). Whilethe issue was recognised as important, there was relatively little action betweenthe third and fourth World Parks Congresses. The methodology presented at theThird Congress and subsequent book do not appear to have been followed up inthe literature or the further work of IUCN. In the fourth (Caracas) Congress,effective management was identi®ed as one of the four major protected area issuesof global concern (McNeely, 1993). One of the Congress workshops aimed to:

d develop further and obtain agreement on the categories and types of protectedareas;

d develop an international system for categorising the effectiveness of protectedarea management;

d develop an agreed system of categories of threats to protected areas.

(McNeely 1993, p. 162).

A discussion paper with a possible methodology for assessing managementeffectiveness was prepared for the workshop (Foster, 1991). The proposedmethodology was discussed but not adopted at the workshop and therecommendations of the Caracas Congress included a call (Recommendation17) for IUCN to further develop a system for monitoring management effective-ness of protected areas. At the same time, a number of evaluation methodologies(e.g. Chrome, 1995; Hockings, 1998; The Nature Conservancy, 1998; Courrau,1999; Cifuentes et al., 2000) were being independently developed around theworld. However none had resulted in widespread adoption outside of the

158 M. Hockings et al.

organisation, protected area system or region in which they were developed (seeHockings (2000) for a review of these systems).

There was little action on the issue within IUCN until early 1996 when thedecision was taken to develop a methodology for the CNPPA that would bewidely applicable around the world. One of us (MH) was asked to lead a TaskForce within the Commission to carry this work forward and all the authors werekey participants in this process. This paper outlines the framework for assessingmanagement effectiveness of protected areas that has been developed by IUCNand examines its application in case studies in Australia and Africa.

What is Meant by Management Effectiveness?

There are three main components that should be considered in assessingmanagement effectiveness of protected areas:

d design/planning issues;d adequacy and appropriateness of management resources, systems and pro-

cesses; andd delivery of protected-area objectives.

Design/Planning

considers how design issues such as the size and shape of protected areas; theexistence and management of buffer zones and links between protected areas,affect the capacity of sites to achieve their stated function. Design failures can, forexample, lead to problems of protected areas that are too small to be effective, arefragmented and fail to provide capacity to adapt to environmental change.Planning considers the existence and adequacy of planning undertaken for theprotected areas.

Adequacy/Appropriateness

looks at how management is resourced and conducted. This component considersboth whether there are suf®cient management resources and whether manage-ment processes and actions are appropriate. Management failures therefore rangefrom complete lack of implementation (so-called `paper parks') through tostrategic errors about where to focus effort or how management is conducted.

Delivery

assesses whether protected areas are achieving their stated objectives. Measuresinclude both biological elements (such as whether key species are surviving,recovering or declining) and social aspects (such as recreational use or theattitudes and behaviour of local human communities towards the protected area).

Why Assess Management Effectiveness?

There are many reasons why people want to assess management effectiveness.Three common uses of such evaluation are:

Assessing Management of Protected Areas? 159

d promoting adaptive management (progressive improvement of managementbased on re¯ective learning);

d improving project planning; andd promoting accountability.

Adaptive Management

First and foremost, evaluation should be seen as a normal part of the managementprocess. Adaptive management is based on a circular, rather than a linear,management process, which allows information concerning the past to feed backinto and improve the way management is conducted in future (Holling, 1978;Grumbine, 1994). Evaluation helps management to adapt and improve throughthis learning process.

Improve Program Planning

Evaluation studies can also be used to improve program/project planning ± eitherat the time of initial design or as a review of previous programs where the lessonslearnt will be applied to programs that follow. Within government, systematicevaluation has become a key step in linking program implementation withplanning and budgeting (Knight, 1990). Where common problems are beingaddressed in different ways in a number of protected areas, evaluation data can beused to compare results and allow managers to select the best approach.Managers can use the results of evaluations of management effectiveness indeveloping proposals for additional resources. Such requests are more likely towin support when they can be justi®ed on the basis of evaluation results. Forexample, a study of the 1991±92 Australian Government budget process foundthat over 60 per cent of the major new initiatives and almost half of all newinitiatives (measured by dollar value) announced in the budget were in¯uencedby evaluation ®ndings (Department of Finance 1992).

Promote Accountability

Accountability for performance is being increasingly demanded across all sectorsof society and conservation management is no exception. Governments in manycountries have instituted procedures for reviewing government programs,including conservation programs (O'Faircheallaigh and Ryan, 1992; AuditorGeneral of British Columbia, 1996; Rump, 1996). Traditionally, concerns foraccountability focused on issues of ®nancial and managerial probity, but this hasnow expanded to include concerns for management effectiveness. Viewed in thislight, accountability is not so much about `checking up' on managers to see wherethey are failing, as about developing a professional approach to management.

Planning guidelines commonly specify that `performance indicators' shouldbe developed as part of the plan to provide a means for measuring success of theprogram. The speci®cation of performance indicators is, for example, a keyelement of the logframe planning system used by USAID and many otheragencies (Mason 1997).


The Management Cycle and Evaluation

A management cycle approach which relates evaluation to the process ofmanagement is a common approach to the design of public sector evaluationprograms (Caulley, 1993; Auditor General of British Columbia, 1996). Evaluationmethodologies based on this approach have been criticised because of a perceivedfocus on program inputs, processes and outputs, rather than concern for the realimpact (outcome) on the program in terms of its intended objectives (Caulley,1993). However, there is no inherent reason why a focus on outcomes should notbe included in a system based on this approach. One of the major strengths of thisapproach is the ready match that exists between evaluation information and theplanning and management systems used by managers. Information frommonitoring and assessment programs can therefore be readily analysed andapplied within the planning and management system. This was particularlyimportant in the development of the WCPA approach, given that we sought tolink the evaluation system closely to the manager's needs and perspectives and topromote an adaptive approach to management.

Management consists of several linked, iterative phases of planning, actionand review (Department of Finance, 1989). The starting point of this managementcycle consists of understanding the environment within which management isoperating (context) and then establishing objectives and associated managementstrategies designed to achieve these objectives (planning). Resources of staff andmoney (inputs) are allocated to undertake management activities and actionsaccording to established operational practices and standards (processes). Thismanagement activity produces services and products (outputs) that are intendedto achieve objectives (outcomes).

The WCPA management effectiveness evaluation framework (WCPAFramework) is based around this management cycle (Figure 1). The frameworkelements and the criteria that are used to assess management effectiveness inrelation to each element are explained in Table 1 and the accompanying text.

The context review, although not an analysis of management, provides theinformation that helps put management decisions into context and allows them tobe prioritised based on the biological, cultural and political information gathered.In the case of protected areas, important aspects of context are the signi®cance andvalues of the area that led to its declaration as a conservation area and the threatsand opportunities that the area faces.

Planning involves setting the direction and objectives of management anddeciding on the strategies that are required to achieve the objectives. Planningeffective strategies requires an understanding of both the desired endpoint ofmanagement (the vision and speci®c objectives) and the starting point or contextwithin which management operates.

Allocating funds and staff time to management should be linked to and, inlarge part, directed by planning decisions. Although protected-area managementplans rarely provide speci®c commitments of funds and staff, they establish thebasis for short-term or annual operational planning in which decisions aboutallocation of resources are made. Managers then use these resources to undertaketheir jobs by implementing the actions and strategies indicated by planningdocuments (e.g. management plans, annual operations plans, functional or issue-speci®c plans) and through reactive or opportunistic management actions. Inundertaking these activities, managers are guided by agency policies and


practices, the general norms and standards applied to protected area managementin their regional area and by their own training and experience.

The results of this management activity can be considered in two ways. Firstthere are the direct outputs produced by the management activity that commonlyconsist of a set of products or services (some examples for protected areas are areaof land sprayed for weeds, kilometres of track maintained, numbers of guidedwalks conducted, numbers of anti-poaching patrols undertaken). As well as theseoutputs from management, results can also be considered in terms of the impactor outcome of management activities, especially in relation to the achievement ofobjectives established for the area (some examples for protected areas are extent ofreduction in weed infestation, visitor satisfaction with walking experiences,change in knowledge and attitudes of visitors who attend guided activities, extentto which poaching is controlled).

The management cycle is completed when the manager reviews progress anduses this review information to adjust or correct their planning and management.This review function is often visualised and presented as only being linked tomanagement outcomes. However, evaluation can look at all aspects of themanagement cycle, including the context within which management takes place.The results of evaluating each aspect can be fed back into the management cycle.The criteria that can be used to assess each element of the Framework are speci®ed

Figure 1. The management cycle and evaluation


in general terms below, but it is expected that speci®c criteria, indicators andassessment methods will be adapted to suit individual cases.

Design Issues ± Context and Planning

Context

involves issues that lie outside the direct operations of the protected area manageror management agency. It is the context within which they operate and includesconsideration of the conservation and other values of the protected area whichunderpin the objectives set for management of the site, its current status and theparticular threats and opportunities that are affecting it, including the broadpolicy environment. This is not an analysis of management per se, but providesinformation that helps put management decisions into context. Where assessmentis being used to identify management priorities within a protected area network,or to decide on the time and resources to devote to a particular protected area, thismay be the main area of assessment required.

Planning

focuses on the determination of intended outcomes for the protected area systemor the individual protected area: the vision for which the system or site is beingplanned and the strategies that have been selected to achieve this vision.Assessment may consider the appropriateness of national protected-area legis-lation and policies, the existence and adequacy of plans and strategies forprotected-area systems, the adequacy of design of individual protected areas andplans for their management. Design is considered a planning issue because designof individual reserves or the whole protected area system is essentially a planningprocess undertaken by mangers rather than imposed from outside (althoughmany of the design decisions made may be constrained by the external physical,social and political environment). Assessment may consider the design of aprotected area in relation to the integrity and status of the resource. The selectedindicators for evaluation will depend on the purpose of assessment and,particularly, whether it is looking at a system of reserves or at an individualprotected area. With systems, issues of ecological representativeness andconnectivity are especially important. The focus of assessment of individualprotected areas will be on the shape, size, location and detailed managementobjectives and plans. System assessments should consider, for example, ifprotected-area systems omit or under-represent certain habitat types; while siteassessments ask questions such as whether the protected area is too small toprotect biodiversity over the long term.

Adequacy and Appropriateness of Management Resources, Systems andProcesses ± Inputs and Processes

The evaluation of inputs assesses the adequacy of resources available tomanagement, focusing primarily on measures of staff numbers and skills,funds, equipment and facilities required at either agency or site level. Theadequacy of resourcing needs to be measured in relation to the size of themanagement task and within the standards of the national and regional area.


Management demands are affected by the management purpose and strategyapplying to the area. For example, areas developed for intensive tourism willrequire more resources for recreation management than isolated sites with fewvisitors. The level of threat, and consequent requirements for threat abatement,and the attributes and condition of the natural and cultural resources within thearea will also affect requirements for funds and staff. Regional protected-areamanagement norms will also affect the requirements for management resources.For example, there are regional differences in visitor expectations regarding theprovision of visitor interpretation and the quality of visitor infrastructure. Thesedifferences impact on the resource requirements for management.

Processes

considers the appropriateness of management processes and systems in relationto the management objectives for a system or a site. Differing regional norms forthe way management of protected areas is conducted will affect the assessment ofthis element of the evaluation framework in much the same way as regionalnorms affect the assessment of management inputs. Assessment will involveconsideration of diverse management processes such as facility maintenance,methods of interaction with local communities, visitor management, proceduresfor natural and cultural resource management, and ®nancial and of®ce manage-ment systems.

Delivery of Protected Area Objectives ± Outputs and Outcomes

Output

evaluation considers what has been done by management, and examines theextent to which targets, work programs or plans have been implemented. Targetsmay be set through management plans or a process of annual work programming.The focus of output monitoring is not so much on whether these actions haveachieved their desired objectives (this is the province of outcome evaluation), buton whether the activities have been carried out as scheduled and the progressmade in implementing long-term management plans.

Outcome

evaluation assesses whether management has been successful with respect to theobjectives in a management plan, legislation, national plans and ultimately theaims of the IUCN category of the protected area. Outcome evaluation is mostmeaningful where concrete objectives for management have been speci®ed eitherin national legislation, policies, or site-speci®c management plans. Approaches tooutcome evaluation may involve long-term monitoring of the condition of thebiological and cultural resources of the system/site, socio-economic aspects ofuse, and the impacts of the management of the system/site on local communities.In the ®nal analysis, outcome evaluation is the true test of managementeffectiveness. However, the monitoring required is signi®cant, especially sincelittle attention has been given to this aspect of protected area management in thepast. Protected area management plans typically establish many objectives formanagement. Establishing outcome-based monitoring programs for each object-


ive is likely to be beyond the capacity of most management agencies. Thus, theselection of indicators to be monitored is critical. Attention should be directedtowards the most important objectives and, where possible, indicators should beselected that do not require elaborate or expensive monitoring programs to collectrelevant data.

Evaluating Management Effectiveness Using the Framework

Table 1 sets out each of the framework elements (context, planning, inputs, processes,outputs and outcomes), explains the issues covered within each element, and listssome of the criteria that can be used to evaluate each element.Ideally, systems for assessing management effectiveness of protected areas willincorporate components that cover each of the elements of evaluation outlinedhere. Because each type of evaluation has a different focus, they are complemen-tary rather than alternative approaches to evaluating management effectiveness.Time series data for both inputs and outputs within a protected area or system canbe particularly valuable in assessing changes in the ef®ciency of management andmay enable a judgement to be made about the effectiveness of a change inmanagement practice or policy. However, assessments will be driven byparticular needs and a partial evaluation can still provide very useful information.In general, the effort required to collect the relevant monitoring data increasesfrom left to right across the model in Table 1. But so, too, does the value of theinformation collected.

Selection of Indicators for Environmental Performance Monitoring

As it is not practical to measure directly all the attributes that relate to protectedarea management (either the condition of the environment itself or aspects ofmanagement action), a limited number of representative indicators need to beselected. The selection of priority issues, and hence indicators, for monitoringshould be guided by the natural, cultural and social values of the area, which, inturn, can be guided by an assessment of the context within which the site or systemis operating.

The selection of indicators is a complex process often involving trade-offsbetween partially incompatible attributes. As Bernstein (1992, p.1112) points out,

`developing indicators that successfully re¯ect ecological effects and aremanagerially useful requires reconciling two sets of often con¯icting constraints¼ [that] ¼ emerge from the separate ecological and management contexts thatindicators must be responsive to'.

Key attributes of useful environmental indicators have been identi®ed byvarious authors (e.g. Centre for Coastal Management, 1993; Briggs et al., 1996;Abbot and Guijt, 1998). They suggest that, to the greatest extent possible,indicators to measure management effectiveness should:

d have an unambiguous, predictable and veri®able relationship to the attributebeing assessed;

d be sensitive to change in the attribute being assessed;d integrate environmental effects over time and space (i.e. re¯ect enduring change

rather than short-term or localised ¯uctuations in conditions);


Tab

le1.

WC

PA

Fra

mew

ork

for

asse

ssin

gm

anag

emen

tef

fect

iven

ess

of

pro

tect

edar

eas

and

pro

tect

ed-a

rea

syst

ems

Ele

men

tso

fev

alu

atio

nD

esig

nis

sues

Ap

pro

pri

aten

ess

of

man

agem

ent

syst

ems

and

pro

cess

esD

eliv

ery

of

pro

tect

edar

eao

bje

ctiv

es

Co

nte

xt

Pla

nn

ing

Inp

uts

Pro

cess

esO

utp

uts

Ou

tco

mes

Ex

pla

na

tio

nW

here

are

we

now

?W

here

dow

ew

ant

tobe

?W

hat

dow

en

eed?

How

dow

ego

abou

tit

?W

hat

wer

eth

ere

sult

s?W

hat

did

we

achi

eve?

Ass

essm

ent

of

imp

ort

ance

,th

reat

san

dp

oli

cyen

vir

on

men

t

Ass

essm

ent

of

PA

des

ign

and

pla

nn

ing

Ass

essm

ent

of

reso

urc

esn

eed

edto

carr

yo

ut

man

agem

ent

Ass

essm

ent

of

the

way

inw

hic

hm

anag

emen

tis

con

du

cted

Ass

essm

ent

of

the

imp

lem

enta

tio

no

fm

anag

emen

tp

rog

ram

san

dac

tio

ns;

del

iver

yo

fp

rod

uct

san

dse

rvic

es

Ass

essm

ent

of

the

ou

tco

mes

and

the

exte

nt

tow

hic

hth

eyac

hie

ved

ob

ject

ives

Cri

teri

ath

at

are

use

dto

ass

ess

ma

na

ge

me

nt

eff

ect

ive

ne

ss

Sig

ni®

can

ceT

hre

ats

Vu

lner

abil

ity

Nat

ion

alco

nte

xt

Pro

tect

edar

eale

gis

lati

on

and

po

licy

Pro

tect

edar

easy

stem

des

ign

Res

erv

ed

esig

nM

anag

emen

tp

lan

nin

g

Res

ou

rcin

go

fag

ency

Res

ou

rcin

go

fsi

teC

on

trib

uti

on

sfr

om

par

tner

s

Su

itab

ilit

yo

fm

anag

emen

tp

roce

sses

Res

ult

so

fm

anag

emen

tac

tio

ns

Ser

vic

esan

dp

rod

uct

s

Imp

acts

:ef

fect

so

fm

anag

emen

tin

rela

tio

nto

ob

ject

ives

Fo

cus

of

ev

alu

ati

on

Sta

tus

Ap

pro

pri

aten

ess

Eco

no

my

Ef®

cien

cyA

pp

rop

riat

enes

sE

ffec

tiv

enes

sE

ffec

tiv

enes

sA

pp

rop

riat

enes

s


d re¯ect changes and processes of signi®cance to management (includingbiophysical, social, cultural, economic, political and managerial attributes);

d re¯ect changes at spatial and temporal scales of relevance to management;d be cost-effective in terms of data collection, analysis and interpretation;d be simple to measure and interpret; andd be able to be collected, analysed and reported on in a timely fashion.

It is important that data-collection programs for the selected indicators can besustained in terms of budgets and staff skills. Simple indicators are generallypreferable to complex ones. If assessments are to be reported widely, the extent towhich indicators are understandable by the non-specialist is also a consideration.The criteria and possible indicators relating to each of the Framework elementshave been outlined in more detail in Hockings et al. (2000).

Application of the WCPA Framework

Two examples of the application of the WCPA Framework are outlined below.The information provided about these examples focuses on the process ofdevelopment and application of the evaluation systems, and on the ways in whichthe results have been applied rather than the content and conclusions of theevaluation work which has been published elsewhere in papers and reports. Thetwo case studies provide examples of how the WCPA Framework can be used todevelop evaluation systems focused on different aspects of management andrequiring differing levels of time and effort to conduct. The Fraser Island WorldHeritage Area case study was designed as a long-term monitoring programfocused strongly on meeting the information needs of site managers. The DjaReserve case study was designed as a rapid assessment system designed to meetthe information needs of multiple audiences (managers, local communities andinternational donors).

Fraser Island World Heritage Area

Fraser Island, claimed to be the largest sand island in the world, stretches for over120 kilometres along the southern Queensland coast (Department of the Arts,1991). Its long, uninterrupted eastern beaches, magni®cent heaths, swamps,forests and lakes are of great biological and recreational signi®cance. The islandwas included on the World Heritage list in 1992. Management of the public land(national park, recreation management area and adjacent marine park) is theresponsibility of the Queensland Parks and Wildlife Service (QPWS). Thedevelopment and implementation of the evaluation strategy and associatedmonitoring programs was carried out in close consultation with QPWS managersof the island.

The work on management effectiveness evaluation for Fraser Island wasconducted in two overlapping phases. It began in 1994 with initial development ofan evaluation system for Fraser Island, primarily focused on assessing outputs andoutcomes (Phase 1). Based on the early stages of this work, one of us (MH) wasasked by IUCN to develop a general approach to assessing managementeffectiveness that could be applied around the world. This led to developmentof the draft WCPA Framework which was released for comment in 1997(Hockings, 1997). During the process of developing the draft Framework, the


potential utility of assessing elements other than outputs and outcomes becameapparent and assessment elements focussing on inputs and processes weresubsequently added to the monitoring and evaluation program for FraserIsland (Phase 2).

In Phase 1 of the program, the statements of desired outcomes for each of themajor topics in the management plan formed the starting point for the evaluationstrategy. It was recognised that it would not be possible to establish outcome-oriented monitoring programs for all 55 topic areas in the management planbecause of limitations on staff time and resources. The areas addressed in theinitial phase were selected in consultation with managers. They represent theareas considered to be most potentially important in terms of maintenance ofconservation values, quality of visitor experiences and extent of managementeffort. The issue areas from the management plan selected for inclusion in the casestudy were Camping, Vehicular use of beaches, Marine and terrestrial wildlife, Fire,Roads and Water quality. For each of these topics, potential items for performanceassessment were identi®ed on the basis that they would re¯ect achievement of keyattributes of the desired outcome. The information required to make theseperformance assessments was then speci®ed along with ideas on an appropriatemethodology for collecting this information and notes on problems or issues to beconsidered in relation to data collection. In some cases, data could be used toprovide assessment of performance for more than one issue. For example,monitoring changes in ¯oristic composition of habitats is relevant to outcomesrelating to Marine and Terrestrial Wildlife and Fire.

The second component of Phase 1 of the evaluation strategy monitored theextent of implementation of the policies and actions speci®ed in the managementplan. The objective of this component of the strategy was to provide managers andthe broader community with data on the general status of plan implementationand speci®c information on progress of individual components of the plan. Amanagement information system was developed to provide a mechanism for thismonitoring. It contained all of the policies and guidelines from the managementplan and provided for annual recording of status and expenditure information onthese actions. The system was designed to provide forward planning as well asmonitoring and reporting functions and allowed for proposed work programsand budgets to be recorded and analysed.

In Phase 2 of the Fraser Island monitoring program, evaluation elements formanagement inputs and processes were added to the existing outputs and outcomesmonitoring programs. Accurate and comparable input data proved dif®cult toobtain because of changing management systems. It was possible to obtaincomparable budget data from 1994/95 to 1998/99, but not to get staf®ng detailsfor this period. However, data on State operational and special funds is a partialsurrogate for staf®ng numbers as this was the source of salaries for the majority ofQPWS employees on Fraser Island. The assessment of management processes wasbased on the scoring system developed as part of the draft WCPA Framework. Asintended in the framework, the scoring instrument was amended, in consultationwith managers, to better re¯ect the circumstances and management demandsapplicable to Fraser Island. This involved adding additional elements relating totourism and visitor management.

The evaluation system for Fraser Island was designed as a long-termassessment program that would provide feedback to managers on an on-goingbasis. Hence, reporting has been primarily directed at managers and associated


management committees which represent key stakeholder groups with interestsin the area. This reporting has been conducted primarily on a program-by-program basis. The results of camping, vehicular use of beaches and road-monitoring programs were immediately fed back to managers through work-shops and reports. Results of the camping and vehicular use of beaches studieswere also published in the scienti®c literature (e.g. Hockings and Twyford, 1997;Fisher et al., 1998). Annual reports were prepared on the monitoring ofmanagement plan implementation. Analysis and reporting of Phase 2 elementsof the case study (input and process evaluation) and the vegetation, fauna andwater quality monitoring programs were included in a major report on the casestudy that was completed in 2000 (Hockings and Hobson, 2000), althoughpreliminary ®ndings from the vegetation and fauna monitoring had beenpreviously contributed to relevant workshops conducted by QPWS. Reportingto the Community Advisory Committee, which represents stakeholders involvedin the area, has resulted in some dissemination of the results to wider stakeholdergroup membership, for example through reports on the results of monitoringprograms in the newsletter of the Fraser Island Defenders Organisation, aconservation NGO associated with the island.

The ways in which QPWS has used the information from the monitoringprograms (Table 2) has been identi®ed against one or more of the three commonuses for evaluation; adaptive management (am), improving planning (pl), andaccountability (ac). The use of information has been identi®ed as contributing toadaptive management where it has prompted speci®c action or policy changes.Accountability has been identi®ed as a use where there has been a process ofreporting on the results of the monitoring to stakeholders or information frommonitoring programs has been incorporated into public reports. Planning hasbeen identi®ed as a use where results have been applied in later planningexercises.

Of the 26 identi®ed instances where information from the monitoringprograms has been used, planning was the most common purpose (46 per cent),followed by accountability (35 per cent) and adaptive management (27 per cent).This assessment of use is likely to underestimate use of information for adaptivemanagement purposes because of the dif®culty in recognising this use wheremonitoring information indicates that current management activity is effectiveand hence no management change has been required. The most active use ofassessment information by managers has been in relation to outputs and outcomes,with input and process information only used in a minor way for reporting to thejoint meetings of the advisory committees. This bias towards output and outcomemeasures may be, in part, a re¯ection of the origins of the evaluation system whichinitially only looked at these elements of the framework, but it may also re¯ect afocus by managers on these measures of performance.

IUCN/WWF Forest Innovations ± Assessing Protected-Area ManagementEffectiveness in the Congo Basin

The draft WCPA Framework (Hockings 1997) was used by two of us (SS and ND2)to develop an assessment of management effectiveness for two pilot sites in theCongo Basin (Stolton et al., 2001). The results of the trial have been used in theCongo region to demonstrate the value of assessing management effectiveness,and were used to re®ne the ®nal version of the WCPA framework published in


Table 2. Application of results and subsequent monitoring work undertaken byQPWS for each element of evaluation program (R.Hobson, pers.comm. January

2002; L. Fullerton, pers.comm., January 2002)

Monitoring program Instances where program information has beenused*

Inputs (budget allocation monitoring) Report sent to members of the World HeritageArea Advisory Committees** (ac)

Process (Management process evaluation)Report sent to members of the World HeritageArea Advisory Committees (ac)Outputs (management plan implementationmonitoring)

Report sent to members of the World HeritageArea Advisory Committees (ac)Reporting to Minister and use in Ministerialreplies (ac)Revision of the management plan beingundertaken by QPWS (ac, pl)

Outcomes

Development of Camping Management Plan (pl)Camping Beach camping site management (am)

Results presented to World Heritage AreaAdvisory Committees (ac)Development of rehabilitation program (am)Development of Camping Management Plan (pl)Beach camping site management (am)

Vehicular use of beaches Results presented to World Heritage AreaAdvisory Committees (ac)Results used to assess and manage impactsassociated with the annual Fishing Expo (am).

Vegetation (Marine and terrestrial wildlife) Development of draft Fire Management Plan (pl)Development of Camping Management Plan (pl)Development of prescribed burning program (am)Development of draft Fire Management Plan (pl)Development of Camping Management Plan (pl)Development of Dingo Management Plan (pl)

Fauna (Marine and terrestrial wildlife) Management decisions relating to coastal campingand other site management decisions (am)Development of prescribed burning program (am)Information used in preparing Ministerial replies(ac)Monitoring results were a major informationsource for the review of World Heritage values ofFraser Island (Ingram et al. in press) (ac, pl)

Roads Partial stimulus for and input to the transportmanagement study (pl)Results used in preparing the review of WorldHeritage values of Fraser Island (Ingram et al. inpress) (ac, pl)

Water quality Input to more detailed research program (pl)

* ac=accountability; am=adaptive management; pl=planning** There are three World Heritage Area

Advisory Committees for Fraser Island ± a Community Advisory Committee consisting of stakeholders

representatives from all sectors; a Scienti®c Advisory Committee consisting of scientists from relevant

disciplines and a Management Committee consisting of representatives of relevant federal, state and local

governments.


2000. The methodology put particular emphasis on social aspects and increasingthe participation of a wide range of stakeholders in assessment, and included theconcept of exploring the context within which the protected area is operating as animportant addition to the assessment process. The methodology used aparticipatory rapid-appraisal approach and an assessor who was independentof, but familiar with, the protected areas under consideration.

Testing of the methodology took place in the Dja reserve, which covers 5,260km2 in Cameroon and MinkeÂbeÂ Forest Reserve in Gabon. Dja was protected as a`ReÂserve de faune et de chasse' in 1950, a `Faunal Reserve' in 1973, a Biosphere Reservein 1981 and was inscribed on the World Heritage list in 1984. Ecologically, the DjaReserve is characterised by a deciduous and semi-deciduous forest mixed withextensive swamps. Dja harbours 109 mammal species including threatenedspecies such as the gorilla (Gorilla gorilla gorilla) and elephant (Loxodonta africana).About 30,000 Bantou and Pygmy (Baka) people depend directly on the resourcesof the reserve. There is no commercial timber exploitation within the reserve,although logging and mining take place close by. Poaching is common for homeconsumption and commercial purposes. The MinkeÂbeÂ Forest Reserve, covering6,000 km2, includes wetlands, a variety of primary forest types, and ancientpatches of secondary forest. This diversity of habitat, coupled with the low humanactivity, provides an incredibly rich and varied environment for a large number ofanimal species. MinkeÂbeÂ is considered to be one of the few remaining regionalstrongholds for forest mammals.

The evaluation methodology was developed through the identi®cation ofcomponents of protected-area management relevant to the Central Africansituation within the context of the WCPA framework. These were used to developcriteria and indicators that formed the basis for consultation with managers,project staff and community members. Assessment was carried out using twoquestionnaires (one for staff and one for local communities), coupled withbackground research that included a literature review, site visits, interviews andanalysis. The results of these consultations were summarised in a SWOT(strengths, weaknesses, opportunities and threats) analysis and using a scorecard± based on a template provided in WCPA Framework document but modi®ed tore¯ect local circumstances. In addition to the scoring, a key part of the assessmentwas the requirement to record information on why a particular score wasallocated and comments on current issues or problems relating to the particularaspect of management. Whether these issues and problems were able to becontrolled by managers and current and potential management initiatives inrelation to each issue was also noted. After collection of data and information,results were analysed to formulate conclusions and recommendations for futuremanagement.

The methodology collected data on all elements of the WCPA Framework.Although it did not involve ®eld monitoring of outputs and outcomes, theassessment relied on working closely with a range of stakeholders, such asvillagers from the area, who had some knowledge of these aspects. Of the 128conclusions recorded in the SWOT analyses for the two sites that could beassociated with one of the WCPA framework elements, the majority related to theplanning and process elements of the Framework (each 25 per cent), followed byoutcome elements (19.5 per cent), input and context (each 12 per cent) and outputs(6.5 per cent).

The ®eld testing resulted in a number of recommendations both for the


management of the site and for the development of management effectivenessassessment. In relation to the process of evaluation, the main limitation recordedduring the process of ®eld-testing the system for the management effectivenesswas the suspicion manifested by the protected area management team toward theassessment. Introductory meetings were necessary to explain the objectives andthe importance of assessing management effectiveness. Involving protected areastaff in the development of the assessment process allowed for both an increasedawareness of management effectiveness as an issue and the building of con®dencebetween the assessor and staff. However, the bene®ts of working with an externaland independent assessor were evident in the community consultation phase ofthe assessment. Conducting discussions with the local communities in the absenceof protected area staff created an environment of open and interactive discussions.Communities raised a number of issues concerning management of the site which,in the opinion of the assessor, would not have been raised if management staff hadbeen present.

Discussion

Management effectiveness has grown to be a prominent issue in relation toprotected areas over the past two decades. During this time, a number ofevaluation systems have been proposed but these have not been widely adoptedby management agencies, although the interest by management agencies is high.The WCPA Framework has been developed as a ¯exible design tool for preparingevaluation systems that are responsive to the needs, capacities and circumstancesof protected areas around the world. The two case studies reported in this paperrepresent very different systems for evaluation although both were developedfrom the WCPA Framework. The Fraser Island case study involves a detailed andon-going monitoring and assessment of management that has been used to adjustfuture planning and management of the reserve, as well as to report onmanagement outcomes to stakeholders. The park managers were directlyinvolved in the conduct of the monitoring programs. The Congo Basin casestudy was much less resource intensive and involved an external assessorworking with park managers and the community to assess current management,with a particular focus on the adequacy and appropriateness of planning andmanagement systems and processes.

Acknowledgments

Many people contributed to the discussions leading the development andre®nement of the WCPA Framework for evaluating management effectiveness ofprotected areas. Comments and inputs from the members of the WCPAManagement Effectiveness Task Force are particularly acknowledged. Financialsupport was provided by WWF and the World Bank for Task Force operationsand meetings. QPWS ®eld staff and University of Queensland staff and studentscontributed to the conduct of monitoring programs on Fraser Island. Specialthanks are due to Rod Hobson, Niki Turner and Keith Twyford. The Congo Basincase study was developed by the BMZ-funded joint WWF/IUCN ForestInnovations project with contributions from IUCN and WWF administrativeand technical staff in Central Africa.


Notes

1. Later in 1996, the Commission on National Parks and Protected Areas changed its name to theWorld Commission on Protected Areas (WCPA).

2. Field assessments and staff and community consultations in this project were undertaken by ElieHakizumwami, who was familiar with protected-area management issues in the two sites and hadworked extensively on conservation issues in the Congo Basin.

References

Abbot, J. & Guijt, I. (1998) Changing views on change: participatory approaches to monitoring theenvironment (London, UK: The International Institute for Environment and Development (IIED)).

Auditor General of British Columbia (1996) Enhancing accountability for performance: a framework andimplementation plan (Victoria, British Columbia: Of®ce of the Auditor General).

Bernstein, B. B. (1992) A framework for trend detection: coupling ecological and managerialperspectives, in: D. H. McKenzie, D. E. Hyatt & V. J. McDonald (Eds) Ecological Indicators (ElsevierScience).

Briggs, D., Tantram, D. & Scott, P. (1996) Improving information for management and decision making innational parks (Northampton, UK: Nene Centre for Research).

Caulley, D. (1993) `What is problematic about government program evaluation in Australiasia today?'Evaluation News and Comment, 2(1), pp. 2±8.

Centre for Coastal Management (1993) Monitoring the ef®ciency and effectiveness of the Wet TropicsManagement Plan (Lismore: The University of New England-Northern Rivers).

Chrome, F. (1995) A regional monitoring program for the Wet Tropics of Queensland World Heritage Area(Cairns, Wet Tropics Management Agency).

Cifuentes, M., Izurieta, A. & de Faria, H. (2000) Measuring protected area management effectiveness(Turrialba, Costa Rica: IUCN/WWF Forest Innovations Project, WWF Centroamerica).

Courrau, J. (1999) Strategy for monitoring and management of protected areas in Central America (USA:PROARCA-CAPAS Program, The Nature Conservancy).

Croze, H. (1984) Monitoring within and outside protected areas, in: J. A. McNeely and K. R. Miller(Eds) National parks, conservation and development: the role of protected areas in sustaining society,proceedings of the World Congress on National Parks, Bali, Indonesia, 11±22 October, 1982(Washington DC: Smithsonian Institution Press), pp. 628±633.

Department of Finance (1989) Program evaluation: a guide for program managers (Canberra, Departmentof Finance).

Department of Finance (1992) The role of evaluation in decision-making in the 1991±92 budget context(Canberra, Department of Finance), p. 21.

Department of the Arts, the Environment, Tourism and Territories (1991) Nomination of Fraser Islandand the Great Sandy Region by the Government of Australia for inclusion in the World HeritageList. (Canberra: Department of the Arts, Sport, the Environment, Tourism and Territories), p. 68plus appendices.

Deshler, W. O. (1982) A systematic approach to effective management of protected areas (Gland: IUCNCommission on National Parks and Protected Areas).

Fisher, F., Hockings, M. & Hobson, R. (1998) Recreational impacts on waders on Fraser Island, Sunbird,28(1), pp. 1±11.

Foster, J. (1991) An international review system for categorising protected areas, their managementeffectiveness and threats to them. Framework paper (Gland: IUCN).

Grumbine, R. E. (1994) `What is ecosystem management?', Conservation Biology, 8(1), pp. 27±38.

Hockings, M. (1997) Evaluating Management Effectiveness: A framework for evaluating managementof protected areas. Discussion Paper (Cambridge, UK: IUCN World Commission on ProtectedAreas), p. 80.

Hockings, M. (1998) `Evaluating management of protected areas: integrating planning andevaluation', Environmental Management, 22(3), pp. 337±345.

Hockings, M. (2000) Evaluating protected area management: a review of systems for assessing managementeffectiveness of protected areas (Lawes, Qld: University of Queensland, School of Natural and RuralSystems Management).

Hockings, M. & Hobson, R. (2000) Fraser Island World Heritage Area monitoring and managementeffectiveness project report (Brisbane: University of Queensland).

Hockings, M. & Twyford, K. (1997) `Assessment and management of beach camping impacts within


Fraser Island World Heritage Area, south-east Queensland', Australian Journal of EnvironmentalManagement, 4(1), pp. 26±39.

Hockings, M., with Stolton, S. & Dudley, N. (2000) Evaluating effectiveness: a framework for assessing themanagement of protected areas (Gland: IUCN).

Holling, C. S. (1978) Adaptive environmental assessment and management (Chichester; New York:International Institute for Applied Systems Analysis: Wiley).

Ingram, G., Brown, T., Hegerl, E., Hockings, M., Lamb, D., Thompson, C. & Haig, J. (In press) FraserIsland ± Review of outstanding universal value (Brisbane: Fraser Island World Heritage Area Scienti®cAdvisory Committee), p. 109.

IUCN Commission on National Parks and Protected Areas (1988) New challenges for the world'sprotected area system: proceedings of the 30th Working Session of IUCN's Commission on National Parksand Protected Areas, San JoseÂ, Costa Rica, 30±31 January 1988 (Gland: Switzerland, IUCN).

Knight, H. (1990) `Evaluation in Commonwealth administration: purposes and progress', CanberraBulletin of Public Administration, 63, pp. 18±20.

MacKinnon, J. & MacKinnon, K. (1986a) Managing protected areas in the tropics (Gland: Switzerland,IUCN).

MacKinnon, J. & MacKinnon, K. (1986b) Review of the protected areas system in the Afrotropical realm(Gland, Switzerland: IUCN and UNEP).

MacKinnon, J. & MacKinnon, K. (1986c) Review of the protected areas system in the Indo-Malayan realm(Gland, Switzerland: IUCN and UNEP).

Mason, D. (1997) Performance monitoring of programs to conserve biological diversity: a briefintroduction, with examples. IBAMA/GTZ International Workshop on the Biodiversity ProtectionMonitoring System in Conservation Units., BrasõÂlia e PirenoÂpolis, 22±25 June 1997, IBAMA/GTZ.

McNeely, J. A. (1993) Parks for life: report of the IVth World Congress on National Parks and ProtectedAreas, 10±21 February 1992 (Gland, Switzerland: IUCN in collaboration with WWF).

Miller, K. R. (1984) The Bali action plan: A framework for the future of protected areas, in: J. A.McNeely and K. R. Miller (Eds) National parks, conservation, and development: the role ofprotected areas in sustaining society: proceedings of the World Congress on National Parks, Bali,Indonesia, 11±22 October 1982 (Washington, DC: Smithsonian Institution Press), pp. 756±764.

O'Faircheallaigh, C. & Ryan, B. (1992) Program evaluation and performance monitoring: An Australianperspective (Melbourne: MacMillan).

Rump, P. C. (1996) State of the environment reporting: source book of methods and approaches (Nairobi,Kenya: United Nations Environment Program).

Stolton, S., Dudley, N. & Hakizumwami, E. (2001) Assessing Protected Area Management Effectiveness inthe Congo Basin (Gland, Switzerland: Forest Innovations Project, WWF/IUCN).

Strong, A. L., Durning, M., Foster, C., Rowntree, R., Schulman, S. & Stoddard, C. (1972) The nationalparks system: ways and means. National Parks for the future, TC Foundation. (Washington DC: TheConservation Foundation).

The Nature Conservancy (1998) Measures of conservation success: revised draft handbook (Washington,DC: The Nature Conservancy).

Thorsell, J. W. (1982) Evaluating effective management in protected areas: an application to Arusha NationalPark, Tanzania (Gland, Switzerland: IUCN Commission on National Parks and Protected Areas).


Management Effectiveness: Assessing Management of Protected Areas

Documents

Transcript of Management Effectiveness: Assessing Management of Protected Areas