PLEASE SCROLL DOWN FOR ARTICLE

This article was downloaded by: [Voets, Joris]On: 12 December 2008Access details: Access Details: [subscription number 905024300]Publisher RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Public Management ReviewPublication details, including instructions for authors and subscription information:http://www.informaworld.com/smpp/title~content=t713926128

A Framework for Assessing the Performance of Policy NetworksJoris Voets; Wouter Van Dooren; Filip De Rynck

Online Publication Date: 01 January 2008

To cite this Article Voets, Joris, Van Dooren, Wouter and De Rynck, Filip(2008)'A Framework for Assessing the Performance of PolicyNetworks',Public Management Review,10:6,773 — 790

To link to this Article: DOI: 10.1080/14719030802423129

URL: http://dx.doi.org/10.1080/14719030802423129

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf

This article may be used for research, teaching and private study purposes. Any substantial orsystematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply ordistribution in any form to anyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae and drug dosesshould be independently verified with primary sources. The publisher shall not be liable for any loss,actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directlyor indirectly in connection with or arising out of the use of this material.

A FRAMEWORK FORASSESSING THEPERFORMANCE OFPOLICY NETWORKS

Joris Voets,Wouter Van Dooren andFilip De Rynck

Joris VoetsPublic Management InstituteKatholieke Universiteit Leuven

Leuven, BelgiumE-mail: joris.voets@soc.kuleuven.be

Wouter Van DoorenDepartment of Political SciencesUniversity of Antwerp

Antwerp, Belgiumand

Public Management Institute

Katholieke Universiteit LeuvenLeuven, BelgiumE-mail: wouter.vandooren@ua.ac.be

Filip De RynckDepartment of Business Administration andPublic Administration

University College GhentGhent, BelgiumE-mail: filip.derynck@hogent.be

Abstract

This article deals with the question of how to

assess policy network performance. We

propose three dimensions of policy network

performance (production, process, and re-

gime) that are derived from both performance

and network literature. Each of these dimen-

sions can be linked to the costs of policy

networks. The framework goes beyond the

one-dimensional new public management

conceptualization of performance that

focuses on efficiency and effectiveness. A

policy network assessment needs to take into

account other public principles such as

democratic quality and capacity building.

Key wordsnetwork performance, performance assess-

ment, policy network

Vol. 10 Issue 6 2008 773–790

Public Management Review ISSN 1471-9037 print/ISSN 1471-9045 online

� 2008 Taylor & Francis

http://www.tandf.co.uk/journals

DOI: 10.1080/14719030802423129

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

INTRODUCTION

It is a well-founded assumption that contemporary government is increasinglynetworked. This phenomenon is characterized as the era of collaboration by Agranoffand McGuire (2003) and the network society by Castells (1996). In particular, policynetworks (of which some are collaborative networks) have become more important inpublic governance to deal with wicked issues. As a result, we need to comprehend theperformance of such networks (Agranoff and McGuire 2001a). In order to develop thisunderstanding, we need to leave the beaten track of the traditional performanceliterature. This research tradition has conceptualized performance mainly at the macrolevel (performance of a country) or at the micro level (performance of an organizationor individual). Policy networks however fall between macro and micro and thereforetraditional approaches do not simply work. Thinking about policy network performancechallenges our conventional understanding of performance (see also Gray 1989;O’Toole 1997; Mandell 1999; Agranoff 2005; Provan and Kenis 2008). Publicadministration increasingly takes place in settings of networked actors who necessarilyrely on each other and cannot compel compliance on the part of the rest. Yetthe standard writings to which most administrators turn for advice to improveperformance devote relatively little attention to acting effectively in such situations(O’Toole 1997).

In this article, the leading research question is how to assess policy networkperformance. In the first section, we start with the conventional definition ofperformance, which relies heavily on the new public management (NPM) discourse(Hood 1991). We discuss two limitations of this NPM definition of performance,namely its narrow focus on efficiency and effectiveness, and its preoccupation withorganizational (micro) performance. We conclude the first section with somesuggestions to overcome these limitations. In the second section, the notions ofperformance and policy networks are joined together. This is the core of the argument,resulting in an assessment framework for network performance. The third section is adiscussion of how to put the framework to work as well as of the limitations of theapproach to policy network performance developed in this article.

BUILDING BLOCKS

Performance the NPM way

Performance is about intentional behaviour, what is done or what is believed to bedone (Dubnick 2005). The now dominant definition of performance is inspired bynew public management thinking. In response to fiscal and other crises, governmentspledged to do more with fewer resources – a government that works better and costsless (Gore 1993). Government reformed in the name of performance. In particular in

774 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

the UK, this led to cutback management and a reduction of the size of government(Dunleavy 1986). Other countries followed other trajectories (Pollitt and Bouckaert2004).

The NPM interpretation of performance is focused on effectiveness and efficiency.An organization is effective when it attains predetermined goals. This should be done inan efficient way, which means maximizing outputs while minimizing inputs. Performingorganizations thus keep it lean and purposeful; they match resources to defined tasks.Frugality of resource use in relation to given goals is the criterion of success, whilefailure is counted in terms of instances of avoidable waste and incompetence (Hood1991: 12). Performance in this view follows a production logic where inputs aretransformed to outputs and outcomes (Van Dooren 2006). In the remainder of the text,we will refer to this interpretation of performance as production performance. Productionperformance constitutes the first dimension of the framework to assess policy networkperformance presented in this article.

The NPM interpretation of performance is one dimensional and restricted to themicro level. In order to discuss performance of policy networks (but probably also toimprove discussions about the performance of single organizations, Van Dooren 2008),we need to extend the performance concept and make it multidimensional andmultilevel.

From a narrow focus on efficiency and effectiveness, to a multidimensionalperformance concept

Although production is an important aspect of public performance, other dimensionsexist. Unlike the profit principle in the private sector, the public sector has nobottom line unless a vague one in terms of serving the public interest (Moore 1995).Private value is expressed in monetary terms and determined through the invisiblehand of the market. In contrast, public value can seldom be expressed in monetaryterms only, and is determined through a complex and imperfect system of politicalrepresentation and bureaucracy. As a result, there is more to public performance thansetting targets and objectives, producing products and services, and evaluating theirachievement. Based on a seminal article by Hood (1991), two dimensions can beadded to the production logic, namely process performance (focused on values likefairness, honesty, and mutuality) and regime performance (focused on values likerobustness and resilience).

An important second dimension of public performance is focused on values likefairness, honesty, and mutuality. Government has to pursue honesty, fairness, andmutuality through the prevention of distortion, inequity, bias, and abuse of office(Hood 1991: 13). Government has to act in a democratic way, and democratic isbroadly defined as ‘with respect for the power of the people’ – as in the Greekroot word: demos (people) and kratos (rule or strength) (Graubard 2003).

Voets et al.: A framework for assessing the performance of policy networks 775

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

The concept of democracy thus covers more than the traditional institutions ofdemocracy. These values are institutionalized in appeal mechanisms, public reportingrequirements and ethical codes. Success and failure are measured by levels of trustand entitlement (i.e. consent, legitimacy). This performance dimension is about theprocesses of governance. This view strongly opposes the claim of productionperformance that only results matter. We will label this dimension as processperformance.

A third dimension of performance is linked to robustness and resilience of theregime. Government has to keep operating even in adverse ‘worst case’ conditions, andshould be able to adapt rapidly in a crisis (Hood 1991: 14). Reliability is often anargument for choosing public production instead of private production. In times ofstress, governments cannot go broke and start over with another business plan. On thecontrary, government has to perform when other actors do not perform. Robustnessand resilience are not only about the capacity to resist a crisis, but also about thecapacity to deal with future challenges. For instance, will governments be able to dealwith ageing populations or unrelenting globalization? Success and failure in thisdimension are counted as survival and security. We label this performance dimension asregime performance.

From a narrow focus on the micro level, to a multi-level performance concept

Micro and macro levels of analysis have different meanings in different disciplines(Schelling 1978; Hedstrom and Swedberg 1998). Here, we use a crude distinctionwhere micro is the organizational level and macro is the jurisdiction-wide level. Theperformance literature has traditionally focused on either macro or micro performance(Van Dooren 2007). Macro performance is expressed in statistical series, often underthe banner of quality of life and social indicators. Micro performance, as we argued above,has been a central discussion in the managerial reforms of the 1980s and 1990s and isfound in scorecards and management tools. Although in theory micro performance shouldadd up to macro performance, the two perspectives are seemingly decoupled. Theseparation is not so much conceptual, but institutional. Macro performance is mainlystudied by highly specialized professionals in statistical bureaus and think tanks, whilemicro performance is assessed by well-informed practitioners in a rough and readymode.

The question then is how to approach the intermediary network level (in terms ofnot referring to a state system or a single organization within a state system). Twoseemingly evident strategies are to start from the macro or the micro perspectiverespectively. To start from the macro level, we would have to identify some cross-cutting, jurisdiction-wide issues (for instance poverty), then single out that segment ofthe issue that is relevant for the network (for instance poverty in a particular region),and finally attribute network activities to the issue (what is the impact of the network

776 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

activities on poverty reduction). Each step seems to be a daunting task, and such astrategy might be criticized for being too bureaucratic and technocratic, or simplyunrealistic.

A more realistic and useful approach is to start from the micro level. The networkswe study are policy networks. Such networks commonly refer to set(s) of horizontalrelationships with a certain level of stability/endurance and of structuring, betweenmultiple actors that are relatively autonomous and faced with resource dependency,involving processes of resource exchange through bargaining and negotiation, to achievepublic purpose (see Voets 2008 for an overview). As such, policy networks do not referto state systems, but to multi-organizational structures in which certain aspects ofpublic policy are dealt with. We contend that the assessment of policy networkperformance is more closely related to micro performance assessments of organizationsthan to the statistical series of the macro studies. Therefore, we believe that the mainstep is to broaden the narrow micro view of organizational performance to includeperformance of policy networks.

Multiple levels of assessment

While it is already challenging to evaluate performance of single organizations, policynetwork performance is even more complex. According to Provan and Milward (2001:416), ‘most problematic is the fact that multiple organizations require dealing withmultiple sets of constituencies’. The question ‘effectiveness for whom’ is veryprominent in policy network performance (Provan and Kenis 2008). In a policynetwork, the private actors for instance might applaud the results of the networkbecause it takes over their agenda, while it might have negative effects for otherstakeholders or governmental actors.

For this reason, a policy network performance framework should not only includedifferent dimensions of performance, but also multiple levels of assessment. Provan andMilward (2001) suggest that network performance can be assessed at three levels: thelevel of the participants/organization; the level of the network; and the level of thecommunity.

The first level of assessment fits the micro level, in the sense that it focuses on theperformance of a policy network in relation to individual organizations. Policy networkperformance is assessed from a single actor perspective, which assesses the value of thenetwork in relation to its organizational aims. Autonomous actors participate in anetwork for their own purposes, based on an assessment of perceived costs andbenefits. The network performs as long as they feel network participation can servetheir interests (although participants can also be driven by external motivation, likelegislation (as part of the shadow of hierarchy) or the prospect of subsidies). This levelof assessment is very prominent in the network literature (Kickert et al. 1997; Provanand Milward 2001; Mandell and Keast 2007).

Voets et al.: A framework for assessing the performance of policy networks 777

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

The policy network itself is the second level to evaluate performance. This levelrefers to the network structure and formal members in the network; it is multi-organizational and focuses on the network as a whole. Provan et al. (2007) argue thatthis level is understudied in network and performance literature.

The community level as a third level takes the action range of the policy network asthe point of departure. The community level can be interpreted in terms of policysectors (e.g. welfare), target groups (e.g. drug users, children), and from a territorialperspective (e.g. stakeholders resident in a geographical focus area). A policy networkfocused on tackling drug abuse for instance involves a community made up of drugusers, care organizations, justice departments, police, and schools. These actors are partof the community on which the network activities are focused, but communitymembers are not necessarily network members or represented directly in the policynetwork. The focus on the community level reflects an increased need to get supportfrom a wide range of stakeholders (Stoker 2006: 47).

A policy network performance framework should take into account the three levelsbecause performance may differ substantially depending on the level that an analyst islooking at.

PUTTING IT ALL TOGETHER: A FRAMEWORK FOR NETWORK PERFORMANCE

Based on the arguments developed in the previous section, this section develops ageneral framework to assess policy network performance. It does so by building onexisting frameworks, dimensions and criteria present in the literature (e.g. Kickert et al.1997; Provan and Milward 2001; Boyne 2002; Mandell and Keast 2007; Skelcher et al.2008).

The framework is developed along three dimensions. The three dimensionsreiterate the types of performance discussed in the previous section, namelyproduction performance, process performance, and regime performance. These threedimensions need to be set against the costs of policy networks. The frameworkessentially structures the arguments for a proper performance assessment of policynetworks.

Production performance: Policy networks need to deliver

This dimension closely follows the NPM interpretation of performance where goalattainment is a key criterion. In this view, only results matter (Hatry 1999). Policynetworks are only successful if they are effective. Through policy networks,government has to be able to do more (e.g. more outcomes), preferably with fewerresources. Goal attainment is important for policy networks. Research shows thatwithout goal attainment of some sort, policy networks lose trust and ultimately become

778 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

irrelevant (Huxham and Vangen 2005). Goal attainment is an issue at the level of thecommunity, the policy network, and the participating organizations.

At the level of participating organizations, the key criterion is the fulfilment of self-interest (Khator and Brunson 2001: 156). This criterion can be defined more broadly aspartner satisfaction. This can be measured by asking actors to what extent they aresatisfied with the realized or intermediate solutions (Koppenjan and Klijn 2004: 126).Actors are satisfied if they can reduce uncertainty in relation to their position in thenetwork environment or if they are able to acquire (access to) resources of others.

Goal attainment at the level of the policy network adds up the level of goalattainment of participating organizations. The higher the number of participants thatfeel that their self-interest is being served by the policy network, the higher thenetwork performance is. Following the production logic, even overall networkobjectives should be seen as goals of individual organizations that are howeverpurposively shared by other organizations.

At the level of the community, a criterion for goal attainment in the eyes of thestakeholders is defined by Provan and Milward (2001) in terms of public perceptionsthat problems are being solved.

Goal attainment at one level does not imply goal attainment at another. The networkmembers can consider the policy network to perform because it fully serves their self-interest (e.g. iron triangles1), while community members outside the network mightfeel the opposite or are confronted with negative side-effects (de Rynck and Voets2006). The role of government in this respect is to ensure a balance of performance onthe different levels. If we accept representative democracy as the cornerstone ofgovernment, the evaluation of goal attainment by network or community membersshould not take precedence over the evaluation of the elected bodies of the competentgovernments. We will discuss this democratic aspect in more detail below.

Production performance in a network context does not imply that goals have to beset a priori. More likely than not, objectives will change throughout a process (Klijn andTeisman 1997: 114). The crucial question for assessing production performance iswhether goals are attained in an efficient way. These goals may be the initial goals aswell as goals that are defined during the process. They even may be goals that areformulated during the performance assessment itself. In the latter case, processes of expost rationalization or social pressure may be at stake (Koppenjan and Klijn 2004: 126).

Some scholars argue that performance of policy networks should not be related tothe idea of goal attainment at all. Nelson (2001: 93) for instance argues that goalattainment is not a performance measure in networks that formulate strategies, becausethe network interactions are strategic, not operational. The interactions within thenetwork aim at developing goals (see also Klijn and Teisman 1997: 113). We touch amiddle ground by contending that goal attainment can be an important criterion inassessing policy network performance, but it is certainly not the only one. Performancealso refers to, among others, changing systems of interactions, empoweringstakeholders, or developing new relationships (Mandell and Keast 2007). We regroup

Voets et al.: A framework for assessing the performance of policy networks 779

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

these performance arguments in the two additional categories; process performance andregime performance.

Process performance: Ensuring democratic quality

A second important dimension of network performance is related to the democraticquality of policy networks (Boyne 2002; De Rynck and Voets 2006). Policy networksare multi-organizational, and therefore the institutional arrangements of representativedemocracy are insufficient for a policy network to function democratically. Thedemocratic performance of the policy network is mostly ingrained in the processes ofpolicy making and less in the products the network delivers.

According to Skelcher et al. (2008) democratic performance of policy networksconsists of three components: legitimacy; accountability; and accordance. Eachcomponent suggests a number of criteria to assess the democratic quality of a network.

LegitimacyFor a policy network to perform democratically, it needs formal and informallegitimacy. Formal legitimacy refers to the formal authorization for the network to act oncertain policy matters (Skelcher et al. 2008). Because a policy network is consideredpart of public policy and involves public resources, it should be formally mandated todo so. An exemplary criterion is the existence of a formal declaration of support for thenetwork and its agenda. For the governmental actors in the network, a criterion is aformal decision by political representatives to set up or to engage the organization in apolicy network. At the level of the community, a measure then might be the number ofstakeholders that have issued such a declaration. A similar measure can be defined at thelevel of the network members, although their membership as such might be considereda measure of formal legitimacy. While formal legitimacy refers to a formal mandate toact, informal legitimacy is the extent to which actors also act supportive in relation to theactual functioning of the network (Skelcher et al. 2008). While such legitimacy mightbe more important, it is more difficult to assess than formal legitimacy (Pressman andWildavsky 1973). In this respect, Provan and Milward (2001: 416) define the criterion‘member commitment to network goals’ at the network level. At the community level,a similar criterion is ‘stakeholder behaviour in relation to the network’ (2001: 416).

AccountabilityAccountability is a second component of process performance (Boyne 2002). Criteriafor this component should consider two aspects, namely giving account and being heldaccountable. The policy network needs to give account to its members as well as to thecommunity. Account giving also refers to the feedback loops between representatives in

780 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

the network and their parent organizations. Communication can be considered the mainmeasure in relation to account giving. Indicators can be the number of meetings withcommunity members, the rate of participation in such meetings, and the like. Besidesgiving account, the policy network and its members should also be held accountable forthe network activities. This criterion can be measured by looking at mechanisms toappoint, to remove or to sanction network members. A measure might be the presenceof a public evaluation conference discussing the policy network and its members.

AccordanceAccordance is about asking and expressing consent (or denying it) for decisions oractions by the policy network. Accordance covers at least three criteria: issues offeredfor consent; mechanisms to give consent; and status of consent (Skelcher et al. 2008).The first criterion is about issues offered for consent: what is up for discussion? Possiblemeasures are the importance and number of issues that are offered for consent tonetwork members or to stakeholders in the network community. From a democraticperspective, the higher the number and importance of issues offered for consent to awide range of members and stakeholders, the higher the performance on this criterion.The second criterion is related to mechanisms to give consent. In a representativedemocracy, elections are the main mechanism for consent. Additional instrumentsinclude referenda, user panels, and consumer assessments (in case of service deliverynetworks). Policy network performance implies the presence and use of suitableconsent giving mechanisms. The final criterion refers to the status of consent. To whatextent does giving or denying consent have an impact on the policy network? Is there amajority mechanism or does every network member or stakeholder have a veto, or isthe status of the consent only considered advisory? A cautionary note is that privateconsent should be subordinated to consent of governmental actors with democraticbacking. The reconcilement of input/positions of non-governmental and non-electedactors with the general interest is a balancing act.

Regime performance: Networks as capacity-building

A third dimension considers the policy network as an organizational and relationalinfrastructure. This dimension reflects the robustness and resilience of the network aspart of government. Is government equipped to deal with situations of societal stressthanks to the policy network? Is government more ‘future proof’?

The first criterion is membership (Boyne 2002). Policy network performance requiresaccess to a range of resources: information; finances; competencies; productioncapacity; and legitimacy (Koppenjan and Klijn 2004). This means that the higher thenumber of members, the higher the number of relevant resources present in thenetwork. A policy network that grows or at least keeps its number of members is

Voets et al.: A framework for assessing the performance of policy networks 781

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

considered to achieve regime performance; conversely, loss of members is considered aloss of regime performance (Provan and Milward 2001: 418). Regime performancetherefore increases with membership, because the policy network joins more resources,making it more robust and resilient as it has a joint policy capacity.

The expansion of membership can however also lead to a decrease of networkperformance on other dimensions. If a certain threshold in terms of numbers is passed,co-ordination costs for instance might exceed the advantage of joining (resources of)actors in a policy network and lower production performance. Membership neitherimplies a positive behaviour or contribution of an actor in the policy process – an actormight even participate precisely to hamper a policy network to function properly.Therefore, membership as a criterion should include the way they participate in thepolicy network or are committed to the policy network (Provan and Milward 2001).This element shows how regime performance and process performance can be linked.

The second criterion for regime performance is the endurance of the policy networkor network institutionalization (Provan and Milward 2001). Klijn and Teisman (1997:117) argue that network structures themselves are an indication of policy networkperformance, as they express the wish of participants to achieve results. Theircontinued presence in the public arena therefore increases government’s robustness andresilience, as they present coalitions of government with other actors to be moreresponsive to policy challenges.

Provan and Milward (2001) extend this criterion with the degree of development ofa network administration. Regime performance then is not only related to the ability ofthe policy network to acquire an institutional position, but also its capacity to developas a proper organization. They argue that the presence, size and scope of the latterindicate the belief of actors that the network is valuable and they express theirengagement by investing in the network. The presence of a network administration cancontribute to regime performance because the network infrastructure arguably is morerobust. A more cynical view is that such a network organization is another example ofbudget-maximization, which displaces the network goals from the participants’ to theadministration’s goals. While a well-developed network administration might make thepolicy network robust and resilient, it might impede production performance.

The third criterion for regime performance is the quality of the relationships betweenactors (Provan and Milward 2001). To what extent have relationships in the policynetwork improved during the life cycle of the network? Quality can be understood interms of number/density and multiplexity of relations. The idea of density is that anextensive set of linkages in the policy network are better than a small set of linkages.The multiplexity looks at the extent to which these relations are singular (e.g. on oneissue) or plural (e.g. on different issues). Relations between actors that are only built onsingle issues indicate a lower regime performance than relations between actors thatcover multiple issues.

The quality of relations may add substantially to the capacity of government to resistcrises or to adapt to future challenges. In times of societal stress, the traditional chain of

782 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

command and routines often break down. The conventional ways of doing things are nolonger adequate and the capacity of hierarchic and unilateral action of governmentalactors to demonstrate robustness and resilience has decreased substantially (Schneider2005). Governmental actors that have relationships with other governmentalorganizations and private actors are expected to be better able to make things happenin spite of difficult circumstances, as they create groupware2 (Agranoff and McGuire2001b).

Costs: There is no such thing as a free network

Policy network performance along the three dimensions should be related to costs.While the idea of costs is often neglected in policy network literature because it is tooclosely associated with product performance and the NPM-doctrine, cost remains a keyfacet. All networks bring along costs, and they are often very costly (Huxham andVangen 2005). These costs can be related to the resources that an actor has to deploy inrelation to the network, but also the costs of the policy developed by the network.

Costs should not be interpreted narrowly in terms of money; costs also refer to, forinstance, time, production capacities, or legitimacy. A policy network and its activitiesmight cost little in terms of money, but can be very expensive in terms of time.Therefore, the potential benefits of policy networks should be weighted against theexpected costs. According to Huxham and Vangen (2005: 13) ‘seeking collaborativeadvantage is a seriously resource-consuming activity so is only to be considered whenthe stakes are really worth pursuing. Our message to practitioners and policy makersalike is don’t do it unless you have to.’

A cost–benefit analysis ex ante is however more difficult for policy networkscompared to service delivery or public–private partnerships because costs and benefitsare less tangible and typically reveal themselves in the long term. In order to besuccessful in dealing with wicked issues, policy networks therefore also require a certainlevel of risk-taking by actors (Koppenjan and Klijn 2004).

Production, process, and regime performance all come at a cost. While cost isconsidered a typical component of production performance, it is neglected in discussingprocess and regime performance. In our opinion, the cost factor should be a point ofattention for assessing all three network performance types.

In production performance, the evident measure then is cost per unit of outcome oroutput (Boyne 2002). The process performance however also implies costs: what is thetime or money invested to ensure certain levels of legitimacy, accountability, and/oraccordance? Organizing meetings, preparing brochures, doing surveys to measurestakeholder satisfaction, organizing referenda, etc. to improve process performanceobviously are not cost-free. The time required to ask for consent of stakeholders or toorganize communication as part of account giving requires resources that cannot beused elsewhere. The costs of regime performance include the network administration

Voets et al.: A framework for assessing the performance of policy networks 783

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

costs, but might also be a lack of competition between actors at the level of thecommunity.

The cost factor is linked to production, process, and regime performance, but itsconcrete shape can also differ for the three levels of assessment. What are costs at thelevel of individual organizations, at the level of the policy network, and at the level ofthe community? Are costs equally distributed among these levels, or does one levelexternalize costs to another level? Each actor can make its own assessment in terms ofcosts, but costs also need to be monitored at the network level and community level(Provan and Milward 2001).

In policy networks, there is a risk of shifting costs to actors outside the network(Kickert et al. 1997: 172–3). A policy network might achieve high productionperformance at low costs for participants but high costs for the community. Irontriangles for instance serve their members’ interests, while externalizing costs to thelarger community outside the policy network (Schattschneider 1960). Hence, costs forthe community also need to be included in an assessment of policy networkperformance. Government should ensure that there is a proper calculation of costs atthe level of the policy network and the level of the community.

In Table 1, these main dimensions and criteria are summarized.

Table 1: An assessment framework for network performance

Dimensions Components/criteria Cost factor

Production performance Goal attainment and efficiency Costs linked to production for

goal attainment

Process performance Legitimacy

– Formal

– Informal

Accountability

– Account giving

– Being held accountable

Accordance

– Issues offered for consent

– Mechanisms for consent giving

– Status of consent

Costs linked to ensure process

performance (legitimacy,

accountability, and accordance)

Regime performance Membership

Network institutionalization

– Network administrative organization

Quality of relationships

– Density

– Multiplexity

Costs linked to ensuring regime

performance (membership,

instititutionalization, relationship

quality)

784 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

LIMITATIONS AND RESEARCH IMPLICATIONS

The framework to assess policy network performance developed so far has importantlimitations. Some critical comments need to be made in order to use the frameworkproperly.

Assessment is more than measurement

It is clear that the dimensions and criteria forwarded in this article are by no means easyto quantify. It is imperative at this point to stress the distinction between performanceassessment (which can be both qualitative or quantitative) and performancemeasurement (which is quantitative). This article is focused on performance assessmentof policy networks. We contend that an assessment framework needs to start from valuesand principles of government. These values and principles should be developed intodimensions, criteria, and measures where possible. We believe that a framework basedon aspects that can be measured clearly can never be sufficient to assess performance ofpolicy networks. The framework developed above can be used to develop performancemeasures, but the latter goes beyond the scope of this article. The development ofindicators based on the performance dimensions is a main challenge for future research.

A balanced view

In network literature, there appears to be a tendency to neglect production performanceand to stress process criteria. de Bruijn and Ringeling (1997: 158) warn not to focus tooheavily on process criteria. They argue that assessing policy network performance basedon such criteria does not guarantee ‘good policy’. While process performance might behigh, the production performance might be close to zero. The quality of interactions isimportant, but if the latter is overemphasized, a risk is real that substance is neglected;policy networks might become mere chatterboxes and might even degenerate toproceduralism. The challenge for policy networks is to join production and processperformance, without neglecting the relevance of regime performance. The issue of howto balance these dimensions should lead to a lively debate among performance and policynetwork scholars. An important consideration should be taken into account whensearching for this balance, namely the phase in the life cycle of the network.

Different performances for different phases in the life cycle of networks

Policy networks are dynamic relational structures. A policy network can for instancefocus on strategy development in one phase of its life cycle, but can move to service

Voets et al.: A framework for assessing the performance of policy networks 785

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

delivery later on. A policy network is not necessarily confined to one network type forits whole lifetime. Hence, the balance of performance dimensions will shift accordingly.

This is linked to the issue of the frequency of assessment of network performance(Nelson 2001). As policy networks (and the network environment) are dynamic, oneshould define multiple moments to assess their performance. In our opinion, thereshould always be an assessment of the situation at the time the policy network isfounded or created. Too often, the performance of policy networks is questionedwithout referring to (or having a good view of) the situation before the start of a policynetwork.

During the network life cycle, the frequency of assessment then is variable. Whileactors might permanently evaluate network performance in relation to theirorganization, we consider this insufficient. A rule of thumb should be that there areperiodical reviews of network performance during the network life cycle. Howelaborate and what the frequency of such a joint assessment should be, is againdependent on the network type.

Who is the assessor?

An important issue is who the assessor is. Is there a single assessor or are there multipleassessors? If the latter is the case, how are different assessments balanced? Are differentassessments aggregated because all judgements of all actors are considered equal, or aresome more relevant than others (Kickert et al. 1997: 172–3)?

We believe that governmental actors should play a key role. The assessment ofgovernment takes precedence over that of private actors. The main argument is that apolicy network is engaged in public policy making, and therefore needs to respect thespecial position of government, even if non-governmental actors are involved.

This however does not mean that other assessments are not relevant. The idea ofpartnership that often underlies policy networks also requires a sufficient level ofacceptance of non-governmental participants’ views. The weight of governmentassessment should also depend on its portion of resources in the network (does itprovide most resources or is it a minor contributor) and the impact of the networks’policy (does it affect the community in important ways or not?).

Contingencies

It is important to take into account that network performance is also dependent oncontingent factors. As Boyne (2002: 24) stated in his study of local authorityperformance, ‘performance indicators cannot tell the whole story about local authorityperformance, because other relevant variables (such as local socio-economiccircumstances) need to be taken into account when interpreting the figures.’ This is

786 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

similar to what Healey (1997) and others argued: one needs to read and understand the‘politics of space’ to assess (the nature of) performance (or the lack thereof) in anygiven case.

While the contingency argument applies to all levels and types of performance, thedegree of complexity increases in multi-level arrangements as multiple actors, multiplelevels, and/or multiple issues make the impact of multiple contingencies moreplausible. This element should caution one to keep the expectations about policynetwork performance realistic.

Special role of government

The special position of government in policy networks has been touched upon a fewtimes, but needs to be stressed. Governments – consciously or unconsciously – operateas meta-governors in relation to policy networks (Sørensen and Torfing 2006), but theyare often also a participant in the network. While this might lead to conflicting roles,government should guard the (balance of the) three dimensions of performance at alltimes and ensure a proper cost calculation for each dimension at the network andcommunity level. In doing so, the emphasis can differ along network types or indifferent stages of a network life cycle. An important aspect of the role of governmentis to ensure that the bodies of representative democracy are properly involved. Even ifpolicy networks are developed precisely to overcome political conflict and to deal withissues at a certain distance from the political realm, the link with the latter shouldremain clear.

The role of government is also dependent on the dominant ideology. Based on theiranalysis of contemporary governing modes, Pierre and Peters (2005) for instancedistinguish five future models or scenarios, each model encompassing differentprinciples and hence a different understanding of how performance is to be perceived.These models go from an ‘etatiste’-model with government as principal actor on theone end to a ‘governance without government’-model in which states are no more thanarenas filled with self-steering arrangements consisting of private actors at the other.

Network performance in the first model will focus dominantly on the instrumentalityof networks for governmental actors. The State is or wants to be the principal actor(e.g. government as a producer, a service provider). In the latter model, government isnot the principal but a meta-governor that organizes arenas in which policy networksoperate. In such a model, the emphasis is on safeguarding a certain quality of the arena,an openness for stakeholders to participate in the arena, and rules for the policyprocesses taking place in that arena (e.g. government as a regulator) (Sørensen andTorfing 2006).

We acknowledge however that the position of government in relation topolicy network performance is not that straightforward and therefore requires morethought.

Voets et al.: A framework for assessing the performance of policy networks 787

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

CONCLUSION

This article forwarded a framework to assess policy network performance. Theframework joins three performance dimensions (production, process, and regime),suggests a set of assessment criteria, and links them to costs as a key factor.

Production performance is important because policy networks have to deliver results,even if the production logic in such a network is more complex than that of a singleorganization. Because policy networks are multi-actor, process performance is also animportant dimension. The focus on process performance should ensure the democraticnature of the network. Third, regime performance is the extent to which the networkstructure is resilient and robust. Does the policy network strengthen the capacity ofgovernment to deal with crises or future challenges? Finally, all three dimensions of networkperformance should be related to costs, because there is no such thing as a free network.

The framework presented in this article is under construction. It does howeversuggest some next steps. The scholarly agenda includes the debate on the balance ofdimensions, the elaboration for other types of networks, the issue of developingmeasures and indicators, and the role of government. In dealing with these issues, theframework also needs to be tested empirically.

NOTES1 Iron triangles are networks (commonly referred to as sub-governments) of interdependent, autonomous

decision-makers promoting policies that serve their self-interest (Schattschneider 1960).

2 Groupware is ‘group development that reaches mutual understanding and transcends hierarchy-based

communication/interaction that allows multiple cultures, procedures, and divisions of labor to come

together’ (Agranoff 2007: 213).

REFERENCESAgranoff, R. (2005) Managing Collaborative Performance: Changing the Boundaries of the State?, Public

Performance and Management Review. 29: 1 pp18–45.

—— (2007) Managing Within Networks: Adding Value to Public Organizations, Washington: Georgetown University

Press.

Agranoff, R. and McGuire, M. (2001a) ‘After the Network Is Formed: Process, Power and Performance’ in

M. Mandell (ed.) Getting Results through Collaboration: Networks and Network Structures for Public Policy and

Management, Westport, CT: Quorum Books.

—— (2001b) Big Questions in Public Network Management Research, Journal of Public Administration Research and

Theory. 11: 3 pp295–326.

—— (2003) Collaborative Public Management: New Strategies for Local Governments, Washington: Georgetown

University Press.

Boyne, G. A. (2002) Concepts and Indicators of Local Authority Performance: An Evaluation of the Statutory

Framework in England and Wales, Public Money and Management. 22: 2 pp17–24.

Castells, M. (1996) The Information Age, Vol I: The Rise of the Network Society, Oxford: Blackwell Publishers.

788 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

de Bruijn, J. A. and Ringeling, A. B. (1997) ‘Normative Notes: Perspectives on Networks’ in W. M. J. Kickert,

E.-H. Klijn and J. F. M. Koppenjan (eds) Managing Complex Networks: Strategies for the Public Sector,

London: Sage.

De Rynck, F. and Voets, J. (2006) Democracy in Area-Based Policy Networks: The Case of Ghent, American

Review of Public Administration. 36: 1 pp58–78.

Dubnick, M. (2005) Accountability and the Promise of Performance: In Search of Mechanisms, Public Performance

and Management Review. 28: 3 pp376–417.

Dunleavy, P. (1986) Explaining the Privatization Boom: Public Choice Versus Radical Approaches, Public

Administration. 64: 1 pp13–34.

Gore, A. (1993) From Red Tape to Results: Creating a Government That Works Better & Costs Less, Report of the

National Performance Review, Washington, DC: US Government Printing Office.

Graubard, S. (2003) ‘Democracy’ in Dictionary of the History of Ideas. The Electronic Text Centre at the University

of Virginia Library. Available at http://etext.lib.virginia.edu/cgi-local/DHI/dhi.cgi?id¼dv1-78 (ac-

cessed 20 July 2008).

Gray, B. (1989) Collaborating: Finding Common Ground for Multi-Party Problems, San Francisco, CA: Jossey-Bass.

Hatry, H. P. (1999) Performance Measurement: Getting Results, Washington: Urban Institute Press.

Healey, P. (1997) Collaborative Planning: Shaping Places in Fragmented Societies, Houndmills & London: MacMillan

Press.

Hedstrom, P. and Swedberg, R. (1998) Social Mechanisms: An Analytical Approach to Social Theory, Cambridge:

Cambridge University Press.

Hood, C. (1991) A Public Management for All Seasons, Public Administration. 69: 1 pp3–19.

Huxham, C. and Vangen, S. (2005) Managing to Collaborate: The Theory and Practice of Collaborative Advantage,

New York: Routledge.

Khator, R. and Brunson, N. (2001) ‘Creating Networks for Inter-Organizational Settings: A Two-Year Follow-

Up Study on Determinants’ in M. Mandell (ed.) Networks and Inter-Organizational Settings, Westport, CT:

Quorum Books.

Kickert, W. M. J., Klijn, E.-H. and Koppenjan, J. F. M. (1997) ‘Managing Networks in the Public Sector:

Findings and Reflections’ in W. M. J. Kickert, E.-H. Klijn and J. F. M. Koppenjan (eds) Managing

Complex Networks: Strategies for the Public Sector, London: Sage.

Klijn, E.-H. and Teisman, G. R. (1997) ‘Strategies and Games in Networks’ in W. M. J. Kickert, E.-H. Klijn

and J. F. M. Koppenjan (eds) Managing Complex Networks: Strategies for the Public Sector, London: Sage.

Koppenjan, J. F. M. and Klijn, E.-H. (2004) Managing Uncertainties in Networks: A Network Approach to Problem

Solving and Decision Making, London: Routledge.

Mandell, M. (1999) The Impact of Collaborative Efforts: Changing the Face of Public Policy through Networks

and Network Structures, Policy Studies Review. 16: 1 pp4–17.

Mandell, M. and Keast, R. L. (2007) Evaluating Network Arrangements: Toward Revised Performance

Measures, Public Performance and Management Review. 30: 4 pp574–97.

Moore, M. H. (1995) Creating Public Value, Cambridge: Harvard University Press.

Nelson, L. S. (2001) ‘Environmental Networks: Relying on Process or Outcome for Motivation’ in M. Mandell

(ed.) Getting Results through Collaboration: Networks and Network Structures for Public Policy and Management,

Westport, CT: Quorum Books.

O’Toole, L. J. (1997) Treating Networks Seriously: Practical and Research-Based Agendas in Public

Administration, Public Administration Review. 57: 1 pp45–53.

Pierre, J. and Peters, B. G. (2005) Governing Complex Societies: Trajectories and Scenarios, London: Palgrave

Macmillan.

Pollitt, C. and Bouckaert, G. (2004) Public Management Reform: A Comparative Analysis, Oxford: Oxford University

Press.

Voets et al.: A framework for assessing the performance of policy networks 789

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008

Pressmann, J. and Wildavsky, A. (1973) Implementation, Berkeley, CA: University of California Press.

Provan, K. G. and Kenis, P. (2008) Modes of Network Governance: Structure, Management, and Effectiveness,

Journal of Public Administration Research and Theory. 18: 2 pp229–52.

Provan, K. G. and Milward, H. B. (2001) Do Networks Really Work? A Framework for Evaluating Public-Sector

Organizational Networks, Public Administration Review. 61: 4 pp414–23.

Provan, K. G., Fish, A. and Sydow, J. (2007) Interorganizational Networks at the Network Level: A Review of

the Empirical Literature on Whole Networks, Journal of Management. 33: 3 pp479–516.

Schattschneider, E. E. (1960) Semisovereign People: A Realist’s View of Democracy in America, New York: Holt,

Rinehart & Winston.

Schelling, T. (1978) Micro-Motives and Macro Behavior, New York: Norton.

Schneider, S. K. (2005) Administrative Breakdowns in the Governmental Response to Hurricane Katrina, Public

Administration Review. 65: 5 pp515–16.

Skelcher, C., de Rynck, F., Klijn, E.-H. and Voets, J. (2008) ‘Designing Democratic Institutions for

Collaborative Economic Development: Lessons from Belgian and Dutch Cases’ in M. Considine and

S. Giguere (eds) Government and Communities in Partnership: The Theory and Practice of Local Governance and

Economic Development, London: Palgrave.

Sørensen, E. and Torfing, J. (2006) ‘Theoretical Approaches to Metagovernance’ in E. Sørensen and J. Torfing

(eds) Theories of Democratic Network Governance, London: Palgrave MacMillan.

Stoker, G. (2006) Public Value Management: A New Narrative for Networked Governance?, American Review of

Public Administration. 36: 1 pp41–57.

Van Dooren, W. (2006) Performance Measurement in the Flemish Public Sector: A Supply and Demand Approach, Leuven:

Katholieke Universiteit Leuven.

—— (2007) ‘Quality and Performance Management: Toward a Better Integration?’ in P. Delancer Julnes, F.

Stokes Berry, M. P. Aristigueta and K. Yang (eds) International Handbook of Practice-Based Performance

Management, Thousand Oaks, CA: Sage.

—— (2008) Government and the GDP; the Politics of National Accounts Statistics, Journal of Comparative Policy

Analysis, (forthcoming).

Voets, J. (2008) Intergovernmental Relations in Multi-Level Arrangements: Collaborative Public Management in Flanders,

Leuven: Katholieke Universiteit Leuven.

790 Public Management Review

Downloaded By: [Voets, Joris] At: 15:25 12 December 2008