http://evi.sagepub.com/Evaluation

http://evi.sagepub.com/content/3/3/345The online version of this article can be found at:

 DOI: 10.1177/135638909700300306

1997 3: 345EvaluationCaroline Costongs and Jane Springett

Towards a Framework for the Evaluation of Health-related Policies in Cities  

Published by:

http://www.sagepublications.com

On behalf of: 

  The Tavistock Institute

can be found at:EvaluationAdditional services and information for    

  http://evi.sagepub.com/cgi/alertsEmail Alerts:

 

http://evi.sagepub.com/subscriptionsSubscriptions:  

http://www.sagepub.com/journalsReprints.navReprints:  

http://www.sagepub.com/journalsPermissions.navPermissions:  

http://evi.sagepub.com/content/3/3/345.refs.htmlCitations:  

What is This? 

- Jul 1, 1997Version of Record >>

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

345

Towards a Framework for theEvaluation of Health-related Policiesin Cities

CAROLINE COSTONGS AND JANE SPRINGETTInstitute for Health, Liverpool John Moores University

The aim of this paper is to develop an evaluation framework which will offersome guidance in evaluating health-related policies at the city level. Theemphasis of the framework lies on the evaluation of the processes andconditions under which health-related policies are formulated, implemented,facilitated or restrained. Several key issues that underpin the proposedevaluation framework are discussed, such as the politics associated with theevaluation process, the use of indicators, participation of city inhabitants,validity issues and the skills evaluators should have. It is argued that theevaluation process itself is an important element in the formulation and theimplementation of these policies and that the particular nature of health-related urban policy demands an open, negotiated and process-orientatedapproach.

Introduction

Health policy is often associated with policy that deals primariiy with health-careservices and the treatment of disease and illness and, in the United Kingdom, withpolicies relating to the National Health Service. Since the early 1980s with a revival ofpublic health there has, however, been a return to an awareness of the impact of theenvironment, lifestyles and social inequalities on health (Blackman, 1995) and a

resulting demand for health-related public policies (Ottawa Charter for Health

Promotion, 1986: WHO, 1987). In 1992 the British government responded with itsnational long-term health strategy in ’The Health of the Nation’ White Paper as acontribution to the reorientation of health policy away from curative services towardsdisease prevention and health promotion (Blackman, 1995). However, already in theUnited Kingdom and other parts of Europe, as well as in Canada and elsewhere, therewas emerging through networks such as the UK Health for All Network (established in1987) and the WHO Healthy City project an acknowledgement by municipal author-ities of their role in creating healthy public policy (Blackman, 1995). Thus it is

EvaluationCopynght © 1997

SAGE Publications (London,Thousand Oaks and New Delhi)

[ 1356-3890 (199707) 3:3]Vol 3(3) 345-362

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

346

increasingly accepted that there are various public sectors in a city which influence thehealth of its inhabitants, for example employment, education, housing, water and airpollution and poverty.

Accordingly, this paper favours a broad, holistic and ecological approach to healthwhich takes a whole range of different urban factors into account to explain health inthe urban context. This broader definition of health implies that a focus on health policyalone is not sufficient. Therefore, the theme ’health-related urban policy’ will be

emphasized throughout this paper and will be the focus of the evaluation framework.First, the benefits of an evaluation framework will be explained, followed by a

discussion of the different theoretical issues that are important to consider beforesetting up an evaluation of health-related policies. Based on this discussion, the

evaluation framework will be presented in the form of a table that consists of an eight-step evaluation process which is then applied to the three phases of the policy cycle,that is, policy formulation, policy implementation and policy outcome. The frameworkis seen as a contribution to the development of participatory approaches to evaluation(Fetterman et al., 1996). It is also seen as providing an aid to the development of health-related policies in practice.

The Benefits of an Evaluation Framework

One approach to the evaluation of health-related policies is the development of a set ofstandards or criteria by which polices may be judged. An example is the checklistdeveloped by WHO for assessing the quality of health promotion services and

programmes (WHO, 1994). In it there are 70 questions which reflect the key attributesof the Ottawa Charter. While such a checklist may contribute to the development ofindicators of good practice, it cannot be used in isolation. A good evaluation shouldprovide information for the development and improvement of urban policies andtherefore can have a positive influence on health factors in the city. Through carryingout an evaluation within an evaluation framework that sets general guidelines andincludes essential steps, it is possible to understand the conditions favouring andimpeding the policy process within the city (Ziglio, 1991). Within such an evaluationframework, there are a range of possible evaluation questions that might be addressed.For example, what barriers must policies for health overcome and what structures andprocesses make it possible to achieve changes in the complex urban environment(Tsouros and Draper, 1993)? Why are policies for health effectively planned andimplemented in one city with a particular context and environment, and yet the samestrategies may prove inapplicable in another city (Ziglio, 1991)? How is the publicresponding to changed policy initiatives in the city? Is there a question of an’implementation gap’ (Gill and Thrasher, 1985), that is, the existence of majordifferences between theory and practice related to urban policies relevant to health? Aproper evaluation framework is needed in order to search for answers to these questionsin such a way as to enhance the decision-making process (Long and Wilkinson, 1984)and thereby to develop and to improve policy formulation and implementation in theurban context.

In addition to providing essential information, evaluation in itself can be a useful tool

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

347

to facilitate communication within the policy process. According to Minogue (1983)policy-makers cannot have direct first-hand knowledge of the problems they are calledupon to solve.

On the other hand, the implementers, who know these problems, do not have enoughpower to adjust and to improve this policy. In order to overcome this gap, theevaluation framework includes strategies to involve policy-makers as well as imple-menters, to facilitate their communication, to find solutions to their problems and toprovide them both with information.

Finally, within an evaluation framework it is possible to assess policy effectiveness, tominimize the waste of resources (Downie et al., 1990) and to decide whether or not thepolicy should continue. The knowledge of the extent to which health factors are changedpositively in a city is essential for the improvement of health-related urban policies.

In practice, the political context of the evaluation intertwined with the beliefs of theevaluator her/himself and the theory underlying the policy initiative (Chen, 1990) willguide the selection of a particular approach to evaluation, what questions are asked andhow outcomes are measured. Nonetheless it is possible to develop guidelines as to whatsorts of questions should be asked during the evaluation process. Thus, at best, a

general evaluation framework, such as will be developed here, is merely a set ofguidelines that includes questions to be addressed at different stages of the process ofevaluation. The approach used here has some similarities to the generic programmelogic model which has been used in the development of an evaluation framework forhealth promotion programmes (Rootman et al., 1996).

Issues in the Evaluation of PolicyIt must be stressed that the evaluation framework presented here is not a ’cookbook’method for doing policy evaluation, but takes account of the assumptions andunderlying theories (Chen, 1990) of the health-related policy process. O’Neill andPederson (1992) have argued that policy analysis is often driven more by a preferencefor technique (e.g. cost-benefit analysis, survey research, multiple regression, systemanalysis etc.) than by the underlying theory upon which the techniques are based. Theyalso argued that the complexity of healthy public policy puts additional demands on thepolicy process. This requires greater use of qualitative methods supplemented asnecessary by quantitative data, for more attention to values, for wider participation inpolicy-making, for an explicit assessment of policy impact on health and for a greaterunderstanding of the policy process.The framework here tries to be sensitive to different research paradigms, the political

and organizational contexts, and to hidden agendas of stakeholders. It considers intendedas well as unintended implications of policy. Because of the nature of the decision-making and policy implementation process and because of the broad area of health-related policies in cities which includes various public sectors, the development of anappropriate framework is dependent on the consideration of a number of key issues.

Positivist-only evaluationThe positivist paradigm, in which logical-deductive, quantitative measurements,

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

348

randomized controlled experiments and a valid and a value-free objectivist science arehighly rated, is still the dominant paradigm in health sciences and often for thoseworking in the field of urban health. It is reflected in the current emphasis on evidence-based practice, performance indicators and economic appraisal in the health sector(DOH, 1995). The latter are essentially managerial tools for resource control wherebymanagers are seen as central actors in the policy process. Implicit is the assumption thatsuch systems of performance measurement are sources of policy improvement and thatthis can be achieved by managers alone without reference to other actors in the processor to external changes (Considine, 1994: 248). Interpretation of the former has beendominated by a medical science perspective reflecting power structures within thehealth-care sector (Springett et al., 1995). In other sectors, however, there is now agrowing awareness of the limitations of a positivist approach and there is a move awayfrom a reliance on rigid ’scientific’ methodologies to more interpretative, process-oriented methods with an emphasis on learning (for example: Bamekov et al., 1990;Fetterman et al., 1996; Fuerstein, 1986; Hoggart et al., 1994; Knox and McAlister,1994; Mathison, 1994; Means and Smith, 1988; Winje and Wewell, 1992). In theUS, for example, there has developed an increasing disillusionment with positivistapproaches in the evaluation of urban policy. Many of the evaluative studies

undertaken in the 1970s and early 1980s have had little impact on policy or practice.This resulted in a retreat from quantitative studies to qualitative assessments, use of aneclectic mix of evaluative methods and a concern for developing learning networks aswell as actively involving all those who participate in policy development andimplementation (Hart, 1992).

Ethically, it can be argued that a positivist-style evaluation collects data withoutmaking changes in the situation of the respondents or giving them something in return(Reason, 1994). The aim is to test a hypothesis which the respondents do not knowabout. To avoid bias it is necessary to keep them in the dark about the purpose of theevaluation.From a technical perspective it can be argued that the more rigorous the method-

ology, the greater the artificial control and manipulation. This leads to even morereactivity from the respondents, which consequently means that the outcomes of theevaluation will be more misleading (Chen, 1988). Furthermore, because of this

continuing emphasis on rigour and objectivity, positivist research ignores that all

science is value-bound (Guba and Lincoln, 1989) and that the evaluator is not a neutralscientific observer (Thompson, 1992).

In terms of policy relevance, it is argued that rigorously designed evaluations willnot influence the policy process just for the reason that they are scientific, objective orvalid. According to Rist (1994), this linear relation of evaluation to action does notexist. In practice the real world of policy-making and implementation often throws upconditions which resist positivist methods since policy systems are complex structuresfor political learning comprising of many subcultures and values (Considine, 1994;Weiss, 1972). It is important to consider the values, aspirations and motivations of thepeople involved, which cannot be quantified (Leeuw, 1989; Ziglio, 1991).

Given these considerations the framework advocates a flexible, negotiated andprocess-oriented approach to the evaluation of urban health-related policies. This does

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

349

Figure 1. Kolb’s Learning Cycle (after Kolb, 1984)

not mean that outcome measures, qualitative or quantitative, are abandoned but ratherall the participants in the process are encouraged to enter into a dialogue about what isappropriate and what is possible, thus encouraging reflection on existing perspectivesand ideologies. This is the approach which lies at the centre of action research

(Gustavsen, 1992; Springett and Leavey, 1995; Winter, 1990), see Figure 1.

PoliticsThe political context and power structures within a city crucially influence the design,the implementation and the findings of an evaluation of a health-related policy. Politicsdetermine the theory underlying the policy, and therefore affect what can be evaluated,how it is evaluated and whether recommendations will be adopted (Chen, 1990;Downie, et al., 1990). For example, an assumption can be made that ill health is relatedto lifestyles of individuals (victim-blaming) rather than to factors outside the individual(such as poverty). As a consequence, Conservative governments are more likely to putemphasis on individuals and changes in lifestyles than on external, urban factors whenevaluating health issues (Ham, 1991). Furthermore, policy-makers are not very eager tohave too critical an examination of their failures (Minogue, 1983); thus, even whenevaluators are not asking for it, they will always have political influence (Greene,1994). In order to carry out an evaluation of health-related policies, the evaluator has tobe sensitive to the stories behind the policies, power struggles, personality conflicts andfeelings of mutual distrust. The evaluation framework will recognize that political and

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

350

personal preferences affect the direction and utilization of the evaluation process(Patton, 1982).

ParticipationBelief in the evaluation findings is increased when people ’own’ the results and

understand them, which is enhanced by their involvement in the process of evaluation(Patton, 1982). Therefore, public participation is one of the key concepts of theproposed framework. Those who are affected by a policy initiative are often the peoplewho know the problems and how to act upon them. However, in practice it is difficultto involve city inhabitants in evaluation projects, especially at the policy level. AsStarmans (1993) showed in his action research project, local people become frustratedwith the lack of action on matters that concern them and can be unwilling to participate.According to Quellet et al. (1994), local people will get involved only when they see aglimmer of a solution or when they feel empowered. Participation plays a role ingenerating solidarity and continuous reconciliation of conflicting values inside a policysystem. It has an instrumental as well as developmental value. It is a means to improvedecision-making and can bind people together. In the academic literature that

developmental role has tended to be undervalued (Considine, 1994) despite its inherentcapacity-building functions in terms of knowledge development and understanding astrust solidarity and conflict resolution. The evaluation framework recognizes the idealof participation and the practical realities. The evaluators themselves have to findcreative solutions (Reason, 1994).

IndicatorsOne way of gaining information on the policy process is through developing andinterpreting evaluation indicators. However, Hancock argues:

While there has been some agreement that we need a relatively limited number of indicatorsthat are holistic, subjective as well as objective, focus on both process and status, are

relatively easy to collect and use at both the city and small-area level, and carry social andpolitical punch, there has been little or no agreement on what those indicators should be.(Hancock, 1993: 23)

There has been little agreement because many indicators of all kinds are alreadyavailable to measure almost any dimension of health-related policy concepts (O’Neill,1993). Moreover, the indicators to be used depend on the purpose of the evaluation andits context. They vary according to who requests the indicators to be selected, who paysfor such a selection, who uses it and what are their agendas (O’Neill, 1993). Hence, theprocess of developing and using indicators to evaluate health-related policies in cities ismore a political problem than a technical one. In any city, people have different visionsof the world, different definitions of health and different power to impose it on others,which will suggest different ways to define indicators (Baum, 1992; O’Neill, 1993).The evaluation framework proposed asserts that indicators are context-bound and thateach city will have to develop its own set of indicators to meet its own unique needs,and that it is not possible to develop a set of broadly agreed upon indicators that allowinter-city comparison (Hancock, 1993) or permit generalizing. For example, at the timeof writing the Joint Public Health Team in Liverpool are currently developing a set of

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

351

health impact indicators to monitor the Liverpool Health Plan which reflects the aimsand objectives and underlying theories which drive the plan. An attempt to developgeneralizable indicators to be used in the rest of the North West Region provedimpossible. In short, it is important that indicators are not developed in isolation fromthe political and theoretical basis underlying the specific policy to be evaluated. Thecomplexity involved in developing indicators needs to be acknowledged.

Validity, reliability and generali,zability _Although issues of validity, reliability and generalizability are not the key focus in aprocess-oriented approach to evaluation (Chen, 1988), any evaluator still wants to

know if he/she is measuring what he/she intends to measure (validity), if there will besimilar results when using alternative methods or the same method twice (reliability),and if the evaluation findings are applicable to a more general population (general-izability). However, the latter does not fit comfortably with a process-orientedevaluation of often uniquely identified health-related policy initiatives. According to deLeeuw (1989) the concept of generalizability does not have much potency in

qualitative policy analysis. Also, she argues, the achievement of high validity in

qualitative evaluation is problematic, since crucial variables and conditions will

change, especially in dynamic settings such as cities. Indicators, for example, areessentially static variables that try to measure a dynamic process. In this context, Ziglio(1991) warns against overemphasizing measurement; the indicators selected may givea simplified and misleading representation of the reality. Validity and reliability arealso in danger when people are involved in evaluating their own work since this canintroduce bias (Smithies and Adams, 1993). As Quellet et al. (1994: 158) point out: ’itis not easy to criticise a project in which you believe and have invested a great deal oftime and effort.’

Several authors argue for triangulation techniques in order to enhance the credibilityof the evaluation findings (Denzin and Lincoln, 1994; Greene, 1994; Leeuw, 1993;Means and Smith, 1988; Pederson et al., 1988). Triangulation involves the use ofdifferent evaluators, multiple methods and/or diverse data sources, since each datasource tends to be interest- and ideology-bound (Means and Smith, 1988). De Leeuw(1993) adds that an ongoing feedback to all stakeholders during the evaluation is

another acceptable form of validation. Greene (1994) mentions member checks, peerdebriefs and audits. Finally, Beattie (1991) argues that the best guarantee of validityand reliability in an evaluation is to involve both an internal and external element in theevaluation of an urban policy initiative, an argument also supported by Baum (1992).

Evaluator’s skillsThe conceptual framework presented here takes into account that evaluators havecompeting and often contradictory tasks to fulfil. The evaluator should be involvedwith the policy initiative in order to collaborate effectively with local workers, but atthe same time he/she must keep enough distance to be critical and ask the rightquestions (Sweep, 1993). Also the evaluator must find the balance between the public’s sright to know and the individual’s right to privacy (Greene, 1994). Therefore, a wholerange of skills such as communication skills, negotiation, creativity, political and

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

352

facilitation skills are needed by evaluators (Quellet et al., 1994; Reason, 1994). AsLong and Wilkinson (1984) pointed out, they must have the ability to carry out anevaluation, while coping with the complexities of social programmes and conflictingpersonalities. In addition, evaluators should play a more active role in the policyprocess (Greene, 1994); they must go beyond their evaluation to sell their ideas topolicy-makers (Ham, 1991). Only publishing in journals is not enough. This is

especially true with qualitative evaluations which are often underread in comparisonwith the compact tables and summaries of quantitative work (Richardson, 1994). Theevaluator has to present his/her ideas and recommendations clearly and attractively tomake an impact. Ham (1991) remarks that communication and salesmanship must gohand in hand with good academic techniques if the evaluator wants to influence

policy.

The Health-related Policy ProcessIt is almost impossible to provide a complete analysis of the entire policy process.Flynn (1993) therefore recommends concentrating on key elements and strategicissues. However, in order to be able to make health-related changes visible on all

levels, the proposed framework will cover the whole process from health-relatedpolicy-making until its actual outcome. The smaller individual evaluation projects thatwill be carried out within this framework have then the task to focus and concentrate on

pertinent issues. For example, a city developing a health plan may decide to use onespecific part of the framework and evaluate only the plan’s outcome in a particular areathat is most relevant to the development of future policy.One possible model for demonstrating how the policy process can be evaluated and

how evaluation can be used within the policy process is Rist’s policy cycle (1994). Ristcharacterizes the policy cycle as consisting of three phases (Rist, 1994), namely: policyformulation, policy implementation and policy outcome.’

I

There is some disagreement amongst policy researchers whether formulation, imple-mentation and outcome should be isolated as three separate processes, since policy isshaped while being implemented (Hupe, 1990; O’Toole, 1986) and in complex policysituations it is difficult to isolate the relationship between specific policies and specificoutcomes. Barrett and Fudge (1981) have talked about a policy-action continuum,without making a distinction between the different phases. Sabatier (1986) howeverargues that obliterating the distinction between policy formulation and implementationprecludes policy evaluation and the analysis of change, as there is never a defined policyat tO which changes into another defined policy at tl. Much depends on whether a top-down or bottom-up perspective on policy is adopted. Most policy-makers workingwithin current institutional frameworks adopt a top-down perspective, so embedded areWeberian ideas about hierarchical organization and management (Barrett and Fudge,1981). If the framework proposed is to be modified and developed to aid evaluation inpractice, such a framework needs to start from that perspective, but at the same timeencourage a movement away to other ways of viewing the world. A strength of Rist’s(1994) model is that it makes a complex concept such as policy understandable and itprovides a clear overview that can serve as a basis for a framework which also adopts a

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

353

process approach and thus can go some way to combining the two perspectives. Whilethere is an important area of overlap between the three phases, it is useful to evaluatethem relatively independently. Each of these phases has its own policy actors andinformation requirements, and in each phase different evaluation questions can beaddressed. Notwithstanding the potential shortcomings of adopting such a linear pro-cess, the framework takes Rist’s (1994) model of the policy cycle and applies it to

policies that have an impact on the health of a city’s inhabitants.

The Evaluation Framework

Given the above discussion, the framework developed here and presented in Table 1

builds on a process-oriented approach to the evaluation of health-related policies in thecity. The evaluation process is considered as a continuous activity and consists of thefollowing eight basic steps: 1. Observe and do some groundwork. 2. Establish thepurpose of the evaluation. 3. Design the evaluation protocol. 4. Decide whether or notto use indicators. 5. Carry out the data collection. 6. Interpret the data obtained. 7. Givefeedback to all people involved. 8. Undertake the actions derived from the evaluationfindings. Throughout every phase of the policy-cycle (formulation, implementation andoutcome), there are activities that need to be undertaken and key questions that need tobe addressed at each evaluation step. To make sure that the same activities are not

repeated for every policy phase, the framework identifies those elements that arecommon to all phases of the policy process and puts them into the first column of thetable. The other three columns deal with those issues that a -e particular to each of thethree phases of the policy process.

Phase One in the Policy Cycle: Policy FormulationIn the first phase of the policy cycle, evaluation is needed to understand and to defineclearly the health-related issue. Evaluation can unfold the problems or conditionspolicy-makers are faced with when trying to formulate a policy that takes health intoaccount. In addition, it is important for policy-makers to be aware of the most

appropriate policy strategy to achieve improved health. A way of doing that is to

evaluate previous policies that have been formulated in the past in response to thisparticular health issue, or the policy instruments that have been used at that time, andthe consequences these policy initiatives have had. This would help policy-makers tochoose among present-day options. However, as Rist (1994: 549) noted: ’the windowfor policy formulation is frequently very small and open only a short time. Theinformation that can be passed through has to be ready and in a form that enhancesquick understanding.’

Phase Two in the Policy Cycle: Policy ImplementationThe focus in the second phase of the policy cycle is on the day to day realities ofbringing a new policy into existence (Rist, 1994). For example, is the policy beingimplemented as planned (Pirie, 1990)? Have new formal or informal inter-organiz-ational networks developed (Milio, 1988)? An evaluation can judge if there is enoughinstitutional capacity and expertise to respond effectively to the new policy, if this

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

354

x

03aro

w

0’S

’5

>MHE-o-a,

SE!:,~

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

355

bM

Ir_r_0u

I.N-Q

E!~ll

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

356

T30

c0u

I~lk?

3E!Z&dquo;

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

357

’0u

-Ec0U

I-a!::t5E~

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

358

0

ë0U

I-~

vi~

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

359

policy is acceptable to the implementing officials and what aspects of the policy are notoperable. Since target populations and conditions related to health within a city arechanging continuously, an ongoing and process-oriented evaluation is able to helpimplementers to match and adjust their activities to the present circumstances.

Phase Three in the Policy Cycle: Policy OutcomeRist (1994) argues that when the policy is sufficiently implemented and (health)outcomes are given enough time to show up, an evaluation of impacts and outcomescan be carried out. However, it is not easy to see whether the impact or outcome can beascribed to the particular policy initiative or to other factors. It will be difficult to

isolate effects of one particular policy, because often there is a variety of policyinitiatives at the same time. Moreover, there are historical, environmental, social,economical and political factors in a city that will also influence the specific outcome ofthe policy initiative.

Conclusions

Little has been published on the extent to which attempts to reorientate local policiestowards a health dimension have been successful or to what extent they are beingevaluated. Such attempts to move from health-care policy to health policy are in theearly stages of development, and are mostly apparent in cities who have had Health forAll on the political agenda for five or more years (Goumans, 1995; Goumans andSpringett, forthcoming). The evaluation framework presented here is offered as a

contribution to understanding the process of health-relattd policy development, toproviding a learning tool for those developing such policies and to the process ofgetting health on other agendas. It rests on two elements: the policy cycle-that is,policy formulation, implementation and outcome-and a process-oriented approach toevaluation. Several issues important to the evaluation of health-related urban policieshave been reviewed and taken into account in this framework. The evaluation

framework provides guidelines and ways of understanding how the health-relatedpolicy process might work within a city and what conditions facilitate or hinder it.

However, the policy process is very complicated, and while evaluation is useful, it willnot explain necessarily all decisions, activities and outcomes that have taken place, norwill the findings influence the policy process automatically. Rather the frameworkproposed here is a first attempt to provide policy-makers and policy managers, in thewidest sense, in urban areas with supportive information on how to evaluate and

improve health-related policies in a flexible, negotiated and process-oriented way. Ithas yet to be used in practice but the ideas behind it are currently helping to inform thedevelopment of an evaluation framework for Liverpool’s City Health Plan. Implement-ing and testing the framework in full in a variety of contexts will form the basis offuture research.

AcknowledgementsThe authors would like to thank their colleagues from the City Health Research Network, funded

by the EC, and Lmdsey Dugdill for their comments on an earlier draft of this paper

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

360

Note

1. Rist (1994) called this phase ’policy accountability’.

References

Barnekov, T., D. Hart and W. Benfer (1990) US Experience in Evaluating Urban Regeneration.London: HMSO.

Barrett, S. and C. Fudge, eds (1981) ’Examining the Policy-Action Relationship’, in Policy andAction, 3-32. London: Methuen.

Baum, F. (1992) ’Research for Healthy Cities: Some Experiences from Down-under’, in E. deLeeuw, M. O’Neill, M. Goumans and F. de Bruijn (eds) Healthy Cities Research Agenda.Proceedings of an Expert Panel. RHC Monograph Series No. 2. Maastricht.

Beattie, A. (1991) ’The Evaluation of Community Development Initiatives in Health Promotion:a Review of Current Strategies’, in Roots and Branches, papers from the OU/HEA 1990Winter School on Community Development and Health. Milton Keynes: The Open Uni-versity.

Blackman, T. (1995) Urban Policy in Practice. London: Routledge.Chen, H. T. (1988) ’Validity in Evaluation Research: a Critical Assessment of Current Issues’,

Policy and Politics 16(1): 1-16.Chen, H. T. (1990) Theory-driven Evaluations. Newbury Park, CA: Sage.Considme, M. (1994) Public Policy, a Critical Approach. Melbourne: Macmillan.Denzin, N. K. and Y. S. Lincoln, eds (1994) Handbook of Qualitative Research. London: Sage.DOH (1995) Policy Appraisal and Health: A Guide from the Department of Health. London: UK

Department of Health.Downie, R. S., C. Fyfe and A. Tannahill (1990) Health Promotion, Models and Values, pp.

65-82. Oxford: Oxford University Press.Fetterman, D. M., S. J. Kaftanan and Wandersman, eds (1996) Empowerment Evaluation,

Knowledge and Tools for Self Assessment and Accountability. London: Sage.Flynn, R. (1993) ’Restructuring Health Systems: a Comparative Analysis of England and the

Netherlands’, in M. Hill (ed.) New Agendas in the Study of the Policy Process. London:Harvester Wheatsheaf.

Fuerstein, M. T. (1986) Partners in Evaluation. New York: Macmillan.Gill, M. and M. Thrasher (1985) ’Problems in Administering Community Policing: Some

Lessons from Implementation Analysis’, Policy and Politics 13(1): 37-52.Goumans, M. (1995) ’Putting Concepts into Health Policy’, in N. Bruce, J. Springett, J. Hodgkiss

and A. Scott Samuel (eds) Research and Change in Urban Community Health. Aldershot:Avebury.

Goumans, M. and J. Springett (forthcoming) When Projects Become Policies: the Experience ofHealthy Cities in the UK and the Netherlands.

Greene, J. C. (1994) ’Qualitative Program Evaluation, Practice and Promise’, in N. K. Denzinand Y. S. Lincoln (eds) Handbook of Qualitative Research. London: Sage.

Guba, E. G. and Y. S. Lincoln (1989) Fourth Generation Evaluation. London: Sage.Gustavsen, B. (1992) Dialogue and Development. Assen: Van Gorcum.Ham, C. (1991) ’Analysis of Health Policy; Principles and Practice’, Scandinavian Journal of

Social Medicine Supplement No 46: 62-6.Hancock, T. (1993) ’The Healthy City from Concept to Application. Implications for Research’,

in J. K. Davies and M. P. Kelly (eds) Healthy Cities, Research and Practice. London:

Routledge.

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

361

Hart, D. (1992) ’US Urban Policy Evaluation in the 1980s: Lessons from Practice’, RegionalStudies 25(3): 255-65.

Hoggart, R., S. Jeffries and L. Harrison (1994) ’Reflexivity and Uncertainty in the ResearchProcess’, Policy and Politics 22(1): 59-70.

Hupe, P. L. (1990) ’Implementing a Meta-policy: The Case of Decentralization in the

Netherlands’, Policy and Politics 18(3): 181-91.Knox, C. and D. McAlister (1994) ’Evaluation: Incorporating User Views’, Public Administra-tion 73: 413-15.

Kolb, D. A. (1984) Experiential Learning: Experience as a Source of Learning and Develop-ment. Englewood Cliffs, NJ: Prentice Hall.

Leeuw, E. de (1989) Health Policy; an Exploratory Inquiry into the Development of Policy forthe New Public Health in the Netherlands. Maastricht: Savannah.

Leeuw, E. de (1993) ’Epiloog: Wat is goed Gezonde Steden-onderzoek?’, in J. Ten Dam and E.de Leeuw (eds) Gezonde Steden en onderzoek: reikwijdte, methoden, toepassingen. Assen/Maastncht: Van Gorcum.

Long, R. J. and W. E. Wilkinson (1984) ’A Theoretical Framework for Occupational HealthProgram Evaluation’, Occupational Health Nursing, May, pp. 257-9.

Mathison, S. (1994) ’Rethinking the Evaluator Role: Partnerships Between Organizations andEvaluators’, Evaluation and Planning 17(3): 299-304.

Means, R. and R. Smith (1988) ’Implementing a Pluralistic Approach to Evaluation in HealthEducation’, Policy and Politics 16(1): 17-28.

Milio, N. (1988) ’Making Healthy Public Policy; Developing The Science by Learning the Art:an Ecological Framework for Policy Studies’, Health Promotion: an International Journal2(3): 263-74.

Minogue, M. (1983) ’Theory and Practice in Public Policy and Administration’, Policy andPolitics 11(2): 63-85.

O’Neill, M. and A. P. Pederson (1992) ’Building a Methods Bridge Between Public PolicyAnalysis and Healthy Public Policy’, Canadian Journal of Public Health Supplement 1 83:S25-30.

O’Neill, M. (1993) ’Building Bndges between Knowledge and Action: the Canadian Process ofHealthy Communities Indicators’, in J. K. Davies and M. P. Kelly (eds) Healthy Cities,Research and Practice. London: Routledge.

O’Toole, L. J. (1986) ’Policy Recommendations for Multi-actor Implementation: an Assessmentof the Field’, Journal of Public Policy 6(2): 181-210.

Patton, M. Q. (1982) Practical Evaluation. London: Sage.Pederson, A. P., R. K. Edwards, M. Kelner, K. R. Allison and V. W. Marshall (1988)

Coordinating Healthy Public Policy: An Analytic Literature Review and Annotated Bibliogra-phy, working paper. Ottawa: Minister of Supply and Services; Health Services and PromotionBranch.

Pirie, P. L. (1990) ’Evaluation’, in N. Bracht (ed.) Health Promotion at the Community Level.London: Sage.

Quellet, F., D. Durand and G. Forget (1994) ’Preliminary Results of an Evaluation of ThreeHealthy Cities Initiatives in the Montreal Area’, Health Promotion International 9(3):153-9.

Reason, P. (1994) ’Three Approaches to Participative Inquiry’, in N. K. Denzin and Y. S.Lincoln (eds) Handbook of Qualitative Research. London: Sage.

Richardson, L. (1994) ’Writing: a Method of Inquiry’, in N. K. Denzin and Y. S. Lincoln (eds)Handbook of Qualitative Research. London: Sage.

Rist, R. C. (1994) ’Influencing the Policy Process with Qualitative Research’, in N. K. Denzinand Y. S. Lincoln (eds) Handbook of Qualitative Research. London: Sage.

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from

362

Rootman, I., M. Goodstadt, L. Potvin and J. Springett (1996) ’A Framework for HealthPromotion Evaluation’, a background paper for the WHO/Euro Working Group on Evaluationof Health Promotion Approaches.

Sabatier, P. A. (1986) ’Top-down and Bottom-up Approaches to Implementation Research: aCntical Analysis and Suggested Synthesis’, Journal of Public Policy 6: 21-48.

Smithies, J. and L. Adams (1993) ’Walking the Tightrope. Issues in Evaluation and CommunityParticipation for Health for All’, in J. K. Davies and M. P. Kelly (eds) Healthy Cities,Research and Practice. London: Routledge.

Springett, J., C. Costongs and L. Dugdill (1995) ’Towards a Framework for Evaluation in HealthPromotion: the Importance of Process’, Journal of Contemporary Health 2: 61-5.

Springett, J. and C. Leavey (1995) ’Participatory Action Research: The Development of aParadigm, Dilemmas and Prospects’, in N. Bruce, J. Springett, J. Hodgkiss and A. ScottSamuel (eds) Research and Change in Urban Community Health. Aldershot: Avebury.

Starmans, I. (1993) ’Migranten en gezondheid’, in J. Ten Dam and E. de Leeuw (eds) GezondeSteden en onderzoek: reikwijdte, methoden, toepassingen. Assen/Maastncht: Van Gorcum.

Sweep, A. (1993) ’Process-evaluatie’, in J. Ten Dam and E. de Leeuw (eds) Gezonde Steden enonderzoek: reikwijdte, methoden, toepassingen. Assen/Maastricht: Van Gorcum.

Thompson, J. C. (1992) ’Program Evaluation within a Health Promotion Framework’, CanadianJournal of Public Health Supplement 1 83: 67-71.

Tsouros, A. and R. A. Draper (1993) ’The Healthy City Project: New Developments andResearch Needs’, in J. K. Davies and M. P. Kelly (eds) Healthy Cities, Research and Practice.London: Routledge.

Weiss, C. H. (1972) Evaluation Research. Methods for Assessing Program Effectiveness. NewYork: Prentice Hall.

WHO (1987) ’Ottawa Charter for Health Promotion (1986)’, in Health Promotion 1(4): iii-v.WHO (1994) Auditing Health Promotion, Quality Standards and Check-lists. Working paper,

Granada, 23-24 October. WHO Regional Office for Europe, Health Promotion Unit.Winje, G. and H. Wewell (1992) ’Influencing Public Health Policy through Action Research’,

Journal of Drug Issues 22: 169-78.Winter, R. (1990) Action Research and the Nature of Social Inquiry. Aldershot: Avebury.Ziglio, E. (1991) ’Indicators of Health Promotion Policy: Directions for Research’, in B. Badura

and J. Kickbush (eds) Health Promotion Research; Towards a New Social Epidemiology.Copenhagen: WHO Regional Publications. European Series No. 37.

C A R O L I N E C O S T O N G S is currently a research project officer withNOC*NSF Sector Sport en Gezondheid, The Netherlands. She was previously aresearch assistant with the Institute for Health, Liverpool John Moores University,funded by the European Capital and Mobility Fund. Her interests include theevaluation of health promotion policy and practice. Institute for Health, LiverpoolJohn Moores University. JANE S P R I N G ETT is Professor in Health andCoordinator of the Institute for Health School of Human Sciences, Liverpool JohnMoores University. She is a member of the Joint Public Health Team which isresponsible for the development of the Liverpool City Health Plan. Her interestsinclude health policy and program evaluation and multisectoral collaboration. Sheis strongly committed to participatory research and action learning. Addresscorrespondence to Institute for Health School of Human Sciences, Liverpool JohnMoores University, Trueman Building, 15-21 Webster Street, Liverpool L3 2ET,UK [email: r.j.springett@livjm.ac.uk].

at UNIVERSITY OF ALBERTA LIBRARY on April 28, 2014evi.sagepub.comDownloaded from