Evaluability Assessment: A Catalyst for Program Change and Improvement

16
10.1177/0163278703252264 ARTICLE Evaluation & the Health Professions / June 2003 Thurston et al. / EVALUABILITY ASSESSMENT Using a local cross-cultural health service program as a framework, the authors describe the process of an evaluability assessment (EA) and illustrate how it can be a catalyst for pro- gram change. An EA is a process that improves evaluation. The key product was a logic model, which traces the links between objectives, activities, and outcomes. Four key insights emerged. First, the distinction of who was included and excluded in the target popu- lation, originally ambiguous, was clearly defined. Second, through the development of the logic model, staff members were able to analyze their goals and assumptions and criti- cally explore possible gaps between expected outcomes and activities. Third, the EA enabled reflection on and clarification of both process and outcome measures. Finally, global goals were pared down to better match the project capacity. Developing an evaluability assessment was a cost-effective way to collaborate with staff to develop a clearer, more evaluable project. Keywords: evaluability; mental health; cul- tural competence; immigrant 206 EVALUATION & THE HEALTH PROFESSIONS, Vol. 26 No. 2, June 2003 206-221 DOI: 10.1177/0163278703252264 © 2003 Sage Publications AUTHORS’ NOTE: We would like to thank the staff and clients of the Calgary Cross- Cultural Mental Health Consultation Project. Several colleagues provided feedback and asked helpful questions throughout the evaluability assessment. We thank Kathy Dirk for copyediting. Finally, we thank the anony- mous reviewers and the editor for very helpful suggestions. Correspondence may be sent to Dr. Wilfreda E. Thurston, Associate Profes- sor, Department of Community Health Sci- ences, Faculty of Medicine, University of Cal- gary, 3330 Hospital Dr. NW, Calgary, Alberta, Canada, T2N 4N1; e-mail: thurston@ ucalgary.ca. EVALUABILITY ASSESSMENT A Catalyst for Program Change and Improvement WILFREDA E. THURSTON JENNIFER GRAHAM University of Calgary JENNIFER HATFIELD Hatfield Consulting Group, Calgary, Alberta

Transcript of Evaluability Assessment: A Catalyst for Program Change and Improvement

10.1177/0163278703252264

ARTICLEEvaluation & the Health Professions / June 2003

Thurston et al. / EVALUABILITY ASSESSMENT

Using a local cross-cultural health service

program as a framework, the authors describe

the process of an evaluability assessment (EA)

and illustrate how it can be a catalyst for pro-

gram change. An EA is a process that

improves evaluation. The key product was a

logic model, which traces the links between

objectives, activities, and outcomes. Four key

insights emerged. First, the distinction of who

was included and excluded in the target popu-

lation, originally ambiguous, was clearly

defined. Second, through the development of

the logic model, staff members were able to

analyze their goals and assumptions and criti-

cally explore possible gaps between expected

outcomes and activities. Third, the EA

enabled reflection on and clarification of both

process and outcome measures. Finally,

global goals were pared down to better match

the project capacity. Developing an

evaluability assessment was a cost-effective

way to collaborate with staff to develop a

clearer, more evaluable project.

Keywords: evaluability; mental health; cul-

tural competence; immigrant

206

EVALUATION & THE HEALTH PROFESSIONS, Vol. 26 No. 2, June 2003 206-221

DOI: 10.1177/0163278703252264

© 2003 Sage Publications

AUTHORS’ NOTE: We would like to thank

the staff and clients of the Calgary Cross-

Cultural Mental Health Consultation Project.

Several colleagues provided feedback and

asked helpful questions throughout the

evaluability assessment. We thank Kathy Dirk

for copyediting. Finally, we thank the anony-

mous reviewers and the editor for very helpful

suggestions. Correspondence may be sent to

Dr. Wilfreda E. Thurston, Associate Profes-

sor, Department of Community Health Sci-

ences, Faculty of Medicine, University of Cal-

gary, 3330 Hospital Dr. NW, Calgary, Alberta,

Canada, T2N 4N1; e-mail: thurston@

ucalgary.ca.

EVALUABILITY

ASSESSMENT

A Catalyst for Program Change

and Improvement

WILFREDA E. THURSTON

JENNIFER GRAHAMUniversity of Calgary

JENNIFER HATFIELDHatfield Consulting Group,

Calgary, Alberta

Evaluability assessment (EA) is not new. Authors have described it

as “a set of procedures for planning evaluations so that stake-

holders’ interests are taken into account in order to maximize the util-

ity of the evaluation” (Rossi & Freeman, 1993, p. 104; Wholey, 1977).

Early writers on health program evaluation recommended “a rapid

feedback evaluation” to determine the evaluability of a program

(Horst, Nay, Scanlon, & Wholey, 1979). It has been argued that this

body of work needs to be rejuvenated and that the evaluation process

should start with the program planning process (Thurston & Potvin,

2002).

The desired products of an EA are a thorough description of the

program, the key questions to be addressed by the evaluation, an eval-

uation plan, and an agreement among the stakeholders on all of these

(Rutman, 1977; Wholey, 1977). In the best of worlds, an EA would be

conducted as a parallel process to program planning from the proposal

writing stage to the end of the program. In this way, the two processes

of EA and program planning would be linked and would inform each

other (Thurston & Potvin, 2002). One of the aims of EA is to prevent

“useless evaluation attempts” (Horst et al., 1979), but as this article

will show, program improvement may be the most significant outcome.

BACKGROUND

The Calgary Cross-Cultural Mental Health Consultation Project

(CMHP) was established to increase access to mental health services

(particularly upstream, nonacute services) for immigrants and to

increase the cultural competence of mental health practitioners in Cal-

gary. The hospital that the CMHP originated from is located in a part

of the city known to have a high proportion of residents who recently

immigrated to Canada; hence, this hospital has sponsored a number of

initiatives regarding ethnocultural diversity in that setting.

The CMHP complements and builds upon an earlier pilot project,

the Multicultural Awareness Program (MAP), initiated at the same

hospital between 1999 and 2000. The MAP focused on improving

access to and delivery of culturally competent care to immigrant pop-

ulations across a wide range of health care services in the Northeast

only, constituting a model for the CMHP. The CMHP focuses specifi-

cally on increasing awareness and beginning the process of providing

Thurston et al. / EVALUABILITY ASSESSMENT 207

culturally competent mental health service across the region. Both

projects were stimulated by a series of needs assessments conducted

regionally that showed that immigrants tended to underutilize ser-

vices, making use of acute care services in emergency departments

rather than earlier preventative or community services (V. Wasil, per-

sonal communication, February 2002).

In the MAP pilot project, a cultural awareness model was used as a

point of entry. Planners believed that awareness activities would be

nonthreatening and would build trust and buy-in so that future projects

could address more critical thinking and skill development. More

recently, the CMHP has integrated a more complex model of cultural

competency into its project philosophy. Whereas formerly the focus

was on increasing knowledge that was, in turn, believed to increase

sensitivity (Chin, 1999; Hoang & Erickson, 1985; Welch, 1997), the

focus has evolved to incorporate knowledge, attitudes, and skill devel-

opment for culturally competent service delivery (Chin, 1999; Welch,

1997).

Cultural awareness operates on the premise that understanding cul-

tural norms of people from specific ethnocultural groups would

enable providers to be more sensitive to clients from a particular back-

ground (Hoang & Erickson, 1985; Ogunranti, 1995). Sensitivity and

awareness were based on the notion that “doctors who have alien [ital-

ics added] patients need to know about alien [italics added] cultures”

(Ogunranti, 1995, p. 67). Several assumptions underlie the cultural

awareness model, which can lead to an artificial sense of confidence

and sensitivity but which may actually be counterproductive.

METHOD

To meet program expected outcomes efficiently, the CHMP com-

missioned an evaluation. The purpose of conducting the EA with the

CHMP was to increase the utility of the evaluation. A framework was

used for conducting the EA developed by Thurston (1991) and based

on the work of Wholey (1977), Rutman (1977), and Rossi and Free-

man (1989). The framework includes seven elements: (a) bounding

208 Evaluation & the Health Professions / June 2003

the program by identifying goals, objectives, and activities that make

up the program; (b) reviewing documents; (c) modeling resource

inputs, intended program activities, intended impacts, and assumed

causal links; (d) scouting the program or getting a firsthand look at

how it operates; (e) developing an evaluable program model; (f) iden-

tifying evaluation users and other key stakeholders; and (g) achieving

agreement to proceed on an evaluation.

Several methods were used to gather data. The written proposal for

the CMHP as well as any reports from CMHP and the earlier MAP

pilot were collected and reviewed. These documents provided a

framework for subsequent interviews and guided the development of a

first draft logic model. A logic model is a chart that traces the flow of

reasoning linking goals to activities to outcomes (both expected and

unexpected) in order to identify any discrepancies or gaps (Mohr,

1995; Rossi & Freeman, 1993; Unrau, 1993).

By seeking multiple viewpoints, the authors attempted to deter-

mine both expected and unexpected outcomes from project activities

and the priorities of different stakeholders. Semistructured interviews

were conducted (and tape-recorded) with all project staff. These inter-

views enabled the authors to understand how the program implement-

ers viewed and prioritized the activities, goals, and outcomes of their

program. Small focus group sessions, lasting over an hour, were held

with clients from the previous MAP pilot project to gain insight into

program activities and outcomes, as understood by program clients.

Focus groups with health practitioners were difficult to schedule due

to the busy timetables of these individuals; therefore, group sizes were

reduced to approximately three participants. Throughout the data col-

lection, researchers were in frequent contact with CMHP staff to ask

for clarification and elaboration.

The authors met frequently to discuss findings and to decide on

subsequent steps in gathering information from and/or reporting

information to key stakeholders. In this way, the EA was iterative.

This iterative process allowed project staff to verify the interpretations

made, thus increasing validity of the data. Two of the authors worked

part-time for approximately one and one-half months to collect all the

data necessary for the EA.

Thurston et al. / EVALUABILITY ASSESSMENT 209

RESULTS

BOUNDING THE PROGRAM

The interviews, focus groups, and document review provided the

evaluators with a solid description of what the goals and objectives

and the planned activities of the project were. This enabled the evalua-

tors to work with the staff to try to bound the project or to delimit the

boundaries of the sphere of influence of the CMHP. Although pro-

gram staff kept in mind the overall goal of creating an accessible, cul-

turally competent mental health care system, close critical examina-

tion during the bounding of the program led to refocusing on a smaller

scope for the project. Bounding the program required a balance of the

tension between the capacity of the small program (small number of

staff and budget) and the desire to achieve the overall goal. The CHMP

has two distinct branches of activities, those targeting recent immi-

grants and those targeting mental health practitioners. The CMHP was

also intended to serve as a means of linking the mental health sector

and another local initiative, the Culturally Competent Professionals

Network.

IDENTIFYING KEY STAKEHOLDERS

One of the main elements of an EA is identifying the people who

have a central role to play in the program and who may use a future

evaluation in making decisions about the program. There were four

categories of key stakeholders involved with the CMHP: the project

staff, external support agencies, people in the mental health sector,

and immigrant populations.

One full-time and two part-time staff of the Calgary Health Region

(CHR) were running the CMHP, and a research consultant was con-

tracted to assist in the development of an evaluation and monitoring

framework. The full-time staff member was responsible for planning

of initiatives, facilitating workshops, and serving as a liaison between

practitioners and immigrant service organizations. The part-time staff

members provided administrative and program support.

External support came from funding organizations, political lead-

ers who supported the project through policy, other regional health

authorities, and other mental health projects. The mental health sector,

210 Evaluation & the Health Professions / June 2003

for the purposes of this project, was comprised of mental health practi-

tioners, volunteers, and interpreters within the Calgary Health

Region. The immigrant population was represented by the Culturally

Competent Professionals Network, a group of visible minority physi-

cians dedicated to providing culturally competent care for the diverse

populations of Calgary; community organizations with culturally

competent strategies; and immigrant-serving agencies.

DEVELOPING THE LOGIC MODEL

Various staff, because of their discipline and focus, identified com-

ponents of the logic model that others may have missed or stated dif-

ferently. As they were being identified, components of the model were

linked together to follow the logic of the project developers. The logic

model identified several areas in which there were possible gaps.

These are identified in Table 1. The logic model was combined with a

summary of the background of the CMHP, and this was presented to

the project staff for their feedback. The staff and evaluators used the

logic model as a tool for looking at the program with a critical lens. It

allowed them to deconstruct program assumptions, recommend pro-

gram changes, and revise problematic objectives.

Uncovering and articulating assumptions helped the staff to

explore the connections between planned activities and program

objectives. For example, to achieve a culturally competent mental

health system, CMHP staff identified the need to eliminate barriers to

access. The program, however, had up to that point focused on

increasing practitioner knowledge and skills (limited to skills in work-

ing with a translator), clearly not adequate in achieving systemwide

cultural competence (Table 1). As a result of discussions, it was

decided that workshops should at least include opportunities for more

critical self-reflection by practitioners on their own attitudes.

LESSONS LEARNED DURING THE EA

The EA acted as a catalyst for change for the CMHP. There were

four key insights gained through doing the EA: (1) target clients of the

program were not clearly defined; (2) the logic model contained gaps

between planned activities and expected outcomes, and underlying

assumptions required review and clear articulation; (3) a shift from a

Thurston et al. / EVALUABILITY ASSESSMENT 211

TABLE 1

Analysis of the Causal Model Reflected in Proposed Activities Prior to the EA

Objective/Activitya

Purposeb

Assumptions Made in Linking the Purpose to the Planned Activity

Establishing mental health

service links

To increase awareness among practitioners

of the existence of the CMHP

If practitioners know about the services of the CMHP they will make

use of them. There exists a lack of dialogue and support for mental

health workers (and providing that support will lead to better service).

Education and consultation

with/for health providers

To assess current levels of awareness/

knowledge and develop a curriculum and

modules for training

Through teaching the front line workers, the mental health system

will become more culturally competent. Multicultural education

complements the client-centered approach, a policy adopted by the

region.

Culturally competent mental

health services

To develop a culturally competent mental

health service through informal strategic

interventions and teaching modules

Resistance to change by practitioners has not been planned for at this

point. Change can be driven up from the front line providers. An

initiative sanctioned by administration will effect change in the

system (there is a role for administration in the sustainability of

the initiative)

Establishing a community

network with organizations

providing services for

immigrants

To establish contacts with groups and

organizations working with immigrants

in Calgary

Community organizations will be a valuable source of information

about immigrant clients for mental health practitioners.

212

Education and consultation with

communities

To educate different ethnocultural groups

about the kinds of mental services in

Calgary and how to access them and to

provide a vehicle for feedback on the

mental health system from ethnocultural

groups.

If immigrants know about services and how to access them they will

make more timely use of services (i.e., lack of knowledge is a key

barrier to accessing services). By targeting four ethnocultural

groups—South Asian, Chinese, Southeast Asian, and Newcomer—

the majority of the immigrant population currently underutilizing

the mental health system will be reached. Recent immigrants will be

able to articulate their needs in a group setting in English.

Promotion of mental health

services and prevention of

mental health crises

To educate immigrant populations about

prevention of mental health crises and to

promote noncrisis services

The current service delivery mechanisms will be appropriate for

immigrants (there is a recognition that services must adapt to better

meet needs, and the project is seeking to identify these needs).

Preventing/averting crises is a better use of resources and easier on

individuals and families. Lack of knowledge is effecting utilization

of services.

Improved confidence in mental

health services

To increase the level of confidence of

immigrant families in the competence

and effectiveness of the mental health

services available in Calgary

Increased knowledge and increasingly culturally competent mental

health practitioners will lead to increased confidence in the services

by immigrants. Increased confidence will lead to increased (and

more timely) utilization.

NOTE: EA = evaluability assessment; CMHP = Calgary Cross-Cultural Mental Health Consultation Project.

a. The objective or activity stated in the project documentation. Objectives were similar to activities; therefore, the first column contains only the objectives as

originally stated.

b. The purpose or expected outcome of the activity.

213

process orientation to an outcome orientation was needed; and (4)

broad goals needed to be pared down to make them more congruent

with available resources.

1. Defining the Target Clients

The project staff revealed that they were interested in increasing the

level of awareness and skills of practitioners to enable them to better

deal with diversity; however, the project goals and mission statement

specified a focus on dealing with immigrants. Project staff eventually

identified their target population as immigrants, and after further dis-

cussion, the target population was focused even more to recent immi-

grants, particularly from non-English speaking countries.

2. Gaps in the Logic Model

For the CMHP, evaluators and staff used the logic model to bring to

light several gaps between planned activities and what the goals and

objectives identified as expected outcomes. The project activities used

a community development model to a) improve immigrant and refu-

gee knowledge and utilization of mental health services and b)

develop culturally competent mental health care providers. The

planned activities, however, focused on the development of skills,

knowledge, and attitudes surrounding a much broader population.

Thus, although the target population specified was recent immigrants,

the activities focused on visible minorities, particularly people from

diverse linguistic, religious, and ethnocultural backgrounds. There

was, therefore, a gap between the specified target group and the

expected outcomes of planned activities. Other gaps in theory devel-

opment were also identified.

Table 1 illustrates some of the assumptions uncovered in the con-

struction of the logic model. The first problematic assumption was

that mental health practitioners who knew about the CMHP services

would use them. This is problematic, as it may hold true only for prac-

titioners who value cultural competence and are motivated to provide

culturally competent care. Those practitioners, however, who have not

214 Evaluation & the Health Professions / June 2003

yet reached a level of awareness where they feel culturally competent

care is necessary and who would therefore benefit greatly from the

CMHP might not access the program.

The EA provided a mechanism to investigate if the CMHP was

making assumptions that had been found to be problematic in similar

projects in other jurisdictions (e.g., a common assumption that educa-

tion leads to behavior change). Behavior change is a complex process,

of which knowledge is only a minor component (Crosby, DiClemente,

Wingood, & Harrington, 2002; Duncan, Jones, & Moon, 1996). One

common assumption made in projects with this emphasis is that if

immigrants only knew about services, they would use them. Utiliza-

tion of services will not increase, however, if they are not provided in a

culturally competent way, transportation is impossible, or culturally-

based stigma exists against using such services. Although project

developers were aware of this problematic assumption, the only activ-

ities planned at this point were educational in nature.

Another issue brought forward during the EA was that cultural

awareness building activities, although potentially motivating and

generating interest in the project, could have negative unintended con-

sequences of stereotyping and generating negative cultural compari-

sons. The latter could actually act to compromise the cultural compe-

tence of individuals.

Closely linked to the assumption that practitioners who know about

CMHP services will use them was the final problematic assumption

that needed to be critically analyzed by project staff. The assumption

was that resistance to change by practitioners would be minimal and

not act as a barrier to the systemic change hoped for in the project.

Although this assumption would hold true for many mental health

practitioners, not all practitioners will be open to cultural competency

training or the idea of culturally competent care. The individuals who

resist the notion of cultural competency are arguably the most impor-

tant people for the project. There has been a recognition that cultural

competence can be a threatening subject for some people, and cultural

awareness models have been used in the pilot project as an entry point

to try to increase enthusiasm and trust to lead to more analytical, skill-

focused workshops.

Thurston et al. / EVALUABILITY ASSESSMENT 215

3. Shifting From a

Process to Outcome Orientation

A major development that came about through the EA was the real-

ization that the project was focused on measuring processes rather

than outcomes. Process-oriented indicators measure whether activi-

ties have taken place. Outcome-oriented indicators measure the effect

of project activities on target groups. It is important to record pro-

cesses for replicability of projects; however outcome-oriented indica-

tors will provide information on the effectiveness of the interventions.

Although the previous MAP pilot project had focused on processes

(number of people trained, number of consultations, or whether activi-

ties had taken place), the staff foresaw the need for measuring out-

comes to assist them in understanding the effects of their interven-

tions. Knowledge of the outcomes of the project was sought to help in

securing funding for the future. The staff also wanted information that

would help them improve their project activities.

During the EA, the project staff reworked their goals to focus more

on expected outcomes rather than processes, for example, changes in

the ways mental health practitioners use community service agencies

as part of their overall treatment plan. The staff were able to clearly

articulate the expected outcomes of the project activities, which led to

outcome-focused objectives.

4. Realistic Goal Development

The logic model, in particular, was a useful tool for assisting project

staff in paring down the global goals of the CMHP. Although the pro-

ject aimed to create a culturally competent mental health system in the

region, the logic model allowed the staff to see the discrepancy

between planned activities, staff and other resources, and the over-

reaching ultimate goal of the project. With only three staff and 2 years

of funding, the scope of the global goals was inappropriately large. A

gap was also identified between the planned activities and the original

goals that specified the expected outcomes. The project staff realized

that either the activities or the goals had to change in order to bridge

the gap. After the submission of a draft of the EA, the project staff

began to examine the scope of the CMHP, and by the completion of the

216 Evaluation & the Health Professions / June 2003

EA, the goals had been pared down to better suit the resource capabili-

ties of the project.

The EA also enabled project staff to realize that whereas cultural

competence training is used to develop skills and awareness for

diverse working environments, the skills and attitudes necessary for

working with recent immigrant clients are somewhat different. The

educational component for the mental health practitioners, planned to

start the following year, was based on broader models of cultural com-

petence without identifying skills specific to practitioners working

with recent immigrant clients. The project staff realized that, although

improving practitioners’ skills to competently deal with immigrant

clients was a starting point for increasing the cultural competence of

the organization, the CMHP was not capable of dealing with such an

enormous task with only one full-time program staff. At the end of the

EA, project staff members were conducting a needs assessment with

mental health practitioners to learn about their readiness for cultural

competence training and were reviewing current literature to develop

a more specific list of key skills and attitudes important in working

with recent immigrants.

DISCUSSION

Mohr (1995) argued that a good analysis of the impact of a program

is partially dependent on understanding the problem to be addressed,

the ultimate outcomes sought, the outcomes of interest to the program,

and the outcomes associated with each program activity and how

these are related to the outcomes of interest. In fact, what Mohr and

others (Chen, 1990; Macaskill et al., 2000; Milstein, Wetterhall, &

CDC Evaluation Working Group, 2000; Rossi & Freeman, 1993;

Unrau, 1993; Wimbush & Watson, 2000) required of evaluators is a

good understanding of program theory. Mohr defined a program the-

ory as that which “tells what is to be done in the program and why—

what is to result from the program and how. It is, in short, a testable

assertion that certain program activities and subobjectives will bring

about specified results” (p. 18).

Problems arise in program development and subsequently in evalu-

ation when people fail to adequately describe the problem to be

Thurston et al. / EVALUABILITY ASSESSMENT 217

addressed. This failure can lead to a number of missteps (Mohr, 1995):

(a) becoming fixated on outcomes that are not related to the problem,

(b) inability to see that one outcome may actually represent several

problems that require alternative solutions or activities, (c) stating and

therefore measuring outcomes that do not represent the real possibili-

ties of the program, (d) not addressing the issue of how extensive a

program must be to solve the stated program, and (e) losing sight alto-

gether of the social good that was intended to result from the program.

Clearly, the program theory articulated through a logic model is criti-

cal to developing the evaluation plan (Wimbush & Watson, 2000). It is

also critical for utilization of evaluation findings by people outside of

the program who need to know what worked, for whom, and where.

The results of the evaluability assessment described here were sim-

ilar to the results of a formative evaluation and hopefully will prevent

the missteps described above. The EA itself and the iterative process

of writing the EA acted as a catalyst for change within the CMHP.

Through the building of a logic model, gaps between what the objec-

tives would achieve and the overall goal of the project were exposed.

The process of writing the EA facilitated a parallel process by project

staff of tailoring objectives. The result was the ongoing examination

of objectives and goals to pare them down to be realistic, achievable,

and outcome based.

Bounding the project, partly through identification of key stake-

holders, allowed the staff of the CMHP to realize the project strengths

and limitations. A snapshot of the planned activities, the way activities

matched with objectives was crucial for critical analysis of the

intended direction for the CMHP. Through the EA, project staff were

able to more clearly articulate their target populations and expected

outcomes. Measuring the effectiveness of a project is only possible

when objectives are outcome based. The EA allowed CMHP project

staff to better tailor their objectives to the expected outcomes. Work

was ongoing at the end of this EA to develop indicators that would

measure the achievement of project objectives and expected

outcomes.

For the CMHP, developing a logic model highlighted several

important gaps between activities and anticipated outcomes, which

needed to be addressed in order for the project to work toward achiev-

ing the goals stated in their project literature. By bringing to light the

underlying assumptions of the CMHP, project staff were able to

218 Evaluation & the Health Professions / June 2003

examine the assumptions, challenge them, and make appropriate

changes to the activities or goals. The link between activities and goals

is bridged with assumptions that require critical analysis before pro-

gram activities proceed, and the logic model was a tool that facilitated

the analytical process. The need for a clearer understanding of the

complex web of barriers that are restricting access to mental health

services for recent immigrants was identified.

There were some difficulties encountered during the EA. Commu-

nication was sometimes difficult because both program staff and eval-

uators had many responsibilities other than this evaluation. It may

have been more effective to have regular meetings with program staff

and all evaluators. Furthermore, as the EA progressed, the context

changed as the health authority to which the hospital was responsible

announced a new diversity initiative. This increased the desire on the

part of CMHP champions that it be seen as a positive program. Among

themselves, the evaluators discussed the need for sensitivity and to

plan their communications with the CMHP so as to minimize the

chances that staff commitment, skills, and values were being

questioned.

The CMHP, although based on an earlier pilot program, was under

development and changing and adapting to the environment. The EA

provided valuable insight, and staff implemented changes to resolve

problems making written reports somewhat redundant. The process

rather than the EA report was effective, causing some to question the

need for and value of a report. The evaluators limited the time spent on

writing an in-depth report of the EA but saw it as a historical document

that might be important for future reference. Further, the evaluators

had to clarify more than once the purpose of an EA. The discourse of

best practices and evidence-based medicine has heightened awareness

of the need for outcomes evaluation, further obscuring the need for the

type of evaluation work described here. An EA cannot assess whether

a program is meeting its goals but focuses more on whether the goals

are appropriate.

There was disagreement among program staff and evaluators as to

whether some of the gaps in logic identified during the EA warranted

attention. In a climate in which there is great ambiguity in terms relat-

ing to culture, race, and ethnicity (Krieger, 1992), it is easy for pro-

gram planners to overlook important differences in terms and there-

fore subpopulations. The evaluators questioned whether a focus on

Thurston et al. / EVALUABILITY ASSESSMENT 219

immigrants was appropriate and thought this could lead mental health

practitioners to confuse the needs of immigrants with those of other

populations of ethnic minorities. It was not possible for evaluators and

program staff to agree on all issues, however, and the evaluators took

the position that they could provide an evaluation framework that pro-

gram staff could use to make strategic decisions but were not in a posi-

tion to insist on a given approach. In the end, the EA led to a clear defi-

nition of target clients, something that is imperative for developing

activities that will meet their needs and achieve the overall goals of the

project.

CONCLUSION

EA is a cost-effective strategy for organizational change. It cannot

ignore the political tensions that frequently surround health programs

or program evaluation and requires the same attention to human rela-

tionships as other evaluation activities. Because of its iterative nature

and focus on program development, it creates a relatively safe space

for the critical examination of assumptions and can operate as a cata-

lyst for change. Whether the changes are optimal and the program is

successful remains to be assessed in other evaluation activities. Some-

times the evaluators and program planners will disagree. For the rela-

tively small cost of two part-time staff over 2 months, the project dis-

cussed in this article was able to clearly identify client groups, refocus

goals and objectives, and work toward developing indicators that

would measure expected outcomes rather than simply processes, all of

which would result in an improved project. EA is a cost-effective solu-

tion recommended by the authors for projects in formative stages

looking for assistance in direction, evaluation, and setting up a moni-

toring system.

REFERENCES

Chen, H. T. (1990). Theory-driven evaluation. Newbury Park, CA: Sage.

Chin, J. L. (1999). Cultural competence and health care in Massachusetts: Where are we?

Where should we be? (Issue Brief 5). Massachusetts: Massachusetts Health Policy Forum.

Retrieved Feb. 12, 2002, from www.ihp.brandeis.edu/mhpf/pages/publications.htm

220 Evaluation & the Health Professions / June 2003

Crosby, R. A., DiClemente, R. J., Wingood, G. M., & Harrington, K. (2002). HIV/STD preven-

tion benefits of living in supportive families: A prospective analysis of high risk African-

American female teens. American Journal of Health Promotion, 16, 142-145.

Duncan, C., Jones, K., & Moon, G. (1996). Health-related behavior in context: A multilevel

modeling approach. Social Science & Medicine, 37, 725-733.

Hoang, G. N., & Erickson, R. V. (1985). Cultural barriers to effective medical care among

Indochinese patients. Annual Review of Medicine, 36, 229-239.

Horst, P., Nay, J. N., Scanlon, J. W., & Wholey, J. S. (1979). Program management and the fed-

eral evaluator. In H. C. Schulberg & F. Baker (Eds.), Program evaluation in the health fields

(Vol. 2, pp. 73-109). New York: Human Sciences Press.

Krieger, N. (1992, Winter). The making of public health data: Paradigms, politics, and policy.

Journal of Public Health Policy, 412-427.

Macaskill, L., Dwyer, J. J. M., Uetrecht, C., Dombrow, C., Crompton, R., Wilck, B., et al. (2000).

An evaluability assessment to develop a restaurant health promotion program in Canada.

Health Promotion International, 15, 57-69.

Milstein, B., Wetterhall, S., & Centers for Disease Control Evaluation Working Group. (2000).

A framework featuring steps and standards for program evaluation. Health Promotion Prac-

tice, 1, 221-228.

Mohr, L. B. (1995). Impact analysis for program evaluation (2nd ed.). Thousand Oaks, CA:

Sage.

Ogunranti, J. O. (1995). Cultural and biological diversity in medical practice. World Health

Forum, 16(1), 66-68.

Rossi, P. H., & Freeman, H. E. (1989). Evaluation: A systematic approach. Newbury Park, CA:

Sage.

Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury

Park, CA: Sage.

Rutman, L. (1977). Formative research and program evaluability. In L.Rutman (Ed.), Evaluation

research methods: A basic guide (pp. 59-71). Beverly Hills, CA: Sage.

Thurston, W. E. (1991). Program planning and evaluation. Unpublished course documents,

University of Calgary, Canada.

Thurston, W. E., & Potvin, L. (2002). Evaluability assessment: A tool for incorporating evalua-

tion in social change programs. Manuscript submitted for publication.

Unrau, Y. (1993). A program logic model approach to conceptualizing social service programs.

Canadian Journal of Program Evaluation, 8, 117-134.

Welch, M. (1997). Culturally competent care for diverse populations. Retrieved February 12,

2002, from http://dgim.ucsf.edu/pods/html/divpeople.html

Wholey, J. S. (1977). Evaluability assessment. In L. Rutman (Ed.), Evaluation research meth-

ods: A basic guide (pp. 41-56). Beverly Hills, CA: Sage.

Wimbush, E., & Watson, J. (2000). An evaluation framework for health promotion: Theory,

quality and effectiveness. Evaluation, 6, 301-321.

Thurston et al. / EVALUABILITY ASSESSMENT 221