Evaluability Assessment: A Catalyst for Program Change and Improvement
Transcript of Evaluability Assessment: A Catalyst for Program Change and Improvement
10.1177/0163278703252264
ARTICLEEvaluation & the Health Professions / June 2003
Thurston et al. / EVALUABILITY ASSESSMENT
Using a local cross-cultural health service
program as a framework, the authors describe
the process of an evaluability assessment (EA)
and illustrate how it can be a catalyst for pro-
gram change. An EA is a process that
improves evaluation. The key product was a
logic model, which traces the links between
objectives, activities, and outcomes. Four key
insights emerged. First, the distinction of who
was included and excluded in the target popu-
lation, originally ambiguous, was clearly
defined. Second, through the development of
the logic model, staff members were able to
analyze their goals and assumptions and criti-
cally explore possible gaps between expected
outcomes and activities. Third, the EA
enabled reflection on and clarification of both
process and outcome measures. Finally,
global goals were pared down to better match
the project capacity. Developing an
evaluability assessment was a cost-effective
way to collaborate with staff to develop a
clearer, more evaluable project.
Keywords: evaluability; mental health; cul-
tural competence; immigrant
206
EVALUATION & THE HEALTH PROFESSIONS, Vol. 26 No. 2, June 2003 206-221
DOI: 10.1177/0163278703252264
© 2003 Sage Publications
AUTHORS’ NOTE: We would like to thank
the staff and clients of the Calgary Cross-
Cultural Mental Health Consultation Project.
Several colleagues provided feedback and
asked helpful questions throughout the
evaluability assessment. We thank Kathy Dirk
for copyediting. Finally, we thank the anony-
mous reviewers and the editor for very helpful
suggestions. Correspondence may be sent to
Dr. Wilfreda E. Thurston, Associate Profes-
sor, Department of Community Health Sci-
ences, Faculty of Medicine, University of Cal-
gary, 3330 Hospital Dr. NW, Calgary, Alberta,
Canada, T2N 4N1; e-mail: thurston@
ucalgary.ca.
EVALUABILITY
ASSESSMENT
A Catalyst for Program Change
and Improvement
WILFREDA E. THURSTON
JENNIFER GRAHAMUniversity of Calgary
JENNIFER HATFIELDHatfield Consulting Group,
Calgary, Alberta
Evaluability assessment (EA) is not new. Authors have described it
as “a set of procedures for planning evaluations so that stake-
holders’ interests are taken into account in order to maximize the util-
ity of the evaluation” (Rossi & Freeman, 1993, p. 104; Wholey, 1977).
Early writers on health program evaluation recommended “a rapid
feedback evaluation” to determine the evaluability of a program
(Horst, Nay, Scanlon, & Wholey, 1979). It has been argued that this
body of work needs to be rejuvenated and that the evaluation process
should start with the program planning process (Thurston & Potvin,
2002).
The desired products of an EA are a thorough description of the
program, the key questions to be addressed by the evaluation, an eval-
uation plan, and an agreement among the stakeholders on all of these
(Rutman, 1977; Wholey, 1977). In the best of worlds, an EA would be
conducted as a parallel process to program planning from the proposal
writing stage to the end of the program. In this way, the two processes
of EA and program planning would be linked and would inform each
other (Thurston & Potvin, 2002). One of the aims of EA is to prevent
“useless evaluation attempts” (Horst et al., 1979), but as this article
will show, program improvement may be the most significant outcome.
BACKGROUND
The Calgary Cross-Cultural Mental Health Consultation Project
(CMHP) was established to increase access to mental health services
(particularly upstream, nonacute services) for immigrants and to
increase the cultural competence of mental health practitioners in Cal-
gary. The hospital that the CMHP originated from is located in a part
of the city known to have a high proportion of residents who recently
immigrated to Canada; hence, this hospital has sponsored a number of
initiatives regarding ethnocultural diversity in that setting.
The CMHP complements and builds upon an earlier pilot project,
the Multicultural Awareness Program (MAP), initiated at the same
hospital between 1999 and 2000. The MAP focused on improving
access to and delivery of culturally competent care to immigrant pop-
ulations across a wide range of health care services in the Northeast
only, constituting a model for the CMHP. The CMHP focuses specifi-
cally on increasing awareness and beginning the process of providing
Thurston et al. / EVALUABILITY ASSESSMENT 207
culturally competent mental health service across the region. Both
projects were stimulated by a series of needs assessments conducted
regionally that showed that immigrants tended to underutilize ser-
vices, making use of acute care services in emergency departments
rather than earlier preventative or community services (V. Wasil, per-
sonal communication, February 2002).
In the MAP pilot project, a cultural awareness model was used as a
point of entry. Planners believed that awareness activities would be
nonthreatening and would build trust and buy-in so that future projects
could address more critical thinking and skill development. More
recently, the CMHP has integrated a more complex model of cultural
competency into its project philosophy. Whereas formerly the focus
was on increasing knowledge that was, in turn, believed to increase
sensitivity (Chin, 1999; Hoang & Erickson, 1985; Welch, 1997), the
focus has evolved to incorporate knowledge, attitudes, and skill devel-
opment for culturally competent service delivery (Chin, 1999; Welch,
1997).
Cultural awareness operates on the premise that understanding cul-
tural norms of people from specific ethnocultural groups would
enable providers to be more sensitive to clients from a particular back-
ground (Hoang & Erickson, 1985; Ogunranti, 1995). Sensitivity and
awareness were based on the notion that “doctors who have alien [ital-
ics added] patients need to know about alien [italics added] cultures”
(Ogunranti, 1995, p. 67). Several assumptions underlie the cultural
awareness model, which can lead to an artificial sense of confidence
and sensitivity but which may actually be counterproductive.
METHOD
To meet program expected outcomes efficiently, the CHMP com-
missioned an evaluation. The purpose of conducting the EA with the
CHMP was to increase the utility of the evaluation. A framework was
used for conducting the EA developed by Thurston (1991) and based
on the work of Wholey (1977), Rutman (1977), and Rossi and Free-
man (1989). The framework includes seven elements: (a) bounding
208 Evaluation & the Health Professions / June 2003
the program by identifying goals, objectives, and activities that make
up the program; (b) reviewing documents; (c) modeling resource
inputs, intended program activities, intended impacts, and assumed
causal links; (d) scouting the program or getting a firsthand look at
how it operates; (e) developing an evaluable program model; (f) iden-
tifying evaluation users and other key stakeholders; and (g) achieving
agreement to proceed on an evaluation.
Several methods were used to gather data. The written proposal for
the CMHP as well as any reports from CMHP and the earlier MAP
pilot were collected and reviewed. These documents provided a
framework for subsequent interviews and guided the development of a
first draft logic model. A logic model is a chart that traces the flow of
reasoning linking goals to activities to outcomes (both expected and
unexpected) in order to identify any discrepancies or gaps (Mohr,
1995; Rossi & Freeman, 1993; Unrau, 1993).
By seeking multiple viewpoints, the authors attempted to deter-
mine both expected and unexpected outcomes from project activities
and the priorities of different stakeholders. Semistructured interviews
were conducted (and tape-recorded) with all project staff. These inter-
views enabled the authors to understand how the program implement-
ers viewed and prioritized the activities, goals, and outcomes of their
program. Small focus group sessions, lasting over an hour, were held
with clients from the previous MAP pilot project to gain insight into
program activities and outcomes, as understood by program clients.
Focus groups with health practitioners were difficult to schedule due
to the busy timetables of these individuals; therefore, group sizes were
reduced to approximately three participants. Throughout the data col-
lection, researchers were in frequent contact with CMHP staff to ask
for clarification and elaboration.
The authors met frequently to discuss findings and to decide on
subsequent steps in gathering information from and/or reporting
information to key stakeholders. In this way, the EA was iterative.
This iterative process allowed project staff to verify the interpretations
made, thus increasing validity of the data. Two of the authors worked
part-time for approximately one and one-half months to collect all the
data necessary for the EA.
Thurston et al. / EVALUABILITY ASSESSMENT 209
RESULTS
BOUNDING THE PROGRAM
The interviews, focus groups, and document review provided the
evaluators with a solid description of what the goals and objectives
and the planned activities of the project were. This enabled the evalua-
tors to work with the staff to try to bound the project or to delimit the
boundaries of the sphere of influence of the CMHP. Although pro-
gram staff kept in mind the overall goal of creating an accessible, cul-
turally competent mental health care system, close critical examina-
tion during the bounding of the program led to refocusing on a smaller
scope for the project. Bounding the program required a balance of the
tension between the capacity of the small program (small number of
staff and budget) and the desire to achieve the overall goal. The CHMP
has two distinct branches of activities, those targeting recent immi-
grants and those targeting mental health practitioners. The CMHP was
also intended to serve as a means of linking the mental health sector
and another local initiative, the Culturally Competent Professionals
Network.
IDENTIFYING KEY STAKEHOLDERS
One of the main elements of an EA is identifying the people who
have a central role to play in the program and who may use a future
evaluation in making decisions about the program. There were four
categories of key stakeholders involved with the CMHP: the project
staff, external support agencies, people in the mental health sector,
and immigrant populations.
One full-time and two part-time staff of the Calgary Health Region
(CHR) were running the CMHP, and a research consultant was con-
tracted to assist in the development of an evaluation and monitoring
framework. The full-time staff member was responsible for planning
of initiatives, facilitating workshops, and serving as a liaison between
practitioners and immigrant service organizations. The part-time staff
members provided administrative and program support.
External support came from funding organizations, political lead-
ers who supported the project through policy, other regional health
authorities, and other mental health projects. The mental health sector,
210 Evaluation & the Health Professions / June 2003
for the purposes of this project, was comprised of mental health practi-
tioners, volunteers, and interpreters within the Calgary Health
Region. The immigrant population was represented by the Culturally
Competent Professionals Network, a group of visible minority physi-
cians dedicated to providing culturally competent care for the diverse
populations of Calgary; community organizations with culturally
competent strategies; and immigrant-serving agencies.
DEVELOPING THE LOGIC MODEL
Various staff, because of their discipline and focus, identified com-
ponents of the logic model that others may have missed or stated dif-
ferently. As they were being identified, components of the model were
linked together to follow the logic of the project developers. The logic
model identified several areas in which there were possible gaps.
These are identified in Table 1. The logic model was combined with a
summary of the background of the CMHP, and this was presented to
the project staff for their feedback. The staff and evaluators used the
logic model as a tool for looking at the program with a critical lens. It
allowed them to deconstruct program assumptions, recommend pro-
gram changes, and revise problematic objectives.
Uncovering and articulating assumptions helped the staff to
explore the connections between planned activities and program
objectives. For example, to achieve a culturally competent mental
health system, CMHP staff identified the need to eliminate barriers to
access. The program, however, had up to that point focused on
increasing practitioner knowledge and skills (limited to skills in work-
ing with a translator), clearly not adequate in achieving systemwide
cultural competence (Table 1). As a result of discussions, it was
decided that workshops should at least include opportunities for more
critical self-reflection by practitioners on their own attitudes.
LESSONS LEARNED DURING THE EA
The EA acted as a catalyst for change for the CMHP. There were
four key insights gained through doing the EA: (1) target clients of the
program were not clearly defined; (2) the logic model contained gaps
between planned activities and expected outcomes, and underlying
assumptions required review and clear articulation; (3) a shift from a
Thurston et al. / EVALUABILITY ASSESSMENT 211
TABLE 1
Analysis of the Causal Model Reflected in Proposed Activities Prior to the EA
Objective/Activitya
Purposeb
Assumptions Made in Linking the Purpose to the Planned Activity
Establishing mental health
service links
To increase awareness among practitioners
of the existence of the CMHP
If practitioners know about the services of the CMHP they will make
use of them. There exists a lack of dialogue and support for mental
health workers (and providing that support will lead to better service).
Education and consultation
with/for health providers
To assess current levels of awareness/
knowledge and develop a curriculum and
modules for training
Through teaching the front line workers, the mental health system
will become more culturally competent. Multicultural education
complements the client-centered approach, a policy adopted by the
region.
Culturally competent mental
health services
To develop a culturally competent mental
health service through informal strategic
interventions and teaching modules
Resistance to change by practitioners has not been planned for at this
point. Change can be driven up from the front line providers. An
initiative sanctioned by administration will effect change in the
system (there is a role for administration in the sustainability of
the initiative)
Establishing a community
network with organizations
providing services for
immigrants
To establish contacts with groups and
organizations working with immigrants
in Calgary
Community organizations will be a valuable source of information
about immigrant clients for mental health practitioners.
212
Education and consultation with
communities
To educate different ethnocultural groups
about the kinds of mental services in
Calgary and how to access them and to
provide a vehicle for feedback on the
mental health system from ethnocultural
groups.
If immigrants know about services and how to access them they will
make more timely use of services (i.e., lack of knowledge is a key
barrier to accessing services). By targeting four ethnocultural
groups—South Asian, Chinese, Southeast Asian, and Newcomer—
the majority of the immigrant population currently underutilizing
the mental health system will be reached. Recent immigrants will be
able to articulate their needs in a group setting in English.
Promotion of mental health
services and prevention of
mental health crises
To educate immigrant populations about
prevention of mental health crises and to
promote noncrisis services
The current service delivery mechanisms will be appropriate for
immigrants (there is a recognition that services must adapt to better
meet needs, and the project is seeking to identify these needs).
Preventing/averting crises is a better use of resources and easier on
individuals and families. Lack of knowledge is effecting utilization
of services.
Improved confidence in mental
health services
To increase the level of confidence of
immigrant families in the competence
and effectiveness of the mental health
services available in Calgary
Increased knowledge and increasingly culturally competent mental
health practitioners will lead to increased confidence in the services
by immigrants. Increased confidence will lead to increased (and
more timely) utilization.
NOTE: EA = evaluability assessment; CMHP = Calgary Cross-Cultural Mental Health Consultation Project.
a. The objective or activity stated in the project documentation. Objectives were similar to activities; therefore, the first column contains only the objectives as
originally stated.
b. The purpose or expected outcome of the activity.
213
process orientation to an outcome orientation was needed; and (4)
broad goals needed to be pared down to make them more congruent
with available resources.
1. Defining the Target Clients
The project staff revealed that they were interested in increasing the
level of awareness and skills of practitioners to enable them to better
deal with diversity; however, the project goals and mission statement
specified a focus on dealing with immigrants. Project staff eventually
identified their target population as immigrants, and after further dis-
cussion, the target population was focused even more to recent immi-
grants, particularly from non-English speaking countries.
2. Gaps in the Logic Model
For the CMHP, evaluators and staff used the logic model to bring to
light several gaps between planned activities and what the goals and
objectives identified as expected outcomes. The project activities used
a community development model to a) improve immigrant and refu-
gee knowledge and utilization of mental health services and b)
develop culturally competent mental health care providers. The
planned activities, however, focused on the development of skills,
knowledge, and attitudes surrounding a much broader population.
Thus, although the target population specified was recent immigrants,
the activities focused on visible minorities, particularly people from
diverse linguistic, religious, and ethnocultural backgrounds. There
was, therefore, a gap between the specified target group and the
expected outcomes of planned activities. Other gaps in theory devel-
opment were also identified.
Table 1 illustrates some of the assumptions uncovered in the con-
struction of the logic model. The first problematic assumption was
that mental health practitioners who knew about the CMHP services
would use them. This is problematic, as it may hold true only for prac-
titioners who value cultural competence and are motivated to provide
culturally competent care. Those practitioners, however, who have not
214 Evaluation & the Health Professions / June 2003
yet reached a level of awareness where they feel culturally competent
care is necessary and who would therefore benefit greatly from the
CMHP might not access the program.
The EA provided a mechanism to investigate if the CMHP was
making assumptions that had been found to be problematic in similar
projects in other jurisdictions (e.g., a common assumption that educa-
tion leads to behavior change). Behavior change is a complex process,
of which knowledge is only a minor component (Crosby, DiClemente,
Wingood, & Harrington, 2002; Duncan, Jones, & Moon, 1996). One
common assumption made in projects with this emphasis is that if
immigrants only knew about services, they would use them. Utiliza-
tion of services will not increase, however, if they are not provided in a
culturally competent way, transportation is impossible, or culturally-
based stigma exists against using such services. Although project
developers were aware of this problematic assumption, the only activ-
ities planned at this point were educational in nature.
Another issue brought forward during the EA was that cultural
awareness building activities, although potentially motivating and
generating interest in the project, could have negative unintended con-
sequences of stereotyping and generating negative cultural compari-
sons. The latter could actually act to compromise the cultural compe-
tence of individuals.
Closely linked to the assumption that practitioners who know about
CMHP services will use them was the final problematic assumption
that needed to be critically analyzed by project staff. The assumption
was that resistance to change by practitioners would be minimal and
not act as a barrier to the systemic change hoped for in the project.
Although this assumption would hold true for many mental health
practitioners, not all practitioners will be open to cultural competency
training or the idea of culturally competent care. The individuals who
resist the notion of cultural competency are arguably the most impor-
tant people for the project. There has been a recognition that cultural
competence can be a threatening subject for some people, and cultural
awareness models have been used in the pilot project as an entry point
to try to increase enthusiasm and trust to lead to more analytical, skill-
focused workshops.
Thurston et al. / EVALUABILITY ASSESSMENT 215
3. Shifting From a
Process to Outcome Orientation
A major development that came about through the EA was the real-
ization that the project was focused on measuring processes rather
than outcomes. Process-oriented indicators measure whether activi-
ties have taken place. Outcome-oriented indicators measure the effect
of project activities on target groups. It is important to record pro-
cesses for replicability of projects; however outcome-oriented indica-
tors will provide information on the effectiveness of the interventions.
Although the previous MAP pilot project had focused on processes
(number of people trained, number of consultations, or whether activi-
ties had taken place), the staff foresaw the need for measuring out-
comes to assist them in understanding the effects of their interven-
tions. Knowledge of the outcomes of the project was sought to help in
securing funding for the future. The staff also wanted information that
would help them improve their project activities.
During the EA, the project staff reworked their goals to focus more
on expected outcomes rather than processes, for example, changes in
the ways mental health practitioners use community service agencies
as part of their overall treatment plan. The staff were able to clearly
articulate the expected outcomes of the project activities, which led to
outcome-focused objectives.
4. Realistic Goal Development
The logic model, in particular, was a useful tool for assisting project
staff in paring down the global goals of the CMHP. Although the pro-
ject aimed to create a culturally competent mental health system in the
region, the logic model allowed the staff to see the discrepancy
between planned activities, staff and other resources, and the over-
reaching ultimate goal of the project. With only three staff and 2 years
of funding, the scope of the global goals was inappropriately large. A
gap was also identified between the planned activities and the original
goals that specified the expected outcomes. The project staff realized
that either the activities or the goals had to change in order to bridge
the gap. After the submission of a draft of the EA, the project staff
began to examine the scope of the CMHP, and by the completion of the
216 Evaluation & the Health Professions / June 2003
EA, the goals had been pared down to better suit the resource capabili-
ties of the project.
The EA also enabled project staff to realize that whereas cultural
competence training is used to develop skills and awareness for
diverse working environments, the skills and attitudes necessary for
working with recent immigrant clients are somewhat different. The
educational component for the mental health practitioners, planned to
start the following year, was based on broader models of cultural com-
petence without identifying skills specific to practitioners working
with recent immigrant clients. The project staff realized that, although
improving practitioners’ skills to competently deal with immigrant
clients was a starting point for increasing the cultural competence of
the organization, the CMHP was not capable of dealing with such an
enormous task with only one full-time program staff. At the end of the
EA, project staff members were conducting a needs assessment with
mental health practitioners to learn about their readiness for cultural
competence training and were reviewing current literature to develop
a more specific list of key skills and attitudes important in working
with recent immigrants.
DISCUSSION
Mohr (1995) argued that a good analysis of the impact of a program
is partially dependent on understanding the problem to be addressed,
the ultimate outcomes sought, the outcomes of interest to the program,
and the outcomes associated with each program activity and how
these are related to the outcomes of interest. In fact, what Mohr and
others (Chen, 1990; Macaskill et al., 2000; Milstein, Wetterhall, &
CDC Evaluation Working Group, 2000; Rossi & Freeman, 1993;
Unrau, 1993; Wimbush & Watson, 2000) required of evaluators is a
good understanding of program theory. Mohr defined a program the-
ory as that which “tells what is to be done in the program and why—
what is to result from the program and how. It is, in short, a testable
assertion that certain program activities and subobjectives will bring
about specified results” (p. 18).
Problems arise in program development and subsequently in evalu-
ation when people fail to adequately describe the problem to be
Thurston et al. / EVALUABILITY ASSESSMENT 217
addressed. This failure can lead to a number of missteps (Mohr, 1995):
(a) becoming fixated on outcomes that are not related to the problem,
(b) inability to see that one outcome may actually represent several
problems that require alternative solutions or activities, (c) stating and
therefore measuring outcomes that do not represent the real possibili-
ties of the program, (d) not addressing the issue of how extensive a
program must be to solve the stated program, and (e) losing sight alto-
gether of the social good that was intended to result from the program.
Clearly, the program theory articulated through a logic model is criti-
cal to developing the evaluation plan (Wimbush & Watson, 2000). It is
also critical for utilization of evaluation findings by people outside of
the program who need to know what worked, for whom, and where.
The results of the evaluability assessment described here were sim-
ilar to the results of a formative evaluation and hopefully will prevent
the missteps described above. The EA itself and the iterative process
of writing the EA acted as a catalyst for change within the CMHP.
Through the building of a logic model, gaps between what the objec-
tives would achieve and the overall goal of the project were exposed.
The process of writing the EA facilitated a parallel process by project
staff of tailoring objectives. The result was the ongoing examination
of objectives and goals to pare them down to be realistic, achievable,
and outcome based.
Bounding the project, partly through identification of key stake-
holders, allowed the staff of the CMHP to realize the project strengths
and limitations. A snapshot of the planned activities, the way activities
matched with objectives was crucial for critical analysis of the
intended direction for the CMHP. Through the EA, project staff were
able to more clearly articulate their target populations and expected
outcomes. Measuring the effectiveness of a project is only possible
when objectives are outcome based. The EA allowed CMHP project
staff to better tailor their objectives to the expected outcomes. Work
was ongoing at the end of this EA to develop indicators that would
measure the achievement of project objectives and expected
outcomes.
For the CMHP, developing a logic model highlighted several
important gaps between activities and anticipated outcomes, which
needed to be addressed in order for the project to work toward achiev-
ing the goals stated in their project literature. By bringing to light the
underlying assumptions of the CMHP, project staff were able to
218 Evaluation & the Health Professions / June 2003
examine the assumptions, challenge them, and make appropriate
changes to the activities or goals. The link between activities and goals
is bridged with assumptions that require critical analysis before pro-
gram activities proceed, and the logic model was a tool that facilitated
the analytical process. The need for a clearer understanding of the
complex web of barriers that are restricting access to mental health
services for recent immigrants was identified.
There were some difficulties encountered during the EA. Commu-
nication was sometimes difficult because both program staff and eval-
uators had many responsibilities other than this evaluation. It may
have been more effective to have regular meetings with program staff
and all evaluators. Furthermore, as the EA progressed, the context
changed as the health authority to which the hospital was responsible
announced a new diversity initiative. This increased the desire on the
part of CMHP champions that it be seen as a positive program. Among
themselves, the evaluators discussed the need for sensitivity and to
plan their communications with the CMHP so as to minimize the
chances that staff commitment, skills, and values were being
questioned.
The CMHP, although based on an earlier pilot program, was under
development and changing and adapting to the environment. The EA
provided valuable insight, and staff implemented changes to resolve
problems making written reports somewhat redundant. The process
rather than the EA report was effective, causing some to question the
need for and value of a report. The evaluators limited the time spent on
writing an in-depth report of the EA but saw it as a historical document
that might be important for future reference. Further, the evaluators
had to clarify more than once the purpose of an EA. The discourse of
best practices and evidence-based medicine has heightened awareness
of the need for outcomes evaluation, further obscuring the need for the
type of evaluation work described here. An EA cannot assess whether
a program is meeting its goals but focuses more on whether the goals
are appropriate.
There was disagreement among program staff and evaluators as to
whether some of the gaps in logic identified during the EA warranted
attention. In a climate in which there is great ambiguity in terms relat-
ing to culture, race, and ethnicity (Krieger, 1992), it is easy for pro-
gram planners to overlook important differences in terms and there-
fore subpopulations. The evaluators questioned whether a focus on
Thurston et al. / EVALUABILITY ASSESSMENT 219
immigrants was appropriate and thought this could lead mental health
practitioners to confuse the needs of immigrants with those of other
populations of ethnic minorities. It was not possible for evaluators and
program staff to agree on all issues, however, and the evaluators took
the position that they could provide an evaluation framework that pro-
gram staff could use to make strategic decisions but were not in a posi-
tion to insist on a given approach. In the end, the EA led to a clear defi-
nition of target clients, something that is imperative for developing
activities that will meet their needs and achieve the overall goals of the
project.
CONCLUSION
EA is a cost-effective strategy for organizational change. It cannot
ignore the political tensions that frequently surround health programs
or program evaluation and requires the same attention to human rela-
tionships as other evaluation activities. Because of its iterative nature
and focus on program development, it creates a relatively safe space
for the critical examination of assumptions and can operate as a cata-
lyst for change. Whether the changes are optimal and the program is
successful remains to be assessed in other evaluation activities. Some-
times the evaluators and program planners will disagree. For the rela-
tively small cost of two part-time staff over 2 months, the project dis-
cussed in this article was able to clearly identify client groups, refocus
goals and objectives, and work toward developing indicators that
would measure expected outcomes rather than simply processes, all of
which would result in an improved project. EA is a cost-effective solu-
tion recommended by the authors for projects in formative stages
looking for assistance in direction, evaluation, and setting up a moni-
toring system.
REFERENCES
Chen, H. T. (1990). Theory-driven evaluation. Newbury Park, CA: Sage.
Chin, J. L. (1999). Cultural competence and health care in Massachusetts: Where are we?
Where should we be? (Issue Brief 5). Massachusetts: Massachusetts Health Policy Forum.
Retrieved Feb. 12, 2002, from www.ihp.brandeis.edu/mhpf/pages/publications.htm
220 Evaluation & the Health Professions / June 2003
Crosby, R. A., DiClemente, R. J., Wingood, G. M., & Harrington, K. (2002). HIV/STD preven-
tion benefits of living in supportive families: A prospective analysis of high risk African-
American female teens. American Journal of Health Promotion, 16, 142-145.
Duncan, C., Jones, K., & Moon, G. (1996). Health-related behavior in context: A multilevel
modeling approach. Social Science & Medicine, 37, 725-733.
Hoang, G. N., & Erickson, R. V. (1985). Cultural barriers to effective medical care among
Indochinese patients. Annual Review of Medicine, 36, 229-239.
Horst, P., Nay, J. N., Scanlon, J. W., & Wholey, J. S. (1979). Program management and the fed-
eral evaluator. In H. C. Schulberg & F. Baker (Eds.), Program evaluation in the health fields
(Vol. 2, pp. 73-109). New York: Human Sciences Press.
Krieger, N. (1992, Winter). The making of public health data: Paradigms, politics, and policy.
Journal of Public Health Policy, 412-427.
Macaskill, L., Dwyer, J. J. M., Uetrecht, C., Dombrow, C., Crompton, R., Wilck, B., et al. (2000).
An evaluability assessment to develop a restaurant health promotion program in Canada.
Health Promotion International, 15, 57-69.
Milstein, B., Wetterhall, S., & Centers for Disease Control Evaluation Working Group. (2000).
A framework featuring steps and standards for program evaluation. Health Promotion Prac-
tice, 1, 221-228.
Mohr, L. B. (1995). Impact analysis for program evaluation (2nd ed.). Thousand Oaks, CA:
Sage.
Ogunranti, J. O. (1995). Cultural and biological diversity in medical practice. World Health
Forum, 16(1), 66-68.
Rossi, P. H., & Freeman, H. E. (1989). Evaluation: A systematic approach. Newbury Park, CA:
Sage.
Rossi, P. H., & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury
Park, CA: Sage.
Rutman, L. (1977). Formative research and program evaluability. In L.Rutman (Ed.), Evaluation
research methods: A basic guide (pp. 59-71). Beverly Hills, CA: Sage.
Thurston, W. E. (1991). Program planning and evaluation. Unpublished course documents,
University of Calgary, Canada.
Thurston, W. E., & Potvin, L. (2002). Evaluability assessment: A tool for incorporating evalua-
tion in social change programs. Manuscript submitted for publication.
Unrau, Y. (1993). A program logic model approach to conceptualizing social service programs.
Canadian Journal of Program Evaluation, 8, 117-134.
Welch, M. (1997). Culturally competent care for diverse populations. Retrieved February 12,
2002, from http://dgim.ucsf.edu/pods/html/divpeople.html
Wholey, J. S. (1977). Evaluability assessment. In L. Rutman (Ed.), Evaluation research meth-
ods: A basic guide (pp. 41-56). Beverly Hills, CA: Sage.
Wimbush, E., & Watson, J. (2000). An evaluation framework for health promotion: Theory,
quality and effectiveness. Evaluation, 6, 301-321.
Thurston et al. / EVALUABILITY ASSESSMENT 221