A Research Synthesis of the Evaluation Capacity Building Literature

http://aje.sagepub.com/American Journal of Evaluation

http://aje.sagepub.com/content/early/2012/01/27/1098214011434608The online version of this article can be found at:

DOI: 10.1177/1098214011434608

published online 27 January 2012American Journal of EvaluationLesesne

Susan N. Labin, Jennifer L. Duffy, Duncan C. Meyers, Abraham Wandersman and Catherine A.A Research Synthesis of the Evaluation Capacity Building Literature

Published by:

http://www.sagepublications.com

On behalf of:

American Evaluation Association

can be found at:American Journal of EvaluationAdditional services and information for

http://aje.sagepub.com/cgi/alertsEmail Alerts:

http://aje.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

What is This?

- Jan 27, 2012OnlineFirst Version of Record >>

at American Evaluation Association on February 9, 2012aje.sagepub.comDownloaded from

http://aje.sagepub.com/

http://aje.sagepub.com/content/early/2012/01/27/1098214011434608

http://www.sagepublications.com

http://www.eval.org

http://aje.sagepub.com/cgi/alerts

http://aje.sagepub.com/subscriptions

http://www.sagepub.com/journalsReprints.nav

http://www.sagepub.com/journalsPermissions.nav

http://aje.sagepub.com/content/early/2012/01/27/1098214011434608.full.pdf

http://online.sagepub.com/site/sphelp/vorhelp.xhtml


A Research Synthesis ofthe Evaluation CapacityBuilding Literature

Susan N. Labin1, Jennifer L. Duffy2, Duncan C. Meyers2,Abraham Wandersman2, and Catherine A. Lesesne3

Abstract

The continuously growing demand for program results has produced an increased need for eva-luation capacity building (ECB). The Integrative ECB Model was developed to integrate concepts fromexisting ECB theory literature and to structure a synthesis of the empirical ECB literature. The studyused a broad-based research synthesis method with systematic decision rules and demonstrates theviability of the method for producing a reliable analysis of disparate data from a variety of designs.There was a high degree of consistency in what was reported in the empirical literature and thetheoretical literature in terms of strategies and outcomes. Reported outcomes at the individual levelincluded attitudes, knowledge, and behaviors and at the organizational level included practices,leadership, culture, mainstreaming, and resources. Collaborative processes and programmaticoutcomes emerged as important issues for ECB models and practice. The consistency between theempirical and the theoretical literature indicates that the field is ready to develop common mea-sures, use stronger designs, and report more systematically. This synthesis provides an overview ofexisting data and an empirical basis for refining strategies and common measures for enhancing theresearch and practice of ECB to achieve ECB and programmatic goals and outcomes.

Keywords

ECB, evaluation capacity building, research synthesis, capacity building, systematic review

For at least the past decade, evaluation capacity building (ECB) has been attracting the interest of

evaluators committed to increasing stakeholder understanding of evaluation and building evaluation

culture and practice in organizations (Boyle, Lemaire, & Rist, 1999; Compton, Baizerman, & Stockdill,

2002; Fetterman, Kaftarian, & Wandersman, 1996; Milstein & Cotton, 2000). Consequently, there is a

growing theoretical and empirical ECB literature (Compton et al., 2002; Cousins, Goh, Clark, & Lee,

1 Washington, DC, USA2 Department of Psychology, University of South Carolina, Columbia, SC, USA3 ICF Macro, Atlanta, GA, USA

Corresponding Author:

Susan N. Labin, 8517 Rayburn Road, Bethesda, MD 20817, USA

Email: [email protected]

American Journal of Evaluation00(0) 1-32ª The Author(s) 2012Reprints and permission:sagepub.com/journalsPermissions.navDOI: 10.1177/1098214011434608http://aje.sagepub.com



2004; Preskill & Boyle, 2008). However, to date there has not been a systematic review of the empirical

ECB literature. The purpose of this article is to address this gap.

Synthesizing and taking stock of the existing empirical literature on ECB has a number of

intended benefits. A systematic synthesis provides an evidence base about how ECB is being prac-

ticed in the field. The results can illuminate the current landscape of ECB by describing the contexts

and settings where it occurs, the strategies used, the outcomes reported, and how the strategies have

been evaluated. Such a descriptive base is an important step in the development of ECB and can

contribute to the improvement of ECB practice by increasing information sharing for the many

evaluators who have incorporated ECB into their practice (according to an American Evaluation

Association [AEA] membership survey, at least half of those responding reported they included

ECB in their practice; AEA, 2008). This descriptive base can also be of value to organizations and

funders that embark on the journey of ECB for improving programs and organizations.

Building on the work of others who have contributed to developing a language of and indicators

for ECB (Compton et al., 2002; Preskill & Boyle, 2008), this synthesis systematically codifies the

literature. Common terminology and indicators are the basis for describing ECB processes and are

prerequisites for evaluating ECB efforts; all of which will enhance both the practice and science of

ECB. As with all syntheses (Labin, 2008), this synthesis is intended to clarify existing knowledge,

raise questions, and reveal gaps to inform future practice and research.

Defining ECB

Developing a working definition of ECB was an essential first step for synthesizing the literature on

ECB. Various definitions of ECB have been proposed (Boyle et al., 1999; Preskill & Boyle, 2008;

Schaumberg-Muller, 1996; Stockdill, Baizerman, & Compton, 2002). Our review of these

definitions identified common features. For example, each identifies ECB as an activity separate

from actually conducting evaluations. There were also differences among definitions-some focus

primarily on ECB as an activity at the organizational level (Stockdill et al., 2002), while others are

concerned with capacity building at the individual and organizational levels (Preskill & Boyle, 2008;

Schaumberg-Muller, 1996).

In addition to examining explicit definitions of ECB, we drew on the work of collaborative,

participatory, and empowerment evaluation, which are precursors or approaches that include aspects

of ECB. For example, in empowerment evaluation, building evaluation capacity in order to improve

program outcomes has been a central and explicit principle since its inception in the early 1990s

(Fetterman et al., 1996). Empowerment evaluation and other participatory and collaborative approaches

emphasize how goals similar to those of ECB could be achieved through participatory means and thus,

provided much of the foundation for what has became known as ECB (Cousins & Whitmore, 1998;

Fetterman & Wandersman, 2005; Love, 2006; O’Sullivan, 2004; Rodriguez-Campos, 2005). Based

on our review of these frameworks and approaches, we developed the following working definition

of ECB:

Evaluation capacity building (ECB) is an intentional process to increase individual motivation,

knowledge, and skills, and to enhance a group or organization’s ability to conduct or use evaluation.

The Integrative ECB Model

This working definition was used to identify the types of cases to include in the synthesis. In order to

determine what information would be systematically extracted from the cases, we developed an

Integrative ECB Model (Figure 1) based on existing ECB frameworks and a review of both the

2 American Journal of Evaluation 00(0)



theoretical and empirical ECB literature (Baizerman, Compton, & Stockdill, 2002b; Cousins et al.,

2004; Duffy & Wandersman, 2007; Milstein & Cotton, 2000; Owen, 2003; Preskill & Boyle, 2008;

Suarez-Balcazar et al., 2010). In particular, we used Preskill and Boyle’s (2008) Multidisciplinary

Model of ECB. We developed the Integrative ECB Model to ensure that the synthesis reflected and

integrated key elements in existing theory and empirical literature on ECB and did not restrict

information to that from any one existing framework or subset of the literature. Our intention was

to maximize the breadth of information to be extracted, advance the development and use of

common terminology, and operationalize the study’s working definition of ECB.

A basic logic model of Needs-Activities-Outcomes (Kellogg Foundation, 2001; United Way,

1996, 2008; University of Wisconsin-Extension, 2003) was used to organize and portray the key

circumstances, activities, processes, and outcomes of ECB. The logic model structure implies a

causal direction from left to right, that is, needs will affect the strategies and strategies will affect

outcomes achieved. Furthermore, implementation and evaluation descriptors may mediate the

effects of strategies on outcomes.

Individual levelo Attitudeso Knowledgeo Skills/Behaviors

Organizational level o Processes, Policies, and

Practices (PPP)o Leadershipo Organizational Cultureo Mainstreamingo Resources

Program Outcomeso Developmento Implementation o Results

Negative outcomes

Lessons Learned

ShortLong-Term &

SustainableStrategieso Theoryo Modeo

o Typeo Content

Implementationo Target

Population OrganizationDomain

o Timing, Frequency,Dosage

o Mid−CourseCorrectionsBarriers

Evaluation of ECBo Approacho Designo Measureso Data typeo Timeframeo Internal or External

Reasons–Motivationso Audience: Internal−

Externalo Assumptionso Expectations

Goals–Objectives

Contexto Needs Assessmento Tailored

Resources and Strengthso Individual−Attitudeso Organizational

Resources: staff/time/moneyEvaluationexpertisePractices,leadership,

streaming

*

III. RESULTS: OUTCOMES

•ooo

•o

oooo

•ooo

•

•

II. ACTIVITIES: WHAT and HOW

•ooo Level: Individual-

Organizationaloo

•o

o

o

o•oooooo

I. NEED:WHY

•o

oo

•

•oo

•oo

culture, main-

Figure 1. Integrative evaluation capacity-building model.*Collaborative and participatory aspects and processes should be included in defining and operationalizingnearly all elements of the model.

Labin et al. 3



Our goal was to create a model that included key activities and processes in ECB. We realize that

our model is a simplification and may not have identified every interactive process that is part of

ECB. However, we believe we have captured the major activities and processes, thus allowing for

systematic extraction and coding of data.

Need for ECB––Why

The first column of the model relates to the need for conducting ECB and who and what motivates

the interest in ECB (Milstein & Cotton, 2000; Preskill & Boyle, 2008). ECB may be driven by

factors internal to the organization (such as a leader’s desire to increase evaluation within the orga-

nization), by external factors (such as funder requirements), or by a combination of internal or exter-

nal factors. Preskill and Boyle emphasize the importance of considering three elements related to the

need for ECB: (a) motivation for ECB; (b) assumptions and expectations about ECB; and (c) iden-

tification of goals and objectives for ECB. They point out that ‘‘Understanding the organization’s

motivation for engaging in ECB . . . provides insight into who should participate and which teaching

and learning strategies might be most beneficial’’ (p. 446). Related to the motivation for ECB are

assumptions that may underlie the desire to engage in ECB. Preskill and Boyle suggest that when

these assumptions are not shared among the key people involved in ECB, the success of the effort

may be inhibited. They also note that the explication of specific objectives is important for the suc-

cessful design and implementation of ECB efforts.

Conducting a needs assessment and tailoring ECB efforts to the particular population and con-

text can affect the selection and implementation of the ECB strategies. The existing characteristics

of an organization have also been hypothesized to affect the type of strategies utilized and their

efficacy. Some of these specific factors noted in the literature include attitudes toward evaluation,

availability of resources for ECB (staff, time, and financial), internal evaluation expertise, and

organizational practices and capacities such as support for evaluation and ECB from leadership,

from the organizational culture, and through mainstreaming or making evaluation a routine part

of the organization (Milstein & Cotton, 2000; Owen, 2003; Preskill & Boyle, 2008; Suarez-

Balcazar et al., 2010). These factors have been hypothesized to affect organizational learning and

the extent to which the outcomes of ECB will become sustainable (Preskill & Boyle, 2008). In the

Integrative ECB Model, many of these preexisting characteristics that can facilitate the ECB pro-

cess are defined as strengths and resources.

ECB Activities––What and How

The second column of the Integrative ECB Model categorizes the activities of the ECB strategies,

implementation specifics, and evaluation of the ECB efforts. Various aspects of strategies are

defined to capture important dimensions that define their nature and effectiveness. Some ECB

efforts are justified by an underlying theory or approach such as empowerment evaluation or orga-

nizational learning. Preskill and Boyle (2008) identified a number of types of theories which can

inform the design and implementation of ECB strategies, including theories about evaluation, adult

learning theory, and theories related to organizational change and development.

ECB strategies may be provided through multiple modes, such as face-to-face meetings, telecon-

ferences or phone calls, e-mail or other web-based mechanisms, and through written materials such

as evaluation manuals. Some may use a mode of exclusively face-to-face efforts, while others might

utilize only distal modes such as phone or web. Strategies can be directed at the individual level for

learning and behavior change and at the organizational level. The ECB literature addresses how the

level at which strategies are directed affects outcomes (Preskill & Boyle, 2008; Stockdill et al.,

2002). Types of strategies refer to the mechanisms of delivery, that is, training, technical assistance

(TA), and experiential involvement or participation in evaluation activities (Duffy & Wandersman,




https://www.researchgate.net/publication/228352197_A_Multidisciplinary_Model_of_Evaluation_Capacity_Building?el=1_x_8&enrichId=rgreq-05bf3f60-531b-44c1-b27b-b210d5e7b1a1&enrichSource=Y292ZXJQYWdlOzIzNDA2NDYwNztBUzoyMTI4MDY2NTg5OTAwODBAMTQyNzc0ODQ2OTMwMw==



2007; Milstein & Cotton, 2000). Furthermore, there is also discussion in both the empirical and

theoretical literature about the substance or content of the ECB activities. At the individual level,

strategy content focuses on attitudes and evaluation curriculum, for example, designing evaluations

and analyzing data. At the organizational level, strategy content focuses on collective activities such

as using evaluation as part of organizational processes and practices, providing leadership support

for evaluative activities, fostering a learning culture, mainstreaming evaluation by making it a more

routine aspect of how the organization functions, or increasing resources for evaluation (Compton et

al., 2002; Duffy & Wandersman, 2007; Sanders, 2003; Suarez-Balcazar et al., 2010).

Implementation variables are not only important descriptors of ECB efforts, but they also may

mediate effects of the strategies on outcomes (Durlak & DuPre, 2008; Rapkin & Trickett, 2005).

Standard implementation variables were included in the model—population, organizational setting,

domain or area—in order to both describe existing ECB practice and to determine whether these

factors mediate the effects of strategies on outcomes (Preskill & Boyle, 2008). As with most inter-

ventions, timing, frequency, and dosage of the interventions may also affect the outcomes. Mid-

course corrections to accommodate population and contextual issues are often cited as an

important implementation variable for increasing the likelihood of positive outcomes (Wandersman,

Imm, Chinman, & Kaftarian, 2000). Conversely, barriers encountered may hinder the likelihood of

achieving positive outcomes.

The activities of the ECB efforts include not only strategies but also evaluations of those efforts.

The nature of the evaluations is important for developing indicators, measuring, and interpreting out-

comes (Preskill & Boyle, 2008). The descriptive categories of the evaluations represent basic com-

mon and important features of evaluations including the approach, design, measures, data type, time

frame, and who conducted the evaluation.

ECB Results—Outcomes

Both individual and organizational outcomes of ECB have been hypothesized. Individual-level

outcomes include improved attitudes, knowledge, and skills as evidenced in behaviors such as enga-

ging in various evaluation activities (Duffy & Wandersman, 2007; Owen, 2003; Preskill & Boyle,

2008; Suarez-Balcazar et al., 2010). Preskill and Boyle hypothesize that individual learning is

affected by the organizational context and Taylor-Ritzler, Suarez-Balcazar, and Garcia-Iriarte

(2010) found that supportive organizational characteristics were necessary for individual learning

and behavior change. Organizational-level outcomes in the Integrative ECB Model consist of five

organizational characteristics that are important for a successful ECB effort. Processes, policies, and

practices (PPP) relate to the doing and using of evaluation, both hallmarks of ECB (GAO, 2003;

Gibbs, Napp, Jolly, Westover, & Uhl, 2002; Preskill & Boyle, 2008; Suarez-Balcazar et al.,

2010). Leadership is included because of its well-established importance for organizational change

(Kotter, 1996; Milstein & Cotton, 2000). Organizational culture is the collective values, attitudes,

goals, and practices that can support or hinder organizational change and is considered an essential

outcome or indicator of successful ECB (Boyle, 1999; Cousins et al., 2004; GAO, 2003; Owen,

2003; Preskill & Torres, 1999; Suarez-Balcazar et al., 2010). We selected indicators of

organizational culture that have been identified as conducive for ECB such as being committed

to learning and using data (Preskill & Torres, 1999; Robinson & Cousins, 2004). Mainstreaming

or the routinization of evaluation is considered an essential element in the sustainability of ECB,

which is considered by some the most important long-term goal (Boyle, 1999; Preskill & Boyle,

2008; Sanders, 2003; Stockdill et al, 2002). Resources to support evaluation are hypothesized to

be important if ECB is going to be successful (Boyle et al., 1999; Gibbs et al., 2002; Milstein &

Cotton, 2002; Preskill & Boyle, 2008). The five organizational characteristics are elements in the

Labin et al. 5



https://www.researchgate.net/publication/5529147_Implementation_Matters_A_Review_of_Research_on_the_Influence_of_Implementation_on_Program_Outcomes_and_the_Factors_Affecting_Implementation?el=1_x_8&enrichId=rgreq-05bf3f60-531b-44c1-b27b-b210d5e7b1a1&enrichSource=Y292ZXJQYWdlOzIzNDA2NDYwNztBUzoyMTI4MDY2NTg5OTAwODBAMTQyNzc0ODQ2OTMwMw==

https://www.researchgate.net/publication/223218993_Increasing_evaluation_capacity_within_community-based_HIV_prevention_programs?el=1_x_8&enrichId=rgreq-05bf3f60-531b-44c1-b27b-b210d5e7b1a1&enrichSource=Y292ZXJQYWdlOzIzNDA2NDYwNztBUzoyMTI4MDY2NTg5OTAwODBAMTQyNzc0ODQ2OTMwMw==



Integrative ECB Model and are derived from the ECB literature as intended and necessary outcomes

to document successful ECB practice (Duffy & Wandersman, 2007).

Program outcomes refer to the outcomes that are the mission of the organization that is hosting

the ECB effort. For example, a health program may want to improve prenatal and newborn health

outcomes. Program outcomes were neither extracted from the literature nor included in the synth-

esis, but rather were added to the model because of reports in the empirical literature that program

outcomes improved as a result of the ECB. Improved program outcomes have been a long-standing

rationale for conducing ECB (Wandersman, et al., 2005). Negative outcomes were included in order

to track any unintended negative consequences of ECB. Lessons learned as comments by the authors

were collected from the cases.

Research Questions

Using the Integrative ECB Model as a guide, we identified the following research questions:

1. What are the needs preceding ECB efforts?

2. What strategies are being used for ECB and what implementation variables are being reported?

3. What evaluation approaches and methods are being used to assess ECB efforts?

4. What outcomes of ECB are being reported at the individual and organizational levels?

In addition to these four descriptive questions, we examined several relationships posited by the

logic model structure, that is, the causal direction from left to right.

1. How do strategies vary by the presence of pre-existing resources?1

2. How do the outcomes vary by strategies?

Method

We employed a broad-based research synthesis method (Labin, 2008) that uses systematic decision

rules that are derived from principles of meta-analysis and that distinguish synthesis from traditional

literature reviews. However, meta-analysis usually includes only randomized control trials (RCTs;

Cooper & Hedges, 1994; Higgins & Green, 2011; Labin, 2008), whereas broad-based synthesis

methods include studies with a variety of designs and types of data. Broad-based synthesis methods

have been developed and used by the Government Accountability Office (GAO, 1987, 1989, 1992a,

1992b) and the Centers for Diseases Control and Prevention (The Guide to Community and Preven-

tive Services, 2011). While broad-based methods are appropriate in well-developed fields to incor-

porate data from a variety of designs, they are especially and perhaps uniquely well suited for fields

early in their development such as ECB.

Research synthesis, both meta-analysis and broad-based, involves a series of steps, each of which

is governed by systematic decision rules:

1. Define the research questions.

2. Collect information sources.

3. Select information sources based on inclusion criteria.

4. Extract and code data.

5. Analyze data.

6. Present findings.

The Integrative ECB Model displayed in Figure 1 served as the source of the research questions.

To guide the collection and selection of sources, the research team conducted electronic searches




using the nine phrases identified in Table 1. Seven social science electronic databases were searched

including ERIC, Psychinfo, and Sociological Abstracts. Team members also conducted a manual

review of the tables of contents (from 1998 through August 2008) for seven evaluation journals.

We were aware that including only published articles might create ‘‘publication bias’’ (Begg,

1994), which assumes significant findings are more likely to be published. Our inclusion criteria sti-

pulated that the documents must be empirical examples of ECB that also met our definition of ECB.

The search was wide and included studies published as articles or book chapters using all types of

data and methods including case studies, narrative descriptions, qualitative data, and quantitative

methods. Articles did not have to use the term ‘‘ECB.’’ However, articles that were solely about ECB

theory, theoretical frameworks, assessment methods, fictional examples, or descriptions of ECB in

general were excluded. We also excluded articles describing evaluation courses and articles discuss-

ing ECB efforts directed at building national capacity, which were determined to be substantially

different and beyond the scope of this study. However, we did include studies of ECB funded by

federal agencies. The initial searches yielded over 500 results and a manual scan of titles and

abstracts reduced this number to 149 (Table 2). A more in-depth reading of abstracts and in some

instances the full text of the documents produced a final sample of 61 unique cases that met the

inclusion criteria and were coded.

For the purpose of this study, the authors identified each case as a single real-world example of

the practice of ECB. Some cases focused on a single ECB practitioner’s work with a single organi-

zation (Appendix A, e.g., Cohen, 2006). Others focused on a team of ECB practitioners working

with multiple organizations as part of a single ECB initiative, such as a multiyear project intended

to build evaluation capacity throughout a statewide network of substance abuse prevention agencies

(Appendix A, e.g., Stevenson, Florin, Mills, & Andrade, 2002). Sometimes one example of ECB

practice was discussed in multiple articles (Appendix A, e.g., Huffman, Lawrenz, Thomas, &

Table 1. Collect Information Sources: Searches

Databases Searched: Dissertations and ThesesEBSCO Business Source PremierERICPsychInfoSociological AbstractsSocial Work AbstractsWeb of Science

Search terms used: Developing evaluation capacityEmpowerment evaluationEvaluation capacity buildingEvaluation capacity developmentEvaluation skill buildingEvaluation technical assistanceEvaluation trainingEvaluative inquiryInsourcing

Journals for which tables of contents weremanually reviewed (from 1998 to 2008)

American Journal of EvaluationCanadian Journal of Program EvaluationEvaluationEvaluation and Program PlanningEvaluation Journal of AustralasiaJournal of Multidisciplinary EvaluationNew Directions for Evaluation

Labin et al. 7



Clarkson, 2006; Huffman, Thomas, & Lawrenz, 2008), for which information from all the related

articles were coded as a single case. Other times a single article presented information on more than

one example of ECB practice (sometimes comparing and contrasting the approaches and results);

each of these examples was coded as a separate case (Appendix A, e.g., Hoole & Patterson,

2008). The determination of whether ECB examples were treated as a single case including multiple

organizations or multiple separate cases was based on whether the authors described their ECB prac-

tice as part of a single effort or multiple separate efforts. Appendix A provides a list of all the cases

used in the analysis and the articles reviewed in order to code each case.

Coding was conducted by a team of three researchers (authors of this article including one

senior evaluation researcher and two advanced doctoral-level students). The Integrative ECB

Model provided the major concepts that were to be coded. A coding form was developed to

structure the systematic extraction and coding of data from each case. The coding form oper-

ationalized the elements in the model and provided indicators and response categories for each

element.2

The items on the coding form were either binary (e.g., whether or not the ECB effort was eval-

uated) or categorical (e.g., whether the target population consisted of professional evaluators, pro-

gram practitioners/staff, a combination of the two). Many items on the coding form also included an

‘‘other’’ response option that allowed the coder to write-in additional relevant information. These

‘‘other’’ responses were included for nearly every element in the model, for example, preexisting

strengths, strategies, target populations, evaluation descriptors, outcomes, and barriers. Coders uti-

lized the ‘‘other’’ option to write in a response only if the case reported information that was related

to an item on the coding form but was not captured by any of the response options on the coding

form. Each of the ‘‘other’’ responses was assessed by several researchers and was either collapsed

into one of the existing coding categories or remained a true ‘‘other’’ response. With a few

Table 2. Select Information Sources Based On Inclusion Criteria

Selection Stage Included Excluded

1. Initial search of databases andtables of contents

149 articles identified based ondefinition of ECB

Articles that did not meet ECBdefinition were excluded fromfurther review.

2. Review of abstracts 101 items met preliminaryinclusion criteria

66 items excluded because they did notmeet criteria for an empirical case ofECB

3. Coding of articles 85 book chapters and articleswere coded, yielding a totalof 79 cases of ECBa

7 dissertations were excluded(precluded from coding process dueto length)6 items were excluded because onreview of full article they did notmeet ECB case criteria3 items were excluded becauseinsufficient information to code thecase was provided

4. Exclusion of classroom-basedcases

61 cases were included in finalanalyses after excludingclassroom-based cases.

18 cases were excluded after decisionthat classroom-based casesrepresented were substantivelydifferent from other types of ECBcases.

aSome cases of ECB had multiple articles or chapters written about them, and some articles described multiple distinct casesof ECB. Multiple articles on a single case were treated as one case and multiple cases were coded within a single article whenappropriate. See Appendix A.




exceptions, the vast majority of the information fell into the existing response categories. Coders

also collected comments that related to lessons learned and conclusions by the authors.

A coding manual that operationalized each of the items on the coding form was used to ensure

that coders followed systematic coding rules to extract data for each case from the associated arti-

cle/articles. The manual directed coders to avoid making assumptions and to only code material that

was explicitly reported. While no assumptions were made about the presence of a concept or item,

coders generally could code the presence of a given concept even if the authors did not use the same

terminology as the coding manual. For example, mainstreaming would be coded as an outcome if it

was reported that the ECB effort was successful in making evaluation an integral part of how the

organization functions even if none of the specific indicators of mainstreaming or the specific term

was used. In a few instances, use of the same terminology as the coding manual was required. For

example, use of ‘‘written documents’’ was only coded if there was explicit mention of manuals or

other written documents created for or accessed by the project.

Both the coding form and the accompanying coding manual were refined through an iterative

piloting process. On several occasions at least two of the coding team rated a subsample of cases

and all discrepancies were discussed and decision rules were established, refined, and recorded in

the coding manual. During this piloting process, we dropped several items in the model because

there was insufficient detail in the reviewed articles for them to be reliably coded. These dropped

items were (a) the reasons for conducting the ECB effort; (b) the implementation variables of time,

frequency and dosage; and (c) whether it was an internal or external evaluation. A subset of 10 cases

was selected as a purposeful sample on which to assess intercoder reliability and agreement. Purpo-

seful sampling was used in order to obtain a group of cases differing on major concepts of interest

and varying in terms of the level of detail reported. Having such a diverse sample of cases, as sug-

gested by Lipsey and Wilson (2001), provided an opportunity for coders to discuss difficulties

encountered and develop additional decision rules for any discrepancies in how items were coded.

This process was the basis for further refinement of the coding manual.

All three researchers participated in coding the reliability sample; two raters coded each case. The

overall interrater reliability calculated for Cohen’sk among all raters, interpreted using the ranges estab-

lished by Landis and Koch (1977), was in the substantial range (k ¼ .66), with specific results ranging

from moderate (k ¼ .46) to almost perfect (k ¼ .83). Cohen’s k coefficient was chosen as a statistical

measure of intercoder agreement given the categorical nature of the items in the coding form.3

The k statistic may understate interrater agreement and result in a misclassification to a lower

rating when there is little or no variability in ratings. To address this possibility of misclassified rat-

ings, percent agreement was calculated as an additional measure of interrater agreement. Percent

agreement was calculated by dividing the number of agreements between coders by the sum of the

number of agreements plus the number of disagreements. The resulting percent agreement numbers

are much higher than those for the k statistics. The overall percent agreement was 85% with specific

levels ranging from 70.3% to 94.8%.

After coder reliability and agreement were established, and adjustments were made to the coding

process, the remaining cases were coded by one coder on the team. Each coder extracted data via the

coding form and the data were scanned and compiled electronically to ensure accuracy.

Notwithstanding the relatively strong overall levels of reliability and interrater agreement

achieved in this review, there were several issues that posed challenges. First and foremost was the

high degree of variability in the level of detail with which the ECB cases were described in the orig-

inal articles. A second factor was that for many articles the narratives did not follow the structure of a

traditional research article—they did not have methods or results sections––and concepts of interest

were dispersed throughout the articles in such a way that the coder had to review the article multiple

times in order to code information for that case. Notwithstanding these challenges, it is likely that the

coder agreement statistics understate the reliability of the coding because the coding procedure was

Labin et al. 9



https://www.researchgate.net/publication/22310512_The_Measurement_Of_Observer_Agreement_For_Categorical_Data?el=1_x_8&enrichId=rgreq-05bf3f60-531b-44c1-b27b-b210d5e7b1a1&enrichSource=Y292ZXJQYWdlOzIzNDA2NDYwNztBUzoyMTI4MDY2NTg5OTAwODBAMTQyNzc0ODQ2OTMwMw==

refined during the coding of the reliability sample, and the coding manual was improved with more

decision rules. Due to the enhanced detail of the revised manual—based on the resolution of discre-

pancies through group discussions––it is reasonable to assume that the coding of the remaining cases

(n ¼ 51; 83%) was characterized by an even higher degree of intercoder agreement and a higher

degree of accuracy than reflected in the k statistics and the percent agreement percentages for the

reliability sample.

The data from this systematic synthesis, as data for syntheses in general, are only as comprehensive

as the reporting for each case in the published articles. It seems reasonable to assume that the major

elements such as strategies and outcomes would have been reported and described by the authors. Nev-

ertheless, it is possible there was some underreporting of major issues and their relevant detail.

The coding results for the elements in the model were the basis of descriptive analyses—fre-

quencies, percentages, mean number of responses—to answer the first four research questions

(Tables 3–10). These distributions were assessed for the appropriateness of cross-tab analyses

to address the relational research questions (Questions 5 and 6). Limited variation in distributions

and small cell frequencies posed analytical constraints. However, the data for several of the

descriptive variables was judged adequate for relational analyses (Tables 11–14). Nevertheless,

given the limitations of the data the findings from the relational analyses are considered

exploratory.

Results

Goals, Contexts, and Strengths

The first column of the model refers to establishing the need for the ECB effort or why ECB was

initiated. In spite of the importance of this information to the design of ECB efforts (Preskill &

Boyle, 2008), this information was infrequently and incompletely reported. A majority of the cases

mentioned having goals and objectives (Table 3), but there were not enough specifics to reliably

code any further details. While needs assessments were not common (30%), a majority did report

tailoring the ECB effort to fit the context (64%).

A relatively small percentage of cases (slightly less than 15%) reported positive individual-level

attitudes toward evaluation as a strength preceding the ECB effort. However, a majority of cases

indicated some organizational-level strengths (62%) preceding the ECB effort. Indicators defining

organizational strengths are specified in Table 3. ‘‘Other’’ strengths noted included relationship

issues, such as those between the evaluators and the organizations.

Strategies Used and Implementation Variables

In the majority of cases (80%; Table 4), some theory or principles were reported as guiding the ECB

strategies used and about half of these provided some specifics on that theory. Among those that

reported specifics on the theory, two thirds explicitly mentioned participatory, collaborative, or

empowerment evaluation, of which empowerment evaluation was the most frequently reported

(about half). The type of strategy most commonly used was training (77%; Table 5). There was also

frequent use of multiple strategies and frequent use of involvement in an evaluation. Additional

information sometimes provided details on how the strategies were implemented, especially

including collaborative and interactive processes such as involving stakeholders or creating a

‘‘learning team.’’

Nearly all cases (91.8%) reported specific content for the individual strategies. Virtually all of

these efforts were directed toward multiple competencies in evaluation. Of those that targeted

‘‘doing an evaluation,’’ more focused on collecting data than analyzing data (77% compared to

50%). The single most frequently mentioned ‘‘other’’ content that strategies targeted was




collaborative and interactive processes. The least frequently reported content of individual-level

strategies was directed toward influencing attitudes about evaluation.

Half of the cases reported some content for organizational-level strategies of which the most fre-

quently reported (over one third) was ‘‘organizational practices’’ and the least frequently reported

was ‘‘how to build leadership support’’ (Table 6).

Three types of barriers were coded: external, individual, and organizational (Table 7). About half

of the cases reported individual-level barriers of which attitudes were the most prevalent (44%). This

was most often specified as the conflict between evaluating and providing service. However, other

details about the individual-level barriers indicated that not only was there difficulty applying new

knowledge, but there was also difficulty learning the evaluation subject matter. Organizational-level

barriers were the most commonly reported type of barriers (60%); and the most common of these

were resources (30/37¼ 81.8% of the organizational barriers). ‘‘Other’’ responses focused on collabora-

tive and interactive challenges such as those between stakeholders, evaluators and clients, funders, orga-

nizations, and the community.

Evaluation Approaches and Methods Used To Assess ECB Efforts

As shown in Table 8, about half of the cases reported some evaluation of their ECB efforts. More

than half of the cases were classified as case studies, which were in-depth studies or descriptions.

Only two cases mentioned a comparison group. Only one case mentioned instruments. Discussion

of measurement of the outcomes was extremely rare. Forty percent indicated data collections were

conducted over more than a 2-year period, which is consistent with the duration of ECB projects.

A range of data collection methods was reported, however, almost a third did not mention any

specific data collection methods (Table 8). The majority of data collection methods were used in

Table 3. Goals and Resources, Context, and Strengths

Ma n %

ECB goals or objectivesb

Any goals or objective reported 45 73.8Context

Needs assessment conducted 18 29.5ECB effort tailored to fit context 39 63.9Not reported 16 26.2

Strengths: Individual-level 9 14.8Individual positive attitudes 9 14.8

Strengths: Organization-level 2.3 38 62.3Resources (staff/time/money) 22 36.1Internal evaluation expertise 12 19.7Organizational practices 8 13.1Leadership support 20 32.8Evaluation culture 13 21.3Mainstreaming 7 11.5Other strengthsc 4 6.6

The total number of cases is 61. Totals may exceed 61 because subcategories are not mutually exclusive and multipleresponses are possible.aMean number of organizational strengths reported from total number of possible strengths (seven) for cases where anorganizational strength was reported.bGoals and objectives were not sufficiently reported to allow further specification.cOther strengths focused on relationships such as between evaluators and the organizations.

Labin et al. 11



combination with other methods. For example, the most prevalent data collection method was a sur-

vey and two thirds of those cases reporting using surveys also used additional data collection meth-

ods. ‘‘Other’’ methods were used to provide feedback and monitoring such as narratives, field notes,

photos, visuals, as well as briefings and discussions. Notwithstanding the variety of data collection

methods reported, a minimal amount of quantitative data was reported.

Outcomes Reported at the Individual and Organizational Levels

Nearly all (about 90%; Table 9) of the cases reported some individual-level outcomes in terms of

changes in attitudes, knowledge, or behaviors, of which positive attitude outcomes were the least

frequently reported (36%). About half of the cases (51%) reported knowledge outcomes and a

Table 4. Theory and Modes of ECB Strategies

n %

Underlying theory/principles guiding strategies 49 80.3Underlying theory/principles guiding strategies not reported 12 19.7Total 61 100Modes of strategies reported

Face-to-face only 29 47.5Face-to-face combined with other modes 27 44.3Combination not including face-to-face 1 1.6Not reported 4 6.6

Total 61 100

TA ¼ technical assistance.

Table 5. Type, Level, and Content of ECB Strategies

Any of thisType of Strategy

Reported

StrategyReportedExclusively

n % n %

Type of strategies 59 96.7Training 47 77.0 10 16.4TA/coaching/support 38 62.3 1 1.6Involvement in evaluation 41 67.2 5 8.2

Individual-level content reported 56 91.8Attitudes 14 23.0 0 0.0Terms, approaches, or methods 23 37.7 0 0.0Logic models 32 52.5 1 1.6Evaluation plan 50 82.0 2 3.3How to do an evaluation? 44 72.1 1 1.6Interpretation and use of data 32 52.5 0 0.0

Organization-level content reported 31 50.8Organizational practices 24 39.3 8 13.1Building leadership support 6 9.8 0 0.0Building evaluation culture 15 24.6 4 6.6Mainstreaming 15 24.6 2 3.3

The total number of cases is 61. Totals may exceed 61 because subcategories are not mutually exclusive and multipleresponses are possible.




significant majority reported behavioral outcomes (80%). The most frequently reported knowledge

outcomes were learning elements of an evaluation plan, how to do an evaluation, and terms,

approaches, and methods. ‘‘Other’’ knowledge outcomes were interpersonal and included group

facilitation, leadership, and how to work in a group or team. Planning and doing an evaluation were

the most common behavioral outcomes and were reported more frequently than the corresponding

knowledge outcomes of planning and doing. In knowledge outcomes, collecting and analyzing data

as part of doing an evaluation were specified with about the same frequency, but in behavior out-

comes there were more reported outcomes for collecting than for analyzing data. ‘‘Other’’ behavior

outcomes focused on collaborative skills such as involving stakeholders.

In the majority of cases, some organization-level outcomes were reported (77%; Table 10) and

93% of these cases also reported individual outcomes. Organizational outcomes were defined and

divided into five organizational characteristics as shown in Table 10. ‘‘Other’’ outcomes mentioned

pertained not to the host organization or target of the ECB but rather to the programs under their

purview. These program outcomes included developing the program’s theory of change and improv-

ing program implementation and program results.

Table 6. Implementation––Target Population of ECB

Characteristics of Target Population N %

Intended target of ECBa

Individual only 16 26.2Organization only 9 14.8Individuals and organizations 36 59.0Not reported 0 0.0

Types of individual participantsStaff only 47 77.0Students only 0 0.0Staff with evaluators/other 9 14.8Students with staff/evaluators 5 8.2

Types of organizationsNonprofit Only 24 39.3Government Only 7 11.5School/school district only 9 14.8University only 4 6.6Other type 1 1.6Multiple types reported 9 14.8Not reported 7 11.5

DomainEducation 15 24.6Health 18 29.5Child and youth development 4 6.6Justice 1 1.6Community development 6 9.8Other domain 3 4.9Multiple domainsb 12 19.7Not reported 2 3.3

CountryUnited States 42 68.9Outside United States 18 29.5Not reported 1 1.6

aFive of the cases targeting organizations included communities.bEight of the multiple domain cases included education and/or health and/or other domains.

Labin et al. 13



How Type and Number of Strategies Vary by the Presence of Preexisting Resources

To explore the data for patterns of relationships between Needs in Column I and Activities in

Column II in the Model (Figure 1), we examined how the type and number of ECB strategies

employed varied by the reported presence of preexisting resources––financial, staff, and technolo-

gical.4 About 36% of the cases reported the presence of resources, but there does not appear to be a

discernable pattern of relationship between the reporting of preexisting resources and the number of

strategies (Table 11). This may suggest that the number and type of strategies was decided for rea-

sons other than material resources or perhaps there are differences in perceived project resources

versus resources from a larger entity that is funding the ECB effort.

However, there is some data suggesting that using TA cooccurs with a greater reporting of

resources (Table 12). For example, for those cases not reporting resources, those using TA have a

lower percentage (62%) compared to those using training (80%) or experiential involvement

(72%). Furthermore, if we calculate the percentage of cases reporting resources for those using each

type of strategy, those using TA report existing resources (14/38 ¼ 37%) with a slightly greater fre-

quency than those who reported using training (15/46 ¼ 33%) and those using experiential

involvement (13/41 ¼ 32%). While the percentage differences are small, these findings suggest that

TA may occur in situations with more resources. Given the potential importance of TA for sustain-

able outcomes of ECB, the need for resources for TA warrants further exploration.

How Types of Outcomes Vary by Type of ECB Strategies

We also examined the relationship between Strategies in Column II with Outcomes in Column III of the

Model. For cases reporting individual-level outcomes, over a third utilized all three types of strategies—

training, TA, and experience (Table 13); and for these cases, behavioral outcomes were the

Table 7. Implementation Adjustments and Barriers Reported

Ma n %

Midcourse corrections were made 17 27.9External barriers reported 1.3 9 14.8

Timelines too limited 6 9.8External evaluation requirements don’t fit internal needs 7 11.5

Individual-level barriers reported 1.4 34 55.7Attitudes/beliefs as barriers 27 44.3Lack of participation 11 18.0Difficulty applying new knowledge/skills 7 11.5Other individual-level barrier 3 4.9

Organization-level barriers reported 2.1 37 60.7Lack of resources 30 49.2Staff turnover 14 23.0Lack of internal evaluation expertise 8 13.1Lack of organizational practices 5 8.2Lack of leadership support 7 11.5Lack of evaluation culture 5 8.2Lack of mainstreaming 0 0.0Other 7 11.5

The total number of cases is 61. Totals may exceed 61 because subcategories are not mutually exclusive and multipleresponses are possible.aMean number of barriers reported from total number of possible barriers for cases where a barrier of that type wasreported, that is, total possible for external barriers (2), individual-level (5), organization-level (8).




most frequent type of individual outcome. If only one type of strategy was utilized, it was most

often training; and for these cases, knowledge outcomes were the most frequent type of indi-

vidual outcome.

Looking at the organizational outcomes for those cases that utilized all three types of strategies,

we see that PPP is the most frequent type of outcome followed by mainstreaming and resources. Cul-

ture and leadership exhibit the lowest frequencies of outcomes. This same pattern for the most (PPP,

mainstreaming, and resources) and least (culture and leadership) frequent outcomes is observed for

those cases that utilized only training as a strategy as well as for most of the other cases.

How Types of Outcomes Vary by Content of Strategies?

Organizational outcomes are reported with a higher frequency for those cases that report

organizational content than those that report only individual-level content (Table 14). Individual-

Table 8. Evaluation of ECB

Characteristics of Evaluations of ECB N %

Evaluation of ECB workAny evaluation reported 30 49.2Not reported 31 50.8

Evaluation approach/ theory specifiedAny approach or theory reported 12 19.7Not reported 49 80.3

Evaluation designPrepost with comparison group 1 1.6Post only with comparison group 1 1.6Prepost with no comparison group 9 14.8Retrospective pre-post, no comparison group 2 3.3Post only with no comparison group 8 13.1Case study 35 57.4Not reported 5 8.2

Time period of data collectionLess than or equal to 6 months 6 9.8Between 6 months and 1 year 4 6.6Between 1 year and 2 years 10 16.4Between 2 years and 3 years 8 13.1More than 3 years 16 26.2Not reported 17 27.9

Data collection methodsSurvey 26 42.6Interview 19 31.1Third party observation 12 19.7Document review 11 18.0Focus groups 7 11.5Other methods 7 11.5Not reported 20 32.8

Quantitative data reportedIndividual-level outcomes 13 21.3Organization level-outcomes 6 9.8

The total number of cases is 61. Totals may exceed 61 because subcategories are not mutually exclusive and multipleresponses are possible.

Labin et al. 15



level outcomes of positive attitudes and behavioral changes are reported with a lower frequency for

those reporting only individual-level content compared to those cases reporting both individual- and

organizational-level content.

Lessons Learned

Coders extracted ‘‘other’’ responses related to lessons learned. The most frequent of these ‘‘other’’

responses fell into four areas: (a) barriers due to time and resource constraints, (b) collaborative and

experiential principles of the strategies, (c) collaborative and relationship building content of stra-

tegies, and (d) appropriate evaluation content of strategies. Various dimensions of time were men-

tioned and the most frequently mentioned was how ECB and organizational change is something

that happens over time, for example, 2 years.

In the area related to theories and principles underlying the strategies, a notable finding is the high

frequency of positive reporting of empowerment evaluation and other experiential, participatory,

and active learning approaches. Building relationships, collaborating, and networking were cited

as needed content for the individual-level ECB strategies. Other issues related to evaluation content

for individual-level strategies were often mentioned with the most prevalent aspect reported being

Table 9. Individual-Level Outcomes

Outcome Categories Ma n %

Any individual-level outcomes reported 56 91.8Attitudes 22 36.1Knowledge 2.5 31 50.8

Terms, approaches, and methods 15 24.6How to hire/work with an evaluator? 3 4.9Logic models 11 18.0Elements of an evaluation plan 17 27.9How to do an evaluation? 15 24.6

How to manage evaluation? 1 1.6How to collect data? 11 18.0How to analyze data? 10 16.4Reporting data? 5 8.2

Interpretation and use of data 8 13.1Other knowledge-related outcome 3 4.9General mention of knowledge 6 9.8

Behaviors and Skills 2.8 49 80.3Hired &/or worked with an evaluator 19 31.1Developed logic models 22 36.1Planned/designed an evaluation 35 57.4Did an evaluation 38 62.3

Managed an evaluation 5 8.2Collected data 29 47.5Analyzed data 19 31.1Reported data 17 27.9

Interpreted and used data 24 39.3Other behavior-related outcome 0 0.0General mention of behavior 1 1.6

The total number of cases is 61. Totals may exceed 61 because subcategories are not mutually exclusive and multipleresponses are possible.aMean number of outcomes reported from total number of possible outcomes for cases where an outcome of that type wasreported, that is, total possible outcomes for knowledge (8), behaviors and skills (7).




the importance of using process evaluation and immediate findings to show program progress and

outcomes, for example, with the use of logic models, outputs, short-term, and intermediate out-

comes. The importance of tailoring strategies to the situation and attending to the organizational cul-

ture were both mentioned as important lessons learned.

Table 10. Organization-Level Outcomes

Outcome Categories Ma n %

Any organization-level outcomes 47 77.0Processes, policies, and practices 2.7 44 72.1

Improved Organizational Capacity 15 24.6Improved Ability to Get Funding 16 26.2Increase in Doing Evaluation 26 42.6Increase in Using Evaluation 24 39.3Plan for future ECB Activities 14 23.0Plan for future evaluations 20 32.8Other process practice or policies 5 8.2

Leadership: Increased Support 1 8 13.1Organizational Culture 1.6 17 27.9

Learning, participation, collaboration 10 16.4Evaluation/data use 16 26.2Other organizational culture 2 3.3

Mainstreaming 1.8 33 54.1Evaluation is more routine 11 18.0Jobs redesigned to include evaluation 16 26.2New relationships among organizations 18 29.5Ongoing learning opportunities 13 21.3Other mainstreaming outcome 0 0.0General mention of mainstreaming 2 3.3

Resources 1.9 28 45.9Written evaluation materials 6 9.8Technology 14 23.0More resources for evaluation 12 19.7Internal evaluation expertise 18 29.5Other resource/resources 2 3.3

The total number of cases is 61. Totals may exceed 61 because subcategories are not mutually exclusive and multipleresponses are possible.aMean number of outcomes reported from total number of possible outcomes for cases where an outcome of that type wasreported, that is, total possible outcomes for PPP (7); leadership (1); culture (3); mainstreaming (6); resources (5).

Table 11. Number of Strategies (Training, Technical Assistance, & Experiential Involvement) by the Presenceof Preexisting Resources

Resources: StaffTime, Money

All CasesNone

Reported

Any OneStrategyReported

Any TwoStrategiesReported

All ThreeStrategiesReported

Total % WithinResources Categoriesn % n % n % n % n %

Resources reported 22 36.1 2 9.1 6 27.3 6 27.3 8 36.4 100.0Resources not reported 39 63.9 0 0.0 10 25.6 13 33.3 16 41.0 100.0Total 61 100.0 2 3.3 16 26.2 19 31.1 24 39.3 100.0

Labin et al. 17



Discussion

The findings of this synthesis of the empirical ECB literature reflect the emergent nature of the field

but also suggest that ECB efforts have a relatively common set of principles and strategies and have

achieved some short-term and intermediate-term outcomes at both the individual and organizational

levels. In spite of the variation in the narrative reporting of ECB, there is a high degree of consis-

tency between the concepts in the theoretical literature and those found in the empirical literature.

This consistency confirms that the ECB field has developed beyond its infancy and demonstrates the

field’s need and readiness for more systematic reporting practices and evaluation methods. In the

discussion below, we offer additional suggestions for using the findings from the synthesis in imple-

menting and evaluating ECB.

Collaboration

As mentioned earlier, ECB has deep roots in collaborative, participatory, and empowerment evalua-

tion approaches (Cousins et al., 2004; Cousins & Whitmore, 1998; Fetterman & Wandersman, 2005;

Preskill & Torres, 2009). In this empirical synthesis, collaboration emerged as the essential thread in

the fabric of ECB efforts, warranting its explicit inclusion as a key concept in ECB models, efforts,

and evaluations. We found collaborative issues reported as an aspect of existing strengths, strategies,

barriers, individual-level outcomes, and organizational-level outcomes. Collaborative issues were

not explicitly included in the original Integrative ECB Model but have been added. Collaborative

and participatory processes involve the ways in which people interact and can be considered part

of the human relations of ECB. There was also convergence of evidence from the synthesis that sug-

gests the need for more attention to the other human relation dimensions of ECB such as targeting

attitudes, leadership, and a supportive organizational culture.

Collaboration between funders and projects may also be something to explore. Funders were not

reported as being participants in the ECB efforts, but there was mention of their importance to the

efforts. Adequate resources are needed not only to begin ECB efforts, but also to sustain them. If

funders were included as target participants in the ECB efforts, it could increase their first-hand

knowledge of ECB efforts and requirements, which, in turn, could affect expectations and funding

cycles and reduce related resource and staff-turnover barriers. These hypotheses merit further

exploration.

Strategies and Outcomes

The synthesis findings confirm the importance of participatory processes in ECB strategies. Consis-

tent with the emphasis in the literature (Fetterman & Wandersman, 2005; Patton, 2012), experiential

learning through involvement or participation in an evaluation was a major type of strategy or

Table 12. Types of Strategies by the Presence of Preexisting Resources

Resources: Staff, Time, Money

All CasesNone

Reported TrainingTechnicalAssistance

ExperientialInvolvement

n % n % n % n % n %

Resources reported 22 36.1 2 9.1 15 68.2 14 63.6 13 59.1Resources not reported 39 63.9 0 0.0 31 79.5 24 61.5 28 71.8Total 61 100.0 2 3.3 46 75.4 38 62.3 41 67.2

The total number of cases is 61. Totals may exceed 61 because categories are not mutually exclusive and multiple responsesare possible.




Tab

le13.

Typ

esofO

utc

om

esby

Typ

eofSt

rate

gies Indiv

idual

-Lev

elO

utc

om

esO

rgan

izat

ional

-Lev

elO

utc

om

es

All

Cas

esA

ttitude

Know

ledge

Beh

avio

rPPP

Lead

ersh

ipC

ulture

Mai

nst

ream

ing

Res

ourc

es

Num

ber

&T

ype

ofSt

rate

gies

n%

n%

n%

n%

n%

n%

n%

n%

n%

Only

One

Stra

tegy

Tra

inin

g10

16.4

220.0

880.0

660.0

660.0

220.0

220.0

440.0

330.0

TA

11.6

00.0

1100.0

1100.0

1100.0

00.0

00.0

1100.0

1100.0

Exper

ience

58.2

240.0

120.0

5100.0

360.0

00.0

120.0

360.0

360.0

Any

two

stra

tegi

es19

31.1

631.6

842.1

13

68.4

14

73.7

210.5

526.3

10

52.6

842.1

All

thre

est

rate

gies

24

39.3

11

45.8

13

54.2

23

95.8

18

75.0

312.5

729.2

13

54.2

11

45.8

None

report

ed2

3.3

150.0

00.0

150.0

2100.0

150.0

2100.0

2100.0

2100.0

Tota

l61

100.0

22

36.1

31

50.8

49

80.3

44

72.1

813.1

17

27.9

33

54.1

28

45.9

PPP¼

pro

cess

es,polic

ies,

and

pra

ctic

es;T

A¼

tech

nic

alas

sist

ance

.T

he

tota

lnum

ber

ofca

ses

is61.T

ota

lsm

ayex

ceed

61

bec

ause

outc

om

eca

tego

ries

are

not

mutu

ally

excl

usi

vean

dm

ultip

lere

sponse

sar

eposs

ible

.

19 at American Evaluation Association on February 9, 2012aje.sagepub.comDownloaded from


Tab

le14.

Typ

esofO

utc

om

esby

Conte

nt

ofSt

rate

gies

Indiv

idual

Outc

om

esO

rgan

izat

ional

Outc

om

es

Conte

nt

ofEC

Bst

rate

gy

All

Cas

esW

ith

Conte

nt

Att

itude

Know

ledge

Beh

avio

rPPP

Lead

ersh

ipC

ulture

Mai

nst

ream

Res

ourc

es

N%

n%

n%

n%

n%

n%

n%

n%

n%

Indiv

idual

Only

29

48.3

931.0

19

65.5

22

75.8

15

51.7

00.0

26.9

10

34.5

724.1

Indiv

idual

and

org

aniz

atio

nal

27

45.0

13

48.1

10

37.0

26

96.3

24

88.9

829.6

13

48.1

20

74.1

17

63.0

Org

aniz

atio

nal

only

46.7

00.0

125.0

125.0

4100.0

00.0

250.0

375.0

4100.0

Not

report

ed1

00

00

00

00

00

00

00

00

0T

ota

l61

100

22

36

30

49

49

80

43

70

813

17

28

33

54

28

46

EC

B¼

eval

uat

ion

capac

ity

build

ing;

PPP¼

pro

cess

es,polic

ies,

and

pra

ctic

es.

The

tota

lnum

ber

ofca

ses

is61.T

ota

lsm

ayex

ceed

61

bec

ause

outc

om

eca

tego

ries

are

not

mutu

ally

excl

usi

vean

dm

ultip

lere

sponse

sar

eposs

ible

.



delivery mechanism and was associated with a high frequency of individual behavioral outcomes as

was the use of the combination of all three types of strategies, that is, experiential, training, and TA.

Training was associated with a high frequency of individual knowledge outcomes. This suggests that

the optimal use of multiple strategies—training, participating in an evaluation, and TA—for achiev-

ing individual knowledge and behavioral outcomes.

Multiple strategies may also be optimal for achieving organizational outcomes such as PPP, cul-

ture, and mainstreaming. TA may play a critical role in developing and sustaining organizational

changes. Therefore, various stakeholders, including funders, may want to consider the need for

ongoing resources for TA.

In this synthesis, we examined specific components of evaluation such as planning, doing, and

using data––key elements in definitions of ECB. Stevenson, Florin, Mills, and Andrade (2002) have

concluded that ECB should focus on selected components of evaluation (i.e., planning and designing

evaluation) and less on other skills (i.e., data analysis). The findings in this synthesis suggest support

for Stevenson and his colleagues. For example, collecting and analyzing data were targeted with

about the same frequency in the strategies as they were reported as knowledge outcomes. However,

behavior outcomes of collecting data were more frequent than those for analyzing data. Therefore,

while the content may equally address collecting and analyzing data (and knowledge outcomes are

commensurately equal), there are more behavioral outcomes of collecting data than of conducting

analyses. Furthermore, there were reports of barriers or difficulties in learning the subject matter and

of needing help with analyzing data. Thus, more attention in the future to which evaluation skills are

most important and feasible seems warranted.

Notwithstanding the conceptual literature that stresses the importance of motivation (Preskill &

Boyle, 2008) and attitudes (Owen, 2003) toward evaluation and ECB, this empirical synthesis did

not find much attention directed to targeting such attitudes. Attitudes were not often reported as a

preexisting strength, as content for strategies, or as an outcome. Conversely, negative attitudes

were frequently noted as a barrier and the importance of attitudes was mentioned in other com-

ments. All of these findings suggest the need for ECB efforts to pay more attention to targeting

the development of positive attitudes toward evaluation and understanding how to overcome neg-

ative attitudes.

Individual, Organizational, and Program Outcomes

Of the five types of organizational ECB outcomes examined––PPP, leadership, culture, mainstreaming,

resources—PPP is the outcome that occurred with the greatest frequency. The indicators for

PPP included ‘‘doing’’ and ‘‘using’’ evaluation, which are essential elements in the working defini-

tion of ECB used in this synthesis as well as in other definitions and frameworks of ECB. Main-

streaming was the second most frequently cited organizational outcome and its indicators reflect

changes needed for the sustainability of ECB such as making evaluation more routine and changing

the way people do their jobs. The frequency with which change occurs for each of the five charac-

teristics of organizational capacity may reflect varying levels of difficulty in achieving change. For

example, that mainstreaming occurs less frequently than PPP suggests that it is more difficult to

achieve. Similarly, leadership is the least frequently reported outcome and may be one of the more

difficult to affect.

While all the factors reinforce each other, leadership, culture, and resources may be factors that

play a particularly important role in supporting the other organizational factors. It is difficult to envi-

sion PPP or mainstreaming advancing without at least certain minimal levels of these other factors.

Yet, leadership was the least frequently targeted organizational factor and the least frequently

reported organizational outcome. This suggests that more attention should be paid to defining, tar-

geting, developing and measuring leadership.

Labin et al. 21



There may be optimal sequencing regarding the targeting of particular organizational character-

istics before others. For example, perhaps PPP should be targeted before mainstreaming, consistent

with the concept that ECB is developmental and occurs in stages (Gibbs et al., 2002; Love, 1991;

Owen, 2003).

Another issue to consider is the sequence and relationship of individual outcomes to organiza-

tional outcomes. The findings here indicate that individual outcomes of attitudes and behaviors were

more frequent when both individual and organizational content or change was targeted in the stra-

tegies than when only individual content or change was targeted (Table 14). This is consistent with

Preskill and Boyle’s (2008) discussion of the importance of organizational factors for individual

learning to take place. It also supports Taylor-Ritzler, et al. (2010) who found that an organizational

environment conducive to evaluation was necessary to increase individual motivation and behavior

change.

The organizations that are the target of the ECB efforts are generally service organizations

that are responsible for developing and implementing programs, predominately in the health

and education fields. An important purpose of ECB is to support the efforts of organizations

to improve their programs and program outcomes for their service populations. While program

outcomes were not in the original Integrative ECB Model, they emerged as an important issue

from the synthesis and were added to the final Integrative ECB Model (Figure 1). Achieving

program outcomes has long been an important and explicit goal of the evaluation field, of

empowerment evaluation (Fetterman & Wandersman, 2005), and of ECB efforts in general. The

explicit inclusion of program outcomes to the model is empirically justified and hopefully their

inclusion will be helpful in orienting ECB theory, models, and strategies toward achieving pro-

gram outcomes.

Evaluation of ECB

The information in the original articles describing the evaluations of each ECB case was limited and

often communicated in qualitative narratives. A strength of the evaluations reported was the large

number of cases that used a variety of data collection methods. However, there are weaknesses in

the evaluations, especially in regard to very limited reporting of measures and quantitative data. This

seems somewhat surprising for a field embedded in evaluation and populated by evaluators. It may

be useful to understand what is driving the current state of evaluation of ECB in order to inform sug-

gestions of how to improve it.

We do know that ECB is a complex phenomenon involving issues of individual learning,

organizational change, sustained change, and program processes and outcomes. It is important to

continue to look at all of these areas and build on existing knowledge (Compton et al., 2002;

Cousins, et al., 2004; Preskill & Boyle, 2008). Nevertheless, the challenges to evaluating ECB

derive from the complex interplay within and across individual and organizational factors, the

difficulty of isolating causal factors of change both for the organization and for program outcomes,

and the long-term goals of sustainable change (Wing, 2004). Furthermore, comparative designs are

difficult to implement long-term because of the need to keep the treatment and comparison groups

sufficiently separate and distinct over time. The complexity of these issues is a challenge for mean-

ingful evaluations of ECB that adequately represent the important aspects and effects of ECB and

not just the easy-to-measure effects.

Added to this complexity is the fact that ECB is a relatively new area, still being defined and

without agreed upon measures. As with all fledging endeavors, it is important to use evaluation

methods that will encourage and not squelch development. However, there are numerous principles

and methods that evaluators have in their toolbox that can enhance the state of ECB evaluation. The

first priority is to focus on developing, using, and reporting on ECB with common terminology,




indicators, and rigorous measures. The second priority is improvement of evaluation designs with

the judicious use of comparison or control groups for short-term and intermediate-term outcomes.

Improving designs also includes planning now for longitudinal studies with longer time frames in

order to address the definitions and goals of ECB as sustainable organizational change (Baizerman,

Compton, & Stockdill, 2002b). Such efforts could propel the field forward and foster an evidence-

based practice for ECB.

The Integrative Model and Synthesis Method: Implications forMeasurement and Practice

The Integrated ECB model proved to be useful and the findings of the synthesis indicated the need

for a couple of additions to the model of important reoccurring themes in the empirical literature.

These additions included explicit inclusion of collaborative and participatory aspects and processes

to most elements of the model and inclusion of program outcomes.

The methodology produced a reliable empirical knowledge base from a diverse body of literature.

Therefore, this study demonstrates the viability of a systematic synthesis research method for a

developing field of inquiry using narrative accounts of programs and evaluations as well as the fea-

sibility of codifying such knowledge. The systematic coding process created an operationalization of

concepts that provides an empirical basis for developing measures to be used in the field and in

future evaluations. The strength of this study is its ability to summarize existing knowledge, but its

inherent dependence on what is reported in the literature is also a limitation of this and all syntheses.

The Integrative ECB Model, the questions derived from it, and the analyses conducted were all

developed with the goal of painting a reliable description of the current empirical ECB field and a

preliminary assessment of some basic relationships. Certainly this, as one piece of research, has its

particular limitations. There are undoubtedly more elements to be added to the model, more analyses

to be conducted to answer additional questions, and more ECB efforts to be reviewed.

One means to address these limitations would be to think of this synthesis as a database. As most

databases, their first iteration is not their last nor is any one set of analyses definitive; rather it is a

resource to be refined, added to, and accessed for further analyses. Expanding on the idea of a data-

base are other means of developing knowledge and practice that are being used in other areas and

that could be used to develop the ECB field. One is a clearinghouse for ECB program materials and

measures, whereas another is online access to detailed information about ECB efforts, implementa-

tion, and outcomes such as those used in projects and sites compiling effective practices for specific

areas, for example, substance abuse or HIVAIDS (GAO, 2009; KIT Solutions, 2011; Tanglewood

Research, 2011). An ECB website could be established and users could submit entries of additional

ECB examples for review. The coding categories for the elements of the Integrative ECB Model

could be posted to encourage common language and collaborative input, and to contribute to the

refinement of existing measures (Preskill & Torres, 2009; Stevenson, et. al, 2002; Taylor-Ritzler,

Suarez-Balcazar, & Garcia-Iriarte, 2010; Volkov & King, 2007). Focusing on developing, using, and

reporting with common measures would allow the field to define and assess strategies and outcomes

and empirically examine the causal sequences and relationships hypothesized in the various theore-

tical ECB models. This would enhance the empirical data to guide ECB practice and evaluation.

In summary, we have demonstrated that the field of ECB has reported many outcomes, but it still

has much room for growth and refinement. Our analysis highlights the frequency and types of indi-

vidual and organizational outcomes reported while also noting some preliminary results of relation-

ships between strategies and outcomes. Next steps for the field include more complete articulations

of the ECB process and more rigorous measurement and evaluation of these processes. This would

contribute to the ECB field becoming increasingly effective at achieving its long-term goals includ-

ing positive program outcomes.

Labin et al. 23



Cas

eN

um

ber

Num

ber

of

Art

icle

s/C

hap

ters

for

This

Cas

eA

rtic

le/A

rtic

les

and/o

rC

hap

ter/

Chap

ters

Use

dto

Code

Cas

es

Num

ber

ofca

ses

Bas

edon

Eac

hA

rtic

le

1.

1A

ndre

ws,

A.B.,

Mote

s,P.S.

,Fl

oyd

,A

.G

.,Fl

erx,V

.C

.,&

Lopez

-De

Fede,

A.(2

006).

Build

ing

eval

uat

ion

capac

ity

inco

mm

unity-

bas

edorg

aniz

atio

ns:

refle

ctio

ns

ofa

nem

pow

erm

ent

eval

uat

ion

team

.Jou

rnal

ofCom

mun

ityPr

actic

e,13,85–104.

1

2.

1A

rnold

,M

.E.(2

006).

Dev

elopin

gev

aluat

ion

capac

ity

inex

tensi

on

4-H

field

facu

lty:

Afr

amew

ork

for

succ

ess.

Am

eric

anJo

urna

lof

Eva

luat

ion,

27,257.

1

3.

1A

stra

movi

ch,R

.L.

,C

oke

r,J.

K.,

&H

osk

ins,

W.J.

(2005).

Tra

inin

gsc

hoolco

unse

lors

inpro

gram

eval

uat

ion.Pr

ofes

sion

alSc

hool

Cou

nsel

ing,

9,49–54.

1

4.

1A

tkin

son,D

.D

.,W

ilson,M

.,&

Avu

la,D

.(2

005).

Apar

tici

pat

ory

appro

ach

tobuild

ing

capac

ity

of

trea

tmen

tpro

gram

sto

enga

gein

eval

uat

ion.Eva

luat

ion

&Pr

ogra

mPl

anni

ng,28,329–334.

1

5.

1Bee

re,D

.(2

005).

Eva

luat

ion

capac

ity

build

ing:

Ata

leofva

lue

addin

g.Eva

luat

ion

Jour

nalo

fAus

tral

asia

,5,

41–47.

1

6.

1Bra

ndon,P.R

.,&

Hig

a,T

.A

.(2

004).

An

empir

ical

study

ofbuild

ing

the

eval

uat

ion

capac

ity

ofK

-12

site

man

aged

pro

ject

per

sonnel

.The

Can

adia

nJo

urna

lof

Prog

ram

Eva

luat

ion,

19(1

),125–142.

1

7.

1Bro

wn,N

.L.

,Lu

na,

V.,

Ram

irez

,M

.H

.,V

ail,

K.A

.,&

Will

iam

s,C

.A

.(2

005).

Dev

elopin

gan

effe

ctiv

ein

terv

ention

for

IDU

wom

en:A

har

mre

duct

ion

appro

ach

toco

llabora

tion.AID

SPr

even

tion

and

Edu

catio

n,17(4

),317–333.

1

8.

1Bro

wn,R

.E.&

Ree

d,C

.S.

(2002).

An

inte

gral

appro

ach

toev

aluat

ing

outc

om

eev

aluat

ion

trai

nin

g.Am

eric

anJo

urna

lof

Eva

luat

ion,

23,1–17.

1

9.

1C

ampbel

l,R

.,D

ore

y,H

.,N

aege

li,M

.,G

rubst

ein,L.

K.,

Ben

net

t,K

.K

.,Bonte

r,F.

,..

.D

avid

son,W

.S.

(2004).

An

empow

erm

ent

eval

uat

ion

model

for

sexual

assa

ult

pro

gram

s:Em

pir

ical

evid

ence

of

effe

ctiv

enes

s.Am

eric

anJo

urna

lof

Com

mun

ityPs

ycho

logy

,34,251–262.

1

10.

1C

arden

,F.

&Ear

l,S.

(2007).

Infu

sing

eval

uat

ive

thin

king

aspro

cess

use

:T

he

case

ofth

ein

tern

atio

nal

dev

elopm

ent

rese

arch

centr

e(I

DR

C).

New

Direc

tions

for

Eva

luat

ion,

116,61.

1

11.

1C

hin

man

,M

.,H

unte

r,S.

B.,

Eben

er,P.,

Pad

dock

,S.

M.,

Still

man

,L.

,Im

m,P.,

&W

ander

sman

,A

.(2

008).

The

gett

ing

tooutc

om

esdem

onst

ration

and

eval

uat

ion:A

nill

ust

ration

ofth

epre

vention

support

syst

em.Am

eric

anJo

urna

lof

Com

mun

ityPs

ycho

logy

,41,143–157.

1

12.

1C

ohen

,C

.(2

006).

Eva

luat

ion

lear

nin

gci

rcle

s:A

sole

pro

pri

etor’

sev

aluat

ion

capac

ity-

build

ing

stra

tegy

.N

ewD

irec

tions

for

Eva

luat

ion,

111,85–93.

1 (con

tinue

d)

Ap

pen

dix

Ao

fE

CB

Case

s



(continued

)

Cas

eN

um

ber

Num

ber

of

Art

icle

s/C

hap

ters

for

This

Cas

eA

rtic

le/A

rtic

les

and/o

rC

hap

ter/

Chap

ters

Use

dto

Code

Cas

es

Num

ber

ofca

ses

Bas

edon

Eac

hA

rtic

le

13.

5C

om

pto

n,D

.,M

.Bai

zerm

an,et

al.(2

001).

Dev

elopin

gev

aluat

ion

capac

ity

while

impro

ving

eval

uat

ion

trai

nin

gin

public

hea

lth:T

he

Am

eric

anC

ance

rSo

ciet

y’s

Colla

bora

tive

Eva

luat

ion

Fello

ws

Pro

ject

.Eva

luat

ion

and

Prog

ram

Plan

ning

,24(1

):33–40.

Com

pto

n,D

.W

.,G

love

r-K

udon,R

.,Sm

ith,I.,

&A

very

,M

.E.(2

002).

Ongo

ing

capac

ity

build

ing

inth

eA

mer

ican

Can

cer

Soci

ety

(AC

S)1995-2

001.N

ewD

irec

tions

for

Eva

luat

ion,

93,47–62.

Com

pto

n,D

.W

.,G

love

r-K

udon,R

.,A

very

,E.,

&M

orr

is,C

.L.

(2001).

The

colla

bora

tive

eval

uat

ion

fello

ws

pro

ject

:Bac

kgro

und

and

ove

rvie

wofth

em

odel

.Can

cer

Prac

tice,

9(1

),S4

–S1

0.

Pre

skill

,H

.,D

.W

.C

om

pto

n,D

.W.,

Smith,M

.,(2

001).

Inte

grat

ing

theo

ryan

dpra

ctic

e—C

once

ptu

alfr

amew

ork

softh

eC

EFP

.Can

cer

Prac

tice

9:S1

7–S2

2.

Bonnet

,D.(

2001).

An

exte

rnal

eval

uat

or’

sper

spec

tive

on

the

colla

bora

tive

eval

uat

ion

fello

ws

pro

ject

.Can

cer

Prac

tice,

9,S7

2–77.

1

14.

1D

iaz-

Puen

te,J

.M.,

Yag

ue,

J.L.

,&A

fonso

,A.(

2008).

Build

ing

eval

uat

ion

capac

ity

inSp

ain:A

case

study

of

rura

ldev

elopm

ent

and

empow

erm

ent

inth

eEuro

pea

nU

nio

n.Eva

luat

ion

Rev

iew

,32,478–506.

1

15.

1D

orr

ingt

on,C

.,&

Solıs

,B.(

2004).

Build

ing

com

munity,

rese

arch

,and

polic

y:A

case

ofco

mm

unity

hea

lth

and

Cen

tral

Am

eric

ans

inLo

sA

nge

les.

InJ.

Mora

&D

.R.D

iaz

(Eds.

),La

tino

soci

alpo

licy:

Apa

rtic

ipat

ory

rese

arch

mod

el(p

p.89–117).

New

York

:H

awort

h.

1

16.

1Fe

bey

,K

.,&

Coyn

e,M

.(2

007).

Pro

gram

eval

uat

ion:T

he

boar

dga

me

anin

tera

ctiv

ele

arnin

gto

olfo

rev

aluat

ors

.Am

eric

anJo

urna

lof

Eva

luat

ion,

28,91–101.

1

17.

1Fe

tter

man

,D

.&

Bow

man

,C

.(2

002).

Exper

iential

educa

tion

and

empow

erm

ent

eval

uat

ion:m

ars

rove

red

uca

tional

pro

gram

case

exam

ple

.Jo

urna

lof

Exp

erie

ntia

lEdu

catio

n,25,286.

18.

1Fe

tter

man

,D

.,Sc

hnei

der

man

,N

.,Sp

eers

,M

.A

.,Si

lva,

J.M

.,T

om

es,H

.,&

Gen

try,

J.H

.(2

001).

Em

pow

erm

ent

eval

uat

ion

and

self-

det

erm

inat

ion:A

pra

ctic

alap

pro

ach

tow

ard

pro

gram

impro

vem

ent

and

capac

ity

build

ing.

InIn

tegr

atin

gbe

havior

alan

dso

cial

scie

nces

with

publ

iche

alth

(pp.321–350).

Was

hin

gton,D

C:A

mer

ican

Psy

cholo

gica

lA

ssoci

atio

n.

19.

20.

1Fe

tter

man

,D.M

.(2005).

Em

pow

erm

ent

eval

uat

ion:F

rom

the

dig

ital

div

ide

toac

adem

icdis

tres

s.In

D.M

.Fe

tter

man

&A

.W

ander

sman

(Eds.

),Em

pow

erm

ent

eval

uatio

npr

inci

ples

inpr

actic

e(p

p.92–122).

New

York

,N

Y:G

uilf

ord

Pre

ss.

2 (con

tinue

d)



(continued

)

Cas

eN

um

ber

Num

ber

of

Art

icle

s/C

hap

ters

for

This

Cas

eA

rtic

le/A

rtic

les

and/o

rC

hap

ter/

Chap

ters

Use

dto

Code

Cas

es

Num

ber

ofca

ses

Bas

edon

Eac

hA

rtic

le

21.

2Fl

aspohle

r,P.,

Wan

der

sman

,A.,

Kee

ner

,D.,

Max

wel

l,K

.N.,

Ace

,A.,

Andre

ws,

A.,

&H

olm

es,B

.(2003).

Pro

moting

pro

gram

succ

ess

and

fulfi

lling

acco

unta

bili

tyre

quir

emen

tsin

ast

atew

ide

com

munity-

bas

edin

itia

tive

:Chal

lenge

s,pro

gres

s,an

dle

ssons

lear

ned

.Jou

rnal

ofPr

even

tion

&In

terv

entio

nin

the

Com

mun

ity,

26,37–52.

Wan

der

sman

,A.,

Flas

pohle

r,P.,

Ace

,A.,

Ford

,L.,

Imm

,P.S

.,C

hin

man

,M.J

.,..

.K

aufm

an,J

.S.(

2003).

PIE

ala

mode:

Mai

nst

ream

ing

eval

uat

ion

and

acco

unta

bili

tyin

each

pro

gram

inev

ery

county

ofa

stat

ewid

esc

hoolre

adin

ess

initia

tive

.N

ewD

irec

tions

for

Eva

luat

ion,

99,33–49.

1

22.

1H

arper

,G

.,C

ontr

eras

,R

.,Ban

gi,A

.,&

Ped

raza

,A

.(2

003).

Colla

bora

tive

pro

cess

eval

uat

ion:en

han

cing

com

munity

rele

vance

and

cultura

lap

pro

pri

aten

ess

inH

IVpre

vention.Jo

urna

lof

Prev

entio

n&

Inte

rven

tion

inth

eCom

mun

ity,26,53–71.

1

23.

24.

25.

1H

oole

,E.,

&Pat

ters

on,T

.E.,

(2008).

Voic

esfr

om

the

field

:Eva

luat

ion

aspar

tofa

lear

nin

gcu

lture

.New

Direc

tions

for

Eva

luat

ion,

119,93–113.

3

26.

2H

uffm

an,D

.,La

wre

nz,

F.,T

hom

as,K

.,&

Cla

rkso

n,L.

(2006).

Colla

bora

tive

eval

uat

ion

com

munitie

sin

urb

ansc

hools

:A

model

ofev

aluat

ion

capac

ity

build

ing

for

STEM

educa

tion.N

ewD

irec

tions

for

Eva

luat

ion,

109,73–85.

Huffm

an,D

.,T

hom

as,K

.,&

Law

renz,

F.(2

008).

Aco

llabora

tive

imm

ersi

on

appro

ach

toev

aluat

ion

capac

ity

build

ing.

Am

eric

anJo

urna

lof

Eva

luat

ion,

29,358–368

1

27.

2K

atz,

S.,Su

ther

land,S.

,&

Ear

l,L.

(2002).

Dev

elopin

gan

eval

uat

ion

hab

itofm

ind.Can

adia

nJo

urna

lof

Prog

ram

Eva

luat

ion,

17,103–119.

Lee,

L.E.(1

999).

Build

ing

capac

ity

for

schoolim

pro

vem

ent

thro

ugh

eval

uat

ion:ex

per

ience

softh

eM

anitoba

schoolim

pro

vem

ent

pro

gram

Inc.

Can

adia

nJo

urna

lof

Prog

ram

Eva

luat

ion

[Spec

ialIs

sue]

,155–178.

1

28.

29.

1K

eener

,D

.C

.,Sn

ell-Jo

hns,

J.,Li

vet,

M.,

&W

ander

sman

,A

.(2

005).

Less

ons

that

influ

ence

dth

ecu

rren

tco

nce

ptu

aliz

atio

nofEm

pow

erm

ent

Eva

luat

ion:R

efle

ctio

ns

from

two

eval

uat

ion

pro

ject

s.In

D.M

.Fe

tter

man

&A

.W

ander

sman

(Eds.

),Em

pow

erm

ent

eval

uatio

npr

inci

ples

inpr

actic

e(p

p.73–91).

New

York

,N

Y:G

uilf

ord

.

2

30.

1K

ing,

J.A

.(2

002).

Build

ing

the

eval

uat

ion

capac

ity

ofa

schooldis

tric

t.N

ewD

irec

tions

for

Eva

luat

ion,

93,

63–80.

1

31.

1K

irsh

,B.,

Kru

pa,

T.,

Horg

an,S

.,K

elly

,D.,

&C

arr,

S.(2

005).

Mak

ing

itbet

ter:

Build

ing

eval

uat

ion

capac

ity

inco

mm

unity

men

talhea

lth.Ps

ychi

atric

Reh

abilita

tion

Jour

nal,

28,234–241.

1 (con

tinue

d)



(continued

)

Cas

eN

um

ber

Num

ber

of

Art

icle

s/C

hap

ters

for

This

Cas

eA

rtic

le/A

rtic

les

and/o

rC

hap

ter/

Chap

ters

Use

dto

Code

Cas

es

Num

ber

ofca

ses

Bas

edon

Eac

hA

rtic

le

32.

1Le

nnie

,J.

(2005).

An

Eva

luat

ion

Cap

acity-

Build

ing

Pro

cess

for

Sust

ainab

leC

om

munity

ITIn

itia

tive

s:Em

pow

erin

gan

dD

isem

pow

erin

gIm

pac

ts.Eva

luat

ion,

11,390–414.

1

33.

1Le

ntz

,B.E.,

Imm

,P.S.

,Y

ost

,J.

B.,

Johnso

n,N

.P.,

Bar

ron,C

.,Li

ndber

g,M

.S.

,&

Tre

istm

an,J.

(2005).

Em

pow

erm

ent

eval

uat

ion

and

org

aniz

atio

nal

lear

nin

g:A

case

study

ofa

com

munity

coal

itio

ndes

igned

topre

vent

child

abuse

and

neg

lect

.In

D.M

.Fe

tter

man

&A

.W

ander

sman

(Eds.

),Em

pow

erm

ent

eval

uatio

npr

inci

ples

inpr

actic

e(p

p.155–82).

New

York

:G

uilf

ord

.

1

34.

1M

acLe

llan-W

righ

t,M

.F.

,Pat

ten,S.

,del

aC

ruz,

A.M

.,&

Flah

erty

,A

.(2

007).

Apar

tici

pat

ory

appro

ach

toth

edev

elopm

ent

ofan

eval

uat

ion

fram

ework

:Pro

cess

,pitfa

lls,a

nd

pay

offs.

Can

adia

nJo

urna

lofPr

ogra

mEva

luat

ion,

22,99–124.

1

35.

1M

aher

,C

.A

.(1

981).

Tra

inin

gofm

anag

ers

inpro

gram

pla

nnin

gan

dev

aluat

ion:C

om

par

ison

oftw

oap

pro

aches

.Jo

urna

lof

Org

aniz

atio

nalB

ehav

ior

Man

agem

ent,

3,45–56.

1

36.

1M

athew

s,M

.,&

Lynch

,A

.(2

007).

Incr

easi

ng

rese

arch

skill

sin

rura

lhea

lth

boar

ds:

anev

aluat

ion

ofa

trai

nin

gpro

gram

from

Wes

tern

New

foundla

nd.Can

adia

nJo

urna

lof

Prog

ram

Eva

luat

ion,

22,41–56.

1

37.

1M

cDonal

d,B.,

Roge

rs,P.,

&K

effo

rd,B.(2

003).

Tea

chin

gpeo

ple

tofis

h?

Build

ing

eval

uat

ion

capac

ity

of

public

sect

or

org

aniz

atio

ns.

Eva

luat

ion,

9,9–29.

1

38.

1M

iller

,W

.,&

Lennie

,J.

(2005).

Em

pow

erm

ent

eval

uat

ion:A

pra

ctic

alm

ethod

for

eval

uat

ing

anat

ional

schoolbre

akfa

stpro

gram

.Eva

luat

ion

Jour

nalo

fAus

tral

asia

,5,18–26.

1

39.

1M

ilste

in,B.,

Chap

el,T

.J.,

Wet

terh

all,

S.F.

,&

Cott

on,D

.A

.(2

002).

Build

ing

capac

ity

for

pro

gram

eval

uat

ion

atth

eC

ente

rsfo

rD

isea

seC

ontr

ola

nd

Pre

vention.N

ewD

irec

tions

forEva

luat

ion,

93,2

7–46.

1

40.

1M

oon,S.

M.(1

996).

Usi

ng

the

Purd

ue

thre

e-st

age

model

tofa

cilit

ate

loca

lpro

gram

eval

uat

ions.

Gift

edChi

ldQ

uart

erly,40,121–128.

1

41.

1M

yric

k,R

.,Le

mel

le,A

.,A

oki

,B.,

Tru

ax,S

.,&

Lem

p,G

.(2005).

Bes

tpra

ctic

esfo

rco

mm

unity

colla

bora

tive

rese

arch

.AID

SEdu

catio

n&

Prev

entio

n,17,400–404.

1

42.

1N

acca

rella

,J.,

Pir

kis,

J.,K

ohn,F

.,M

orl

ey,B

.,Burg

ess,

P.,

&Bla

shki

,G.(

2007).

Build

ing

eval

uat

ion

capac

ity:

Def

initio

nal

and

pra

ctic

alim

plic

atio

ns

from

anA

ust

ralia

nca

sest

udy.

Eva

luat

ion

and

Prog

ram

Plan

ning

,30,231–236.

1

43.

1N

agao

,M.,

Kuji-

Shik

atan

i,K

.,&

Love

,A.J

.(2005).

Pre

par

ing

schoole

valu

ators

:Hir

osh

ima

pilo

tte

stoft

he

Japan

eval

uat

ion

soci

ety’

sac

cred

itat

ion

pro

ject

.Can

adia

nJo

urna

lof

Prog

ram

Eva

luat

ion,

20,125–155.

1 (con

tinue

d)



(continued

)

Cas

eN

um

ber

Num

ber

of

Art

icle

s/C

hap

ters

for

This

Cas

eA

rtic

le/A

rtic

les

and/o

rC

hap

ter/

Chap

ters

Use

dto

Code

Cas

es

Num

ber

ofca

ses

Bas

edon

Eac

hA

rtic

le

44.

2O

’Sulli

van,R

.,&

D’A

gost

ino,A

.(2

002).

Pro

moting

eval

uat

ion

thro

ugh

colla

bora

tion:Fi

ndin

gsfr

om

com

munity-

bas

edpro

gram

sfo

ryo

ung

child

ren

and

thei

rfa

mili

es.Eva

luat

ion,

8,372–387.

O’S

ulli

van,R

.G

.,&

O’S

ulli

van,J.

M.(1

998).

Eva

luat

ion

voic

es:Pro

moting

eval

uat

ion

from

within

pro

gram

sth

rough

colla

bora

tion.Eva

luat

ion

and

Prog

ram

Plan

ning

,21,21–29.

1

45.

1O

wen

,J.

M.(2

003).

Eva

luat

ion

culture

:a

def

initio

nan

dan

alys

isofits

dev

elopm

ent

within

org

aniz

atio

ns.

Eva

luat

ion

Jour

nalo

fAus

tral

asia

,3,43–47.

1

46.

1Plo

eg,J.,

de

Witt,

L.,H

utc

his

on,B.,

Hay

war

d,L.

,&

Gra

yson,K

.(2

008).

Eva

luat

ion

ofa

rese

arch

men

tors

hip

pro

gram

inco

mm

unity

care

.Eva

luat

ion

and

Prog

ram

Plan

ning

,31,22–33.

1

47.

1Port

eous,

N.L.

,Sh

eldri

ck,B.J.,

&St

ewar

t,P.J.

(1999).

Enhan

cing

man

ager

s’ev

aluat

ion

capac

ity:

Aca

sest

udy

from

Onta

rio

Public

Hea

lth.Can

adia

nJo

urna

lof

Prog

ram

Eva

luat

ion

[Spec

ialIs

sue]

,137–154.

1

48.

1R

yan,K

.E.,

Gei

ssle

r,B.,

&K

nel

l,S.

(1996).

Pro

gres

san

dac

counta

bili

tyin

fam

ilylit

erac

y:Le

ssons

from

aco

llabora

tive

appro

ach.Eva

luat

ion

and

Prog

ram

Plan

ning

,19,263–272.

1

49.

1Sc

hnoes

,C

.J.,

Murp

hy-

Ber

man

,V

.,&

Cham

ber

s,J.

M.(2

000).

Em

pow

erm

ent

eval

uat

ion

applie

d:

Exper

ience

s,an

alys

is,an

dre

com

men

dat

ions

from

aca

sest

udy.

Am

eric

anJo

urna

lof

Eva

luat

ion,

21,

53–64.

1

50.

1Se

cret

,M

.,Jo

rdan

,A

.,&

Ford

,A

.(1

999).

Em

pow

erm

ent

eval

uat

ion

asa

soci

alw

ork

stra

tegy

.H

ealth

&So

cial

Wor

k,24,120–127.

1

51.

1St

eich

en,E.M

.,A

.Bhan

dar

i,et

al.(2

006).

‘‘Im

pro

ving

inte

rdis

ciplin

ary

Geo

envi

ronm

enta

lEngi

nee

ring

educa

tion

thro

ugh

empow

erm

ent

eval

uat

ion.’’

Inte

rnat

ional

Journ

alofEngi

nee

ring

Educa

tion

22(1

):171–182.

1

52.

1St

even

son,J

.F.,

Flori

n,P

.,M

ills,

D.S

.,&

Andra

de,

M.(

2002).

Build

ing

eval

uat

ion

capac

ity

inhum

anse

rvic

eorg

aniz

atio

ns:

Aca

sest

udy.

Eva

luat

ion

and

Prog

ram

Plan

ning

,25(3

),233–243.

1

53.

1Su

arez

-Bal

caza

r,Y

.,O

rella

n-D

amac

ela,

L.,P

ort

illo,N

.,Sh

arm

a,A

.,an

dLa

num

,M.(

2003)Im

ple

men

ting

anoutc

om

esm

odel

inth

epar

tici

pat

ory

eval

uat

ion

ofco

mm

unity

initia

tive

s.Jo

urna

lof

Prev

entio

nan

dIn

terv

entio

nin

the

Com

mun

ity,26,5–20.

1

54.

1Su

llins,

C.D

.(2

003).

‘‘Adap

ting

the

Em

pow

erm

ent

Eva

luat

ion

Model

:A

Men

talH

ealth

Dro

p-I

nC

ente

rC

ase

Exam

ple

.’’A

mer

ican

Journ

alofEva

luat

ion

24(3

):387.

1

55.

1T

ang,

H.,

Cow

ling,

D.W

.,K

oum

jian,K

.,R

oes

ler,

A.,

Lloyd

,J.,

&R

oge

rs,T

.(2002).

Build

ing

loca

lpro

gram

eval

uat

ion

capac

ity

tow

ard

aco

mpre

hen

sive

eval

uat

ion.N

ewD

irec

tions

for

Eva

luat

ion,

95,39–56.

1

56.

1T

aut,

S.(2

007).

Studyi

ng

self-

eval

uat

ion

capac

ity

build

ing

ina

larg

ein

tern

atio

nal

dev

elopm

ent

org

aniz

atio

n.Am

eric

anJo

urna

lof

Eva

luat

ion,

28,45–59.

1 (con

tinue

d)



(continued

)

Cas

eN

um

ber

Num

ber

of

Art

icle

s/C

hap

ters

for

This

Cas

eA

rtic

le/A

rtic

les

and/o

rC

hap

ter/

Chap

ters

Use

dto

Code

Cas

es

Num

ber

ofca

ses

Bas

edon

Eac

hA

rtic

le

57.

1T

revi

san,M

.S.

&H

uber

t,M

.(2

001).

Imple

men

ting

Com

pre

hen

sive

Guid

ance

Pro

gram

Eva

luat

ion

Support

:Le

ssons

Lear

ned

.Pr

ofes

sion

alSc

hool

Cou

nsel

ing,

4,225.

1

58.

1V

ander

Pla

at,M

.,Sa

mso

n,Y

.,&

Rav

enP.(

2001).

The

polit

ics

and

pra

ctic

eofe

mpow

erm

entev

aluat

ion

and

soci

alin

terv

entions:

Less

ons

from

the

Atlan

tic

com

munity

action

pro

gram

for

child

ren

regi

onal

eval

uat

ion.Can

adia

nJo

urna

lof

Prog

ram

Eva

luat

ion,

16,79–98.

1

59.

1V

aler

y,R

.,&

Shak

ir,S.

(2005).

Eva

luat

ion

capac

ity

build

ing

and

hum

anitar

ian

org

aniz

atio

ns.

Jour

nalo

fM

ultid

isci

plin

ary

Eva

luat

ion,

3,78–112.

1

60.

1W

iller

,B.S.

,Bar

tlet

t,D

.P.,

Nort

hm

an,J.

E.(1

978).

Sim

ula

tion

asa

met

hod

for

teac

hin

gpro

gram

eval

uat

ion.Eva

luat

ion

and

Prog

ram

Plan

ning

,1,221–228.

1

61.

1Y

awso

n,R

.M.,

Am

oa-

Aw

ua,

W.K

.,et

al.(

2006).

Dev

elopin

ga

per

form

ance

mea

sure

men

tfr

amew

ork

toen

han

ceth

eim

pac

tori

enta

tion

ofth

eFo

od

Res

earc

hIn

stitute

,G

han

a.R&

DM

anag

emen

t,36(2

),161–172.

1

Not

e.Si

ngl

ear

ticl

esor

chap

ters

upon

whic

hm

ultip

lese

par

ate

case

sar

ebas

edar

eid

entifie

dw

ith

ital

ics.



Authors’ Note

Dr. Lesesne was previously with the Centers for Disease Control and Prevention, Division of Reproductive

Health, where she was the Project Officer for this project. She contributed to the initiation and development

of the project during her tenure at CDC and its completion while at ICF Macro. The findings and conclusions

in this article are those of the authors and do not necessarily represent the official position of the U.S. Centers

for Disease Control and Prevention.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publi-

cation of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publica-

tion of this article: This study was supported by CDC contract # 200-2008-M-27814.

Notes

1. The data on organizational strengths other than resources was judged insufficient for relational analyses.

2. Details on coding of items are available by request from the first author.

3. Given that a balanced contingency table (i.e., categories used by one rater are directly comparable to the

categories used by a second rater) is required to properly calculate k via computer-aided statistical software,

pseudo-observations were utilized. Before the pseudo-observations were created, a weight of one (1) was

assigned to each of the codes that were given for each case. To create pseudo-observations, a very small

weight (0.0000000001) was assigned to the range of possible values. Even though the small weight of the

pseudo-observations did not greatly impact the k statistic that was calculated, these observations ensured

that the same categories were used by each coder, which allowed for balanced contingency tables.

4. Information on goals and context (needs assessments and tailoring) were not reported in sufficient detail to

provide meaningful information to conduct relational analyses with strategies.

References

American Evaluation Association. (2008). American evaluation association internal scan report to the

membership, by goodman research group. Retrieved from AEA website: http://www.eval.org

Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002a). Editors’ notes. New Directions for Evaluation, 93,

1–6.

Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002b). New directions for ECB. New Directions for

Evaluation, 93, 109–119.

Begg, C. B. (1994). Publication bias. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis

(pp. 399–409). New York, NY: Sage.

Boyle, R. (1999). Professionalizing the evaluation function: Human resource development and the building of

evaluation capacity. In R. Boyle & D. Lemaire (Eds.), Building effective evaluation capacity (pp. 135–151).

New Brunswick, NJ: Transaction.

Boyle, R., Lemaire, D., & Rist, R. C. (1999). Introduction: Building evaluation capacity. In R. Boyle &

D. Lemaire (Eds.), Building effective evaluation capacity: Lessons from practice (pp. 1–19). New

Brunswick, NJ: Transaction.

Centers for Disease Prevention and Control. (2011). The guide to community and preventive services: The com-

munity guide. Retrieved from http://www.thecommunityguide.org/index.html

Compton, D. W., Baizerman, M., & Stockdill, S. H. (2002). Special issue: The art, craft, and science of evalua-

tion capacity building. New Directions for Evaluation, 93, 1–120.

Cooper, H., & Hedges, L. (1994). The handbook of research synthesis. New York, NY: Russell Sage.




Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. E. (2004). Integrating evaluative inquiry into the organizational

culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19, 99–141.

Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5–23.

Duffy, J. L., & Wandersman, A. (2007, November). A review of research on evaluation capacity-building stra-

tegies. Paper presented at the annual conference of the American Evaluation Association, Baltimore, MD.

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of imple-

mentation on program outcomes and the factors affecting implementation. American Journal of Community

Psychology, 41, 327–350.

Fetterman, D., Kaftarian, S., & Wandersman, A. (Eds.). (1996). Empowerment evaluation. Thousand Oaks, CA:

Sage.

Fetterman, D., & Wandersman, A. (Eds.). (2005). Empowerment evaluation principles in practice. New York,

NY: Guilford Press.

General Accounting Office. (1987). Drinking-age laws: An evaluation synthesis of their impact on highway

safety (GAO/PEMD-87-10). Washington, DC: Author.

General Accounting Office. (1989). Prospective evaluation methods: The prospective evaluation synthesis

(GAO/PEMD-10.1.10). Washington DC: Author.

General Accounting Office. (1992a). Cross-design synthesis (GAO/PEMD-92-18). Washington, DC: Author.

General Accounting Office. (1992b). The evaluation synthesis (GAO/PEMD-10.1.2). Washington, DC: Author.

Government Accountability Office. (2003). Program evaluation: An evaluation culture and collaborative part-

nerships help build agency capacity (GAO-03-454). Washington, DC: Author.

Government Accountability Office. (2009). Program evaluation: A variety of methods can help identify effec-

tive interventions (GAO-30-10, 2009). Washington, DC: Author.

Gibbs, D., Napp, D., Jolly, D., Westover, B., & Uhl, G. (2002). Increasing evaluation capacity within commu-

nity based HIV prevention programs. Evaluation and Program Planning, 25, 261–269.

Higgins, J., & Green, S. (Eds.). (2011, March). Defining types of studies. Cochrane Handbook for Systematic

Reviews of Interventions. Version 5.1.0. Retrieved from www.cochrane-handbook.org

Kellogg Foundation, W. K. (2001). Logic model development guide. Retrieved from: http://www.wkkf.org/

Pubs/Tools/Evaluation/Pub3669.pdf

KIT Solutions. (2011). COMET and iGTO. Retrieved March 15, 2011, from http://www.kitsolutions.net/

our-clients-who-we-serve#all

Kotter, J. (1996). Leading change, Cambridge, MA: Harvard Business School Press.

Labin, S. N. (2008). Research synthesis: Toward broad based evidence. In N. L. Smith & P. R. Brandon (Eds.),

Fundamental issues in evaluation (pp. 89–110). New York, NY: Guilford Press.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics,

33, 159–174.

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.

Love, A. (1991). Internal evaluation: Building organizations from within. Thousand Oaks, CA: Sage.

Love, A. (2006). Developing evaluation capacity in extension 4-H field faculty: A framework for success.

American Journal of Evaluation, 22, 257–269.

Milstein, B., & Cotton, D. (2000). Defining concepts for the presidential strand on building evaluation capacity.

Working paper circulated in advance of the November 2000, meeting of the American Evaluation Association.

O’Sullivan, R. G. (2004). Practicing evaluation: A collaborative approach. Thousand Oaks, CA: Sage.

Owen, J. M. (2003). Evaluation culture: A definition and analysis of its development within organizations.

Evaluation Journal of Australasia, 3, 43–47.

Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Thousand Oaks, CA. Sage.

Preskill, H., & Torres, R. (1999). Evaluative inquiry for learning in organizations. Thousand Oaks, CA: Sage.

Preskill, H., & Torres, R. (2009). Readiness for Organizational Learning and Evaluation (ROLE). In D. Russ-Eft

& H. Preskill (Eds.), Evaluation in organizations (2nd ed., pp. 491–504). Boston, MA: Perseus Books.

Labin et al. 31



Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal

of Evaluation, 29, 443–459.

Preskill, H., Zuckerman, B., & Matthews, B. (2003). An exploratory study of process use: Findings and impli-

cations for future research. American Journal of Evaluation, 24, 423–442.

Rapkin, B. D., & Trickett, E. J. (2005). Comprehensive dynamic trial designs for behavioral prevention research

with communities: Overcoming inadequacies of the randomized controlled trial paradigm. In E. Trickett &

W. Pequenaut (Eds.), Increasing the community impact of HIV prevention interventions (pp. 249–277). New

York, NY: Oxford University Press.

Robinson, T. T., & Cousins, J. B. (2004). Internal participatory evaluation as organizational learning: A long-

itudinal case study. Studies in Educational Evaluation, 30, 1–22.

Rodriguez-Campos, L. (2005). Collaborative evaluations. Tamarac, FL: Llumina Press.

Sanders, J. R. (2003). Mainstreaming evaluation. New Directions for Evaluation, 99, 3–6.

Schaumberg-Muller, H. (1996). Evaluating capacity building: Donor support and experiences. Report for the

DAC (Development Assistance Committee) Expert Group on Aid Evaluation, OECD (Organization for

Economic Cooperation and Development). Copenhagen, Denmark: DANIDA (Danish International Devel-

opment Assistance). Retrieved from: http://www.oecd.org/dataoecd/20/52/16546669.pdf

Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a definition of the ecb process: A conver-

sation with the ECB literature. New Directions for Evaluation, 93, 7–25.

Stevenson, J. F., Florin, P., Mills, D. S., & Andrade, M. (2002). Building evaluation capacity in human service

organizations: A case study. Evaluation and Program Planning, 25, 233–243.

Suarez-Balcazar, Y., Taylor-Ritzler, T., Garcia-Iriarte, E., Keys, C., Kinney, L., & Rush-Ross, H., . . . Curtin, G.

(2010). Evaluation capacity building: A cultural and contextual framework. In F. Balcazar, Y. Suarez-

Balcazar, T. Taylor-Ritzler & C. B. Keys (Eds.), Race, culture and disability: Rehabilitation science and prac-

tice. Sudbury, MA: Jones & Bartlett Learning.

Taylor-Ritzler, T., Suarez-Balcazar, Y., & Garcia-Iriarte, E. (2010). Results and implications of a mixed-methods

ECB model validation study. Paper presented at the American Evaluation Association annual meeting, San

Antonio, TX.

Tanglewood Research Inc. (2011). The evaluation lizard. Retrieved from http://evaluationlizard.com/welcome.

aspx

United Way of America. (1996). Measuring program outcomes: A practical approach. Arlington, VA: Author.

United Way of America. (2008). Logic model handbook. Arlington, VA: Author. Retrieved from: http://www.

vsuw.org/file/logic_model_handbook_updated_2008.pdf

University of Wisconsin Cooperative Extension. (2003). Evaluation logic model bibliography. Retrieved from:

http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

Volkov, B., & King, J., (2007). Checklist for building evaluation capacity. Retrieved http://www.wmich.edu/

evalctr/archive_checklists/ecb.pdf

Wandersman, A., Imm, P., Chinman, M., & Kaftarian, S. (2000). Getting to outcomes: Methods and tools for

planning, evaluation and accountability. Rockville, MD: Center for Substance Abuse Prevention.

Wandersman, A., Snell-Johns, J., Lentz, B. E., Fetterman, D. M., Keener, D. C., Livet, M., . . . Flasphohler, P.

(2005). The principles of empowerment evaluation. In D. Fetterman & A. Wandersman (Eds.), Empowerment

evaluation principles in practice (pp. 27–41). New York, NY: Guilford Press.

Wing, K. T. (2004). Assessing the effectiveness of capacity-building initiatives: Seven issues for the field.

Non-Profit and Voluntary Sector Quarterly, 33, 153–160.




A Research Synthesis of the Evaluation Capacity Building Literature

Documents

Transcript of A Research Synthesis of the Evaluation Capacity Building Literature