The HIV/AIDS Prevention and Control SYNOPSIS ... - USAID

SY

NO

PSIS

HIV/AIDS Prevention and Control Series

CapacityBuilding

Series Editor

M. Ricardo CalderónLatin America and Caribbean Regional Office

AIDSCAP/Family Health International

Project 936-5972.31-4692046Contract HRN-5972-C-00-4001-00The AIDS Control and Prevention

(AIDSCAP) Project, implemented byFamily Health International, is

funded by the United States Agencyfor International Development.

Family Health International (FHI) is a non-governmental

organization that works to improve reproductive health around

the world, with an emphasis on developing nations. Since 1991,

FHI has implemented the AIDS Control and Prevention

(AIDSCAP) Project, which is funded by the United States Agency

for International Development (USAID). FHI/AIDSCAP has

conducted HIV/AIDS prevention programs in more than 40

countries, and the Latin America and Caribbean Regional Office

(LACRO) has implemented interventions in 14 countries within

the region.

December 1997

For further information, contact:Latin America and Caribbean Regional OfficeAIDSCAP/Family Health International2101 Wilson Blvd, Suite 700Arlington,VA 22201Telephone: (703) 516-9779Fax: (703) 516-0839

The HIV/AIDS Prevention and Control SYNOPSIS Series

CAPACITY

BUILDING

Series Editor:M. Ricardo Calderón

AIDSCAP/Family Health InternationalArlington,VA, USA

Prepared by:Joseph Amon

Katherine KotellosAIDSCAP/Family Health International

Arlington,VA

Wendy Githens BenazergaConsultant

Arlington,VA, USA

Project Coordinator:Mary L. Markowicz

AIDSCAP/Family Health InternationalArlington,VA, USA

Published by the Latin America and Caribbean Regional Office ofThe AIDS Control and Prevention (AIDSCAP) Project

Family Health International

The opinions expressed herein are those of the writer(s) and do notnecessarily reflect the views of USAID or Family Health International.Excerpts from this booklet may be freely reproduced, acknowledgingFHI/AIDSCAP as the source.

T A B L E O F C O N T E N T S

Acronyms ii

Acknowledgements iii

Prologue v

Executive Summary ix

Introduction 1

Proposed Theoretic Framework 5

Experiences in Implementing a Capacity Building Evaluation Framework 13

Step 1: Build Consensus 13Step 2: Conduct Assessment 15Step 3: Define Objectives and Benchmarks 16Step 4: Monitor Progress 17Step 5: Measure Outcomes 18Step 6: Analyze and Interpret Results 19

Lessons Learned and Recommendations 21

References 23

TA

BLE

OF C

ON

TE

NT

S

ii Capacity Building

A C R O N Y M S

AIDS acquired immune deficiency syndrome

AIDSCAP AIDS Control and Prevention Project

BCC behavior change communication

CBO community-based organization

CKIIQ comprehensive key informant interview questionnaire

CSW commercial sex worker

FHI Family Health International

HIV human immunodeficiency virus

LAC Latin America and the Caribbean

LACRO Latin America and Caribbean Regional Office

MOH Ministry of Health

NACP National AIDS Control Program

NGO non-governmental organization

PAHO Pan American Health Organization

PVO private voluntary organization

RATE regional AIDS training and education program

ROA rapid organizational assessment

STI sexually transmitted infection

USAID United States Agency for International Development

WHO World Health Organization

Acknowledgements iii

A C K N O W L E D G E M E N T S

We would like to acknowledge and recognize the contributions,work and efforts of the Implementing Agencies — NGOs, PVOs,CBOs, NACPs/MOH, Social Security Institutes, and private sectorenterprises — with whom LACRO has worked and for all that wehave learned and accomplished together in HIV/AIDS preventionand control.

We are especially grateful to Joseph Amon, Evaluation Officer forAIDSCAP/LACRO, Katherine Kotellos,AIDSCAP AssociateEvaluation Officer, and Wendy Githens Benazerga, Consultant, forwriting this booklet.

We take this opportunity to also thank all of the staff members ofAIDSCAP/LACRO, including former staff members, for their over-all support and assistance to LACRO activities: Lee Arnette, MimiBinns, Doria Bouril, Oly Bracho, Marianne Burkhart, Mark Chorna,Rebecca Coleman, Genie Liska, Cathy Mamedes, Mary L.Markowicz, Robert Martínez, Mary Kay McGeown, Steve Mobley,Manuel Mongalo, Polly Mott, Sara Padilla, Marvelín Parsons,Amparo Pinzón, Luis Rodríguez, Melissa Rosenberger, DianaSantos, Isabel Stout, Molly Strachan, and Oscar Viganó. In addi-tion, we express our appreciation to the AIDSCAP LAC ResidentAdvisors: Catherine Brokenshire, Jamaica; Martha Butler de Lister,Dominican Republic; Eddy Génécé, Haiti; Gale Hall, Jamaica; JorgeHiguero Crespo, Honduras; and Maria Eugênia Lemos Fernandes,Brazil; and other AIDSCAP staff for their insights and contribu-tions.We are grateful to Peter Lamptey, FHI Senior Vice Presidentof AIDS Programs and Project Director of AIDSCAP, and TonySchwarzwalder, Deputy Project Director of AIDSCAP, for their continued support of LACRO activities.

Finally, we wish to extend our gratitude and appreciation to theUSAID Global Bureau’s HIV/AIDS Division, field Missions in LatinAmerica and the Caribbean, and to the Population, Health andNutrition Team in the Office of Regional SustainableDevelopment of the LAC Bureau, particularly James B. Sitrick, Jr.,

AC

KN

OW

LE

DG

EM

EN

TS

iv Capacity Building

for the support and funding of the Information DisseminationInitiative and other LACRO programs.

P R O L O G U E

The HIV/AIDS Prevention and Control SYNOPSIS Series is a sum-mary of the lessons learned by the Latin America and CaribbeanRegional Office (LACRO) of the AIDS Control and Prevention(AIDSCAP) Project. AIDSCAP is implemented by Family HealthInternational (FHI) and funded by the United States Agency forInternational Development (USAID). The series is a programactivity of the LACRO Information Dissemination Initiative andwas created with several goals in mind:

to highlight the lessons learned regarding program design,implementation, management and evaluation based on fiveyears of HIV/AIDS prevention and control experience inLAC countries

to serve as a brief theoretical and practical reference regard-ing prevention interventions for HIV/AIDS and other sexual-ly transmitted infections (STIs) for program managers, gov-ernment officials and community leaders, non-governmentalorganizations (NGOs), private voluntary organizations(PVOs), policy and decision makers, opinion leaders, andmembers of the donor community

to provide expert information and guidance regarding cur-rent technical strategies and best practices, including a dis-cussion of other critical issues surrounding HIV/AIDS/STIprogramming

to share lessons learned within donoregion for adaptation orreplication in other countries or regions

to advance new technical strategies that must be taken intoconsideration in order to design and implement more effec-tive prevention and control interventions

to advocate a holistic and multidimensional approach toHIV/AIDS prevention and control as donoonly way to effec-tively stem donotide and impact of the pandemic

Prologue v

PR

OLO

GU

E

vi Capacity Building

AIDSCAP (1991-1997) was originally designed to apply thelessons learned from previous successful small-scale preventionprojects (1987-1991) to develop comprehensive programs toreduce the sexual transmission of HIV, the primary mode of trans-mission of the virus.AIDSCAP applied three primary strategies —Behavior Change Communication (BCC), STD Prevention andControl, and Condom Programming — along with supportingstrategies of Behavioral Research, Policy Development andEvaluation.

The success of this approach, based on the combination of strate-gies and targeted interventions, has been widely documented.The AIDSCAP Project, in fact, has been recognized as among thebest and most powerful international HIV/AIDS prevention pro-grams to date.1 AIDSCAP has worked with over 500 NGOs, gov-ernment agencies, community groups and universities in morethan 40 countries; trained more than 180,000 people; producedand disseminated some 5.8 million printed materials, videos, dra-mas, television and radio programs, and advertisements; reachedalmost 19 million people; and distributed more than 254 millioncondoms.2

However, the pandemic continues to escalate at a rate that out-paces our successes. Thus, we need to build upon these success-es, learn from our experiences, and determine what has workedand what is missing in order to respond with added effect in thefuture.The magnitude and severity of the HIV/AIDS pandemiccalls for boldness, flexibility, wisdom and openness. The worldcannot afford to continue to fight HIV/AIDS only with currentthinking and tools. We must look toward new thinking and strate-gies that complement and carry the current state-of-the-artapproaches forward in the fight against HIV infection.

Therefore, LACRO endorses, promotes and elevates GenderSensitive Initiatives (GSIs), Civil-Military Collaboration (CMC),Religious-Based Initiatives (RBIs), and Care & Management(C&M) as the new prototype of technical strategies that must beincorporated on par with the strategies that have been imple-mented to date. Walls, barriers and biases have to come down in

Prologue vii

order to unlock the strengths, benefits, potential, synergy and/orresources of GSIs, CMC, RBIs and C&M.

More importantly, approaches that compartmentalize strategiescan no longer be justified. Despite the efforts to integrate andcoordinate amongst and between technical strategies and differ-ent sectors of society, prevention programming is barely scratch-ing the surface of what a real comprehensive effort should be.One of the most important lessons learned about HIV/AIDS isthat it is not only a medical problem, nor is it exclusively a publichealth problem. Rather, the pandemic is in addition a socioeco-nomic problem and, as such, threatens the sustainable develop-ment of developing countries and challenges the ethical founda-tions of the developed world. HIV/AIDS has become a challengeto health, development and humanity.

For lasting success, a genuine multidimensional approach isurgently needed. One that demands new forms of wealth distrib-ution, educational opportunities and development; attempts toresolve the inequalities in gender and power; acknowledges theindividual, environmental, structural and superstructural causes ofand solutions for the pandemic; and aims to balance the disparitybetween the “haves”and the “have-nots,” resulting in more sustain-able, equitable, effective and compassionate efforts.

Therefore, the SYNOPSIS Series reaffirms that current HIV/AIDSprevention and control strategies work, and contends that newtechnical strategies are needed and can be effective and comple-mentary. The Series also strongly advocates for, and will discussin a separate issue, the Multidimensional Model (MM) for the pre-vention and control of the pandemic.This model must guidenational, regional and international planning and programming inorder to achieve measurable and significant gains that can trulyeffect changes at the individual, societal, environmental and struc-tural levels.

We trust the reader will be open to our futuristic thinking andwill contribute to the further development of the strategies pre-sented here as well as others.We hope the SYNOPSIS Series will

viii Capacity Building

stimulate discussion and reflection, propel continued dialogue,and encourage the pioneering of new combinations of innova-tive approaches.

M. Ricardo Calderón, MD, MPH, FPMER.Regional DirectorLatin America and Caribbean Regional OfficeAIDSCAP/Family Health International

E X E C U T I V E S U M M A R Y

This SYNOPSIS presents a theoretical and practical means for theevaluation of organizational and institutional development(Capacity Building) in HIV/AIDS prevention and control programsin developing countries. It is based on experiences in measuringcapacity building in more than twenty countries and two hun-dred projects, ranging from communications and behavior changeinitiatives to condom social marketing, to STI/HIV clinical inter-ventions with a range of international non-governmental organiza-tions (NGOs) and Ministries of Health.

Multiple frameworks, methodologies and paradigms for the evalu-ation of organizational and institutional development werereviewed and synthesized, resulting in a pluralistic model for themeasurement of capacity building efforts. This proposed theoreti-cal framework is based on seven capacity-building strategies —technical skill building, management skill building, manage-ment systems development, resource diversification, networkbuilding, organization cross-fertilization, and multi-sectoral col-laboration — designed to enhance individuals, organizations,and institutions to design, manage, evaluate and sustain compre-hensive HIV/AIDS programs and initiatives.

In addition to the theoretical framework, several practical stepsare necessary to implement a monitoring and evaluation plan forcapacity building. These steps include: building consensusamong organization staff and stakeholders around the process ofmonitoring and evaluating capacity building; conducting a base-line assessment of organizational and institutional capacity; defin-ing the objectives and indicators participatively to develop astrategic plan for capacity building; monitoring progress towardsprogram objectives; measuring outcomes; and analyzing andinterpreting results in collaboration with stakeholders.Traditional forms of capacity building evaluation — self-assess-ment, case study, outcome and impact evaluation — are present-ed and a framework for their integration at the level of interpreta-tion and inference is provided.

Executive Summary ix

EX

EC

UT

IVE

SU

MM

AR

Y

There is a growing recognition among international and localorganizations that while technical and financial inputs are oftencritical for improving project performance, this assistance alone isnot sufficient to help groups manage and monitor their growth,define their vision and design effective strategies to adapt to adynamic environment. The capacity building conceptual frame-work discussed in this SYNOPSIS — linking strategies, variablesand outcomes — provides an instructive paradigm for the designand evaluation of HIV/AIDS prevention interventions worldwide.From our experience, we have found that conducting mixed-method evaluation of capacity building efforts allows for a moreinsightful assessment of capacity building and a more comprehen-sive evaluation of HIV prevention programs.

x Capacity Building

I N T R O D U C T I O N

Over the past two decades, many international developmentagencies have gradually begun working in partnership withlocal non-governmental organizations (NGOs), rather thandirectly implementing programs. As an increasing proportion ofhealth assistance and overseas aid resources were channeleddirectly to indigenous organizations in developing countries, ashift in emphasis occurred from identifying program impactsolely in terms of health outcomes to measuring impact interms of both health outcomes and the increased capacity oflocal organizations.3, 4, 5

Initially, capacity building assistance to local organizations focusedprimarily on providing funding and equipment, increasing finan-cial accountability, and strengthening specific technical skills.6, 7

However, there was a growing recognition among internationaland local organizations that while technical and financial inputsare often critical for improving project performance, this assis-tance alone is not sufficient to help groups manage and monitortheir growth, define their vision and design effective strategies toadapt to a dynamic environment.8, 9

While many development programs have been working to buildlocal capacity for over a decade, HIV/AIDS prevention programshave only more recently begun to consider the importance ofbuilding local capacity. Born in a crisis atmosphere, HIV/AIDSprograms deployed resources rapidly and directly intervened inthe delivery of services in critical technical areas, such as control-ling blood supplies, epidemiological reporting, diagnosis and treat-ment of opportunistic infections, and distributing condoms andprevention messages. As the epidemic evolved, this emergency-type response has been replaced by an understanding that theepidemic is a long-term development problem requiring a long-term multi-sectoral response that must involve communities, localorganizations and networks in program planning, implementationand evaluation.10 Developing a comprehensive approach tostrengthening organizations and inter-agency alliances to fight the

Introduction 1

INT

RO

DU

CT

ION

epidemic requires a paradigm shift for many international NGOsbecause, in contrast to earlier interventions, the process of build-ing capacity is slow and requires a new set of indicators whichdefine success in a different manner than before.11, 12 Compre-hensive programs to increase the sustained impacts of HIV pro-grams now must include capacity building strategies focused atthe institutional, organizational and individual levels.

As the strategies to strengthen capacities become more complex,the challenge of measuring effective capacity building efforts alsointensifies.13 For example, a lack of consensus among HIV/AIDSorganizations and donors on defining appropriate capacity build-ing indicators emerges from their different priorities and programobjectives.12, 14 Another difficulty is that current quantitative moni-toring and evaluation systems used by many organizations to mea-sure outcomes of behavior change interventions do not sufficient-ly capture the depth and breadth of capacity building activities.Moreover, many organizations and donors are reluctant to investthe time and resources in baseline and follow-up research forcapacity building, particularly in projects that are expected todemonstrate individual behavioral change or biologic impactwithin a short time frame.15 Complicating further the task of eval-uating capacity building in terms of sustainability is the influenceof political and economic externalities.

Scarce resources and a maturing epidemic have required both thedonor community and international NGOs to pay closer attentionto how the capacities of local organizations and communitygroups are being enhanced to sustain HIV prevention efforts.3

Most international NGOs are now involved in strengtheningcapacities of partner organizations and implementing strategies toevaluate these efforts. The variety of indicators and methodolo-gies used by international NGOs to evaluate capacity buildingefforts reflects the different project priorities, intervention objec-tives, and operational paradigms of the international NGOs andtheir partners, and the reporting, monitoring and evaluationneeds and requirements of the groups involved.

2 Capacity Building

The following discussion presents a mixed-method model ofcapacity building evaluation that balances competing prioritiesand combines multiple complementary methods to measureincreased capacity and sustainability of interventions using anintegrated, holistic and pragmatic approach. The analysis of datacollected through this model provides an important counterbal-ance to behavioral and biologic evaluation methods, addressinglong-term impact, complementary to the measure of short-termtrends or strategy-specific results.

Introduction 3

4 Capacity Building

P R O P O S E D T H E O R E T I CF R A M E W O R K

The proposed framework is based on seven capacity-buildingstrategies (Table 1) designed to enhance individuals, organizations,and institutions to design, manage, evaluate and sustain HIV/AIDSprevention programs and initiatives. These seven strategies arebased on theories of organizational development, institutionaldevelopment and organizational transformation 16, 17 and informedby the practices of community mobilization, participation, andempowerment.18, 19 The core of the framework examines howcapacities are strengthened at each level as well as the synergisticrelationship among the levels.20, 21 At the level of individuals,emphasis is on human resource development through technicaland management skill building. For organizations, the focus is onorganizational development, including systems and structurestrengthening, leadership and governance, resource diversification,and network building. For institutions, organizational cross-fertil-ization and multi-sectoral collaboration are targeted.

The framework distinguishes betweeninstitutional development and organi-zational development. Organizationsare the physical entities with whommany donors work. Institutions tran-scend specific organizations to definethe customs, practices, relationships orbehavioral patterns of importance inthe life of a community or society.22

Organizations, therefore, would includecommunity-based organizations,schools, Ministry of Health (MOH) divi-sions, implementing agencies, and soforth. Institutions usually representcoalitions of organizations and sectorsof society, for example, the media, thesystem of education, religion, and coali-tions of community groups.


Organizations are the physical

entities with whom many

donors work. Institutions

transcend specific organizations

to define the customs, practices,

relationships or behavioral pat-

terns of importance in the life of

a community or society.22

PR

OP

OSE

D T

HE

OR

ET

IC F

RA

ME

WO

RK

6 Capacity Building

Table 1: Capacity Building Strategies

Capacity Building Strategy Definition

Technical skill building The improvement in the skillsnecessary to carry out specifictechnical aspects of programs or initiatives.

Management The improvement in the skillsskill building necessary to effectively manage

programs and efficiently utilize organizational resources.

Management systems The improvement of internaldevelopment systems, operational proce-

dures, or tools that facilitate more effective management.

Resource diversification The diversification of sources of financial and physical resources.

Network building The improvement of organiza-tional ties to constituents,peers, and policy makers to increase support for project activities.

Organizational The improvement in thecross-fertilization exchange of information and

experience between program managers involved in HIV/AIDS programs.

Multi-sectoral The expansion of programcollaboration activities and ties to other pub-

lic and private sectors not actively engaged in addressing the HIV/AIDS epidemic.

Linking the specific strategies to outputs, outcomes and impactsdefines the strategic approach and evaluation framework (Table2). Although differentiated into three levels, there is an importantsynergy among and between each level in the achievement of theobjectives. A training program for NGO program officers in man-agement skills or financial accounting may, for example, lead tobetter program management and accounting systems within theirorganization. As a result, the organization may become a moreproactive partner among NGOs thus strengthening networkswith other organizations and community groups which in turnmay eventually enhance the overall capability of the non-profitsector as a whole.

Despite their somewhat ambiguous delineation, institutions canalso be specifically influenced to improve the sustainability ofHIV/AIDS prevention programs. Increasing capacity at this levelnot only represents the cumulative effects of capacity buildinginitiatives at the individual organizational levels, but attempts tochange the environment, structures and needs that define howorganizations and initiatives are conceived and implemented.21, 23, 24

For many donors, organizational sustainability is a key outcome of capac-ity building efforts.25 Our understand-ing of the meaning of sustainability hasgradually evolved over the last twodecades.26, 27 In the early 1980s, sustain-ability was defined in terms of the con-tinuity of project activities and benefitsin the absence of external funding.28

Currently, more comprehensive andsubtle definitions have emerged. In ourmodel, four distinct aspects of organiza-tional sustainability are proposed: tech-nical sustainability, the ability of anorganization to provide technically

appropriate, state-of-the-art, high-quality services; managementsustainability, the ability to plan and manage all aspects of theoperations; financial sustainability, the ability to generate suffi-


Our model proposes the fol-

lowing four distinct aspects of

organizational sustainability:

technical

management

financial

political

Table 2: Relationship of capacity building strategy to outputs, outcomes, and impacts

Focus Individual Organization Institution

Technical Skill Organizational/ OrganizationalBuilding Systems cross-

Development fertilization

Strategy Management Resource Multi-sectoralSkill Building Diversification Collaboration

Network Building

Outputs Individuals Management Multi-sectoraltrained systems meetings/

established conferences held

Outcome Improved Improved Improved formaltechnical and effectiveness and informalmanagement of financial, coalitions;skills human resource, exchange of

monitoring and lessons learnedevaluation and dissemina-systems; tion ofmultiple funding informationsources;improved stakeholder involvement;policy engagement.

Impact Improved Technical, Sustainability technical and management, of benefitsmanagement financial, and (impact effectiveness political sustainability)

sustainability

cient working capital to continue to produce goods or provideservices; and political sustainability, the ability to maintain thesupport and involvement of the community members, gatekeep-ers, opinion leaders, policy influencers, and key decision makerswhich can affect the viability of the organization. These fouraspects of organizational sustainability are seen as complemen-tary to one another. An organization without any one of the fourcomponents will either be ineffective (lacking technical/manage-ment sustainability), unproductive (lacking financial sustainabili-ty), or irrelevant (lacking political sustainability).

While the sustainability of organizationsworking in HIV/AIDS may lead toreduced HIV transmission, focusing sole-ly on organizational sustainability is notsufficient to prepare organizations toadapt to the changing epidemic anddemands of stakeholders. Consequently,our model defines the sustainability ofbenefits, or impact sustainability, as theultimate goal of capacity buildingefforts. Regardless of the long-term sur-

vival of specific organizations, capacity building efforts thatstrengthen institutions can result in the sustained impact of pro-gram benefits — through the creation of new organizations, theconsolidation of diverse groups, or a shift in social norms.

Specific indicators are defined for evaluation purposes at thelevel of outputs, outcome, and impact (Table 3). Indicators at theoutput level reflect the strategies of enhancing human resourcedevelopment (e.g., number of individuals trained), improvingorganizational development (e.g., mission statement defined, andinternal structure and organizational outputs are congruent withthe mission), and strengthening multi-sectoral collaboration (e.g.,number of multi-sectoral meetings held). Evaluation methods relyupon traditional methods of process evaluation, such as processmonitoring through periodic reporting, key informant interviews,and document analysis.


Our model defines the sustain-

ability of benefits, or impact

sustainability, as the ultimate

goal of capacity building efforts.

10 Capacity Building

Measuring the outcomes of capacity building efforts is more chal-lenging and requires the use of multiple, diverse inquiry methodsto build a robust evaluation approach with appropriate and prag-matic methods. Much as the evaluation of behavior change canbe pursued through the assessment of quantitative, qualitativeand process data, the assessment of increased capacity at the out-come level is triangulated as well. Through this approach, theinsight and validity of data collected is notably improved.29 Bycombining self-assessments with more objective comparisons todefined criteria of organizational development and exit inter-views of beneficiaries, evaluation biases can be mitigated, andchanges in organizational and institutional development can benoted.30

Outcome indicators must be relevant to the specific strategiespursued by each organization but emphasize: 1) the delivery ofhigh-quality, appropriate, care (e.g., percentage of STI patientstreated according to national guidelines for syndromic manage-ment — WHO/GPA Prevention Indicator 6);31 2) the functioningof management systems (e.g., percentage of NGOs engaging instrategic planning with stakeholder involvement); and 3) thedevelopment of institutional networks (e.g., number of NGOs thathave participated in a collaborative project with another NGO inthe last year).

Measuring the impact of capacity building efforts requires similarmixed-method approaches. Evaluating the effectiveness of individ-uals, the sustainability of organizations and the sustainability ofprogram benefits, often requires the detailed, qualitative focus ofcase study analysis, linking advocacy efforts on the part of NGOprogram managers to changes in policies or social norms.Additional appropriate evaluation methods include key informantinterviews, focus group discussions and cost-effectiveness analysis.

The development of national guidelines for syndromic manage-ment in Haiti provides an example that demonstrates thisapproach and the synergistic qualities of capacity-building effortsat multiple levels to measure impact sustainability. This initiativeinvolved building capacity at the level of individuals, organiza-

Table 3: Capacity building indicators and evaluation methods

Level Illustrative indicators Evaluation methods

Output • Number of individuals trained in BCC, STI, condoms,evaluation, etc.

• Percentage of project pro-posals accepted/submitted

• Defined, relevant mission statement

• Fundraising activities conducted

• Number of abstracts accepted to international HIV/AIDS conferences

• Number of multisectoral meetings held

Outcome • Percentage of STD patients treated according to syndro-mic management guidelines

• Percentage of NGOs engaging in strategic planning with stakeholder involvement

• Number of NGOs that are members of a formal coalition

• Number of NGOs that have participated in a collaborative project with another NGO in the last year

Impact • Percentage of individuals trained still working in HIV/AIDS 2 years later

• Percentage of NGOs providing services according to community needs assessment

• Percentage of NGOs with >2 donor organizations providing >10% of overall funding

• Number of NGOs participating in national-level strategic planning

• Number of favorable policiesadopted (STD case manage-ment, condom import policies, etc.)


• Process indicator (monitoring) form

• Document analysis

• Key informant interviews

• Focus group discussions

• Document analysis• Audits• Client/beneficiary

exit interview• Self-assessment• Case studies

• Key informant interviews

• Focus group discussions

• Audits• Qualitative case

study chain-of-eventsanalysis

• Cost-effectivenessanalysis


tions and institutions.32 AIDSCAP initially conducted an ethno-graphic study of beliefs and health seeking behaviors related toSTIs in the neighborhood of Cite Soliel in the capital city of Port-au-Prince in Haiti.33 At the same time, a seroprevalence study wasconducted that found 47 percent of antenatal clinic attendeeshad at least one STI.34 With this information,AIDSCAP was able tobring together representatives from a range of organizations todiscuss the problems and possible appropriateness of a syn-dromic management approach to STI treatment, based upon thedevelopment of a simple, locally derived, algorithm of treatmentalternatives. Collaboration on a case management approach ledto the development of a training manual for use in training STIcare providers. As familiarity with the system of case manage-ment increased, consensus was built around the effectiveness ofthis approach, which was subsequently endorsed for adoption bythe Ministry of Health.35 This process of building capacity in theprovision of STI services demonstrates how increasing individualcapacity, organizational capacity, and ultimately institutionalcapacity creates an entrenched sustainability of program benefitsthat is not dependent upon individual or organizational sustain-ability, but is built on the strengthened interrelated levels.

E X P E R I E N C E S I NI M P L E M E N T I N G A C A P A C I T YB U I L D I N G E V A L U A T I O NF R A M E W O R K

In addition to a theoretical framework, several practical steps arenecessary to implement a monitoring and evaluation plan forcapacity building. These steps include: 1) building consensusamong organization staff and stakeholders around the process ofmonitoring and evaluating capacity building; 2) conducting abaseline assessment of organizational and institutional capacity;3) defining the objectives and indicators participatively to devel-op a strategic plan for capacity building; 4) monitoring theprocess; 5) conducting outcome assessments; and 6) analyzingand interpreting results in collaboration with stakeholders.

Step 1: Build ConsensusFor any evaluation to be successful, it is important that stakehold-ers — including project managers, beneficiaries, organization staffand donors — be involved in the process from the design stage.36, 37 Consensus needs to be built around not only the

process of capacity building, but alsothe process of evaluating capacitybuilding. Organizations may be wary ofparticipating fully in a process, such as aneeds assessment that will expose inter-nal weaknesses as well as strengths, andmay feel the evaluation of capacitybuilding is inherently threatening. It ischallenging to establish trusting relation-ships among organizations that are nec-essary to build networks and shareresources with an aim toward impactsustainability, especially in an atmos-phere which encourages competitionrather than collaboration.38 Therefore, itis critical to build consensus among thegroups involved so they come to view

Implementing a Capacity Building Evaluation Framework 13

EX

PE

RIE

NC

ES I

N I

MP

LE

ME

NT

ING

A

CA

PA

CIT

Y B

UIL

DIN

G E

VA

LU

AT

ION

FR

AM

EW

OR

KIt is critical to build consensus

among the groups involved

so they come to view

capacity building, and the

evaluation of the process of

capacity building, as useful for

the organization.


capacity building, and the evaluation of the process of capacitybuilding, as useful for the organization. When organizations recog-nize there is something positive to be gained from participatingfully in the process, the monitoring and evaluation plan will bethat much more successful and have long-term results.

In the Dominican Republic, the threat that the primary donor, theU.S.Agency for International Development (USAID), would stopall funding and technical assistance to local NGOs involved inHIV prevention was the impetus for local NGOs to come togetherto build consensus around how they were going to work jointlytowards the common aim of preventing the spread of HIV. TheNGOs recognized they had become so focused on the need tocompete for limited funds that they were unable to share ideasand work together. In order to break this cycle of competitivebehavior, the NGOs agreed to engage in a reflective and visioningworkshop. An external consultant was hired to help design andfacilitate the workshop. A primary focus of the workshop was toengage participants in a personal review of the actions, beliefsand spiritual values that are often overlooked when people workin large groups. The workshop succeeded in creating a “safe”non-competitive environment where individuals could listen to eachother, build positive relationships and begin to work together.While a few organizations found such collaboration difficult andwere unwilling to change their competitive behavior, the majorityof NGOs learned that building consensus and working in collabo-ration helped them achieve their common goal as well asstrengthen their own organizations.39

In Honduras,Tanzania, and Indonesia, a collaborative designprocess created integrated prevention programs.40 Because con-sensus building was coupled with practical program design expe-rience, participants from the various organizations learned tounderstand each organization’s role in the comprehensive pro-gram and how to collaborate to achieve the objectives of the pro-gram. At the Honduras design workshop, for example, theMinistry of Health designed a project to strengthen its STD ser-vices. Other participating NGOs then added an STD referral com-ponent to their projects to ensure project staff and volunteers

would encourage members of the target populations to use theseimproved services.

Activities such as those described above, which attempt to fostercollaboration and depict capacity building as a positive processtowards achieving common goals, were the key to building com-mitment around the evaluation of capacity building at both theorganizational and institutional levels. Rather than seen as puni-tive measures — of the inadequacy of financial controls or thelack of expertise in communication materials development —organizations saw capacity building evaluation as a useful part oforganizational planning and growth.

Step 2: Conduct AssessmentThe next step in the implementation of a capacity building evalu-ation plan is to conduct baseline assessments of organizations andinstitutions. Organization assessments serve two primary func-

tions: to assess organizational capacityand to determine individual and organi-zational training and technical assis-tance needs.41 By participating in a sys-tematic diagnostic process, an organiza-tion is better able to critically analyzeits internal strengths and weakness, aswell as its relationship to other govern-mental and non-governmental agencies,community groups, beneficiaries anddonors. Institutional assessmentsreview the environments in which orga-nizations function and the strengthsand weaknesses of the larger contextwithin which HIV/AIDS prevention pro-grams are conducted.24, 42

In 1994, the Tanzania AIDS Project (TAP) conducted a nationalinstitutional needs assessment to determine the scope and needsof NGOs interested in HIV/AIDS/STI programming. This assess-ment revealed a wide variety of organizations involved in differ-ent aspects of HIV control, ranging from small indigenous groups


Organization assessments serve

two primary functions: to

assess organizational capacity

and to determine individual

and organizational training and

technical assistance needs.41


and churches to internationally established development andhealth organizations. Selected on the basis of population densityand HIV prevalence, nine regions were chosen and NGOs withinthese regions were formed into clusters, with the purpose ofengaging a large number of local groups in partnerships to mobi-lize resources and build local capacity. Of the 180 NGOs in thenine regional clusters in the project, some of the NGOs had med-ical expertise, some had purely social objectives, some were affili-ated with religious groups, and others had political agendas.Significant time and resources were devoted to facilitating andguiding the creation of cohesive teams of NGOs working in eachregion. Steering committees, charged with the overall tasks ofplanning and monitoring the network’s activities, were estab-lished for each cluster by the participating NGOs themselves.12, 43

Prior to the project design process, an organization assessmentwas conducted with the targeted NGOs. In each cluster theNGOs together and individually conducted a “SWOT”analysis,which identifies internal and external strengths, weaknesses,opportunities and threats for each organization. In each region,workshops enabled representatives from the participating NGOsto understand the collective strengths and weaknesses of theirorganizations. This information provided the foundation for thenext step.

Step 3: Define Objectives and BenchmarksThe third step involves participatively setting objectives andbenchmarks. This process is based on the identification ofstrengths and weaknesses of the organization, but should also bebased on their vision for the future. Once the organization, orgroup of organizations, has a critical understanding of their inter-nal capacity and how their strengths can be maximized to estab-lish an effective interplay with the external environment, they canformulate a clear plan to enhance their ability to implement bet-ter projects.41 This process is most successful when representa-tives from all levels of the organization participate in theprocess.23 Defining specific benchmarks and indicators (withinthe categories defined in Table 2) is an important part of a strate-gic plan that can help organizations translate their goals into

results-oriented activities. For mostorganizations defining objectives oftentranslates into articulating a desire tobecome more proficient in a technicalor management skill to remedy specificweaknesses or to further enhancestrengths. Organizations can also setobjectives for enhancing networks,donor relationships and stakeholderinvolvement.

Returning to the previous example inTanzania, weaknesses identified duringthe organization assessment were rec-ognized as opportunities for strengthen-

ing specific managerial and technical areas. The NGOs identifiedobjectives to achieve within a given period of time, specifically tobecome proficient in the following management and technicalareas: project design, management, accounting, peer education,materials development, home-based care and counseling, income-generating activities, and condom social marketing for communi-ty-based and peer educators. In addition, they established objec-tives for how they were going to collaborate and work together.To achieve their objectives, they devised a plan for an intensivetraining program. Participatory training workshops were con-ducted that covered the expressed needs, from project designand management to training in income-generating activities.

In Asia, a regional institutional assessment of NGOs and govern-ment ministries revealed there were not enough people with theskills necessary to conduct quality comprehensive HIV preven-tion programs. Based on this need, the groups established objec-tives to improve specific skills, with the overall objective of estab-lishing a sustainable, quality training program for the region. ARegional AIDS Training and Education (RATE) Program was estab-lished in 1993. The RATE Program assisted organizations in Asiato conduct training and education needs assessments and thenprovided learning activities to meet the identified needs.Participants from Bangladesh, Cambodia, India, Indonesia, Laos,


Defining specific benchmarks

and indicators is an important

part of a strategic plan that can

help organizations translate

their goals into results-oriented

activities.

Mongolia, Nepal, the Philippines, South Pacific Islands, Sri Lankaand Thailand have been trained in HIV/AIDS education, manage-ment of sexual transmitted infections, HIV/AIDS policy, qualitynews reporting on HIV/AIDS issues, and training skills.44

Step 4: Monitor ProgressBuilding capacity is a process in which training and technicalassistance play only a part. Developing and internalizing skills,

knowledge, and a clear understandingof complex concepts or procedures —all fundamental elements of capacity —demands nurturing and continual atten-tion.27, 42 Thus, a significant componentof any capacity building strategy isestablishing a pattern of regular and fre-quent supportive and participatorysupervisory visits to the projects.During these visits, implementingagency partner staff together monitorthe progress of building capacity interms of both quantitative process indi-cators and measures of quality. Beyondsimply asking,“how many people havebeen trained in technical or manage-

ment skills?”questions are asked, such as: “how are newlyacquired skills being utilized?” and “how has the strengthenedmanagement system improved productivity?” Support and super-vision of capacity building should be carefully delivered so that itis collaborative, with an understanding that the process of trans-ferring knowledge and skills is two way.

Step 5: Measure OutcomesAs is the case with any evaluation, in evaluating capacity buildingefforts it is critical to determine if the project made a difference.For capacity building evaluation, it is important to determine notonly if objectives were met, but how and how well. A combina-tion of approaches can be effectively used to determine projectoutcomes, including self-assessment by program managers, inter-views with key informants, focus group discussions with organi-


A significant component of any

capacity building strategy is

establishing a pattern of regular

and frequent supportive and

participatory supervisory visits

to the projects.

zation staff, case studies, and audits.AIDSCAP developed two complemen-tary instruments to help program man-agers assess and evaluate capacity build-ing efforts: the Rapid OrganizationalAssessment (ROA) and theComprehensive Key InformantInterview Questionnaire (CKIIQ). TheROA focuses on management systems,structures, staff skills, and external rela-tions. It can be used by an organization

or coalition as a part of project design, monitoring or evaluationto identify organizational strengths and weakness. The CKIIQincludes sections on program management as well as on techni-cal skill building and networking. It focuses on identifying retro-spectively specific examples of increased capacity and lessonslearned. Both instruments were designed to provide qualitativeand quantitative information from program managers and toaddress sustainability issues. Effectiveness related to training andtechnical assistance provided is measured by the program man-agers themselves, either through a self assessment or facilitated byan outside consultant.This participatory methodology defineseffectiveness from their personal perspective. These instrumentswere adapted locally and were used with both governmental andnon-governmental organizations.

The Rapid Organizational Assessment was conducted in Haiti atthe end of the AIDSCAP project funding cycle. All of the imple-menting agencies chose self-assessment rather than having an out-sider conduct the assessment. Ultimately, the organizations foundthat the process of conducting the assessment not only providedthem with a better understanding of their capacity and a vision ofthe future of their organizations, but also with a product usefulfor marketing themselves to other donors.

In Brazil, the Comprehensive Key Informant InterviewQuestionnaire was conducted with each implementing agencyby an outside consultant. The analysis of individual accomplish-ments in technical areas and increased capacity in organizational


For capacity building evalua-

tion, it is important to deter-

mine not only if objectives were

met, but how and how well.


development allowed for an identification of the enhanced net-works between each organization, as well as the strengthenedinstitutions which resulted.

Step 6: Analyze and Interpret ResultsFor the capacity building evaluation plan to be truly participatoryand useful, the organizations must be involved in this last step —the analysis and interpretation of results of data collection efforts.Regardless of whether program managers conducted a self assess-ment, or if an outside consultant facilitated the process, theresults should be used as a foundation for discussion to be sharedwith the entire organization. Together organization staff, or acoalition group, can analyze and interpret results to identify thefactors that have facilitated or hindered increased capacity overthe time period identified. This final step also returns organiza-tions to the first step of renewing consensus towards capacitybuilding and starts the process of planning for the next set ofobjectives and benchmarks.

L E S S O N S L E A R N E D A N DR E C O M M E N D A T I O N S

Enhancing individual, organizational, and ultimately institution-al capacity in HIV prevention is a requirement for the sustain-ability and ultimate success of efforts to reduce HIV incidenceworldwide.

As prevention programs evaluate the success and failures oftheir efforts, the evaluation of capacity building must be anessential component to a comprehensive evaluation strategy.

Conducting mixed-method evaluation of capacity buildingefforts allows for a more insightful assessment of capacitybuilding and a more comprehensive evaluation of HIV preven-tion programs.

The process of evaluating capacity building, when conductedin a participative manner, serves not only to guide programswith information on effective approaches and continuedneeds, but also acts to build capacity in and of itself.

Organizations or coalitions with the capacity to build consen-sus, conduct assessments, define objectives, monitor progress,measure outcomes, and analyze results will begin to be sustain-able in the challenging and dynamic environment of HIV pre-vention.

AIDSCAP’s capacity building conceptual framework — linkingstrategies to outputs, outcomes, and impact across three levelsof interventions — provides an instructive paradigm for thedevelopment of capacity building interventions and the evalua-tion of HIV prevention interventions worldwide. While beingadapted to the local context and specific project priorities, theframework highlights the synergistic benefits from a compre-hensive approach to increasing capacity.

Lessons Learned and Recommendations 21

LE

SSO

NS

LE

AR

NE

D A

ND

RE

CO

MM

EN

DA

TIO

NS

References 23

R E F E R E N C E S

1. Development Associates, Inc. Management Review of theAIDSCAP Project. Washington, DC., 1995.

2. Family Health International/AIDSCAP. Making PreventionWork: Global Lessons Learned from the AIDSCAPProject 1991-1997. Family Health International/AIDSCAP.Washington, DC., 1997.

3. Cassels,A. Aid instruments and Health SystemsDevelopment: An Analysis of Current Practice. HealthPolicy and Planning, 11(4): 354-368. 1996.

4. James, R. Strengthening the Capacity of Southern NGOPartners: A Survey of Current Northern NGOApproaches. INTRAC Occasional Papers Series.Vol 1, No 5INTRAC: Great Britain., 1994.

5. Robinson, M. Development NGOs in Europe and NorthAmerica: A Statistical Profile. Charity Trends, Charities AidFoundation Tonbridge: 154-165. 1991.

6. Oakley, P. Projects With People: The Practice ofParticipation in Rural Development, ILO. Geneva,Switzerland. 1991.

7. Sahley, C. Strengthening the Capacity of NGOs: Cases ofSmall Enterprise Development Agencies in Africa.INTRAC: Great Britain, 1995.

8. Campbell, P. Strengthening Organizations. NGOManagement No. 18: 21-24, International Council of VoluntaryAgencies. July-September 1990. Geneva, Switzerland.

9. Marsden, D.; Oakley, P.; Pratt, B. (eds.) Measuring the Process:Guidelines for Evaluating Social Development.International NGO Training and Research Centre, Oxford. 1994.

RE

FE

RE

NC

ES


10. Mann, J.;Tarantola, D. (eds.) AIDS in the World II: GlobalDimesions, Social Roots, and Responses. The GlobalAIDS Policy Coalition, New York: Oxford University Press,1996.

11. Kanter, R. M. The Measurement of OrganizationalEffectiveness, Productivity, Performance and Success:Issues and Dilemmas in Service and NonprofitOrganizations. PONPO Working Paper No. 8 Yale UniversityInstitution for Social and Policy Studies, Institute ofDevelopment Research. Boston, MA., 1979.

12. WHO/GPA. Consultation of Strengthening NGOHIV/AIDS Umbrella Initiatives. May 29-31, 1995. Geneva.

13. Golembiewski, R.; Proehl, C.; Sink, D. Estimating theSuccess of OD Applications. Training and DevelopmentJournal 36, No 4 : 86-95. April 1982.

14. Carvalho, S.;White, H. Performance Indicators to MonitorPoverty Reduction. Washington, DC: The World Bank, 1993.

15. Brewer, M. B. Evaluation: Past and Present. TheHandbook of Evaluation Research. Beverly Hills: SagePublications, Inc. 1983.

16. Beckhard, R., Harris, R. Organizational Transitions:Managing Complex Change. Reading, MA:Addison-WesleyPublishing Co., 1987.

17. Kim, C.;Whetten, D. (eds) Organizational Effectiveness: AComparison of Multiple Models. Academic Press, NY.1983.

18. Purdey,A.; Adhikari, G.; Robinson, S.; Cox, P. ParticipatoryHealth Development in Rural Nepal: Clarifying theProcess of Empowerment. Health Education Quarterly,Vol.21 (3); 329-343, Fall, 1994.

References 25

19. Finsterbusch, K.;Van Wicklin,W.A. III. The Contribution ofBeneficiary Participation to Development ProjectEffectiveness. Public Administration and Development,Vol. 7pp.1-23, 1987.

20. Israel, B.; Checkoway, B.; Schulz,A.; Zimmerman, M. HealthEducation and Community Empowerment:Conceptualizing and Measuring Perceptions ofIndividual, Organizational, and Community Control.Health Education Quarterly,Vol. 21 (2): 149-170, Summer1994.

21. Kiggundu, M. N. Managing Organizations in DevelopingCountries: An Operational and Strategic Approach.WestHartford, CT: Kumarian Press, Inc. 1989.

22. Fowler,Alan. Capacity Building and NGOs: A Case ofStrengthening Ladles for the Global Soup Kitchen?Institutional Development Vol 1, No 1 PRIA (Society forParticipatory Research in Asia). New Dehli.August 1994.

23. Porras, J.; Silvers, R. Organization Development andTransformation. Annual Review of Psychology 42 : 51-781991.

24. Beer, M.; Walton, E. Developing the CompetitiveOrganization: Interventions and Strategies. AmericanPsychologist. February 1990: 154-61.

25. Seligman,B.; Murray, N. Lessons Learned about Evaluationin the Organizational Development Process. FamilyPlanning Management Development. Newton, MA:Project Management Sciences for Health, 1992.

26. Lafond,A.K. Sustainability in the Health Sector: TheResearch Study. Health Policy and Planning 1995, 10(Sup.):1-5.


27. Kauffman, C.K.; Hue, L.; Randolph, S.; Fee, N. LessonsLearned in Sustainability of Youth Peer HIV/STDPrevention Education: A PVO/NGO Partnership.Vancouver Int’l HIV/AIDS Conference [Tu.C.331] 1996.

28. Bossert,T.J. “Can they get along without us?”Sustainability of Donor-Supported Health Projects inCentral America and Africa. Social Science and Medicine,30: 1015-23 1990.

29. Rugg, D.L.; Nowak, G.;Westover, B.; Monterroso, M.; Pinckney,L.; Kennedy, M.; Shepherd, M. Evaluating the Contributionsof Social Marketing to HIV Prevention in Five U.S.Communities. Vancouver Int’l HIV/AIDS Conference[Tu.D.2880] 1996.

30. Carrol,Thomas. Intermediary NGOs: The SupportingLink in Grassroots Development.West Hartford, CT:Kumarian Press, Inc. 1992.

31. WHO/GPA. A Methods Package: Prevention of HIVInfection. Geneva. 1994.

32. Behets, F.T-M.; Génécé, E., Narcisse, M.; Cohen, M.; Dallabetta,G. Approaches to Control of Sexually TransmittedDiseases in a Difficult Context: The Case of Haiti from1992 through 1995. Bulletin WHO. In press. 1997.

33. Desormeaux, J; Behets, F. M-T.;Adrien, M.; Coicou, G.;Dallabetta, G.; Cohen, M.; Boulos, R. Introduction of PartnerReferral and Treatment for Control of SexuallyTransmitted Diseases in a Poor Haitian Community.International Journal Of AIDS and STDs 1996; 7:502-506.

References 27

34. Behets, F.M-T.; Desormeaux, J.; Joseph, D.;Adrien, M.; Coicou,G.; Dallabetta, G.; Hamilton, H.; Moeng, S.; Davis, H.; Cohen, M.;Boulos, R. Control of Sexually Transmitted Diseases inHaiti: Results and Implications of a Baseline StudyAmong Pregnant Women Living in Cité SoleilShantytowns. Journal Inf Dis 1995; 172: 764-761.

35. Behets, F.T-M.; Génécé, E.; Narcisse, M.; Cohen, M.; Dallabetta,G. Approaches to Control of Sexually TransmittedDiseases in a Difficult Context: the Case of Haiti from1992 through 1995. Bulletin WHO 1997. In Press.

36. Kotellos, K.; Githens,W.; Hartwig, K. IncorporatingEvaluation into Project Design, AIDSCAP Evaluation ToolsModule #3, Family Health International, 1994.

37. Roundtree, E.H.; Diseroad, K.; Pier, K.; Rosenstreich, D.Coalitions: Research on Maximizing Resources for HIVPrevention Education.Vancouver Int’l HIV/AIDS Conference[We.D.3892] 1996.

38. Arygis, C. Inappropriate Defenses Against theMonitoring of Organization Development Practice.Journal of Applied Behavioral Sciences, vol 26, no 3.

39. Mahler, H. Dominican NGOs Move from Competition toCollaboration. AIDScaptions, vol II, no 2 July 1995.

40. Making Prevention Work: Global Lessons Learnedfrom the AIDSCAP Project 1991-1997. FHI/AIDSCAP 1997.

41. Bowers, D.; Franklin, J. Survey-Guided Development: UsingHuman Resources Measurment in OrganizationalChange. Journal of Contemporary Business 1 No 3: 43-55,Summer 1972.


42. Hage, J.; Finsterbusch, K.. Organizational Change as aDevelopment Strategy: Models and Tactics forImproving Third World Organizations,Volume I. Boulder,CO: Lynne Reinner Publishers, 1987.

43. Odhiambo-Ochola, P.; Lutwaza, G.; Shariff, H.; Nguma, J. NGOClustering for HIV/AIDS/STD Prevention and Control:The Tanzania AIDS Project (TAP)/AIDSCAP Experience.Vancouver Int’l HIV/AIDS Conference [Th.D.131] 1996.

44. Burian, Chalintorn. Centers of Excellence in Asia: ARegional Approach to Capacity Building. AIDScaptions,vol II, no 2. July 1995.

The HIV/AIDS Prevention and Control SYNOPSIS ... - USAID

Documents

Transcript of The HIV/AIDS Prevention and Control SYNOPSIS ... - USAID