10TH

EES Biennial ConferenceHelsinki, FINLAND

October 1-5, 2012

Romeo B. Santos

WorkLand M&E Institute

1

How sound is your Theory of Change?

Building skills in developing ToC Model and

understanding its link with rigorous evaluation

and research design

DAY 1 WORKSHOP Room 7 October 1, 2012 (Monday)

2

THEORY OF CHANGE

Why ToC?

a “buzz word”

“results-agenda”

- Complicated

- Confusing

- various orgs ~ different interpretations

3

THEORY OF CHANGE

What is ToC? Its Origin?

… literature says – H. Chen, P. Rossi, M. Patton, and C.

Weiss gave hints…1990s.

Weiss “popularized ToC” [1995 book – “New Approaches to

Evaluating comprehensive Community Initiative”]

-“a theory of how and why an initiative works.”http://www.theoryofchange.org/about/what-is-theory-of-change/

Weiss, C.H. (1995). Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation

for Comprehensive Community Initiatives for Children and Families

4

THEORY OF CHANGE

Criticisms, Arguments, Labels

- Development result is not LINEAR…

-Does not represent COMPLEXITY, w/c is

true nature of development intervention.

-Is there a BETTER Alternative???

5

FOCUS of this WORKSHOP

Formulation of ToC

& ToC Model; know how „SOUND‟

NOT primarily overcoming Controversies or

Arguments…

-BUT, present valid responses & counter

arguments that make sense…

Nature of Intervention

INTERVENTION ≈ CHANGE

- changing state of things for the better -

CHANGE can be:

positive [ + ]negative [ – ]

& measurable

6

ToC Concept

T1

POLICY, PROGRAM, PROJECT

T2

(AP Photo/John Raoux) Mon May 11, 4:24 PM ET 7

Target

[GOAL]

Start

How is intervention

designed?

[Strategic Planning]

How is success of

intervention

measured?

[M&E]

8

All INTERVENTIONS have a THEORY

… how Change will transform state

of things toward a desired state

‘THEORY OF CHANGE’

it is the Logic of Intervention!

INPUTS ACTIVITIES OUTPUTS EFFECTS OUTCOMES IMPACTS

Results of Intervention

short term - Effects

intermediate - Outcomes

long term - Impacts

9

TAKE NOTE!!!

RESULTS are normally referred to

as OUTCOMES as well, and vice-versa(Santos, R. 2010)

Traditional approach

Results

Results-based approach

10

‘THEORY OF CHANGE’

links

OUTPUTS to RESULTS

OUTPUTS = ‘Deliverables’ or

‘Products’ of ACTIVITIES

RESULTS = Effects [immediate],

Outcomes [intermediate],

Impact [long term ≈ Goal]

11

RESULTS or OUTCOMES involve change of

behaviour, change to and/or existence of an

enabling environment that supports the

change.

National GOALs of a Country:

Self sufficiency

Increased employment

Social order

Gender equality

Environmental sustainability

Better quality of life

RESULTS ≈ OUTCOMES: linked to GOAL

12

The clearer the better

Theory of Change ≠ LOGIC Model

Theory of Change Model ≈ LOGIC

Model

Theory of Change ≈ LOGIC of

Intervention

13

TALKING about MODEL

ToC Model is a ‘knowledge model’

– getting a ‘chunk’ of the problem*

and ‘modeling’ it

* [phenomenon, complex instance,

intellectual puzzle]

aid to better understanding; the

simpler the better

14

SOUNDNESS of MODEL

consistent & coherent Articulation

is KEY

– requires convention, rule.

Inconsistency or contradiction kills logic –

spawns confusion!

Coherent Theory of Change is needed!

15

Walk-Thru on currently used

Tools

Related concepts

Schools of thought

16

Logframe

PCM GROUP BVA, Belgium

17

Visual 4x4

Project Purpose, Results, Activities &

corresponding Assumptions and ‘Pre-

conditions‘ in 1st and 4th columns

2nd

& 3rd

Columns – specify objectives

mentioned in 1st

Column with objectively

verifiable Indicators & Means of

Verification

PCM GROUP BVA, Belgium

LFM

18Source; Transparency international

19

Narrative Summary

Verifiable

Indicators

Means of

Verification

Important

Assumption

s

Pre-

Conditi

ons

Narrative Summary

Project Purpose (Outcome)Direct effects on the target group or societythat can be attributed to the project.

Overall Goal (Impact)Intended positive effect or impact produced by the realization of the Project Purpose.

OutputsConditions that are produced through the implementation of Project Activities.

ActivitiesSeries of specific actions through which Inputs are mobilized to produce the Outputs .

Understanding the Project LF

Acknowledgment:: http:/www.jica.go.jp/english/operations/evaluation/tech_and_grant/guides/guidelines.html

20

Narrative

Summary

Objectively

Verifiable

Indicators

Means of

Verification

Important

Assumptions

Overall Goal

Project Purpose

Outputs

Activities Inputs

Pre-Conditions

Causal Relationships

Means

Ends

THEN IF

IF ANDTHEN

IF AND

IF

THENAND

THEN

Understanding the Project LF

Acknowledgment:: http:/www.jica.go.jp/english/operations/evaluation/tech_and_grant/guides/guidelines.html

Narrative Summary

21

Overall GoalCapacity of students in mathematics and science is upgraded.

Project PurposeQuality of education in math and science at secondary school in model district is improved.

Outputs1. In-service education and training system established at Teachers College for

key trainers

2. In-service education and training system established at district level.

3. Role of district center as a resource center established.

Activities1.1 Develop curricula and training materials1.2 Provide training to national trainers..etc2.1 Provide training for teachers at district level2.2 Renovate existing facilities at model schools3.1 Publish books and teaching materials

Inputs

LF: Math & Science Project

Acknowledgment:: http:/www.jica.go.jp/english/operations/evaluation/tech_and_grant/guides/guidelines.html

22

PCM

PCM GROUP BVA, Belgium

23

THE CONVENTIONAL PROJECT CYCLE

[acc. to an international NGO]

IMPLEMENTATION

EVALUATION

PROGRAM

IDENTIFICATION

FORMULATION

APPRAISAL/

COMMITMENT

(Local Livelihoods Ltd., 2006)

24

The Project Cycle & M&E: JICA

Ex-ante

Evaluation

Terminal

Evaluation Mid-term

Evaluation

Project Cycle

Planning

ImplementationPost-

implementation

Feedback

Feedback

Ex-post

Evaluation

Project

Ends

Project

Starts

Acknowledgment:: http:/www.jica.go.jp/english/operations/evaluation/tech_and_grant/guides/guidelines.html

25

(IEG WB, 2012)

RF

26Causal Diagram

(NGO-Ideas, 2012)

27http://web.mit.edu/urbanupgrading/upgrading/issues-tools/tools/problem-tree.html

PROBLEM TREE

Insufficient

maintenance

funds

Bureaucratic

water

administration

Deteriorated water systemFew service

connections

High rate of

diseases

Low productivity of

workersHigh rate of infant mortality

Low income

Lack of sufficient

clean water

Causes

Core Problem

Effects

28(Kellogg Foundation,2004)

ToC

Theory of Change

REDUCED POVERTY

INCREASED ECONOMIC GROWTH

Increased participation of poor population in the formal economy

Registration

system

Number of properties

and business

registrations

50 Radio ads

30 Tv ads

3 mil pamphlets

Feedback

mechanismFeedback

mechanism used

Linkages to

supporting programs

established

Registration system

used by participants

Establishing

a registration

system

Media

awareness

campaign

Development

of a feedback

mechanism

Record

keeping

Coordinating the

supporting

services/program results

OU

TC

OM

ES

Medium-Term

Long-Term

Short-Term

OU

TP

UT

SA

CT

IVIT

IES

Human

resourcesInfrastructure

Financial

resourcesSupplies

Supporting

programs

INP

UT

S

H. Pohoresky, et.al, IPDET 2008)

29

Results-Based M&E

e.g., DRR Program

Outcomes&

Effects

Awareness & active

application of DRR measures

Outputs

# of seminars held, # of trained

communities & on DRR

Activities

Training & campaign to

educate people on DRR

Inputs

Budget, trainees, programs &

other resources for DRR training

Impact

(Goal)

Sustained level of safety from

disaster in the community

Im

ple

me

ntatio

nR

esu

lts

30(Rist & Imas,2008)

31

Basic premises: Conceptual basis

Caring for Logic

Seeing where coherence is weak

WHY refinement is needed

Better semantic~articulation?

ILLUSTRATIONS

32

INTEGRATION

Healthy

communication

Common

language

One line of

thought

INTEGRATION needed in M&E practice

INPUTS ACTIVITIES OUTPUTS EFFECTS OUTCOMES IMPACTS

33

Are Outputs RESULTS???

Yes, but…

RESULTS

OUTPUTS EFFECTS OUTCOMES IMPACTS ENDS CONSEQUENCES UPSHOTS OFFSHOOTS

... are all RESULTS ..

So then, in the ToC concept of RB M&E „doctrine‟, would

RESULTS include OUTPUTS???

RESULTS-based management ... “Outcome-based budgeting”

Result-oriented vs Output-oriented

OUTCOMES/RESULTS are what give real benefits; cannot be „bought‟

Budgets to OUTPUTS, Manage to OUTCOMES

34

PREMISES

35

... all are HUMANS too..

So then again, IF I were to talk about HUMANS of the

FEMININE gender, would I include MEN???

Are Men HUMANS???

Yes, but…

HUMANS

MEN WOMEN BOYS GIRLS WIVES PARENTS GRANDMAS CHILDREN

36

LOGICAL FRAMEWORK applied to EX-POST EVALUATION of a PROGRAM

GOAL

Contribute to an increase in the facilitation and safety of…;

expansion in the capacities of…; and increase in opportunities

for…

PROJECT

PURPOSEImprove the quality of…

OUTPUTS Rating schools will be constructed in _____, _____, ______

INPUTS

Implementation of … works (construction of ….schools)

Procurement of equipment for…

Consulting services

Logframe used in a sample program

(Santos, 2010)

37

CODERESULTS FRAMEWORK

OUTPUTS OUTCOMES IMPACT (GOAL)

L-1

Air pollution from mobile &

stationary sources mitigated

Comprehensive assessment

of DRM status

Fuel quality improved

Emissions from vehicular

used improved

Traffic congestion reduced

through improved traffic flow

Air sector appropriately

legislated, and its

management monitored

Capacity building and

institutional development plan

Public awareness for

cleaner air and support to

air quality-related activities

Improved public health

monitoring of the effects of

air quality

Strengthened capacity of

public health monitoring of

regional offices of the

DOH

Sustainable

improvement in

Metro Manila’s

air-shed quality

LG-1

Retrofit of buildings

13 million CFLs to consumers

Energy efficient lighting

programs

Super ESCO

Certification scheme

Certification process for

energy and

environmentally efficient

commercial buildings

Reduced cost of power

generation

A viable ESCO industry

Reduced cost of

power

generation

(Santos, 2010)

38

PROGRAM

CODE

RESULTS FRAMEWORK

OUTPUTS OUTCOMES IMPACT (GOAL)

LCI-1

Identification of essential

adjustments to

administrative,

institutional, and

regulatory practices and

existing legislations in

order to attract private

investments in the

GOP’s wastewater

sector

Promotion of innovative,

simple and effective

wastewater treatment

techniques

Strengthened

partnership among

agencies

Increased coverage of

sewerage and

sanitation as a

percentage of total

coverage and the

reduction of pollution

load of the Manila Bay

Enhanced inter-agency

consultation and

decision-making

processes

Improved policies,

regulations, plans, and

project

Increased

sustainability of

pollution reduction

activities

Increased

effectiveness o f

the agencies

responsible for

water pollution

control through

improved

coordination

(Santos, 2010)

39

PROGRAM

RESULTS FRAMEWORK

ACTIVITIES OUTCOMESIMPACT

(GOAL)

MM Air Quality

Improvement

Sector Development

Program

Air sector’s appropriate

legislation and its

monitoring system for

management

Capacity building and

institutional

development plan

Public awareness

for cleaner air and

support to air

quality-related

activities

Strengthened

capacity of public

health

monitoring of

regional offices of

the DOH

Improved public

health

Sustainable

improvement

in Metro

Manila’s air-

shed quality

INPUTS OUTPUTS

ODA loan, financial

resources

People

Equipment, others

Drafting of laws

Training for cap

building

Sample results framework of typical intervention program (Results-based approach)(Santos, 2010)

40

Does your ToC make sense?

ToC Model is not just a ‘pile it up

project’, nor a ‘cut & paste’ work!

Theory of Change is built on strong

logic, the stronger the logic the more

coherent and sounder it can be.

http://news.yahoo.com/comics/reality-check

41

CBDRR Program, Cambodia

EDUCATION Program, Czech Republic

N-S Rail Project, Philippines

ToC Model Samples

Logic Model of the Programme

INPUTS ACTIVITIES OUTPUTS RESULTS:

OUTCOMES

RESULTS:

IMPACTS

Lecturers

from P2P

Course

materials

Lectures to

students

Training of

teachers

Students

trained

Teachers

trained

Raised students’

awareness

Schools’ or communities’

increase awareness to

other cultures

Raised students’

tolerance

Schools’ or communities’

increase tolerance to

other cultures

People for People:

Multicultural Education in Czech Republic

Sandra Fox, Australia

Petra Krylova, Czech Republic

Romeo Santos, Philippines

Mark Orkin, South Africa

Jaya S Conhye, Mauritius/France

Enoth Tumukwasibwe, Uganda IPDET 2008

ToC Model of the EVAW Programme

Afghanistan

INPUTS ACTIVITIES OUTPUTS RESULTS:

EFFECTS &

OUTCOMES

RESULTS:

IMPACTS

Financial

support to

NGOs

Training

programmes

for national

& local

NGOs to

implement

initiatives on

EVAW

Effective

mgt of EVAW

special Fund

Training on

gender/ VAW/

SGBV/ male

participation

for xxx NGOs

Knowledge

sharing &

learning

Sessions

Administering

of 1 to 1

technical

support

Training

programmes

held

KS&L

sessions

held

1 to1

technical

support

administered

Increased awareness of

EVAW issues among

NGOS, people

Improved

policy environment on

eliminating

GBV

Increased commitment of

NGOs toward EVAW in

Afghanistan

Improved implementation

of EVAW initiatives

Reduced violence

against women

Improved well-

being of women

and girls in

Afghanistan

(Santos, R. 2008)

Increased

resilience of

communities

to socio -

natural

disasters

Increased

access to

resilient potable

water systems

Reduced

economic

vulnerability

and food

insecurity

Reduced

morbidity to

water borne

diseases

Effects/Outcomes ImpactActivities OutputsInputs

Increased

awareness on

CBDDR

Improved

implementatio

n of DRR

plans

Increased

preparedness

in communities

Communities

mobilized

Preparedness plans

Evacuation shelters

People trained in DP

Canals dug

Loans without interest

made

Crops harvested

Trainings held

Water points

constructed

Filters distributed

Awareness trainings

held

Water committees

organized

Channels constructed

Houses raised

Trees planted

Community

mobilization

DRR plan

formulation

Training on DP

Construction of

shelters

Tree planting

House retrofitting

Digging of canals

Tree planting

Provision of loans

[woi]

Harvesting

Training on

agricultural

extension

Construction of

water points

Distribution of

filters

Awareness raising

on filters

Organization of

water committees

Technical

advisors

Construction

materials and

tools

Cash

Agricultural

inputs

Livelihood

assets

Water filters

Water pumps

Training

modules

DRR model

IEC materials

Cambodia CBDRR - Theory of Change Model

CAMBODIA CBDRR PROGRAMME

Sample Key Evaluation Qs

Questions on Outcomes

1. How are communities better prepared as a result of the

Programme?

2.How successful was the Programme in improving mitigation

infrastructures?

3. How successful was the project in reducing the economic

vulnerability of target communities?

4. Has the Programme resulted in increased access to potable

water?

5. To what extent was the necessary physical and economic

environment in place to support the Programme?

6. Is there evidence of environmentally- responsible behavioural

change and practice as a result of the Programme?

Other Q

7. Have there been unintended consequences as a result of the

Programme?

Question 1. How are communities better prepared for disaster

as a result of the project?

Sub-Question Are individual households better prepared for disaster as a

result of project?

Type of Q Cause and Effect

Measures &

Indicators

% of household representative who can list 4 steps of the evacuation

system

Target &

Standard

Indicator = 65% of household representative who can list the 4 steps of

the evacuation system

Baseline Data Yes – 0% of household representative who can list the 4 steps of the

evacuation system

Design Quasi experimental - Before and After O1 X O2

Data Sources Household representative (adult)

Sample Proportional Random Sampling -(90% confidence level - 3% error

margin)

Data Collection

Instruments

Interviewer administered survey

Focus Group Discussion / informal interview/ Case study

Data Analysis Quantitative: Frequency count, compare to standards.

Qualitative: Content analysis

Comment Graphic

CAMBODIA CBDRR: Design Matrix Sample

Question 6. To what extent was the necessary physical and economic

environment in place to support the Programme?

Sub-Question How effective were the DRR plans in providing

safe/convenient facilities?

Type of Q Descriptive/Normative

Measures & Indicators 1. # of rated cases of DRR implemented plans, 2. % level of compliance

with accepted standards

Target & Standard 1. 10 cases 2. 95% level of compliance

Baseline Data Nil

Design One Shot

Data Sources Key Informants, communities, people under Programme

Sample Snowball sample

Data Collection

Instruments

One- to One Interviews, Observation, Transect

Data Analysis Quantitative: Frequency count, compare to standards.

Qualitative: Content analysis

Comment Graphic

CAMBODIA CBDRR: Design Matrix Sample

48

QUESTION SUB-QUESTION TYPEMeasures &

IndicatorsTarget Baseline

6. Is there evidence

of environmentally-

responsible

behavioural

change and

practice as a result

of the program?

What is the number of

cases of environmentally-

responsible practices

among

people,/communities,

before and after the

Programme; and in

contrast to the comparison

group, before and after?

C&E

Quasi-

Experimental

Census of all

Programme

communities and a

sample of

comparison

providers

Yes Yes

After Programme

completion, do Programme

administrators believe that

it has made a difference in

residents‟ behaviour &

practices?

C&E Survey: yearlyYes Yes

Compared to before the

Programme what is the

evidence of increased

activities & practices

identified as

environmentally-

responsible?

C&E

FGD -randomly

selected trainees/

techno-voc

providers

Yes Yes

M&E Design Matrix Sample

CAMBODIA CBDRR Programme

Programaims

Indicator Type ofindicator

Dataneeded

Baseline

data

Target Data sources

Freq. data collectn,

methods

tools

In charge/ data collectn

Dissemination strategy

Goal

Increased

resilience

of

communitie

s to socio -

natural

disasters

%

villagers

saying

they feel

safer

from

calamitie

s

...

…

Impact

Extent

villagers

saying

they feel

safer from

calamities

Reported

incidents:

deaths,

damage..

(Yes)

-10%

Incre

ase

1st yr

-10%

decre

ase

3rd yr

Civil

Defense

data/

records

Communi

ty reports

Field

reports/

records

Yearly

Desk R

1-shot

survey

Quasi-E

Time

series

Case S

FGD

HQ-

M&E

section

Commu

nity field

teams

Annual

report

Impact

assessmt

report

Intermediat

e result

Improved

implementati

on of DRR

plans

…

# of DRR

plans

administe

red

effectivel

y

…

…

Outcome

Cases of

DRR

plans

admin to

communit

ies

(Yes)

…

…

Civil

Defense

data/

Communi

ty

reports,

Field

reports/

records

Yearly

Desk R

Case S

FGD

HQ-

M&E

section

Commu

nity field

teams

Annual

report

Impact

assessmt

report

(Santos, R. 2009)

M&E Framework – CBDRR

“Ssshhh Zog!... Here

comes one now!”

“Are you sure it ain’t

boo-boo Konk?”...

Adaptation from: Gary Larson, FarWorks Inc., Creators Syndicate, 1982)

50

“Fair use, a limitation and exception to

the exclusive right granted by copyright law to the

author of a creative work, is a doctrine in United

States copyright law that allows limited use of

copyrighted material without acquiring permission

from the rights holders. Examples of fair use

include commentary, criticism, news reporting,

research, teaching, library archiving and

scholarship. …”

[http://en.wikipedia.org/wiki/Fair_use]

FAIR USE LAW

52

1. ADB Publications. http://www.adb.org/projects/rbm/outcomes-diagram.asp: Accessed: Jun12, 2009.

2. Designing a results framework for achieving results: a how-to guide, IEG, WB, 2012.

3. Guide and Proceedings, NEDA.

http://www.neda.gov.ph/progs_prj/ICC/Guide&Proced_june2004/ICC%20Project%20Evaluation%20Form

s%201-6%20(as%20of%2028%20June%202004).pdf.Accessed: Mar 23, 2009.

4. Hatry, Harry and Wholey, Joseph. Performance Measurement: Getting Results. Washington DC, Urban

Institute Press, 1999.

5. Japan International Cooperation Agency,:

http://www.jica.go.jp/english/operations/evaluation/oda_loan/post/2008/index.html. Accessed: Mar 23,

2009.

6. Measuring program outcomes: A practical approach. Arlington, VA: United Way of America, 1996.

7. Monitoring and evaluation Toolkit. Local Livelihoods Ltd., UK, 2006.

8. NGO-IDEAs Tiny Tools – Guide to use “Causal Diagrams” , 2012.

9. Problem Tree. http://web.mit.edu/urbanupgrading/upgrading/issues-tools/tools/problem-tree.html:

Accessed Mar 15, 2009.

10.Santos, R.B. "A look inside development: What the Monitoring and Evaluation Framework Designs of

Foreign-funded Urban Development Projects in MM, Philippines Reveal" 2010 Evaluation Conference,

AEA, San Antonio, Texas, Nov 10-14, 2010.

11.Santos, R.B.“Comparing measures of success’. The monitoring and evaluation (M&E) of Japan’s ODA-

funded projects in the Philippines, Sri Lanka and Indonesia.” Research funded by the Sumitomo

Foundation Research Grant, Mar 2009 - Sep 2010.

12. Strategic Planning. http://www.civicus.org. Accessed: Mar 12, 2009.

References

LONG BEACH, BORACAY ISLAND

Philippines

... if what ToC does is blow away our woes when it clicks...then life’s gonna be

just a walk on the beach!...

53

Thank you!Merci beaucoup!

Salamat po!Mwebare munonga!

Dankie!

Dekujeme!