DFID Social Transfers Evaluation Summary Report

RESEARCH REPORT 60

DFID Social TransfersEvaluation Summary Report

Mark Davies

September 2009

1

DFID Social TransfersEvaluation Summary ReportMark Davies

Institute of Development Studies, Centre for Social Protection

based on reports byArmando Barrientos, Mark Davies, Stephen Devereux,Sam Hickey and Rachel Sabates-Wheeler, Bruce Guenther,Ian Macauslan

September 2009

IDS RESEARCH REPORT 60

Institute of Development Studies at the University of Sussex Brighton BN1 9RE UK


DFID Social Transfers Evaluation Summary ReportMark DaviesIDS Research Report 60First published by the Institute of Development Studies in September 2009© Institute of Development Studies 2009Cover photo: G.M.B. AkashPhoto caption: Nine year-old Shahina Begum (front) with friends on her way to school in the Dawabari river bedarea.

ISSN 0141-1314 ISBN 978 1 85864 778 9

A catalogue record for this publication is available from the British Library.All rights reserved. Reproduction, copy, transmission, or translation of any part of this publication maybe made only under the following conditions:• with the prior permission of the publisher; or• with a licence from the Copyright Licensing Agency Ltd., 90 Tottenham Court Road, London W1P 9HE, UK,or from another national licensing agency; or• under the terms set out below.

This publication is copyright, but may be reproduced by any method without fee for teaching or nonprofit purposes,but not for resale. Formal permission is required for all such uses, but normally will be granted immediately. Forcopying in any other circumstances, or for re-use in other publications, or for translation or adaptation, prior writtenpermission must be obtained from the publisher and a fee may be payable.

Available from:Communication UnitInstitute of Development Studiesat the University of SussexBrighton BN1 9RE, UKTel: +44 (0) 1273 915637Fax: +44 (0) 1273 621202E-mail: [email protected]: www.ids.ac.uk/ids/bookshop

Typeset by Barbara Cheney, Bath, UK. Printed by Nexus, Brighton UK.IDS is a charitable company limited by guarantee and registered in England (No. 877338).


2

3


Keywords: social protection; social transfers; DFID; social protectionprogrammes; design; implementation; influencing; monitoring and evaluation;poverty reduction.

Mark Davies is manager of the Centre for Social Protection at the Institute ofDevelopment Studies, having previously worked with the UK Department forInternational Development in Africa and London as a livelihoods advisor. He haspractical and research experience working in social protection, food security,livelihoods, poverty and vulnerability. He has worked extensively in Ghana, Kenyaand Malawi, working within bilateral agencies, multilateral organisations, in closepartnership with government and NGOs. Mark has in-depth knowledge of thepolicies, institutions and processes that influence both social protection policy andprogramming.


4

5


ContentsKeywords and author note 3

Preface 6

Abbreviations 6

Executive summary 71 Introduction 13

1.1 Scope, objectives and outline of work 13

1.2 Limits of the report 15

1.3 Methodology 15

2 Key lessons learned and recommendations 152.1 Introduction 15

2.2 Effectiveness in implementation of DFID-supported social transfer schemes 16

2.3 Impact of DFID-supported social transfer schemes 20

2.4 DFID and the politics of influencing 25

2.5 Monitoring and evaluating social transfers 28

3 Recommendations for improving the delivery and effectiveness of White Paper commitments 32

Annex 1 Core analytical framework and indicators 34References 36

TablesTable 1.1 DFID-supported social transfer programmes reviewed in detail 14

Table 2.1 Conditionalities applied in programmes reviewed 17

Table 2.2 Levels of transfers in programmes reviewed 18

Table 2.3 Reported food security impacts of programmes reviewed 21

Table 2.4 Reported economic growth impacts of programmes reviewed 22

Table 2.5 Poverty reduction impacts of case-study programmes 23

Table 2.6 Summary information on costs of social transfers 30

BoxesBox 2.1 Community-based targeting in the Bangladesh ‘CHARS Livelihood

Programme’ 19

Box 2.2 Indirect health impacts in the PSNP 25


6

PrefaceThis summary report draws directly and extensively from other reports preparedby Armando Barrientos, Mark Davies, Stephen Devereux, Sam Hickey and RachelSabates-Wheeler, Bruce Guenther, and Ian Macauslan as part of the reviewprocess in preparation for the DFID Social Transfers evaluation planned for2010–2011. I gratefully acknowledge comments and suggestions on previousdrafts from Armando Barrientos and Tim Robertson; however, full responsibility forthe text of this report rests with the author. In common with all evaluation reportscommissioned by DFID’s Evaluation Department, the views contained in thisreport do not necessarily represent those of DFID or of the people consulted.

We would like to thank DFID for funding this work.

Abbreviations BRAC Bangladesh Rural Advancement Committee

DAC Development Assistance Committee (Organisation for EconomicCooperation and Development)

DFID Department for International Development

IDRC International Development Resource Centre (Canada)

ILO International Labour Organization

M&E Monitoring and evaluation

NGOs Non-governmental organisations

OM Outcome mapping

OVC Orphans and Vulnerable Children

PSNP Productive Safety Net Programme

TIP Targeted Inputs Programme (Malawi)

7


Executive summaryOver the past few years DFID have increased their efforts to promote socialtransfers and social protection. The outcomes and impacts of social transfers andsocial protection schemes and DFID-influencing activities vary greatly relative tothe unique conditions that were applied in specific contexts. What has worked wellin one context may work very differently under a different set of conditions inanother context. However, a set of generalised findings can be identified thatsupport the knowledge and evidence base on the range of options available forsocial protection programming and policy debates. Findings, therefore, should beexamined further in specific contexts rather than being used as prescriptive anddefinitive policy options.

Lessons and recommendations

In implementation

The Africa and Asia DFID-supported social protection programmes examined inthis review were as likely to be implemented by government ministries as byinternational NGOs. The United Nations was then the third most used agency forimplementation.

In assessing the effectiveness in implementation of these programmes, there is anencouraging trend towards ‘needs-driven’ schemes (e.g. PSNP in Ethiopia) takingbeneficiary preferences increasingly into consideration. DFID should call for needsassessments in national and multi-donor programmes and engage with thesestakeholders to ensure programmes respond to those needs identified.

The debate about whether conditionalities should be applied to DFID-supportedsocial transfer programmes remains unresolved through our review. In relation topublic works programmes, the limitations of these programmes are evident inNepal, whereas in India, a more positive scheme can be seen. In relation to publicservices, concerns over the quality of the service accessed through socialtransfers can be overcome through initiatives that link social transfers to publicservices, particularly where the service is receiving investment. This approach isbeing considered in Zambia.

Decisions to apply conditionality must be considered carefully and based on anassessment of what seems feasible and appropriate for the particular context intowhich the intervention is being introduced.

The effectiveness of social transfers is largely dependent on their level andregularity. Experience from Zimbabwe’s ‘Protracted Relief Programme’ andEthiopia’s PSNP highlights the need to take account of household need ratherthan size, and to regularly adjust amounts transferred in line with price changes.

Targeting remains one of the most difficult problems facing those trying toimplement social protection. DFID-supported schemes use a variety of methodsfrom administrative schemes in Zambia to community-based schemes inBangladesh, all with some degree of success and failure. This complexity leads


8

us to recommend that while targeting must be implemented carefully andmonitored, the principle of ‘acceptable error’, particularly when building politicalsupport, needs to be considered.

On impact

Most social transfer interventions supported by DFID – unconditional cash, food orasset transfers, public works, school feeding schemes, agricultural inputspackages – contribute to realising the objective of enhanced household foodsecurity in the short term, and to reductions in the severity of poverty, thoughnot necessarily to sustainable poverty reduction in terms of falling povertyheadcounts in the longer term. The potential of social transfers to contribute tobroader economic growth can occur through either (1) multiplier effects or (2)market integration. Only six of the DFID-supported case-study social transferschemes reported any impact – positive, neutral, or even negative – on economicgrowth. It should be noted that the primary purpose of social transfers is to‘protect’ rather than ‘promote’ livelihoods, so a failure to find evidence of economicgrowth impacts is not necessarily an indictment of these programmes.

Governments and donors need to invest adequately in supporting the livelihoodsof poor people, which implies identifying innovative complementary interventionsto social transfers, or building on positively evaluated experiences such asBRAC’s ‘Asset Transfer Programme’ to the ‘ultrapoor’ in Bangladesh.

Social transfers can enhance the health status of programme beneficiariesthrough three pathways (the same as for education): (1) directly – by linking thedelivery of transfers to health services; (2) indirectly – if beneficiaries allocatesome transfer income to purchase healthcare; (3) incrementally – by deliveringcomplementary health interventions to the same individuals who receive socialtransfers. All three were recorded in some projects reviewed, occasionally withimpressive results in terms of health outcomes. Some of the strongest evidencecomes from the ‘Ultra-Poor Programme’ in Bangladesh which ‘directly’ linkstransfers to health services.

Social transfer schemes that promote education and health should intervene notonly on the demand side (by providing incentives or reducing costs), but also onthe supply side, to maximise improvements in both service uptake and outcomes.

On influencing

The analysis of DFID-influencing activities is used to suggest policy-influencingactivities and approaches, which should be continued or extended, whereasothers should be reviewed and/or are subject to revision.

DFID should continue

DFID’s most effective means of influencing government and donor approaches tosocial protection is its ability to facilitate the access of policy actors to the growingevidence base in this field. This capacity has allowed DFID to directly influencedebate over social protection in ways that have increased the number ofbeneficiaries who benefit from social protection schemes (e.g. Ethiopia) and

9


persuaded government officials to adopt social transfer pilot projects (e.g.Zambia). Where DFID has failed to produce and disseminate high-qualityinformation and knowledge in sequence with its facilitating and advocacystrategies within national policy processes it has been less influential (e.g. theearly stage of the process in Uganda). DFID should, therefore, continue to: act asa knowledge interface between key stakeholders and the wider evidence base onsocial protection; adopt a leadership role which in the past has been effective insecuring higher levels of influence; continue the strong support from the centre tocountry advisors.

DFID should review

DFID’s successes in creating policy spaces, which work towards donorharmonisation, have helped policy debates with government officials to movefaster than they might otherwise have done (for example PSNP in Ethiopia).DFID, however, should clarify its position on ownership or influence andreconsider the contradictions between two of its primary objectives – namely itsefforts to promote its own vision of social protection on the one hand, and itsefforts to secure government ownership of social protection policies and strategieson the other.

In some cases, for example Zambia, DFID has used evidence from social transferpilot projects as a basis for arguments to either scale up social transferprogrammes and/or promote broader social protection policies. Pilot projects haveprovided DFID, and other proponents, with something tangible around which tobase their strategies and commitment via their expenditure. Where no suchprojects exist (e.g. Malawi, Uganda), DFID has struggled to build its influence. Onthe down side, with projects implemented by NGOs (e.g. Zambia), this does littleto improve government capacity in this field or increase governmental ownershipof a social protection agenda.

DFID should develop alternative plans to the promotion of pilot cash transferprojects. Broad strategies to influence social protection policies within countriesshould not rest entirely on the promotion of pilot cash transfer projects – at thevery least, a Plan B is required. Pilots are indispensable as a means of learningabout design and delivery, but they are not useful or informative when they areused primarily as an advocacy tool.

Approaches that DFID should change

The ability of DFID to use any increased evidence on cost-effectiveness and widerimpacts to extend social protection programming relies on the creation ofinfluential partnerships. DFID advisors have developed relationships of trust withkey government officials in social welfare ministries, but have been lesssuccessful in building productive relationships with more powerful political andpolicy actors, including parliamentarians, high-level political leaders, and thosewithin the more powerful ministries of finance and planning.

Greater efforts are required in developing close working relations with policyactors operating within finance ministries. It also might be possible to locateprogramme delivery units within ministries of finance (e.g. see the pension


10

scheme in Lesotho) and to run programmes within broader systems of socialservice delivery that have greater political support and institutional capacity,particularly health and education.

Monitoring and evaluation of social transfers

Evidence of impact

Gathering evidence on the impacts of social transfers has, until recently, not beengiven adequate attention in the design of DFID-supported programmes. Fewschemes, either projects or national government schemes, have made anyserious attempt to quantify the impacts of social transfers, especially in terms ofpoverty reduction and economic growth. Social transfer schemes have a tendencyto monitor ‘process’ indicators (inputs and activities) rather than ‘impact’ indicators(outputs and attributable changes in beneficiary wellbeing).

Resources are needed to support the most appropriate evaluation methodologiesthat can attribute costs and attribute impacts to specific social transferprogrammes, and can quantify the full range of impacts (positive and negative), onprimary and secondary beneficiaries. Assessment of long-term impact is crucial.

Monitoring and evaluating influence

There are intrinsic difficulties involved in evaluating policy influence, particularlyconcerning the problem of attribution, as outcomes are fuzzy, and both interveningfactors and contemporaneous events have a considerable influence on otherdonors. Monitoring policy influence is relatively easier, but the current emphasison measuring inputs and not outputs, or the linkages from inputs to outcomes, isproblematic.

It should be recognised that the success of policy-influencing will often take fairlyintangible forms that are difficult to quantify. Any broader effort to quantify andmeasure DFID’s policy-influencing activities should be accompanied by case-study analyses that locate such efforts firmly in particular contexts.

Core analytical framework

A number of monitoring and evaluation lessons learned and recommendationsmade in the review are incorporated in a ‘core analytical framework’ to be used infuture DFID social transfer evaluations. This framework includes guidelines forbest practice and a basic set of standard indicators (that all DFID-supported socialtransfer programmes should monitor and report) to enable comparison acrossschemes, countries and regions.

The core analytical framework needs to have a strong focus on evaluating theimpact of DFID-supported social transfer programmes. Without this, progress onthe White Paper 3 commitments is meaningless as it would be impossible todemonstrate impact, and a lack of attention to impact evaluation would seriouslyundermine policy-influencing activities.

11


Recommendations for improving the delivery and effectiveness ofWhite Paper 3 commitments

Our recommendations for DFID are intended to overcome the ‘dependencydebate’, views that decry social protection as a ‘welfarist, handout approach’ – amajor obstacle facing those supporting the increasing investment and expansionof social protection in Africa and Asia. Opportunities to overcome the dependencydebate include an improved and appropriate evidence base that is founded onrealistic considerations of what can be achieved in each country and that addressthe key concerns of those influential decision-makers who remain unconvinced.

Appropriate methodologies should be supported that can allocate costs andattribute impacts to specific social protection programmes, and can quantify thefull range of impacts, positive or negative, on primary and secondary beneficiaries.Assessment of long-term impact is crucial.

A more detailed evidence base of what works, what does not work and why, indifferent contexts, would provide DFID with a range of context-specific evidence-supported options that would constructively nuance DFID’s advocacy and policyadvice.

Lastly, it is crucial to consider how DFID can use the evidence most effectively. Toenable this, DFID should institutionalise political analysis. Given the importance ofpolitics as highlighted in this review, this provides DFID with the most significantopportunities yet to support the increasing investment and expansion of socialprotection in Africa and Asia.


12

1 Bangladesh, Kenya, Nepal, Pakistan and Tanzania.

13


1 Introduction

1.1 Scope, objectives and outline of work

This report summarises some of the main findings of four reports commissionedby DFID in 2007 as part of the review process in preparation for the evaluation ofDFID-supported work on social transfers planned for 2010–2011 (see Devereuxand Coll Black (2007); Hickey and Sabates-Wheeler (2008); Davies (2008); andBarrientos (2007)).

The review examined evidence of DFID’s effectiveness in supporting theimplementation of over 24 social transfer programmes in 16 countries acrossAfrica, Asia and Europe, and gathered, organised and assessed the availableevidence on the impacts of these programmes, including the impact on growthand poverty reduction. Eighteen of the programmes were examined in detail (seeTable 1.1).

The review also assessed DFID’s progress towards influencing policy on socialtransfer in countries where DFID has sought to promote social protection. DFID’sefforts in Ethiopia, Malawi, Uganda and Zambia were examined in detail with asecondary focus on five other countries.1 Finally, the review developed a baselineand framework designed to monitor and evaluate DFID-supported social transferschemes and progress towards White Paper 3 social protection commitments.

DFID commitments to increasing investment in social transfers and socialprotection as outlined in White Paper 3 commitments include:

significantly increasing spending on social protection in at least ten countriesin Africa and Asia;

doubling the number of people moved from emergency relief to long-termsocial protection programmes in Africa;

supporting partnerships between developing countries to share experience ofexpanding social protection.

Social transfers provide direct, regular, and predictable assistance in cash or kindto poor individuals or households, with the aim of reducing deficits in consumptionand, in some cases, strengthening their productive capacity. Social transfers arean important component of social protection. Social protection includes socialassistance, social insurance, and labour market regulation. This report follows therelative focus of the different review reports (either social transfers or socialprotection) and, therefore, considers both social protection and social transfers atdifferent stages.


14

This purpose of this summary document is to examine each of the fourpreparatory review reports and to draw out key lessons and recommendationsfrom them. The objective is to use this analysis to provide a summary of the mainrecommendations, highlighting the main opportunities and challenges DFID facesin achieving the social protection commitments outlined in White Paper 3.

In Section 2, key lessons learned and recommendations relate to:

the implementation, effectiveness and impact of DFID-supported socialprotection programmes;

DFID’s approach to monitoring the implementation of White Paper 3 socialprotection commitments;

the framework evaluating social transfers;

DFID’s approaches to influencing policy change on social transfers and;

cross-cutting issues.

Table 1.1 DFID-supported social transfer programmes reviewed in detail

Country Project/Programme

Bangladesh BRAC Ultra-Poor Programme

Bangladesh Chars Livelihoods Programme

Croatia Structural Adjustment Loan: Social Policy Reform and Poverty Analysis

Ethiopia Productive Safety Nets Programme

India Community-based Drought Response Programme in Orissa

Kenya Support to World Food Programme Kenya School Feeding Programme

Malawi DFID support to the Inputs for Assets Programme of Malawi

Malawi DFID support to the National Safety Nets Inception Phase programme

Malawi Improving Livelihoods through Public Works in Malawi 2002–2004

Malawi Support to Malawi Government to replenish strategic grain reserve

Malawi Support to 2004/05 Special Agricultural Programme

Malawi 2003/04 Targeted Inputs Programme (TIP)

Mozambique Labour-intensive road programme to mitigate the impact of drought

Nepal Expanded Rural Community Infrastructure Works Programme

Zambia CARE Partnership Programme Agreement

Zimbabwe Vulnerable Farm Workers Relief (Humanitarian Aid) Recovery and Empowerment Programme

Zimbabwe Protracted Relief Programme Phase 1

15


Section 3 presents recommendations for DFID to improve the overall delivery andeffectiveness of the DFID White Paper 3 commitments.

1.2 Limits of the report

Findings are taken from and limited to, the four reports from the initial review anda significant amount of the text is lifted from these reviews. Any lessons learneddrawn from the review of the effectiveness in implementation and impact of DFID-supported social transfer schemes are tentative and preliminary. Because of thelimited availability to date of comprehensive reviews and evaluations, thearguments made were based on a very small number of case studies anddiscussion on impact, therefore, is limited.

Very few lessons and recommendations emerge from the review on how effectiveDFID-supported social protection programmes have been in addressing cross-cutting issues such as gender, social exclusion and HIV. In fact, only gender wasexamined in any detail in one of the reports – the review on the effectiveness andimpacts of DFID-supported programmes.

1.3 Methodology

The study is a desk review of the four reports commissioned as part of the DFIDsocial transfers evaluation. Lessons learned and recommendations are drawndirectly from these reports as examples that have implications for future work onsocial transfers and social protection.

The Terms of Reference for this summary required analysis of the ‘potential forDFID to achieve, or improve, the delivery and effectiveness of White Paper 3commitments’. Analysis, however, is not made against the specific targets outlinedin the commitments as they are limited to increasing spending, transitioningpeople from emergency aid, and supporting southern partnerships. Instead, whenexamining outcomes against White Paper 3 commitments, the term is taken, inthe generic sense, to relate to increasing investment and expansion of socialprotection, including increasing the effectiveness of social protectionprogramming.

2 Key lessons learned andrecommendations

2.1 Introduction

Key lessons and recommendations are drawn from the four areas examined in thefull review, namely: effectiveness in implementation and impact of DFID-supportedsocial transfer schemes; DFID’s approaches to influencing policy change on socialtransfers; DFID’s approach to monitoring the implementation of White Paper 3commitments; and development of a framework for evaluating social transfers.


16

The outcomes and impacts of social transfers and social protection schemes varygreatly relative to the unique conditions applying in specific contexts. There is avariety of experience in DFID-supported programmes. What has worked well inone context may work very differently under a different set of conditions in anothercontext. Aggregating specific key lessons and recommendations is, therefore, achallenge and could be misleading. However, a set of generalised findings hasbeen identified but should be considered as part of the knowledge and evidencebase on a range of options for social transfers and social protection programming,to be evaluated further in other specific contexts, rather than as prescriptive policyoptions.

In relation to influencing, although the political context for social protection mustalso be analysed and considered on a case-by-case basis, a fairly clear andcommon set of constraints emerged against which the relative success of DFIDpolicy-influencing efforts can be judged.

2.2 Effectiveness in implementation of DFID-supported socialtransfer schemes

Key lessons and recommendations relating to the effectiveness in implementationof DFID-supported social transfer programmes include: (1) selection of socialtransfers; (2) conditionalities; (3) level, quality and regularity of social transfers;(4) targeting and delivery strategies.

In each of the programmes supported by DFID, the reasons for selecting thesocial protection instrument were well argued and defensible. The design of thetransfer was motivated in terms of the objectives of the intervention, beneficiaryneeds and preferences, country priorities and/or donors’ and partners’ policies.This signifies an encouraging trend away from ‘resource-driven’ social transferschemes (i.e. delivery of food because food aid was the only resource available)towards ‘needs-driven’ schemes where governments and donors select what theybelieve to be the most relevant transfer, taking beneficiary preferencesincreasingly into consideration. The PSNP in Ethiopia, for example, aims toprovide cash transfers where they are more relevant and effective, consideringbeneficiary preferences whenever possible.

In the context of DFID support to national programmes and multi-donorprogrammes, DFID should call for needs assessments and engage with nationalgovernments to ensure programmes respond to needs.

Social transfers can be delivered either unconditionally (as a free grant orsubsidy), or with conditions that beneficiaries must meet before they collect theirbenefits. The most common conditionality applied to DFID-supported socialtransfer programmes is the work requirement on public works, though there isalso limited experience with promoting access to services (see Table 2.1).However, the debate about whether conditionalities should be applied to DFID-supported programmes remains unresolved, especially in Africa, largely due tolimitations in the evidence base.

17


Arguments against public works include: the heavy labour required excludes manylabour-constrained but highly vulnerable poor people; self-targeting is impossiblein areas with high rates of poverty and unemployment; most assets created bypublic works do not benefit the poor and are rarely maintained and so generatefew sustainable benefits. An example of the limitations of DFID-supported publicworks comes from the Nepal ‘Rural Access Programme’ where the labourrequirement raised concerns, not least that it placed unreasonable demands onpoor people, who were often either time-constrained (e.g. widows caring fororphans) or labour-constrained (e.g. people with disabilities). Given that these areamong the most vulnerable groups in most communities, transfers with labourconditionalities should be complemented by unconditional transfers to people whoare unable to work for food or cash, as we see in the case of Ethiopia’s PSNP.

Arguments in favour of public works rather than unconditional transfers include:work is more dignifying than handouts; the work and time commitment requiredmake this instrument self-targeting; public works can construct or rehabilitateessential infrastructure. The Orissa ‘Drought Response Programme’ (DRP)attempted to ensure that public works projects undertaken were useful andrelevant to participating communities through a strong emphasis on communityownership and empowerment.

In relation to public services, conditionalities are more commonly applied in LatinAmerica where services are generally more accessible and of better quality thanin Africa and Asia. However, the idea of linking social transfers to public servicesin Africa is receiving increasing policy consideration, especially in contexts whereDFID is also supporting simultaneous investments in public services. Noconditionalities were initially applied to a DFID-supported pilot cash transferprogramme in Zambia, but in one pilot district a set of conditionalities relating toaccess to healthcare is being considered (e.g. attendance of under-fives at localhealth clinics). In another district, cash incentives relating to school attendanceare being considered.

Instead of taking a blanket decision for or against conditional transfers, this designchoice must be based on an assessment of what seems feasible and appropriatefor the particular context into which the intervention is being introduced.Conditional cash transfers should be piloted in Africa, preferably linked tomatching investments in the quality of public services (education or health) towhich the conditions for receiving cash grants would be attached. The impacts onboth beneficiary wellbeing and on the public services involved should berigorously evaluated.

Table 2.1 Conditionalities applied in programmes reviewed

None 13Participation in public works 7School attendance 4Health clinic attendance by children under 5 2Participation in income-generating activities 1Attendance at awareness sessions 1

Source: Devereux and Coll Black (2007)


18

The effectiveness of social transfers is largely dependent on their level andregularity. Social transfer programmes often aim to achieve more than merelyincrease or smooth food consumption, but these ambitions are undermined whenpayments are too low or when disbursements are made irregularly rather thanpredictably.

Cash transfer levels are often related to the local wage rate or the cost of asubsistence food basket. However, if the transfers are not adjusted for householdsize, over time and across space, the real value of the cash can be highlyvariable, leaving beneficiaries unable to meet their subsistence needs if transferlevels are set too low or their real value declines. In the PSNP, rapid priceincreases devalued cash transfers and have been cited as one of the factorscompromising the ambition of the PSNP to ‘graduate’ beneficiaries out of chronicfood insecurity. Table 2.2 summarises the transfers delivered in variousprogrammes for which data are available.

Table 2.2 Levels of transfers in programmes reviewed

Cash transfer Stipend = US$3.70/month x 18 months = US$66.50 [Bangladesh]

1–2 OVC = US$14.30/month, 5+ OVC = US$43 [Kenya]

US$6/month for households with no children, US$8 with children [Zambia]

Cash-for-work US$1.30/day [Bangladesh]

US$0.7/day [Ethiopia]

Wage rate set just below the rural wage rate [Malawi]

Wage set at minimum wage = US$50/month [Mozambique]

Cash-for-work Average wage rate = US$1.30/day

Average income = US$200/person per construction season [Nepal]

Food transfer General feeding: 50kg mealie meal, 10kg beans and 1.875 litres of cooking oil

Child supplementary feeding: fortified nutrient dense Corn Soya Blend (CSB) porridge either as a wet feed or as a dry ration; 200kg CSB take-home ration

School feeding: mid-morning drink, lunch of maize and beans [Kenya]

Food-for-work Ration of rice, vegetable oil, pulse [Ethiopia]

4kg rice per person per day, total 280kg rice per person [Nepal]

Asset transfer Valued at US$92 or US$200 [Bangladesh]

Input transfer Vouchers for fertiliser and seed

25kg of fertiliser, 5kg of maize seed, 1kg of legumes

0.1 hectare pack: 10kg fertiliser, 2kg of maize seed, 1kg legume

0.5 hectare pack: Basal fertiliser 50kg; top dressing fertiliser 50kg; maize seed 10kg; cowpea seed 1kg


2 Including: Bangladesh ‘Chars Livelihood Programme’; India ‘Community-based Drought ResponseProgramme’; Nepal ‘Rural Access Programme’; and Zimbabwe ‘Ex-farm Workers Programme’.

19


Amounts transferred on long-term social transfer programmes should be regularlyadjusted for changes in prices, which also implies that relevant prices in localmarkets must be routinely monitored.

Targeting is one of the most difficult problems facing the designers andimplementers of all social protection interventions. In assessing the accuracy andeffectiveness of the targeting strategies of DFID-supported programmes, problemswere identified in all approaches and targeting methodologies adopted in theprogrammes reviewed. Administrative targeting criteria tend to emphasisedemographic vulnerability (‘vulnerable groups’ such as female-headedhouseholds, or people with disabilities) and economic poverty (e.g. landlessness),but these are susceptible to significant exclusion errors (eligibility criteria are oftentoo narrow) and inclusion errors (many proxy indicators are not robust). TheDFID-supported CARE Partnership Programme in Zambia is a case in point.Administrative targeting rules were used to select the poorest 10 per cent ofhouseholds with no productive capacity. The 2006 annual review reports thatcommunities argued that the 10 per cent cut-off point for eligibility excludes manydestitute or incapacitated households who should qualify for benefits.

Community-based targeting avoids both inclusion and exclusion errors (at least intheory), by drawing on local knowledge of individual circumstances. Communityinvolvement, therefore, can enhance community acceptance of targeting decisions(see Box 2.1) although there are risks of ‘elite capture’ and exclusion of sociallymarginalised groups.

Whatever targeting rules and procedures are applied, targeting must beimplemented carefully and it should be monitored. More comparative assessmentsacross programmes are needed to understand both the accuracy and socialconsequences of alternative approaches to beneficiary selection. At the sametime, however, the principle of ‘acceptable error’ needs to be considered. Forexample, building political support for a programme might require some ‘leakage’to non-poor beneficiaries, and a relatively crude set of proxy indicators might bemore cost-effective than rigorous means testing.

In the social protection programmes examined in the review2 that attempt toinclude vulnerable women, gender targeting of social transfers is applied with noanalysis of how context-specific gender relations within communities andhouseholds may influence the distribution of benefits. A worst-case scenario isthat delivering benefits directly to women could lead to intra-household disputes

Box 2.1 Community-based targeting in the Bangladesh ‘CHARSLivelihoods Programme’

Implementation Committees set up in each village are responsible for identifyingparticipants, supervising public works and distributing payments. In the first year ofimplementation, a performance review reported that 80 per cent of asset transferswere accurately targeted and that the selection process was transparent and wellaccepted among programme communities.

3 The CARE Malawi ‘Improving Livelihoods through Public Works’ programme set a quota of 30 per centwomen among beneficiaries, as did the rural access programme in Nepal. In Ethiopia, the PSNPapplied informal quotas.


20

and increase gender-based violence against women. Similarly, there is evidenceto suggest that targeting methods may be influenced by local gender norms.Community-based targeting, as employed by social transfer schemes in Malawi,has resulted in the exclusion of socially marginalised women.

In addition to targeting women as a vulnerable group, some programmes employa quota system to ensure that women benefit (sometimes equally) from socialtransfers.3 Quotas may not be enough, however, to ensure that men and womenbenefit equally from opportunities for employment or other types of social transfer.In addition to gender discrimination in employment practices, women’sresponsibilities for reproductive labour may limit the time they can devote to publicworks, while gender norms in some areas limit women’s ability to travel longdistances, thereby restricting their access to various types of social transfer.

An alternative route to supporting women’s (equal) participation in social transferprogrammes is to engage them in public fora and to involve them in the largerdecision-making structures related to social protection programmes. Thisapproach aims to empower women and to influence social relations between menand women to encourage improved gender equality. In Ethiopia, Food SecurityTask Forces at every level are required to include at least one woman. However,the continued absence of a directive on this issue meant there were still fewfemale members at the district and subdistrict levels, and exceptionally fewwomen in leadership positions, by 2006.

2.3 Impact of DFID-supported social transfer schemes

Some observable lessons and recommendations were identified in the reviewfeaturing four ‘impact’ areas: food security, economic growth, poverty reductionand health.

2.3.1 Food security, economic growth and poverty reduction

Most social transfer interventions supported by DFID – unconditional cash, food orasset transfers, public works, school feeding schemes, agricultural inputspackages – contribute to the objective of enhanced household food security in theshort term, and to reductions in the severity of poverty, though not necessarily tosustainable poverty reduction, in terms of falling poverty headcounts in the longerterm.

Food security

Social transfers almost inevitably have significantly positive impacts on householdfood security, by increasing food availability or access to food. Where socialtransfers are provided in the form of agricultural inputs, the impact on foodsecurity is through production rather than consumption. One notable successstory is the ‘Targeted Inputs Programme’ in Malawi. Finally, well-designed public

21


works projects can improve food security through creating infrastructure thatsupports agricultural production (e.g. irrigation facilities) or market integration (e.g.feeder roads that link isolated communities to trade routes and market centres).Evidence for these impacts is limited, because of methodological difficulties inquantifying and attributing the economic benefits of infrastructure. However, anevaluation of public works in Nepal, which attributed increases in crop yields andmarket access to the construction of irrigation facilities and roads, shows thatmeasuring the impacts of infrastructure is possible.

Economic growth

The potential of social protection to contribute to broader developmentalobjectives of poverty reduction and economic growth can occur through either(1) multiplier effects or (2) market integration. Only six of the DFID-supportedcase-study social transfer schemes reported any impact – positive, neutral, oreven negative – on economic growth (see Table 2.4) and any detailed quantitativedata were not available. It should be noted, however, that the primary purpose ofsocial transfers is to protect rather than promote livelihoods, so a failure to findevidence of economic growth impacts is not necessarily an indictment of theprogrammes reviewed.

Table 2.3 Reported food security impacts of programmes reviewed

Cash transfers Enabled asset accumulation

Protected household assets against distress sales

Public works income was spent on food and invested in agricultural inputs

Public works cash was used to purchase animals and other assets, to diversify household incomes and reduce vulnerability to drought

Cash transfers reduced pressure on community-level support

Reduced pressure to migrate and take loans for food

Food transfers Increased household food consumption

Allowed more retention of own food production

Food-for-work rations bridged the annual ‘hunger gap’

Input transfers Increased food crop production

Free input packages added 2.5 months to household food supplies

Free input packages added 200,000 tonnes of maize to the national harvest and 15 per cent or 2 months of household annual food requirements

Agricultural inputs had little impact on production because of drought

Positive impacts on food security were only short-term, not sustainable

Asset transfers Assets contributed to community-level food security

Household food production deficits fell by 18 per cent in project districts

Asset creation Construction of roads on public works reduced food prices by 25 per cent



22

Poverty reduction

Social transfer schemes can contribute to poverty reduction directly, throughincome transfers, or indirectly, by allowing beneficiaries to generate income andaccumulate assets, or through human capital formation. Social transfers also havethe important objective of assisting poor families to avoid falling deeper intopoverty and becoming destitute. Several DFID-supported schemes reportevidence of poverty reduction, or ‘poverty prevention’, through one or more ofthese routes (see Table 2.5).

Income transfer

The size of social transfers is rarely sufficient to lift beneficiaries above thepoverty line, so the ‘poverty headcount’ is not reduced, but in most cases the‘poverty gap’ is reduced – beneficiaries are less poor than before. In some cases,as in Bangladesh, a kind of graduation is achieved, with beneficiaries reclassifiedfrom ‘ultrapoor’ to ‘moderately poor’. On Ethiopia’s PSNP, ‘direct support’ tolabour-constrained households constituted a sizeable proportion of householdincome – in several cases it was the only recorded source of income.

Income generation

Many social protection interventions transfer not just income, but also the ability togenerate future income – they aim at ‘livelihood promotion’ as well as ‘livelihoodprotection’. There is persuasive evidence that BRAC’s ‘Ultra-Poor Programme’contributed to a reduction in extreme poverty among poor Bangladeshi women.

Table 2.4 Reported economic growth impacts of programmes reviewed

Cash transfers Cash injection from public works stimulated effective demand in the local economy

Shift from food aid to cash transfers removed distortions in food prices

Cash transfers stimulated food production

Traders experienced improved business following the introduction of cash transfers

Cash transfers were too small to affect commercial demand for agricultural inputs

Prices of food and other goods rose following the introduction of cash transfers

Food transfers Food aid encouraged diversification into high-value crops for sale

Food-for-work projects had no impact on local labour markets and wage rates

Input transfers Suppliers benefited from increased demand for fertiliser thanks to input vouchers

Asset creation Infrastructure and assets created under public works were of low quality

Poor quality of public works roads meant that access to markets was not improved and there was little increase in informal trade


23


Table 2.5 Poverty reduction impacts of case-study programmes

Income effect Support for ultra-poor women resulted in significantly improved earnings (40–56 per cent)

Cash transfers were a large proportion of income for poor, elderly or sick beneficiaries

Beneficiaries enjoyed increased capacity to generate income

Public works participation enhanced beneficiaries’ socioeconomic status

Livelihood impacts of public works were undermined by low wage rate and delayed payment

Cash transfers were not enough to cover basic consumption needs

Despite higher incomes, beneficiaries remained poor and vulnerable tomajor shocks

Asset effect Over 95 per cent of 3,274 households that received asset transfers retained these assets

Cash transfers stabilised household asset holdings

25 per cent of beneficiaries acquired new assets

Cash transfer beneficiaries invested in small livestock

Cash transfers protected and, to a lesser extent, built household assets

Production effect Irrigation under public works raised crop yields by 55 per cent

Cropping intensity under public works irrigation doubled from 1 to 2 crops/year

Savings were invested in animal husbandry

Employment effect Employment rates improved: unemployment fell from 13.2 per cent to 9.2 per cent

A slight increase in the operation of non-farm businesses was recorded

Levels of economic activity increased among both men and women

Skills effect Human capital was enhanced by learning construction skills, and literacy training

86 per cent of new skills acquisition by beneficiaries was attributed to public works

Savings effect Increased savings and incomes among poor beneficiaries

Voluntary savings mobilisation increased

Social services Cash transfers enhanced beneficiary access to education and health

General wellbeing Cash transfer beneficiaries avoided loans for food, casual labour, premature harvesting

Living conditions improved in comparison with the baseline and nonpilot districts

Sustainability of recorded improvements in living conditions was questioned


24

Similarly, the ‘Vulnerable Farm Workers Programme’ in Zimbabwe has ‘increasedcapacity to generate income’. In these and other cases, the capacity to generateincome was achieved through the transfer of skills rather than cash or food.

Asset accumulation

A robust indicator of poverty reduction impact is changes in household assetvalues; in fact, this has been proposed as an indicator of ‘graduation’ fromEthiopia’s PSNP. Investment in assets is one of the most popular uses of cashtransfers after consumption needs have been met.

In rural areas, livestock are a store of wealth as well as a productive asset.Participants in Malawi’s ‘Public Works Programme’ and Nepal’s ‘Rural AccessProgramme’ invested some of their public works income in purchasing smallanimals, as did beneficiaries of unconditional cash transfers on the ‘CARE ZambiaPartnership Programme’. Whether this amounts to sustainable poverty reductiondepends on whether the assets are retained and generate future income but, atthe very least, this accumulation of assets builds resilience against futurelivelihood shocks.

Poverty prevention

A robust indicator of ‘anti-poverty’ impacts is whether beneficiaries avoidedadopting damaging ‘coping strategies’, which can best be assessed by comparingbeneficiary behaviour under stress with a non-beneficiary control group, but suchevaluations are rare.

If social protection is to be exploited seriously as an instrument for povertyreduction, and not just as an instrument for social welfare, then changes might beneeded in the design and conceptualisation of many DFID-supported socialprotection interventions that are currently underperforming in terms of thesebroader ambitions.

Social transfers should be raised above basic ‘subsistence’ levels, such that thosebeneficiaries who are able to take advantage of income-generating opportunitiesdo invest in assets and enterprises. Although evidence of this from theprogrammes reviewed is limited, the example of the Ultra-Poor Programme inBangladesh does provide an example of how ‘adequate’ transfers can promotelivelihoods when income-generating opportunities exist. Although this approachmight be more costly, it could be more cost-effective in the long-term if there is anexit strategy linked to graduation.

Similarly, governments and donors need to invest adequately in supporting thelivelihoods of poor people, which implies identifying innovative complementaryinterventions to social transfers, or building on positively evaluated experiencessuch as BRAC’s ‘Asset Transfer Programme’ to the ‘ultra-poor’ in Bangladesh.

Health

Social transfers can enhance the health status of programme beneficiariesthrough three pathways (the same as for education): (1) directly – by linking thedelivery of transfers to health services; (2) indirectly – if beneficiaries allocate

25


some transfer income to purchase healthcare; (3) incrementally – by deliveringcomplementary health interventions to the same individuals who receive socialtransfers. All three were recorded in some projects reviewed, occasionally withimpressive results in terms of health outcomes. Some of the strongest evidencecomes from the ‘Ultra-Poor Programme’ in Bangladesh which ‘directly’ linkstransfers to health services.

18.6 per cent increase in the use of modern contraceptive methods;

installation of 24,690 slab latrines;

45 per cent increase in immunisation coverage (from a baseline coverage of53 per cent in 2002) to over 98 per cent by 2006;

95 per cent of under-fives in the programme area now receive Vitamin A capsules;

96 per cent of pregnant women now receive antenatal care and 93 per centpostnatal care.

However, the most common route in the programmes reviewed is the ‘indirect’route, since very few projects include health conditionalities or linkages to healthservices (see Box 2.2). Indirect health impacts are, however, highly variable,unpredictable and difficult to measure, as these reflect beneficiary spendingchoices rather than programme design to achieve predetermined health objectives.

Social transfer schemes that promote education and health should intervene notonly on the demand side (by providing incentives or reducing costs), but also onthe supply side, to maximise improvements in both service uptake and outcomes.

2.4 DFID and the politics of influencing

2.4.1 Key lessons

Successful activities and approaches

DFID’s most effective means of influencing government and donor approaches tosocial protection is its ability to facilitate the access of policy actors to the growingevidence base in this field. This has been achieved through the provision of experttechnical assistance, the studies that DFID commissions, the study tours that itfacilitates and also through the knowledge that its advisors possess of socialprotection. This capacity has allowed DFID to directly influence debate over socialprotection in ways that have increased the number of beneficiaries who benefitfrom social protection schemes (e.g. Ethiopia) and persuaded government officialsto adopt social transfer pilot projects (e.g. Zambia).

Where DFID has failed to produce and disseminate high-quality information andknowledge in sequence with its facilitating and advocacy strategies within national

Box 2.2 Indirect health impacts in the PSNP

Although improved access to healthcare was not a direct objective, almost 50 percent of PSNP beneficiaries reported using health facilities more in 2005/6 than in theprevious year.


26

policy processes it has been less influential (e.g. the early stage of the process inUganda). In particular, the framing of social protection as ‘an African successstory’, particularly via study tours, has been an important factor in persuadingsome government officials to increasingly adopt the social protection agenda.

DFID should continue

Acting as a knowledge interface

DFID has achieved its most significant levels of influence through acting as aninterface between key stakeholders and the wider evidence base on socialprotection, some of which it has been actively involved in producing.

Taking a leadership role

DFID’s willingness to take a lead role in policy spaces, dedicated to socialprotection, nationally, internationally and in multi-donor processes, has beeneffective in securing higher levels of influence.

Providing strong support from the centre to country advisors

DFID advisors have been powerfully enabled by the level of importance that DFIDHQ and DFID country offices have placed on the social protection agenda, andthe scope that they have given to its advisors to allocate financial and humanresources to these activities on a flexible and responsive basis. Such practicesmust be supported and further embedded and extended.

Donor harmonisation and influence vs. government ownership?

DFID’s successes in creating policy spaces to work towards donor harmonisationhas helped policy debates with government officials to move faster than theymight otherwise have done and presented a common front that seems to havehelped DFID and its partners to gain greater influence over the government.

DFID should, however, clarify its position on ownership or influence andreconsider the contradictions between two of its primary objectives – namely itsefforts to promote its own vision of social protection on the one hand, and itsefforts to secure government ownership of social protection policies and strategieson the other. Each objective may well be different and have possible contradictoryapproaches; for example, increased donor harmony may be good for donorinfluence but can also limit government ownership.

The pilot project-based approach

In some cases, DFID has used evidence from social transfer pilot projects (eithergovernment or NGO-run) as a basis for arguments to either scale up socialtransfer programmes and/or to promote broader social protection policies. Thispilot project-based approach has had mixed success to date. On the plus side,this has led to the implementation of such projects and (sometimes) an improvedcapacity of government institutions to implement them. Pilot projects, for examplein Ethiopia, have provided DFID, and other proponents, something tangible

27


around which to base their broader strategies of promoting social protection andproving their commitment via their expenditure. Where no such projects exist (e.g.Malawi, Uganda), DFID has struggled to build its influence. On the down side,with projects implemented by NGOs (e.g. Zambia), this does little to improvegovernment capacity in this field or increase governmental ownership of a socialprotection agenda.

Although DFID are committed to the Development Assistance Committee (DAC)principles for aid effectiveness, DFID could usefully re-examine its pilot project-based approach, with a particular focus on the following questions and issues:

broad strategies to influence social protection policies within countries shouldnot rest entirely on the promotion of pilot cash transfer projects – at the veryleast, a Plan B is required;

for example, DFID could usefully expend more energy exploring alternatives tosocial protection, including efforts to mainstream social protection interventionswithin higher-capacity sectors such as health and education;

pilots are indispensable as a means to learn about design and delivery, butthey are not useful or informative when they are used primarily as anadvocacy tool.

The DFID factor

The views of key informants in the review suggest that DFID’s influence on socialprotection is shaped by its: high levels of personal commitment and professionalcompetence shown by DFID’s advisors in their promotion of the social protectionagenda; status as a leading bilateral donor; willingness to disburse funds throughdirect budgetary support in support of its favoured policy agendas; relatively longhistory of engagement within each country; and flexible, generous and responsivefinancial management systems that have been critical in enabling DFID to developa degree of comparative advantage in this field.

Build influential relationships

Recognising that politics matters, the ability of DFID to use any increasedevidence on cost-effectiveness and wider impacts to extend social protectionprogramming relies on the creation of influential partnerships.

The official mandate for social protection policies in many countries is often mostclearly held by social welfare-type ministries, and it is with such institutions thatDFID has often sought to forge close partnerships in promoting social protection.The ability of DFID advisors to cultivate relationships of trust with key governmentofficials in social welfare ministries has helped to secure higher levels ofcommitment from such actors and to ensure that social protection debates andpolicy processes have moved forward faster than they might otherwise have done(e.g. Zambia, Uganda). However, such ministries have low levels of funding,capacity and political voice, and have therefore been largely unable to take astrong lead on social protection, thus seriously impeding DFID’s efforts toinfluence national policy, particularly in terms of national-level interventionsbeyond the protection pilot phase.


28

DFID has been less successful in building productive relationships with moreinfluential political and policy actors, including parliamentarians, high-level politicalleaders, and those within the more powerful ministries of finance and planning. Itis this latter group, which can be thought of as part of a broader ‘finance ministrytendency’ that currently constitutes the primary source of resistance to DFID’sefforts in this field.

Build stronger relations with more powerful policy actors

Greater efforts are required in developing close working relations with policyactors operating within the finance ministry tendency on social protection issues.Bridging the gap between different policy actors and sectors should also beconsidered in terms of where social protection programmes are institutionalisedand delivered from. It might be possible to locate programme delivery units withinministries of finance (e.g. see the pension scheme in Lesotho) and to runprogrammes within broader systems of social service delivery that have greaterpolitical support and institutional capacity, particularly health and education.

Develop a broader national constituency

Opportunities to build influential relationships extend beyond ‘formal’ routes. Theevidence presented in the review of influencing suggests that policy-influencingactivities need to be routed through the full range of policy channels in order to beeffective, from formal through semi-formal to more informal channels. In mostcountries, DFID has proved more capable of generating influence through formaland (in particular) semi-formal channels of influence, but has achieved fewergains through more informal channels. DFID’s incipient efforts to persuade civilsociety organisations and parliamentarians in partner countries to play a strongerrole as advocates for social protection could be usefully pushed further, althoughon occasions, indirectly via other agencies.

2.5 Monitoring and evaluating social transfers

Lessons and recommendations for the monitoring and evaluation of socialtransfers are drawn from all four studies of the full review and include therecommendation to adopt a core analytical framework incorporating monitoringand evaluation best practice and a basic set of indicators.

2.5.1 Existing monitoring and evaluation

Many DFID-supported social transfer interventions reviewed were pilots designedto generate lessons on design and impacts. Given this focus on learning lessonsand the limited evidence base to date on the impacts of innovative socialprotection measures, there is a demand for rigorous and robust monitoring andevaluation (M&E) systems. Unfortunately, however, in the DFID programmesreviewed, M&E is often inadequate in fundamental aspects. Even whereinformation is available on, for example, ‘cost-effectiveness’ or ‘poverty reduction’,this does not necessarily mean that the specific indicators were calculated, or canbe derived from project documents. More typically, qualitative conclusions of thiskind are drawn.

4 Programmes developed in 2006, especially Kenya, Pakistan and Ethiopia, gave more consideration tomonitoring and evaluation.

29


2.5.2 Impact

Given the big claims made on behalf of cash transfer and asset transferprogrammes, it is striking how few schemes, either projects or nationalgovernment schemes, have made any serious attempt to quantify the impacts ofsocial transfers, especially in terms of poverty reduction and economic growth.Gathering evidence on the impacts of social transfers has, until recently, not beengiven adequate attention in the design of DFID-supported programmes.4 Overallhowever, in the programmes reviewed, social transfer schemes have a tendencyto monitor ‘process’ indicators (inputs and activities) rather than ‘impact’ indicators(outputs and attributable changes in beneficiary wellbeing). The following keylessons emerge from the review.

Evidence of impact on poverty reduction is stronger on social transferschemes that have evaluation strategies, such as Ethiopia and Bangladesh,and much weaker or non-existent for social transfer schemes without them.Among social transfer schemes with weaker evaluation strategies, estimatesof poverty reduction effects are usually based on estimates of programmeincidence and, in some cases, a basic reflexive comparison of thesocioeconomic status of beneficiary households that cannot provide robustestimates of impact.

In several programmes, log frames refer to poverty reduction as the overridingobjective but provide little guidance on how social transfers will generatepoverty reduction, the size of the expected effects, and their timing.

The effectiveness of social transfers could be improved by measuring theeffects of social transfers on human development. In addition, stronger impactevaluation exercises could help identify factors linking the programme tohuman development outcomes and the existing constraints and limitations.

Upgrade systems

Monitoring and evaluation systems need to be upgraded and adequatelyresourced for both ongoing and future DFID-supported social transferprogrammes. The use and quality of the data both need to be improved.

Resource-appropriate M&E methodology

Resources are needed to support the most appropriate evaluation methodologiesthat can attribute costs and attribute impacts to specific social transferprogrammes, and can quantify the full range of impacts (positive and negative) onprimary and secondary beneficiaries. Monitoring and evaluation systems, therefore,need to use a broader range of indicators than are currently being used in theprogrammes reviewed. Inclusion of indicators to assess long-term impact is crucial.


30

Focus evaluations

In the impact evaluation of social transfer schemes there is very little room for‘trade-offs’ in terms of time or resources. It will be more effective to focus on a fullimpact evaluation for a smaller number of social transfer schemes and seek tovalidate these findings across countries with similar schemes and conditions, thanto attempt scaled-down evaluations across the board.

2.5.3 Cost-effectiveness

Data on programme costs per beneficiary, and on cost-effectiveness moregenerally, were only obtainable for five programmes reviewed (see Table 2.6).Social protection programmes are often advocated on the basis of their cost-effectiveness as compared with alternative delivery methods. The cost-effectiveness of social protection programmes is, however, extremely challengingto determine, partly because full costs are difficult to obtain and partly becauseimpacts (effectiveness) are difficult to attribute and to quantify.

Assessing cost-effectiveness requires that both the tangible and intangiblebenefits in both the short and longer term be calculated. This approach may beparticularly constrained by the limited availability of quantifiable information onprogramme impacts. However, if future evaluations providing estimates of the fullrange of benefits – and negative impacts – are delivered under all components ofthe intervention (as recommended in this review), a comprehensive cost-benefitanalysis, and a more informed comparison across alternative social protectionmechanisms, would then be possible.

2.5.4 Monitoring and evaluation of social protection policy-influencing

Relatively little is known about which activities and strategies are the most effectivefor influencing policy, and still less about the approaches that can evaluate these.There are intrinsic difficulties involved in evaluating policy influence, particularly

Table 2.6 Summary information on costs of social transfers

Programme Cost per beneficiary

BRAC ‘Targeting the Ultra-Poor’ US$287 (total cost, including value of asset transferred plus monthly stipend to beneficiaries for 18 months)

Ethiopia PSNP US $35 (annual cost)

Kenya School Feeding US $7.44 per annum in arid and semi-arid districts;

US $8.33 in urban informal settlements

Malawi 2003/04 TIP US $7 per household (total cost)

Zimbabwe ‘ex-farm workers’ (the food aid US $6.20 per person per month (based on one component of the programme) child equivalent to half an adult)


31


concerning the problem of attribution, as outcomes are fuzzy, and both interveningfactors and contemporaneous events have a considerable influence on otherdonors. To try and narrow down the focus to one particular influence is furthercomplicated here because of the extent to which the promotion of social protectionis pursued by a multiplicity of actors often acting in collaboration with each other.Disentangling what fraction of the growing consensus around social transfers and asocial protection agenda could be directly attributed to DFID’s policy influence is,therefore, no easy matter. Monitoring policy-influencing is relatively easier, but thecurrent emphasis on measuring inputs and not outputs or the linkages from inputsto outcomes is problematic.

Adopt a ‘light-touch’ to evaluating policy influence

There is a clear willingness amongst DFID advisors working on social protectionto be held to account for their policy-influencing work in this area and to learn fromevaluations of their (and their colleagues’) work. However, mechanisms used forthis purpose should be devised and applied via a ‘light-touch’ approach (i.e. theydo not have to be followed too closely). Stringent systems may prevent staff fromtaking risks or from positioning themselves as part of a wider venture that (as withmost advocacy work) inevitably required partnerships with other actors.

Consider adopting the Outcome Mapping approach

The future M&E of DFID’s policy-influencing work could usefully draw on theincreasingly popular Outcome Mapping approach (OM), as promoted by Canada’sInternational Development Resource Centre (IDRC). Here, monitoring is attachedto outcomes rather than inputs.

Keep the (political) context firmly in view

It should be recognised that the success of policy-influencing will often take fairlyintangible forms that are difficult to quantify. If more quantifiable indicators weredeveloped for the purposes of accountability, policy-influencing activities couldbecome skewed towards the achievement of these rather than potentially moreimportant but less tangible outcomes. Any broader effort to quantify and measureDFID’s policy-influencing activities should be accompanied by case-studyanalyses that locate such efforts firmly in particular contexts.

2.5.6 Monitoring the social protection commitments in White Paper 3

Progress towards White Paper 3 commitments has not been monitored due to theabsence of formal reporting and to difficulties in measuring the targets. As aresult, opportunities to chart the increasing recognition and implementation ofsocial protection within national government programmes have been missed,limiting the scope to build lessons from experience for the future.

The second White Paper commitment is, perhaps, the most difficult to measure asthe concept of measuring numbers of people moving from emergency relief tolong-term social protection is problematic. Recipients do not move from oneprogramme to another and a number of external factors may affect the indicator,e.g. years of increased humanitarian need. Given these difficulties, more


32

comparative indicators, principally measuring the expenditure on social transfersin relation to relief, are required, enabling progress to be tracked.

Clearer reporting arrangements across DFID

In reporting against the White Paper commitments; the new DFID CorporateDivisional Performance Frameworks and preparation for the 2010 evaluation,more formal reporting arrangements, outlining roles and responsibilities forcollecting information and reporting against specific commitments with agreedevaluation foci, are required to help ensure information is comprehensive, clearand regular. To ensure each of the three White Paper commitments areunderstood, consistent and measurable, some terms used in them need to bedefined and where necessary quantified with specific targets.

2.5.7 Supporting future evaluations – best practice and a core analyticalframework

A number of M&E lessons learned and recommendations made in the review areincorporated in a ‘core analytical framework’ (see Annex 1) to be used in futureDFID social transfer evaluations. This framework includes guidelines for bestpractice and a basic set of standard indicators (that all DFID-supported socialtransfer programmes should monitor and report) to enable comparison acrossschemes, countries and regions. The objectives of this analytical framework are to:

facilitate measuring progress on the commitments set out in the White Paper;

help demonstrate the impact of DFID assistance to development countriesrelating to social transfers;

support lesson-learning among DFID staff and partners engaged in theestablishment and development of social transfers, as regards the povertyreduction and policy-influencing of their work on social transfers.

The core analytical framework needs to have a strong focus on evaluating theimpact of DFID-supported social transfer programmes. Without this, progress onthe White Paper 3 commitments is meaningless as it would be impossible todemonstrate impact and a lack of attention to impact evaluation would seriouslyundermine policy-influencing activities.

3 Recommendations for improvingthe delivery and effectiveness ofWhite Paper 3 commitments

Our recommendations for DFID are intended to overcome the ‘dependencydebate’ – a major obstacle facing those supporting the increasing investment andexpansion of social protection in Africa and Asia. Opportunities to overcome thedependency debate include an improved and appropriate evidence base foundedon realistic considerations of what can be achieved in each country. This should

33


address the key concerns of those influential decision-makers who remainunconvinced.

In the case-study countries examined for the review of DFID influencing,5

protagonists supporting the implementation of social protection are often faced withviews that decry social protection as a ‘welfarist, handout approach’ that createsdependency on the state and is an expensive, unproductive use of funds. Suchviews anchor the significant degree of resistance exhibited by several ministries offinance towards the promotion of social protection. Efforts to persuadegovernments that social protection does not create dependency remain largelyunconvincing due to an inadequate evidence base surrounding their impact.

Over the time period that DFID has been seeking to promote social protection theevidence base on this policy approach has grown considerably. Whereas in 2002there was relatively little evidence of positive impact to report, there is now awider range of apparent ‘success’ stories that DFID have been able to cite inadvocating social protection. There is a need, however, to improve this evidencebase in two important respects, as follows.

Firstly, there is limited evidence examining social protection and the emergingissues around its wider impacts, i.e. economic growth and influencing. In thecontext of this review, we found that monitoring and evaluation systems need tobe upgraded and adequately resourced for both ongoing and future socialprotection programmes, if these important factors are going to be captured.Appropriate methodologies should be supported that can allocate costs andattribute impacts to specific social protection programmes, and can quantify thefull range of impacts, positive or negative, on primary and secondary beneficiaries.Assessment of long-term impact is crucial.

Secondly, a more detailed evidence base of what works, what does not work, andwhy, in different contexts, would provide DFID with a range of context-specificevidence-supported options that would constructively nuance DFID’s advocacyand policy advice.

Lastly, it is crucial to consider how DFID can use the evidence most effectively.DFID needs to target the influential evidence at those actors and institutions mostable to support change. To enable this, DFID should institutionalise politicalanalysis. Some of the most serious problems with DFID’s strategic approachreflect a failure to engage productively and sensitively with established politicaldiscourses on poverty, with policy processes, and with policy actors capable ofsecuring the success or failure of the social protection agenda. It is possible that aprior and in-depth effort to analyse the politics of social protection might havehelped here. DFID is well positioned to adopt such perspectives, having trialleduseful approaches such as the Drivers of Change studies.

5 Case studies include Ethiopia, Malawi, Uganda and Zambia.


34

Annex 1 Core analytical frameworkand indicators

Core analytical framework for the future evaluation of pilot and national social transferschemes

Objectives Monitoring Evaluation

Evaluation of the impact of social Monitoring processes capable of Impact evaluation: transfer schemes on poverty enabling assessment of progress ex-ante impact simulationsreduction, productive capacity, in implementation, especially: ex-post quasi-experimental and growth (i) programme incidence evaluation supporting difference in

(ii) financial disbursements difference estimates of impact(iii) cost-effectiveness not feasible or cost-effective

everywhere, but validation where deemed not feasible

Pilots to assess scaling up

Evaluation of policy-influencing on social transfers

Generic Reporting on activities and Use assessed strength of linkagesoutcomes existing between inputs andReporting on strength of linkages outcomes to identify and measurefrom inputs to outcomes attribution from influencing

activitiesStrength of linkages is assessed in terms of:(i) collaboration, partnership, and engagement (ii) financial support for joint initiatives

Objective-specific

(i) Engaging with international As above, and especially: As abovepartners to gain greater Identify changes in commitment commitment to the use of social among key international transfers in the poorest countries partners on social transfersincluding the re-focusing of social protection work of key international partners such as the World Bank and the ILO

(ii) Engaging with national As above, and especially: As abovegovernments and other donors to engagement with national support social transfers and the governments Cross-country meta-studies to development of national strategic donor coordination at country identify key lessons and validate frameworks for social protection level attribution

whether DFID-supported social transfer schemes are in place

whether social transfers are embedded in social protection strategies

35


Minimum set of indicators for the evaluation of DFID-supported work on social transferschemes

Indicators for the monitoring of social transfers

Programme incidence Percentage of households participating in the programme as a proportion of the estimated target population

Cost-effectiveness Percentage of programme budget transferred to beneficiaries

Cost of transferring US$1 to programme beneficiaries

Functional distribution of non-transfer programme expenditures

Indicators for the impact of social transfers

Objectives Indicators

Food security Percentage of food-insecure households in programme who become food-secure (adjusted for change in ‘control’ group)

Poverty reduction Percentage reduction in poverty gap among beneficiaries (adjusted for change in ‘control’ group)

Percentage reduction in poverty headcount among beneficiaries (adjusted for change in ‘control’ group)

Human development Percentage increase in school enrolments among beneficiaries (adjusted for change in ‘control’ group)

Percentage of beneficiary children of school age who attend school regularly (usually defined as 80 per cent of term)

Percentage improvement in health status among beneficiaries (adjusted for change in ‘control’ group)

Assets Percentage of beneficiary households reporting an increase in productive assets (adjusted for change in ‘control’ group)

Indicators of policy-influencing

Objectives Indicators

Engaging with international partners ‘to gain Significance of DFID engagement with key international greater commitment to the use of social transfers partners, with the strength of linkages measured by:in the poorest countries including the re-focusing of number and significance of joint initiatives and social protection work of key international partners collaborationsuch as the World Bank and the ILO’ financial support for joint activities

Engaging with national governments and other Quality of engagement with national governments and donors to support social transfers and the donors around social transfers, with the strength of development of national strategic frameworks for linkages measured by:social protection strength of joint initiatives and collaboration, especially

government-donor for, e.g. regularity of and attendance at government-donor social protection committees

financial support for advocacy and evidence dissemination on social transfersFinancial commitments for social transfer schemes, in placeor planned


36

ReferencesBarrientos, A. (2007) ‘Core Analytical Framework and Indicators for the Evaluationof DFID-supported Pilot Social Transfer Schemes’, unpublished

Davies, M. (2008) ‘Identifying Existing and Planned Baseline Information andEvaluation Plans in Connection with DFID’s Social Protection White Paper 3Commitments and Social Transfers Action Plan’, unpublished

Devereux, S. and Coll Black, S. (2007) ‘Review of Evidence and Evidence Gapson the Effectiveness and Impacts of DFID-supported Pilot Social TransferSchemes’, unpublished

Hickey, S. and Sabates-Wheeler, R.; Guenther, G. and Macauslan, I. (2008)‘Promoting SP & ST: DFID: the Politics of Influencing’, unpublished

DFID Social Transfers Evaluation Summary Report

Documents

Transcript of DFID Social Transfers Evaluation Summary Report