encouraging effective evaluation of conflict prevention and - OECD

104 Pages·2007·1.52 MB·English

ENCOURAGING EFFECTIVE
EVALUATION OF
CONFLICT PREVENTION AND
PEACEBUILDING ACTIVITIES
Towards DAC Guidance
A joint project of the
DAC Network on Conflict, Peace and Development Co-operation
and the DAC Network on Development Evaluation
Off-print of the OECD Journal on Development 2007: Volume 8, No. 3
ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT
ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT

The OECD is a unique forum where the governments of 30 democracies work together to
address the economic, social and environmental challenges of globalisation. The OECD is
also at the forefront of efforts to understand and to help governments respond to new
developments and concerns, such ascorporate governance, the information economy and the
challenges of an ageing population. The Organisation provides a setting where governments
can compare policy experiences, seek answers to common problems, identify good practice
and work to co-ordinate domestic and international policies. The OECD member countries
are: Australia, Austria, Belgium, Canada, the Czech Republic, Denmark,Finland, France,
Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the
Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic, Spain, Sweden,
Switzerland, Turkey, the United Kingdom and the United States. The Commission of the
European Communities takes part in the work of the OECD. OECD Publishing disseminates
widely the results of the Organisation’s statistics gathering and research on economic, social
and environmental issues, as well as the conventions, guidelines and standards agreed by its
members.

Published in French under the title:
Vers un guide du CAD pour l’évaluation des activités de la prévention des conflits et de
la construction de la paix

© OECD 2007
ENCOURAGINGEFFECTIVEEVALUATIONOFCONFLICTPREVENTIONANDPEACEBUILDINGACTIVITIES
Part 1
Encouraging Effective Evaluation of Conflict
Prevention and Peacebuilding Activities:
Towards DAC Guidance
Abstract:Thisreportoutlinesanapproachfordevelopingguidanceonevaluatingconflictprevention
and peacebuilding activities.It draws on a broad document review and interviews with key experts,
andhasbeendevelopedinconsultationwiththeDACNetworksonConflict,PeaceandDevelopment
Co-operationandonDevelopmentEvaluation.
First,thereportspecifieswhichtypesofworkcanbeconsideredconflictpreventionandpeacebuilding
activities,distinguishingbetweenconflictsensitivityandconflictpreventionandpeacebuildingwork.
Thechallengesofevaluatinginthisfieldarediscussedandrecommendationsaremadeonkeyconcepts
andimportantelementsfortheprocess,includingconductingaconflictanalysis;examiningtheories
of change; understanding impacts; programming to support evaluation; and choosing evaluation
methodologiesandapproaches.
The report also looks at the applicability of existing DAC tools, and adapts the DAC Criteria for
EvaluatingDevelopmentAssistancetotheconflictpreventionandpeacebuildingfield.Finally,itoutlines
thenextstepsfortheDACNetworksindevelopingguidance.
Acknowledgements:ThispaperisaresultofajointprojectoftheDACNetworkonConflict,PeaceandDevelopment
Co-operation and the DAC Network on Development Evaluation.The report was prepared by Mary B.Anderson,
DianaChigasandPeterWoodrowofCDACollaborativeLearningProjects(CDA),basedinCambridge,Massachusetts,
USA.ThetwoNetworksreceivedandcommentedonthisdocumentpriortoitspublication.Thefindingsofthisresearch
arebeingusedinthepreparationofdraftDACguidance,whichwillbecompletedbythetwonetworksintheautumn
of2007.Theguidancewillbeusedforevaluationsinthefieldduringaoneyearapplicationphase,thenrevisedand
finalisedattheendof2008.Anycommentsorqueriesshouldbeaddressedto<[email protected]>.
©OECD2007 3
.
ENCOURAGINGEFFECTIVEEVALUATIONOFCONFLICTPREVENTIONANDPEACEBUILDINGACTIVITIES
Table of Contents
ExecutiveSummary..................................................................................................................................................7
Summaryofgeneralfindings................................................................................................................................7
FutureOECD/DACactionsandnextsteps.........................................................................................................10
Introduction.............................................................................................................................................................13
Interviews..............................................................................................................................................................14
Documentreview..................................................................................................................................................14
Policydocuments..................................................................................................................................................15
Organisationofthisreport..................................................................................................................................15
I.AnOverviewofConflictPreventionandPeacebuildingActivities...............................................................17
A. Theproblem:blurredboundariesandlackofdefinitionalconsensus...................................................17
B. Definingconflictpreventionandpeacebuildingforevaluationpurposes..............................................19
C. Clarifyingthedifferencebetweenconflictsensitivityandconflictpreventionandpeacebuilding...21
D. Particularchallengesofconflictprevention................................................................................................22
E. Acoresetofconflictpreventionandpeacebuildingactivities.................................................................23
II.ConflictPreventionandPeacebuildingEvaluations:SpecialIssues..........................................................25
A. Framingissues:Conflictanalysis,demystifyingimpacts,consideringindicators................................25
B. Specialproblems,conditionsandexpectations..........................................................................................29
C. Methodologicalissuesinconflictpreventionandpeacebuildingevaluations......................................32
III.RelevanceofExistingDACEvaluationTools.................................................................................................37
A. Guidanceforevaluatinghumanitarianassistanceincomplexemergencies........................................37
B. TheDACEvaluationCriteria.........................................................................................................................38
IV.RecommendationsandMainFindings..........................................................................................................43
1.PreparingtheEvaluation.................................................................................................................................43
A. Gettingprogrammedesignright:Anaidtogoodevaluation...................................................................43
B. Settinguptheevaluation...............................................................................................................................44
C. Additionalissuestoconsider........................................................................................................................51
2.ConductingtheEvaluation..............................................................................................................................52
A. Obtainorperformaconflictanalysis..........................................................................................................52
B. Identifyinterventiongoalsandassessthetheoryofchange...................................................................54
C. Gatherandanalysedata/information..........................................................................................................55
D. Examinetheeffortagainstvariouscriteria.................................................................................................55
E. Analysetheresultschain...............................................................................................................................58
F. Assessforconflictsensitivity.........................................................................................................................59
G. Examinetherelationshiptopolicies............................................................................................................59
H. Engageinalearningprocess.........................................................................................................................59
Afinalnote:Keepitsimple.................................................................................................................................60
©OECD2007 5
ENCOURAGINGEFFECTIVEEVALUATIONOFCONFLICTPREVENTIONANDPEACEBUILDINGACTIVITIES
Annex1.Interviewees............................................................................................................................................61
Annex2.GeneralBibliography.............................................................................................................................63
Annex3.Bibliography:PolicyDocuments..........................................................................................................69
Annex4.WhoshouldusetheDACconflictpreventionandpeacebuilding
EvaluationGuidanceandforwhichtypesofintervention?............................................................73
Potentialusersandapplicationsoftheguidance............................................................................................73
Thecategoriesofconflictpreventionandpeacebuildingprogramming......................................................76
Annex5.ReflectingonPeacePractice(RPP)CriteriaofEffectiveness............................................................79
Challengesforassessingeffectiveness..............................................................................................................79
Programmeeffectivenessvs.peaceeffectiveness...........................................................................................79
SixCriteriaofEffectiveness.................................................................................................................................79
TheCriteriaarecumulative................................................................................................................................80
Annex6.TheoriesofChange................................................................................................................................81
Evaluatingconflictpreventionandpeacebuildingtheoriesofchange.........................................................81
Example1:EvaluatinggrassrootsconflictpreventioninLiberia...................................................................82
Example2:TheimpactofinternationalpeacebuildingpoliciesandprogramminginKosovo.................83
Annex7.ApplicationofGeneralEvaluationGuidancetoSpecificconflict
preventionandpeacebuildingDomainsofWork.............................................................................89
A.Evaluatingtrainingprogrammes:Specialissues,specialconsiderations................................................89
B.Evaluatingdialogueprogramming:Definitions,specialchallenges,considerations.............................91
Annex8.ConsultantTermsofReferenceDevelopingGuidanceforEvaluatingConflictPreventionand
PeacebuildingActivities:AnApproachPaper....................................................................................97
Introduction...........................................................................................................................................................97
Questionsandchallenges:Towardsdevelopingguidanceforevaluatingconflictprevention
andpeacebuildingactivities...............................................................................................................................97
Tables
Table1.1 Applyingconflictpreventionandpeacebuildingevaluationguidance:
Conflictsensitivity,objectivesandeffectiveness..............................................................................18
Table1.2 Operationalvs.structuralprevention.................................................................................................22
Table4.1 Rangeofconflictpreventionandpeacebuildingapproaches.........................................................77
Table6.1 Commontheoriesofchange................................................................................................................86
Figure
Figure4.1. Mappingconflictpreventionandpeacebuildinggoalsandstrategies........................................54
Boxes
4.1. Guidingprinciplesforevaluations...............................................................................................................44
4.2. FutureDACactionsandnextstepsindevelopingguidance....................................................................60
6 ©OECD2007
ENCOURAGINGEFFECTIVEEVALUATIONOFCONFLICTPREVENTIONANDPEACEBUILDINGACTIVITIES
Executive Summary
Thechangingnatureofwarandtheastoundinglyhighhuman,societalandfinancialcostsofrecent
violentconflicts–coupledwithagrowingsensethatsuchsufferinganddevastationcouldbeavoidedor
atleastmitigated–haveledtoamarkedincreaseinconflictpreventionandpeacebuildingactivity.The
past decade has seen growing numbers of individuals,governments,international organisations,civil
society groups and research institutions devote their efforts, intelligence and financial resources to
interventions intended to avert or end violent conflict.At the same time,concern has grown over the
effectivenessofsuchefforts.
TwoNetworksoftheOECDDevelopmentAssistanceCommittee(DAC)–DevelopmentEvaluationand
Conflict,Peace and Development Co-operation (CPDC) – sensed this growing unmet need for direction,
among their members as well as other actors in the conflict prevention and peacebuilding fields.
In response,they launched an initiative to develop forthcoming DAC guidance on evaluating conflict
prevention and peacebuildingactivities.Theoverall objectiveofthis two-year process has been tohelp
improveevaluation practiceand thereby support thebroadercommunity ofexperts and implementing
organisationsintheireffortstoenhancethequalityofconflictpreventionandpeacebuildinginterventions.
CDA Collaborative Learning Projects (CDA) was asked to draft this report with a fourfold aim: to
establish greater clarity with regard to conflict prevention and peacebuilding definitions; to review
standards against which to evaluate conflict prevention and peacebuilding activities; to identify the
distinguishingcharacteristicsandspecialconsiderationsoftheresultingevaluations;andtooutlinekey
potentialelementsthoseevaluationscouldinclude.Theresearchisbasedonanexaminationofevaluation
and policy documents and interviews with evaluation specialists,donor representatives,multilateral
organisationsandprogrammeimplementers,includingnon-governmentalorganisations(NGOs).CDAalso
participatedinaDACmeetinginParisinOctober2006andthejointworkshopofthetwoDACnetworks
inOsloinDecember2006.Thereportthereforerepresentsausefulstepinthelargerprocessofdeveloping
thoroughlytestedandpracticalconflictpreventionandpeacebuildingevaluationguidance.
The main audiences for this report are the evaluation and policy staff of donor agencies,conflict
prevention and peacebuilding field practitioners and other relevant donor government personnel.
Nonetheless,manyotheractorsintheconflictpreventionandpeacebuildingandevaluationfieldswillfind
theinformationcontainedhereinusefulaswell.
Summary of general findings
Definingconflictpreventionandpeacebuilding
Through comparative analysis of activities across contexts and times,CDA found that a policy or
approachthatwaslabelledconflictpreventionandpeacebuildinginsomeplacesoftenwasnotseenas
conflictpreventionandpeacebuildingelsewhere.Theoriesofpeacebuildingandconflictresolutionabound,
buteachhasbeendisprovedasoftenasithasbeenproved.Donorsalsodifferintheirinterpretationsof
whatconflictpreventionandpeacebuildingis.Theconfusionoverdefinitionsofconflictpreventionand
peacebuildingleadsmanytoassume(falsely)thatbybeing“conflict-sensitive”theyareipsofactoalsodoing
peacebuildingwork.
©OECD2007 7
ENCOURAGINGEFFECTIVEEVALUATIONOFCONFLICTPREVENTIONANDPEACEBUILDINGACTIVITIES
Furthermore,manypeopleassumethatadvancesincriticalstructuralareasalsocontributetothe
reduction of conflict and the promotion of peace. Many donor-funded programmes and policies are
undertakenontheassumptionthatprogresstowardsliberalisation,economicgrowth,prosperity,human
rightsanddemocracycontributetopeace.Evidenceshowsthatwhilesomeofthoseeffortsdocontribute
topeace,othershavenegativeornegligibleeffectsonconflict.
Giventhisconfusion,CDAsuggestedthefollowingdefinitionofconflictpreventionandpeacebuilding
asthebasisforguidance:
Conflict prevention and peacebuilding interventions are efforts that adopt goals and objectives
aimed at preventing conflict or building peace; they are usually (but not always) focused on a
particular conflict zone – an area threatened by, in the midst of, or recovering from serious
intergroupviolence.1
Using an intervention’s goals and objectives as the determining criteria for delineating conflict
prevention and peacebuilding boundaries helps to clarify the difference between “conflict-sensitive
programming”and conflict prevention and peacebuilding activities.All efforts undertaken in a conflict
area should be conflict-sensitive (aware of how they affect the dynamics of conflict). Interventions
intended to prevent conflict and build peace must also be accountable for their impact on the specific
factorsthatdriveandshapethatconflict.
Conflictpreventionandpeacebuildingevaluations
Scope–Whenapolicyorprogrammeadoptsgoalstomakeadifferenceinthekeydrivingfactors2of
conflict,it becomes a legitimate subject for conflict prevention- and peacebuilding-focused evaluating.
Donors may also decide to examine the impacts on peace of interventions that do not have conflict
preventionandpeacebuildinggoals,becausethoseeffortsnonethelesshavethepotentialforinfluencing
thefactorsdrivingconflict.Insettingsvulnerabletoconflict,policymakersmaywishtoensurethatall
interventionsareconsistentwiththeefforttopreventviolence.Thus,eveneffortsthatwerenotoriginally
intended to address conflict may be examined for their actual or potential impacts, using conflict
preventionandpeacebuildingevaluationcriteria.
Conflictanalysisandtheoriesofchange–Policymakers,practitionersanddonorsagreethatinorderto
be effective,conflict prevention and peacebuilding activities must be based on an analysis of the key
drivingfactorsofandactorswithintheconflict.Thereisalsogrowingagreementthatevaluatorsofconflict
prevention and peacebuilding should use such an analysis as the basis for evaluations.Based on that
analysis,conflictpreventionandpeacebuildingprogrammestrategiesortheoriesofchangedescribethe
way practitioners and policy makers believe a particular activity will contribute to achieving peace (or
conflict prevention). These underlying theories need to be uncovered, described and tested during
evaluation.
1. Thereisnointernationallyagreeddefinitionoftheterm“violence”.Theterm“intergroupviolence”isusedhereto
distinguishtheintentionalandillegitimateuseofarmedforce,includingbothinter-andintra-stateconflicts,from
othertypesofviolencesuchascriminalactivityandinterpersonalviolence.Whilesomeconflictpreventionand
peacebuildingactivitiesmayaddresssuchformsofviolence,thisresearchdealsprimarilywithintergroupviolence
(war).
2. Keydrivingfactorsaremajorelementsthatcontributetotheconflict,withoutwhichtheconflicteitherwouldnot
existorwouldbesignificantlydifferent.Thesecanbelong-termstructuralissues,moreimmediatetriggersorany-
thinginbetween,asidentifiedbypeopleinthesituation.
8 ©OECD2007
ENCOURAGINGEFFECTIVEEVALUATIONOFCONFLICTPREVENTIONANDPEACEBUILDINGACTIVITIES
Impacts – There are wide differences of opinion about what is meant by impacts in the context of
conflict prevention and peacebuilding evaluations.Many think that impacts are beyond the immediate
outcomesofconflictpreventionandpeacebuildingprogramming,occurring“atahigherlevel”andoverthe
longterm.IntheviewoftheCDAteam,thispreoccupationwiththeremotenessofimpactscanmistakenly
divertevaluationattentionawayfromassessingthemanyeffects(intendedornot)ofconflictprevention
andpeacebuildingprogrammes.Evaluationsshouldtakeaccountofindividualprogrammeimpactsand
cumulative,multi-programmeimpactsatthestrategicorpolicylevel.Bothlong-andshort-termimpacts
shouldbeexamined,withafocusonimpactsontheconflict.
Specialproblems,conditionsandexpectationsofconflictpreventionandpeacebuildingevaluations–Thereis
broadevidencethatevaluatingconflictpreventionandpeacebuildingfacesspecialchallenges,including:
thepoliticalnatureofpeaceprocesses;thepotentiallyproblematicroleofpartnergovernments;unclear
orunspecifiedgoalsofconflictpreventionandpeacebuildingactivities;implicitandunarticulated(aswell
asuntested)theoriesofchange;theimportanceofprocessesaswellasoutcomesandimpacts;andthe
emotional and ideological commitments of conflict prevention and peacebuilding advocates and
programmers.
Evaluationasanintervention–Becauseaconflictpreventionandpeacebuildingevaluationoccursin
thecontextofconflict,itsapproachesandprocessesareeffectivelyinterventionsintheconflict.Evaluators
whodonotknowhowtoengagewithconflictdynamicsinpositiveways(lackingconflictsensitivityand
know-how)maydoharmastheyevaluateaprogramme.Conflictpreventionandpeacebuildingevaluators
must act in accordance with the field’s established principles and be sure that they have thoroughly
analysedtheconflictcontext.
Approachesandmethodologiesforevaluations–Thereisnoone“right”methodologytoaddressallthe
special circumstances and conditions of conflict prevention and peacebuilding.There are, however,
commonprinciplesthatshouldbefollowed,interalia:inclusiveness;testingoftheunderlyingtheoryof
change;useofamixofbothqualitativeandquantitativemethods;attentiontounexpectedimpacts;and
ethical standards for approaching informants and handling the reporting of findings within conflict
environments.
Adaptation of the DAC Criteria to conflict prevention and peacebuilding – DAC’s Criteria for Evaluating
Development Assistance,3 with additional information from the OECD/DAC “Glossary of KeyTerms in
EvaluationandResultsBasedManagement”,havebeenadaptedtoserveasthebasisforconflictprevention
and peacebuilding evaluation guidance. All activities (projects, programmes, country strategies,
interventionsandpolicies)shouldbeexaminedusingthesecriteria.Specificquestionslinkingeachcriterion
to conflict prevention and peacebuilding issues were also developed. The criteria are: Relevance/
Appropriateness; Effectiveness; Efficiency; Impact; Sustainability; Linkages (replacing“Connectedness”);
Coverage;andConsistencywithconflictpreventionandpeacebuildingvalues.Thereisdisagreementover
whetherCoherenceshouldbeconsideredacriterion;somefeelmuchmoreevidenceisneededtoestablish
whether coherence contributes to conflict prevention and peacebuilding effectiveness.Nor is there full
consensusonaddingConsistencyandCoverage,whichcouldbecoveredbyothercriteria.
Skillsandtoolsnecessaryforconflictpreventionandpeacebuildingevaluators–Threequalificationsappear
tobeofparticularimportanceforevaluatorteams:i)knowledgeofevaluationmethods;ii)knowledgeof
theconflictpreventionandpeacebuildingfield(includingthestateoftheartandtheskillsrequiredfor
performing conflict analysis); and iii) an appropriate and conflict-sensitive style of working (being
comfortableworkingindangeroussituations,havinginterpersonalapproachesthataretransparent,being
gender-andculturallysensitive,andpossessingskillsformanagingconflictandtension).
3. DACCriteriaforEvaluatingDevelopmentAssistance,http://www.oecd.org/dataoecd/15/21/39119068.pdf
©OECD2007 9