Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for Development Practitioners

268 Pages·2004·0.94 MB·English

by  Jody Zall Kusek

A H a n d b o o k f o r D e v e l o p m e n t P r a c t i t i o n e r s
TTeenn SStteeppss
ttoo aa
RReessuullttss--
BBaasseedd
MMoonniittoorriinngg
aanndd
EEvvaalluuaattiioonn
SSyysstteemm
Jody Zall Kusek
Ray C. Rist
THE WORLD BANK

A Handbook for Development Practitioners
Ten Steps
to a
Results-Based
Monitoring
and
Evaluation
System

A Handbook for Development Practitioners
TTeenn SStteeppss
ttoo aa
RReessuullttss--BBaasseedd
MMoonniittoorriinngg
aanndd
EEvvaalluuaattiioonn
SSyysstteemm
Jody Zall Kusek
Ray C. Rist
THE WORLD BANK
Washington, D.C.
© 2004 The International Bank for Reconstruction and Development /
The World Bank
1818 H Street, NW
Washington, DC 20433
Telephone [phone]
Internet www.worldbank.org
E-mail [email protected]
All rights reserved.
1 2 3 4 07 06 05 04
The findings, interpretations, and conclusions expressed herein are those of
the author(s) and do not necessarily reflect the views of the Board of
Executive Directors of the World Bank or the governments they represent.
The World Bank does not guarantee the accuracy of the data included in
this work. The boundaries, colors, denominations, and other information
shown on any map in this work do not imply any judgment on the part of
the World Bank concerning the legal status of any territory or the endorse-
ment or acceptance of such boundaries.
Rights and Permissions
The material in this work is copyrighted. Copying and/or transmitting
portions or all of this work without permission may be a violation of
applicable law. The World Bank encourages dissemination of its work and
will normally grant permission promptly.
For permission to photocopy or reprint any part of this work, please send a
request with complete information to the Copyright Clearance Center, Inc.,
222 Rosewood Drive, Danvers, MA 01923, USA, telephone [phone],
fax [phone], www.copyright.com.
All other queries on rights and licenses, including subsidiary rights,
should be addressed to the Office of the Publisher, World Bank, 1818 H
Street NW, Washington, DC 20433, USA, fax [phone], e-mail
[email protected].
Library of Congress Cataloging-in-Publication Data
Kusek, Jody Zall, 1952–
Ten steps to a results-based monitoring and evaluation system : a hand-
book for development practitioners / Jody Zall Kusek and Ray C. Rist.
p. cm.
Includes bibliographical references and index.
ISBN 0-8213-5823-5
1. Government productivity—Developing countries—Evaluation.
2. Performance standards—Developing countries—Evaluation. 3. Total
quality management in government—Developing countries—Evaluation.
4. Public administration—Developing countries—Evaluation. I. Rist, Ray
C. II. Title.
JF1525.P67K87 2004
352.3′5—dc22 2004045527
Contents
Preface xi
About the Authors xiv
Introduction
Building a Results-Based Monitoring and Evaluation System 1
Part I
New Challenges in Public Sector Management 2
International and External Initiatives and Forces for Change 3
National Poverty Reduction Strategy Approach 8
Internal Initiatives and Forces for Change 10
Part 2
Results-BasedM&E—APowerfulPublicManagementTool 11
Monitoring and Evaluation: What Is It All About? 12
Key Features of Traditional Implementation-Focused and Results-
Based M&E Systems 15
Many Applications for Results-Based M&E 17
Political and Technical Challenges to Building a Results-Based
M&E System 20
Introducing the 10-Step Model for Building a Results-Based
M&E System 23
Where to Begin: Whole-of-Government, Enclave, or Mixed Approach
24
Part 3
M&E Experience in Developed and Developing Countries 27
M&E Experience in Developed and OECD Countries 27
Special M&E Challenges Facing Developing Countries 32
M&E Experience in Developing Countries 35
Chapter 1
Step 1: Conducting a Readiness Assessment 39
Part 1
Why Do a Readiness Assessment? 40
Part 2
The Readiness Assessment: Eight Key Questions 43
Part 3
Readiness Assessments in Developing Countries: Bangladesh, Egypt,
and Romania 48
vi Contents
Part 4
Lessons Learned 49
Chapter 2
Step 2: Agreeing on Outcomes to Monitor and Evaluate 56
The Importance of Outcomes 56
Issues to Consider in Choosing Outcomes to Monitor and Evaluate 57
The Importance of Building a Participatory and Consultative Process
involving Main Stakeholders 58
The Overall Process of Setting and Agreeing upon Outcomes 59
Examples and Possible Approaches 61
Chapter 3
Step 3: Selecting Key Performance Indicators to Monitor
Outcomes 65
Indicators Are Required for All Levels of Results-Based M&E Systems 66
Translating Outcomes into Outcome Indicators 66
The “CREAM” of Good Performance Indicators 68
The Use of Proxy Indicators 70
The Pros and Cons of Using Predesigned Indicators 72
Constructing Indicators 74
Setting Indicators: Experience in Developing Countries 75
Chapter 4
Step 4: Setting Baselines and Gathering Data on Indicators 80
Establishing Baseline Data on Indicators 81
Building Baseline Information 82
Identifying Data Sources for Indicators 83
Designing and Comparing Data Collection Methods 84
The Importance of Conducting Pilots 86
Data Collection: Two Developing Country Experiences 89
Chapter 5
Step 5: Planning for Improvement—Selecting Results Targets 90
Definition of Targets 90
Factors to Consider When Selecting Performance Indicator Targets 91
Examples of Targets Related to Development Issues 93
The Overall Performance-Based Framework 94
Chapter 6
Step 6: Monitoring for Results 96
Part 1
Key Types and Levels of Monitoring 98
Links between Implementation Monitoring and Results Monitoring 101
Part 2
Key Principles in Building a Monitoring System 103
Achieving Results through Partnership 105
Needs of Every Results-Based Monitoring System 106
The Data Quality Triangle: Reliability, Validity, and Timeliness 108
Contents vii
Analyzing Performance Data 111
Pretesting Data Collection Instruments and Procedures 112
Chapter 7
Step 7: The "E" in M&E—Using Evaluation Information to
Support a Results-Based Management System 113
Uses of Evaluation 115
The Timing of Evaluations 118
Types of Evaluations 121
Characteristics of Quality Evaluations 126
Examples of Evaluation at the Policy, Program, and Project Levels 128
Chapter 8
Step 8: Reporting the Findings 129
The Uses of Monitoring and Evaluation Findings 130
Know and Target the Audience 130
Presentation of Performance Data in Clear and Understandable Form
132
What Happens If the M&E System Produces Bad Performance News?
136
Chapter 9
Step 9: Using the Findings 138
Uses of Performance Findings 138
Additional Benefits of Using Findings: Feedback, Knowledge, and
Learning 140
Strategies for Sharing Information 146
Chapter 10
Step 10: Sustaining the M&E System within the Organization 151
Six Critical Components of Sustaining Results-Based M&E Systems
152
The Importance of Incentives and Disincentives in Sustaining
M&E Systems 155
Possible Problems in Sustaining Results-Based M&E Systems 155
Validating and Evaluating M&E Systems and Information 160
M&E: Stimulating Positive Cultural Change in Governments and
Organizations 160
Last Reminders 160
Chapter 11
Making Results-Based M&E Work for You and Your
Organization 162
Why Results-Based M&E? 162
How to Create Results-Based M&E Systems 165
Summing Up 170
viii Contents
Annexes:
Annex I: Assessing Performance-Based Monitoring and Evaluation
Capacity: An Assessment Survey for Countries,
Development Institutions, and Their Partners 174
Annex II: Readiness Assessment: Toward Results-Based Monitoring
and Evaluation in Egypt 178
Annex III: Millennium Development Goals (MDGs): List of Goals
and Targets 200
Annex IV: National Evaluation Policy for Sri Lanka: Sri Lanka
Evaluation Association (SLEva) jointly with the Ministry
of Policy Development and Implementation 204
Annex V: Andhra Pradesh (India) Performance Accountability Act
2003: (Draft Act) (APPAC Act of 2003) 211
Annex VI: Glossary: OECD Glossary of Key Terms in Evaluation and
Results-Based Management (2002) 223
Notes 230
References 231
Useful Web Sites 235
Additional Reading 236
Index 239
Boxes
i.i Millennium Development Goals 4
i.ii Example of Millennium Development Goal, Targets, and
Indicators 5
i.iii Transparency International 6
i.iv The Power of Measuring Results 11
i.v Key Features of Implementation Monitoring versus Results
Monitoring 17
i.vi Australia’s Whole-of-Government Model 29
i.vii France: Lagging Behind but Now Speeding Ahead in
Governmental Reform 30
i.viii Republic of Korea: Well on the Road to M&E 31
i.ix Malaysia: Outcome-Based Budgeting, Nation Building, and
Global Competitiveness 36
i.x Uganda and Poverty Reduction—Impetus toward M&E 37
1.1 The Case of Bangladesh—Building from the Bottom Up 50
1.2 The Case of Egypt—Slow, Systematic Moves toward M&E 51
1.3 The Case of Romania—Some Opportunities to Move toward
M&E 52
3.1 Indicator Dilemmas 71
3.2 The Africa Region’s Core Welfare Indicators 76
3.3 Sri Lanka’s National Evaluation Policy 77
3.4 Albania’s Three-Year Action Plan 78
3.5 Program and Project Level Results Indicators: An Example from
the Irrigation Sector 79
3.6 Outcome: Increased Participation of Farmers in Local Markets 79
4.1 Albania’s Strategy for Strengthening Data Collection Capacity 88