Running Randomized Evaluations PDF
Preview Running Randomized Evaluations
Running Randomized Evaluations Running Randomized Evaluations A PRACTICAL GUIDE RACHEL KUDZAI GLENNERSTER and TAKAVARASHA PRINCETON UNIVERSITY PRESS Princeton and Oxford Copyright © 2013 by Princeton University Press Published by Princeton University Press, 41 William Street, Princeton, New Jersey 08540 In the United Kingdom: Princeton University Press, 6 Oxford Street, Woodstock, Oxfordshire OX20 1TW press.princeton.edu Cover design by Leah E. Horgan All Rights Reserved Library of Congress Cataloging-in-Publication Data Glennerster, Rachel. Running randomized evaluations : a practical guide / Rachel Glennerster and Kudzai Takavarasha. pages cm Includes bibliographical references and index. ISBN 978-0-691-15924-9 (hardcover : alk. paper) — ISBN 978-0-691-15927-0 (pbk. : alk. paper) 1. Evaluation research (Social action programs) 2. Social sciences—Research. I. Takavarasha, Kudzai, 1973– II. Title. H61.G5544 2013 001.4′34—dc23 2013014882 British Library Cataloging-in-Publication Data is available Th is book has been composed in Minion Pro with ITC Franklin Gothic Display by Princeton Editorial Associates Inc., Scottsdale, Arizona Printed on acid-free paper. ∞ Printed in the United States of America 10 9 8 7 6 5 4 3 2 1 CONTENTS Preface vii Abbreviations and Acronyms ix 1 The Experimental Approach 1 2 Why Randomize? 24 3 Asking the Right Questions 66 4 Randomizing 98 5 Outcomes and Instruments 180 6 Statistical Power 241 7 Threats 298 8 Analysis 324 9 Drawing Policy Lessons 386 Appendix 421 Glossary 443 Index 453 PREFACE Over the past 10 years, there has been a dramatic increase in the quan- tity and quality of rigorous impact evaluations that have tested the ef- fectiveness of specifi c approaches to reducing poverty from Britain to Burkina Faso. Randomized impact evaluations have enjoyed a partic- ularly dramatic increase in popularity. Twenty years ago, few random- ized evaluations were carried out by governments (outside the United States), international agencies, or nongovernmental organizations— except as a way to test the eff ectiveness of medicines or vaccines. Today hundreds of randomized evaluations are being undertaken by all sorts of diff erent organizations and on virtually every subject of impor- tance to policy, including education, health, governance, the environ- ment, and empowerment. Th ese randomized impact evaluations are improving our under- standing of the fundamental processes that underlie the persistence of poverty and the pathways out of it. Each evaluation answers some questions and raises new ones, the next building on the last, succes- sively adding to our knowledge of what works, knowledge we need to design better programs and more eff ectively fi ght poverty. We have written this book for those who want to help generate a rigorous evidence base on ways to reduce poverty wherever it exists in the world by running (or taking part in) rigorous impact evaluations. Running a randomized evaluation requires making hundreds of prac- tical decisions and constant trade-off s, especially in resource-poor set- tings where the best-laid plans can be quickly overtaken by events. Much of this book draws on the practical experience of the Abdul Latif Jameel Poverty Action Lab (J-PAL), a research network based at vii the Massachusetts Institute of Technology with affi liated research cen- ters and researchers at universities around the world. Th e camaraderie within J-PAL means that every publication, con- ference, or talk has many contributors, including those who never attract the limelight. So many people have provided input and advice on this book that it is almost disingenuous to isolate just two authors. Pascaline Dupas and Michael Kremer have answered hundreds of our questions and commented on numerous draft s over the several years it has taken to write this book. Th ey are also responsible for getting us into the randomization business in the fi rst place. Marc Shotland gave valuable input and advice on virtually every chapter and is the creator of the online tools that accompany this book. Alison Cappellieri cre- ated the appendix, glossary, and bibliography and copyedited the entire book. Claire Walsh fact checked all mentions of existing stud- ies. Leah Horgan designed the fi gures and the cover, and John Floretta gave detailed input on Chapter 5. Ben Feigenberg and Ben Marx checked the technical precision of our nontechnical language, espe- cially in Chapters 4, 6, and 8, and Ben Marx generated all the power graphs. Catlin Tulloch and Anna Yalouris helped with the examples on cost-eff ectiveness. Michael Kremer and Ben Olken patiently helped us work through some of the tougher econometric issues. We are deeply grateful for the detailed comments of many anonymous referees and are particularly grateful to Guido Imbens and Jörn-Steff en Pischke for their very thorough and thoughtful reviews, which led to a much im- proved book. Finally, without Mary Ann Bates this book would never have seen the light of day. Her fresh eyes, good judgment, creative re- writes, and merciless blue pencil whipped the book into shape. None of the work described in this book would have been possible without the dedication of all the implementing organizations that opened their programs up to evaluation. Th ey spent countless hours discussing what should be tested and how, fi guring out how to bring an element of randomization into their programs, and advising on how diff erent outcomes could be measured. Th ey also demonstrated considerable courage, knowing that an evaluation might fi nd that their programs did not work. Th e nongovernmental organizations and their leaders who were among the pioneers of this work in developing countries deserve special mention: Chip Bury at International Child Support Africa, Neelima Khetan at Seva Mandir, and Madav Chavan and Rukmini Banerji at Pratham. viii / PREFACE ABBREVIATIONS AND ACRONYMS 3ie International Initiative for Impact Evaluation AHTS Agricultural Household Tracking Survey BMI body mass index CCT conditional cash transfer CDD community-driven development CDF cumulative distribution function DIME Development Impact Evaluation ETP Extra Teacher Program FICA Federal Insurance Contributions Act FMB First Macro Bank FWER family-wise error rate IAT implicit association test ICS International Child Support IID independent and identically distributed IRB institutional review board ITN insecticide-treated (bed) net ITT intention-to-treat J-PAL Abdul Latif Jameel Poverty Action Lab LLIN long-lasting insecticide-treated net MDE minimum detectable eff ect MFI microfi nance institutions MIT Massachusetts Institute of Technology NERICA New Rice for Africa NGO nongovernmental organization NONIE Network of Networks on Impact Evaluation ix
The list of books you might like
