Advocacy and policy change evaluation: theory and practice

288 Pages·2017·1.234 MB·English

by  Brindis, Claire D.;Gardner, Annette L

ADVOCACY AND POLICY CHANGE EVALUATION
This page intentionally left blank
ANNETTE L. GARDNER
AND CLAIRE D. BRINDIS
A DVO CAC Y A N D
P O LI C Y C H A N G E
EVA LUAT I O N
Theory and Practice
STANFORD BUSINESS BOOKS
AN IMPRINT OF STANFORD UNIVERSITY PRESS
STANFORD, CALIFORNIA
Stanford University Press
Stanford, California
© 2017 by the Board of Trustees of the Leland Stanford Junior University.
All rights reserved.
No part of this book may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying and recording, or in any information
storage or retrieval system without the prior written permission of Stanford University
Press.
Special discounts for bulk quantities of Stanford Business Books are available to
corporations, professional associations, and other organizations. For details and discount
information, contact the special sales department of Stanford University Press.
Tel: [phone], Fax: [phone].
Printed in the United States of America on acid-free, archival-quality paper.
Library of Congress Cataloging-in-Publication Data
Names: Gardner, Annette L., author. | Brindis, Claire D., author.
Title: Advocacy and policy change evaluation : theory and practice / Annette L. Gardner
and Claire D. Brindis.
Description: Stanford, California : Stanford Business Books, an imprint of Stanford
University Press, 2017. | Includes bibliographical references and index.
Identifiers: LCCN 2016039746 | ISBN 9780804792561 (pbk. : alk. paper) | ISBN
9781503602335 (ebook)
Subjects: LCSH: Political planning—Evaluation. | Policy sciences—Evaluation. | Social
advocacy—Evaluation.
Classification: LCC JF1525.P6 G37 2017 | DDC 320.6—dc23
LC record available at https://lccn.loc.gov/2016039746
CONTENTS
Preface vii
Acknowledgments xv
List of Illustrations xix
PART 1 Useful Theories and Conceptual Models
1 Policy and Policymaking: Making a Difference 3
2 Advocacy: Influencing Decision-Making 27
PART 2 Appropriate Designs, Outcomes, and Methods
3 Designing Advocacy and Policy Change Evaluations 63
4 Outcomes and Methods in Advocacy and
Policy Change Evaluation 97
5 Unique Instruments for Advocacy and Policy
Change Evaluation 151
PART 3 Leveraging Wisdom from the Field
6 Evaluator Roles and Relationships with Stakeholders 175
7 Advancing Advocacy and Policy Change
Evaluation Practice 196
Appendix A: Six Evaluation Cases 221
Appendix B: Advocacy and Policy Change
Evaluation Resources 241
References 245
Index 255
This page intentionally left blank
PREFACE
SEVERAL factors have fueled the need for skilled evaluators who can de-
sign appropriate evaluations to meet diverse stakeholder needs: increased
foundation interest in supporting advocacy and policy change (APC)
initiatives to achieve systems change; evaluation of democracy-building
initiatives worldwide; and diffusion of advocacy capacity beyond the tra-
ditional advocacy community (such as service providers). Evaluators have
met these needs with great success, building a new field of evaluation
practice, adapting and creating evaluation concepts and methods, and
shaping advocate, funder, and evaluator thinking on advocacy and policy
change in all its diverse manifestations. The field will only continue to
grow and evolve.
This book is designed to build on this groundswell of evaluation
thought and practice and to be insightful and instructive. We combine
the plethora of concepts, definitions, designs, tools, empirical findings,
and lessons learned thus far into one practice-focused and easy-to-use
resource. This book addresses the varied evaluation needs of stakeholders
by presenting a wide array of options specific to evaluating advocacy and
policy change initiatives. It also addresses the challenges associated with
evaluation practice, such as the complexity and moving target of the con-
text in which advocacy activities occur and the challenge of attribution
issues and identification of causal factors.
There are several academic and practical reasons for developing this
book. Current advocacy and policy change evaluation practice lacks a
deep understanding of the existing research and models from the politi-
cal science, public policy and nonprofit management disciplines, includ-
ing organized interests, influence, agenda setting, media, and models of
the policy process. Consequently, evaluators often do not incorporate a
robust, theory-based foundation into their evaluation practice, limiting
viii Preface
their effectiveness in designing advocacy and policy change evaluations
and informing stakeholder learning. Increased understanding of core
principles and scholarly research will enable evaluators to make them-
selves heard more broadly and to contribute to the knowledge base of
political representation, influence, and systems change.
At the practical level, this book provides useful, real-world examples
of developing appropriate evaluation designs and applying the findings to
advocacy practice and decision-making. Our review of available resources
is broad and deep and includes an examination of relevant evaluation
strategies, as well as an analysis of the findings from the 2014 Aspen/
UCSF APC Evaluation Survey of tested evaluation designs and data-
collection instruments. Completed by 106 members of the American
Evaluation Association (AEA)1 and evaluators of advocacy and policy
change initiatives of all types, our understanding of actual APC evalua-
tion practice has been greatly expanded by the results—advocacy tactics
evaluated, evaluation strategies used, and detailed information about gaps
in the APC evaluation field.
Additionally, throughout the book, we describe and compare six eval-
uation cases that speak to the diversity of advocacy and policy change
evaluations, including a range of evaluation designs, conventional and
unique evaluation methods, and approaches to informing advocate and
funder strategy. They were identified by individual Aspen/UCSF APC
Evaluation Survey respondents as being conducted in the past five years
and containing an interesting methodology or significant lesson. A pri-
mary reason for developing the six cases of evaluation practice was to
surface design models in a variety of advocacy and policy contexts. It is
helpful to see how evaluators of advocacy and policy change initiatives
mix and match different methods and link them to evaluation questions
and a theory of change and/or logic model while being mindful to re-
1. The twenty-three-item survey was administered electronically in May 2014 by the
Aspen Planning and Evaluation Program to the 585 members of the Advocacy and Policy
Change (APC) Topical Interest Group (TIG) and 1,000 randomly selected members of
the American Evaluation Association. The survey was completed by 106 evaluators, a 7
percent response rate. The response rate of APC TIG members was 9 percent. All respon-
dents had been involved in evaluating advocacy and policy change initiatives within the
last five years.
Preface ix
source constraints and a quickly evolving context. How evaluators bal-
ance stakeholder information needs that may go beyond strategic learning
early on while addressing challenges to validity, such as an evolving ini-
tiative, small sample size, and limitations in resources may be the “art” of
advocacy and policy change evaluation. All six initiatives were sponsored
by philanthropic organizations or nonprofit public charities and speak to
funder willingness to invest in different strategies to achieving a policy
change as well as commitment to achieving long-term systems change,
and include: (1) the Initiative to Promote Equitable and Sustainable Trans-
portation (2008—2013) was funded by the Rockefeller Foundation Board
to support adoption of policies for equitable and sustainable transporta-
tion options largely through the reauthorization of the Federal Surface
Transportation Bill in 2009 and through support of commensurate state
policies in key, influential states; (2) the United Nations Foundation pro-
vided support for the Let Girls Lead program (2009—present) to create a
global movement of leaders and organizations advocating for adolescent
girls’ rights. The Let Girls Lead initiative strengthens the capacity of civil
society leaders, girl advocates, and local organizations to promote girl-
friendly laws, policies, programs, and funding in Guatemala, Honduras,
Liberia, Malawi, and Ethiopia; (3) Oxfam funded the GROW Campaign
(2012—present), a multinational campaign to tackle food injustice and
to build a better food system that sustainably feeds a growing popula-
tion, and it included a six-month campaign targeting World Bank policy
on large-scale land acquisition; (4) the Pew Charitable Trusts launched
campaigns in Canada and Australia targeting regional and locally based
land-use planning processes, as part of its International Lands Conserva-
tion Program (1999—present) to conserve old-growth forests and extend
wilderness areas; (5) funded by ClearWay Minnesota, the Tribal Tobacco
Education and Policy (TTEP) Initiative (2008–2013) provided resources
and assistance to five tribal communities to pass or expand formal and
informal smoke-free policies while increasing community awareness
of secondhand smoke; and (6) Project Health Colorado (2011–2013), a
public-will-building campaign was launched by the Colorado Trust to
engage individuals and organizations in a statewide discussion about
health care and how it can be improved. By encouraging people across