Program Evaluation Alternative Approaches and Practical Guidelines FOURTH EDITION

559 Pages·2017·4.05 MB·English

Program Evaluation
Alternative Approaches
and Practical Guidelines
FOURTH EDITION
Jody L. Fitzpatrick
University of Colorado Denver
James R. Sanders
Western Michigan University
Blaine R. Worthen
Utah State University
Boston Columbus Indianapolis New York San Francisco Upper Saddle River
Amsterdam Cape Town Dubai London Madrid Milan Munich Paris Montreal Toronto
Delhi Mexico City São Paulo Sydney Hong Kong Seoul Singapore Taipei Tokyo
Vice President and Editor in Chief:Jeffery W. Johnston
Senior Acquisitions Editor:Meredith D. Fossel
Editorial Assistant:Nancy Holstein
Vice President, Director of Marketing:Margaret Waples
Senior Marketing Manager:Christopher D. Barry
Senior Managing Editor:Pamela D. Bennett
Senior Project Manager:Linda Hillis Bayma
Senior Operations Supervisor:Matthew Ottenweller
Senior Art Director:Diane Lorenzo
Cover Designer:Jeff Vanik
Cover Image:istock
Full-Service Project Management:Ashley Schneider, S4Carlisle Publishing Services
Composition:S4Carlisle Publishing Services
Printer/Binder:Courier/Westford
Cover Printer:Lehigh-Phoenix Color/Hagerstown
Text Font:Meridien
Credits and acknowledgments borrowed from other sources and reproduced, with permission, in this
textbook appear on appropriate page within text.
Every effort has been made to provide accurate and current Internet information in this book.
However, the Internet and information posted on it are constantly changing, so it is inevitable that
some of the Internet addresses listed in this textbook will change.
Copyright © 2011, 2004, 1997 Pearson Education, Inc., Upper Saddle River, New Jersey 07458.
All rights reserved. Manufactured in the United States of America. This publication is protected by
Copyright, and permission should be obtained from the publisher prior to any prohibited
reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic,
mechanical, photocopying, recording, or likewise. Toobtain permission(s) to use material from this
work, please submit a written request to Pearson Education, Inc., Permissions Department,
501 Boylston Street, Suite 900, Boston, MA 02116, fax: [phone],
email: [e-mail].
Library of Congress Cataloging-in-Publication Data
Fitzpatrick, Jody L.
Program evaluation: alternative approaches and practical guidelines / Jody L. Fitzpatrick, James R.
Sanders, Blaine R. Worthen.
p. cm.
ISBN 978-0-205-57935-8
1. Educational evaluation—United States. 2. Evaluation research (Social action programs)—
United States. 3. Evaluation—Study and teaching—United States. I. Sanders, James R.
II. Worthen, Blaine R. III. Worthen, Blaine R. Program evaluation. IV. Title.
LB2822.75.W67 2011
379.1’54—dc22
2010025390
10 9 8 7 6 5 4 3 2
ISBN 10: 0-205-57935-3
ISBN 13: 978-0-205-57935-8
About the Authors
Jody Fitzpatrickhas been a faculty member in public administration at the Uni-
versity of Colorado Denver since 1985. She teaches courses in research methods
and evaluation, conducts evaluations in many schools and human service settings,
and writes extensively about the successful practice of evaluation. She has served
on the Board of the American Evaluation Association and on the editorial boards
of the American Journal of Evaluationand New Directions for Evaluation.She has also
served as Chair of the Teaching of Evaluation Topical Interest Group at the American
Evaluation Association and has won a university-wide teaching award at her
university. In one of her recent publications, Evaluation in Action: Interviews with
Expert Evaluators, she uses interviews with expert evaluators on one evaluation to
talk about the decisions that evaluators face as they plan and conduct evaluations
and the factors that influence their choices. She is currently evaluating the chang-
ing roles of counselors in middle schools and high schools and a program to help
immigrant middle-school girls to achieve and stay in school. Her international
work includes research on evaluation in Spain and Europe and, recently, she has
spoken on evaluation issues to policymakers and evaluators in France, Spain,
Denmark, Mexico, and Chile.
James Sanders is Professor Emeritus of Educational Studies and the Evaluation
Center at Western Michigan University where he has taught, published, consulted,
and conducted evaluations since 1975. A graduate of Bucknell University and the
University of Colorado, he has served on the Board and as President of the American
Evaluation Association (AEA) and has served as Chair of the Steering Committee
that created the Evaluation Network, a predecessor to AEA. His publications in-
clude books on school, student, and program evaluation. He has worked exten-
sively with schools, foundations, and government and nonprofit agencies to
develop their evaluation practices. As Chair of the Joint Committee on Standards
for Educational Evaluation, he led the development of the second edition of The
Program Evaluation Standards. He was also involved in developing the concepts of
applied performance testing for student assessments, cluster evaluation for program
evaluations by foundations and government agencies, and mainstreaming evaluation
for organizational development. His international work in evaluation has been
concentrated in Canada, Europe, and Latin America. He received distinguished ser-
vice awards from Western Michigan University, where he helped to establish a PhD
program in evaluation, and from the Michigan Association for Evaluation.
iii
iv About the Authors
Blaine Worthenis Psychology Professor Emeritus at Utah State University, where
he founded and directed the Evaluation Methodology PhD program and the West-
ern Institute for Research and Evaluation, conducting more than 350 evaluations
for local and national clients in the United States and Canada. He received his PhD
from The Ohio State University. He is a former editor of Evaluation Practice and
founding editor of the American Journal ofEvaluation.He served on the American
Evaluation Association Board of Directors and received AEA’s Myrdal Award for
Outstanding Evaluation Practitioner and AERA’s Best Evaluation Study Award. He
has taught university evaluation courses (1969–1999), managed federally man-
dated evaluations in 17 states (1973–1978), advised numerous government and
private agencies, and given more than 150 keynote addresses and evaluation
workshops in the United States, England, Australia, Israel, Greece, Ecuador, and
other countries. He has written extensively in evaluation, measurement, and as-
sessment and is the author of 135 articles and six books. His Phi Delta Kappanarti-
cle, “Critical Issues That Will Determine the Future of Alternative Assessment,”
was distributed to 500 distinguished invitees at the White House’s Goals 2000
Conference. He is recognized as a national and international leader in the field.
Preface
The twenty-first century is an exciting time for evaluation. The field is growing.
People—schools, organizations, policymakers, the public at large—are interested
in learning more about how programs work: how they succeed and how they
fail. Given the tumult experienced in the first decade of this century, many peo-
ple are interested in accountability from corporations, government, schools, and
nonprofit organizations. The fourth edition of our best-selling textbook is designed
to help readers consider how evaluation can achieve these purposes. As in previ-
ous editions, our book is one of the few to introduce readers to both the different
approaches to evaluation and practical methods for conducting it.
New to This Edition
The fourth edition includes many changes:
• A new chapter on the role of politics in evaluation and ethical considerations.
• A new and reorganized Part Two that presents and discusses the most current
approaches and theories of evaluation.
• An increased focus on mixed methods in design, data collection, and analysis.
• Links to interviews with evaluators who conducted an evaluation that illus-
trates the concepts reviewed in that chapter, as they discuss the choices and
challenges they faced.
• A discussion of how today’s focus on performance measurement, outcomes,
impacts, and standards have influenced evaluation.
• New sections on organizational learning, evaluation capacity building,
mainstreaming evaluation, and cultural competence––trends in evaluation
and organizations.
Evaluation, today, is changing in a variety of ways. Policymakers, managers,
citizens, and consumers want better tracking of activities and outcomes. More
importantly, many want a better understandingof social problems and the programs
and policies being undertaken to reduce these problems. Evaluation in many forms,
including performance measurement and outcome or impact assessments, is ex-
panding around the globe. People who work in organizations are also interested in
evaluation as a way to enhance organizational learning. They want to know how
well they’re doing, how to tackle the tough problems their organizations address,
and how to improve their performance and better serve their clients and their
v
vi Preface
community. Many different methods are being developed and used: mixed meth-
ods for design and data collection, increased involvement of new and different
stakeholders in the evaluation process, expanded consideration of the potential uses
and impacts of evaluation, and more effective and diverse ways to communicate
findings. As evaluation expands around the world, the experiences of adapting eval-
uation to different settings and different cultures are enriching the field.
In this new edition, we hope to convey to you the dynamism and creativity
involved in conducting evaluation. Each of us has many years of experience in
conducting evaluations in a variety of settings, including schools, public welfare
agencies, mental health organizations, environmental programs, nonprofit organ-
izations, and corporations. We also have years of experience teaching students
how to use evaluation in their own organizations or communities. Our goal is, and
always has been, to present information that readers can use either to conduct or
to be a participant in evaluations that make a difference to their workplace, their
clients, and their community. Let us tell you a bit more about how we hope to do
that in this new edition.
Organization of This Text
The book is organized in four parts. Part One introduces the reader to key concepts
in evaluation; its history and current trends; and ethical, political, and interper-
sonal factors that permeate and transcend all phases of evaluation. Evaluation dif-
fers from research in that it is occurring in the real world with the goal of being
used by non-researchers to improve decisions, governance, and society. As a
result, evaluators develop relationships with their users and stakeholders and
work in a political environment in which evaluation results compete with other
demands on decision makers. Evaluators must know how to work in such envi-
ronments to get their results used. In addition, ethical challenges often present
themselves. We find the ways in which evaluation differs from research to be both
challenging and interesting. It is why we chose evaluation as our life’s work. In
Part One, we introduce you to these differences and to the ways evaluators work
in this public, political context.
In Part Two, we present several different approaches, often called models or
theories, to evaluation. (Determining whether objectives or outcomes have been
achieved isn’t the only way to approach evaluation!) Approaches influence how
evaluators determine what to study and how they involve others in what they
study. We have expanded our discussions of theory-based, decision-oriented, and
participatory approaches. In doing so, we describe new ways in which evaluators
use logic models and program theories to understand the workings of a program.
Participatory and transformative approaches to empowering stakeholders and
creating different ways of learning are described and contrasted. Evaluators must
know methodology, but they also must know about different approaches to eval-
uation to consciously and intelligently choose the approach or mix of approaches
that is most appropriate for the program, clients, and stakeholders and context of
their evaluation.
Preface vii
In Parts Three and Four, the core of the book, we describe how to plan and
carry out an evaluation study. Part Three is concerned with the planning stage:
learning about the program, conversing with stakeholders to learn purposes and
consider future uses of the study, and identifying and finalizing evaluation
questions to guide the study. Part Three teaches the reader how to develop an eval-
uation plan and a management plan, including timelines and budgets for conduct-
ing the study. In Part Four, we discuss the methodological choices and decisions
evaluators make: selecting and developing designs; sampling, data collection, and
analysis strategies; interpreting results; and communicating results to others. The
chapters in each of these sections are sequential, representing the order in which
decisions are made or actions are taken in the evaluation study. We make use of
extensive graphics, lists, and examples to illustrate practice to the reader.
This Revision
Each chapter has been revised by considering the most current books, articles, and
reports. Many new references and contemporary examples have been added.
Thus, readers are introduced to current controversies about randomized control
groups and appropriate designs for outcome evaluations, current discussions of
political influences on evaluation policies and practices, research on participative
approaches, discussions of cultural competency and capacity building in organiza-
tions, and new models of evaluation use and views on interpreting and dissemi-
nating results.
We are unabashedly eclectic in our approach to evaluation. We use many
different approaches and methods––whatever is appropriate for the setting––and
encourage you to do the same. We don’t advocate one approach, but instruct you
in many. You will learn about different approaches or theories in Part Two and
different methods of collecting data in Parts Three and Four.
To facilitate learning, we have continued with much the same pedagogical
structure that we have used in past editions. Each chapter presents information on
current and foundational issues in a practical, accessible manner. Tables and
figures are used frequently to summarize or illustrate key points. Each chapter
begins with Orienting Questions to introduce the reader to some of the issues that
will be covered in the chapter and concludes with a list of the Major Concepts and
Theories reviewed in the chapter, Discussion Questions, Application Exercises,
and a list of Suggested Readings on the topics discussed.
Rather than using the case study method from previous editions, we thought
it was time to introduce readers to some real evaluations. Fortunately, while
Blaine Worthen was editor of American Journal of Evaluation,Jody Fitzpatrick wrote
a column in which she interviewed evaluators about a single evaluation they had
conducted. These interviews are now widely used in teaching about evaluation.
We have incorporated them into this new edition by recommending the ones that
illustrate the themes introduced in each chapter. Readers and instructors can
choose either to purchase the book, Evaluation in Action (Fitzpatrick, Christie, &
Mark, 2009), as a case companion to this text or to access many of the interviews
viii Preface
through their original publication in the American Journal of Evaluation.At the end
of each chapter, we describe one to three relevant interviews, citing the chapter in
the book and the original source in the journal.
We hope this book will inspire you to think in a new way about issues—in a
questioning, exploring, evaluative way—and about programs, policy, and organi-
zational change. For those readers who are already evaluators, this book will pro-
vide you with new perspectives and tools for your practice. For those who are new
to evaluation, this book will make you a more informed consumer of or participant
in evaluation studies or, perhaps, guide you to undertake your own evaluation.
Acknowledgments
We would like to thank our colleagues in evaluation for continuing to make this
such an exciting and dynamic field! Our work in each revision of our text has
reminded us of the progress being made in evaluation and the wonderful insights
of our colleagues about evaluation theory and practice. We would also like to
thank Sophia Le, our research assistant, who has worked tirelessly, creatively, and
diligently to bring this manuscript to fruition. We all are grateful to our families
for the interest and pride they have shown in our work and the patience and love
they have demonstrated as we have taken the time to devote to it.
Contents
PART ONE • Introduction to Evaluation 1
1
Evaluation’s Basic Purpose, Uses,
and Conceptual Distinctions 3
Informal versus Formal Evaluation 5
A Brief Definition of Evaluation and Other Key Terms 6
Differences in Evaluation and Research 9
The Purposes of Evaluation 13
Roles and Activities of Professional Evaluators 16
Uses and Objects of Evaluation 18
Some Basic Types of Evaluation 20
Evaluation’s Importance—and Its Limitations 32
2
Origins and Current Trends in Modern
Program Evaluation 38
The History and Influence of Evaluation in Society 38
1990–The Present: History and Current Trends 49
3
Political, Interpersonal, and Ethical Issues
in Evaluation 64
Evaluation and Its Political Context 65
Maintaining Ethical Standards: Considerations,
Issues, and Responsibilities for Evaluators 78
ix