Evaluation in the Face of Uncertainty: Anticipating - Jonny Morell

46 Pages·2010·0.78 MB·English

Evaluation in the Face of Uncertainty:
Anticipating Surprise and Responding to the
Inevitable
AEA/CDC Summer Evaluation Institute
June 13 – 16, 2010
Atlanta GA
Jonathan A. Morell, Ph.D.
[email protected]
[phone]
© 2010 Guilford Publications
© 2010 Jonathan Morell
What is this workshop about?
Response to surprise Adding “surprise” to evaluation
planning
 Crisis response  advance
planning  Funding
Disseminating knowledge  Deadlines
 Tactics for adding surprise to the  Logic models
evaluation mix
 Measurement
Community building
 Program theory
 More and better tactics  Research design
 More and better theory  Information use plans
 Archive of cases  Defining role of evaluator
 Logistics of implementation
 Planning to anticipate and
respond to surprise
In this workshop we will go heavy on tricks and tips, light on theory,
explanation, or analysis of collected cases.
© 2010 Jonathan Morell 2
The goal is informed commitment to practical action
 When is the likelihood of surprise high?
 When will surprise disrupt evaluation?
 If probability of disruption is high, what can we do about it?
1
Experience
Design in
2
practice
Theory
3
 No formula but theory and experience help
 No magic bullet but we can chip away at the
∞
problem
 Many choices, one actual
design
 All have pros and cons
 Tradeoffs are inescapable
© 2010 Jonathan Morell 3
Some historical background
We know why unexpected events But what to do about it as evaluators?
occur
Evaluation
 Goal free evaluation emphasizes what a
?
program does, not what it claims
 Interactivity between evaluation and the
program being evaluated
Explanations embedded in domain
 Marketing, education, drinking
regulation, tobacco control, product
development, welfare, and many others,
I have no doubt.
Guaranteed solution
 Post-test only
Complex systems
 Experimental group only
 Uncertain environments, cross linkages,
self organization, adaptation, feedback  Unstructured data collection
loops with different latencies, etc.
But we want to do a lot better
© 2010 Jonathan Morell 4
You can never tell the future but some surprises are more
foreseeable than others
Foreseeable Unforeseeable
· Get lucky · Complex system behavior
· Knowledge from stakeholders makes prediction impossible no
· Good program theory matter how clever we are.
· Use research literature PS – do not assume that complex
· Use experts systems are always unpredictable!
Theory
Limiting time frames
Exploiting past experience
Forecasting & program monitoring
System based logic modeling
Retooling program theory
Agile methodology
Data choices
© 2010 Jonathan Morell 5
Jonny’s favorite metaphor
We don’t know exactly where the cats are but we can sweep them
toward one side of the landscape, and tame the one’s that escape.
© 2010 Jonathan Morell 6
Programs and their evaluations have an essential
similarity
 What will help us with unexpected program outcomes
will also
 Help us with unexpected problems in conducting an evaluation
because
 Both are similar social constructions
 Resources (time, people, $)
 Processes
 Embedded in a social setting
 To accomplish specific objectives
© 2010 Jonathan Morell 7
What are the practical and political reasons for
surprise?
 Any single organization has limited
If people are smart enough
money, political capital, human
to know that the world
capital, authority and power
looks like this
 Narrow windows of opportunity
Why are they forced
 Competition requires bold claims
to design programs
 Resource owners have parochial
like this?
interests
 Design expertise limited
Internal program Outcomes Internal program Outcomes
operations operations
 Collaboration across agency
boundaries is very difficult
 Short term success is rewarded
 Partial solutions can accrue to
major success over time
 Pursuing limited success with
Program limited resources is justifiable.
Objective
Goal
Result
 Narrow programs
 Simple program theories
© 2010 Guilford Publications  Small set of outcomes
Planners may know better but they are doing the best job they can. Evaluators
have to follow.
8
What might an unforeseen but predictable outcome
look like?
Program Innovation Results
 Post-natal care in Niger  NGO provides drugs and Patients: drug hoarding (patients
supplies learned from previous programs)
 Formal fees
 Informal fees integrated
into (hidden in) overall fee
 Remove fees Staff: game system, new fees
structure
 Experience with similar programs Something like this will happen, even
if we can’t say exactly what.
 Psychology of self interest
 Common sense
© 2010 Jonathan Morell 9
What might unforeseeable outcomes look like?
The problem is not sensitive to scale. We run into the same trouble with large
and small problems.
If you built a logic model here Would it be valid here?
Tutoring Dept. of Tutoring Textbook Charter Dept. of
services Ed. services publishers schools Ed.
Textbook Dept. of Reform Textbook
Unions Legislators Legislators Unions Legislators
publishers Ed. advocates publishers
Reform Charter Charter Reform Tutoring
Unions
advocates schools schools advocates services
Dept. of Ed. Dept. of Ed. Reform
Legislators Legislators Legislators
Tutoring School
Charter Textbook Charter
Textbook Reform Tutoring
Unions Unions Charter Unions Tutoring
Reform Reform Dept. of Ed. Textbook
Health Agenda of Family Health Agenda of Family
measures? other NGOs structure? measures? other NGOs structure?
Similar Replace lost Government Niger
Similar
programs? income? policy? Government
programs Replace lost
policy?
income?
Fluctuating Environments
© 2010 Jonathan Morell 10