ERIC EJ1104005: Evaluative Thinking: Using Results-Oriented Reasoning to Strengthen Collaboration

2016·0.77 MB·English

by  ERIC

Evaluative
Courtney L.
Malloy, PhD, is Thinking:
director of research at
Vital Research, a research
and evaluation
consulting firm based in Using Results-Oriented
Los Angeles, California.
She has conducted
numerous projects
Reasoning to Strengthen
designed to assess the
effect of interventions on
students and teachers.
Collaboration
Malloy received her PhD
from the Rossier School
of Education at the
University of Southern
California, where she
By Courtney L. Malloy, Janet S. Lee, and Stephanie W. Cawthon
teaches courses focused
on assisting practitioners
in using research to make
meaningful changes to
programs, policies, and
organizations. In today’s world, much of what used to be individual work has
become collaborative. Moreover, complex change initiatives often
Janet S. Lee, EdM,
require individuals within and across organizations to team up to set
a senior research associate
and achieve meaningful goals. Our role as researchers and evaluators
at Vital Research,
participates in every is to offer support that can be used to strengthen the work of
aspect of evaluation, organizations and interdisciplinary teams. We focus on four broad
from design and
categories:
implementation to the
reporting and
1.identifying existing evidence in the field that can be used to
dissemination of
inform the quality of a project,
findings. She holds a
bachelor’s degree in
2.gathering information from stakeholders to identify key needs in
psychology from Pomona
the field,
College, a master’s
degree in education from
3.conducting formative and summative evaluation for programs
Harvard University, and
and initiatives, and
she has completed all but
her dissertation for her
4.offering technical assistance support regarding how to use data.
doctoral degree from the
Graduate School of
Education at the Our work with pepnet 2, in particular, has afforded us several lessons regarding
University of California, the significance of evaluative thinking to collaborative endeavors. Evaluative
Los Angeles.
Photos courtesy of pepnet 2
62 ODYSSEY 2016
Stephanie W.
Cawthon, PhD, an
associate professor in the
Department of
Educational Psychology
at the University of Texas
inL Aeftu: sAt ifnat,h leera adnsd the
Rdesaeuaghrctehr ainnvdo lvEevdi dinence
Sytnhet hDeesaisf Rteoalem M foodrel
pePprongerta m2. leSahren ihso awl stoo
disriegnc tIo lro voef ytohue. Deaf and
Hard of Hearing
InFsatri tleuftt:e Aat m tohteh eMreadows
Celenatrenrs tfhoer sPigrne vfaemnitlying
Edfruocma tai odneaafl rRolies km.o Hdeel.r
background is on access
and equity in deaf
education, with an
emphasis on standardized
assessments used for
high-stakes decision
making. Cawthon holds a
bachelor’s degree and a
master’s degree in
psychology from Stanford
thinking—a process that involves collaborativeincreasingly widespread, it University and a doctoral
systematic results-oriented reasoning—is may be tempting to view endeavors degree from the
essential for collaborations to be successful. undertaken through group cooperation as a University of Wisconsin
Evaluative thinking focuses individuals panacea, the single way to effect complex at Madison.
and their discussion on the outcomes that change. Certainly, many of the persistent
are expected from an endeavor, how those and intractable social problems are more The authors welcome
outcomes might be achieved, the research likely to be solved by the collective action questions and comments
and/or evidence that informs results, and of key individuals with diverse about this article and can
ongoing examination and reflection on perspectives. be reached at
data regarding the progress of the Collaboration, however, is a means to an [email protected],
collaboration. (Patton, 2014). end, not the end itself. Simply bringing [email protected], and
When infused into the culture and together people with common interests to stephanie.cawthon@austin.
activities of a collaboration, evaluative work on a challenging initiative is not a utexas.edu, respectively.
thinking propels individuals forward in failsafe approach. In fact, it is insufficient.
the same direction and increases the Collaboration is more likely to be fruitful
likelihood of success down the road. when team members are able to exercise
evaluative thinking systematically about
Lesson 1—Collaboration Alone the change they want to achieve and
is Insufficient properly ground efforts in evidence.
With terms like cross-sectoral, public-private In our work, we have seen the benefits of
partnership, professional learning community, evaluative thinking as a catalyst for
social network, interdisciplinary, multi- progress towards complex problem solving.
disciplinary, alliance, consortium, and As teams begin to think analytically
2016 ODYSSEY 63
Figure 1: Pepnet 2 Theory of Change
Below is an overview of the theory of change that underlies the work of pepnet 2.
through their purpose and arrive at (Annie E. Casey Foundation, 2004): pepnet 2 is that numerous needs in
common goals, they generate the field should be addressed in
1.Long-term and short-term
momentum, enthusiasm, and greater order to achieve greater
outcomes—The long-term goal for
commitment to work. Moreover, when postsecondary success for deaf and
pepnet 2 is to improve access,
teams have a plan that is rooted in hard of hearing individuals,
opportunity, and postsecondary
evidence and includes indicators to including stronger evidence-based
success for students who are deaf or
monitor progress, they can begin the knowledge and tools, greater
hard of hearing. In order for these
meaningful work that is required for cultural competence, better
long-term changes to occur, short-
change. By contrast, when teams fail to transition and advocacy, and
term changes requiring new
craft a coherent plan for moving forward integrated services.
evidence-based practices, policies,
that is grounded in systematic, results-
and programs are required. (See
oriented evaluative thinking, they often By focusing on select strategies that
Figure 1.)
struggle to move beyond convening and are driven by needs in the field, our
toward collective action. assumption is that short-term outcomes
2.Strategies—In order to accomplish
(i.e., greater knowledge regarding
these short-term outcomes, pepnet
Lesson 2—Begin with a effective transition) will lead to long-
2 engages in several strategies and
Theory of Change term outcomes (i.e., deaf students’
activities, including technical
A theory of change provides a useful success in the postsecondary
assistance, training, research, and
starting point for effective collaboration. environment).
convening.
It explains the principles underlying an Stakeholders who are engaged in
initiative and outlines how desired 3.Explicit assumptions— collaborative change efforts often can
outcomes will be produced. This theory Assumptions about why strategies identify the various strategies and
can come in various shapes and sizes, but will work should be explicit. The activities that they hope to implement.
it generally has three main components assumption underlying the work of However, without a theory of change,
64 ODYSSEY 2016
the desired outcomes and assumptions 1.Impact—Changes in people’s lives, strategies are clarified and connected to
underlying those strategies are not made including knowledge, skills, outcomes, a chain of logical reasoning
explicit, and the connection between behaviors, health, or conditions for should begin to emerge. (See Figure 2.)
strategies and outcomes remains unclear. communities.
Taking the time to generate a theory of Lesson 3—Ground
change ensures a chain of reasoning 2.Influence—Changes in Discussion in Research and
grounding the new initiative and institutions, service systems, Evidence
promotes a shared understanding of how community norms, partnerships, Ideally, a theory of change includes a
to move forward. Moreover, the theory policies, or regulations. clear articulation of assumptions about
can be used to explain to others what the why the proposed activities will lead to
3.Leverage—Changes in public or
collaboration is about and provide the the desired short- and long-term
private funding and resource
foundation to determine how to measure outcomes. Without a critical look at the
allocation.
its progress. available research and evidence base,
When crafting a theory of change, it is decisions about planned strategies and
Once the short- and long-term
often useful to begin with all activities may be based on intuition and
outcomes are mapped out, strategies and
stakeholders at the table and address the individual experiences or even on
activities can be generated that are likely
following questions: What is the group’s political demands. Furthermore,
to lead to the change envisioned. As
desired long-term change?What will look
different after the group’s strategies have been
implemented?After addressing these
questions, teams should work to identify
Figure 2: Chain of Logical Reasoning
the short-term changes required to
accomplish the long-term goal. The Below is an example of a chain of logical reasoning for a potential
Annie E. Casey Foundation (2004), an outcome focused on improving transition.
organization dedicated to helping
disadvantaged children in the United
States, provides a practical guide for
structuring theories of change and
suggests three main types of outcomes:
Taking the time to
generate a theory of
change ensures a
chain of reasoning
grounding the new
initiative and
promotes a shared
understanding of how
to move forward.
2016 ODYSSEY 65
collaborative teams may be A second example of where
drawing individual evidence played a role in the
knowledge from different development of the theory of
perspectives, with resulting change was in the
divergent assumptions articulation of the potential
about why particular short-term outcomes that
strategies may or may not would result from the
be effective. Looking to the proposed activities. For
available research and example, research shows that
evidence is thus an self-determination is
important component in predictive of stronger
developing a shared postsecondary outcomes for
understanding of the students with disabilities
rationale for choosing (Konrad & Test, 2004).
activities. Although this research was
The first step to not conducted with deaf or
reviewing the existing hard of hearing students,
literature is to identify the pepnet 2 made the
evidence that supports the needs of the assumption that fostering students to
project. There are many different kinds lead IEPs might have similar results for
of evidence that may be integrated into a our students. When requested, pepnet
Ongoing reflection
rationale that explains why proposed 2’s Research and Evidence Synthesis
strategies and activities will lead to team provided this kind of information
is critical. This
anticipated short- and long-term to collaborative teams throughout the
outcomes. When developing pepnet 2’s allows collaborators organization.
theory of change, our goal was to
provide both theoretical perspectives to refine strategies Lesson 4—Monitor the
and data. We drew from theoretical Progress
perspectives in the fields of human and to measure Ongoing reflection is critical. This
development, cultural psychology, and allows collaborators to refine strategies
outcomes to
deaf education. We asked what factors and to measure outcomes to determine
were important in understanding effectiveness. A well-defined theory of
determine
potential barriers or supports for deaf change should suggest key indicators
and hard of hearing individuals in that can be used for monitoring and
effectiveness.
achieving their educational, work, and evaluating the collaboration.
personal goals. Data and empirical For example, if teachers are trained to
evidence were important. In our pepnet effectively implement student-led IEPs,
2 work, we synthesized existing then it follows that the two primary
literature on key topics, such as the hearing, and cultural competency is a indicators of progress might be the
effectiveness of accommodations, and we part of creating an open and positive number of trainings conducted and the
gathered information from the field to learning or work environment. Synthesis number of teachers trained. If there is an
answer questions that the extant research of current data was important in laying insufficient number of trainings or if
literature could not provide. the foundation for decisions about future attendance at trainings is low, it is
From a theoretical perspective, the activities or programs. More specifically, unlikely that the changes articulated in
pepnet 2 Research and Evidence we drew upon the current demographic the theory would occur. However, if
Synthesis team felt it was important to data available about current high school those indicators were measured
include cultural competency frameworks completion, college enrollment, frequently and reviewed, stakeholders
in considering access and options for postsecondary persistence, and could examine the data and intervene
individuals who are deaf or hard of employment for individuals who are deaf appropriately before the end of the
hearing. This perspective focuses or hard of hearing. Significant initiative.
attention on building the skills and demographic shifts over the last 10 years It is not sufficient to only examine
attitudes of professionals that serve have occurred, and updating was data related to the strategies and
individuals who are deaf or hard of essential. activities in an initiative; an
66 ODYSSEY 2016
examination of the outcomes is also
warranted to determine whether the
References
collaborative endeavor has produced the
intended changes. Potential indicators in
Annie E. Casey Foundation. (2004). Theory of change: A practical tool for action,
the example above might include pre-
results and learning. Retrieved October 15, 2015, from http://badael.org/wp-
and post-measures of the following
content/uploads/2014/12/aecf_theory_of_change_manual.pdf
outcomes: knowledge about and
confidence in the IEP process of students Konrad, M., & Test, D. W. (2004). Teaching middle school students with
and teachers, the number of student-led disabilities to use an IEP template. Career Development for Exceptional Individuals,
IEPs that are taking place, and the 27, 101-124.
number of IEPs with transition plans
that are aligned to students’ longer-term Patton, M. Q. (2014). Evaluation flash cards: Embedding evaluative thinking in
educational and career goals. organizational culture. Retrieved October 15, 2015, from http://www.ottobremer.
org/sites/default/files/fact-sheets/ OBF_flashcards_201402.pdf
Putting It Together
Taken together, these lessons illustrate
how transparency in planning activities,
naming assumptions behind their agencies with multiple perspectives, purpose of devising or adopting an
effectiveness, and monitoring outcomes diverse training, and a broad range of underlying theory of change and
can assist collaborative teams in the experiences. A collaborative model in combining it with evaluative thinking
development and implementation of effective program and service for the collaboration. Our work with
activities. Preparing individuals who are development therefore requires an pepnet 2 represents an evolving model
deaf or hard of hearing for future anchor that represents the shared beliefs that can provide a platform for those
education and work opportunities often and assumptions about what is being seeking to engage in meaningful
involves the collaboration of people or done, why, and to what end. This is the discussion and collaborative activities.
2016 ODYSSEY 67