ERIC EJ1148461: Reciprocal Accountability for Transformative Change: New Hampshire's Performance Assessment of Competency Education

2017·0.32 MB·English

by  ERIC

Reciprocal Accountability for Transformative
Change: New Hampshire’s Performance
Assessment of Competency Education
Scott F. Marion, Jonathan Vander Els, and Paul Leather
In New Hampshire, a new perfor- For every increment of performance
mance assessment system focuses on I demand from you, I have an equal
responsibility to provide you with the
reciprocal accountability and shared
capacity to meet that expectation.
leadership among teachers and leaders
Likewise, for every investment you
at the school, district, and state levels.
make in my skill and knowledge,
I have a reciprocal responsibility to
demonstrate some new increment in
performance. (Elmore 2002, p. 5)
Scott F. Marion is the executive director of the National Center for the Improvement of Educational
Assessment in Dover, New Hampshire. Jonathan Vander Els is the executive director of the New
Hampshire Learning Initiative. Paul Leather is deputy commissioner of the New Hampshire Depart-
ment of Education.
20 Annenberg Institute for School Reform
This concept of reciprocal measures, and bars set for proficiency.
accountability, developed by Assessment of competency-based
school improvement expert learning almost always requires
Richard Elmore, is at the core of New performance-based assessment, and the
Hampshire’s Performance Assessment of information learned through this
Competency Education (PACE), a process will continue to inform the
competency-based educational ap- design of the accountability system and,
proach designed to ensure that students hopefully, better inform school improve-
have meaningful opportunities to ment (Hargreaves & Braun 2013).
achieve critical knowledge and skills
PACE involves multiple lines of work
(see Marion & Leather 2015; Rothman
and multiple players. Here, we use three
& Marion 2016; New Hampshire
specific perspectives to provide tangible
Department of Education 2016). For
examples of reciprocal accountability in
PACE, reciprocal accountability means
action:
that local educational leaders are
involved in designing and implementing • The first example – of shared
the assessment and accountability leadership – is presented by Paul
systems and receive intense technical, Leather, New Hampshire’s deputy
policy, and practical support and commissioner of education, who as
guidance from the New Hampshire the official leader of the project had
Department of Education (NHDOE) to build a structure based on shared
and other experts in the field. PACE decision making among the state,
attempts to foster organizational districts, and external partners.
learning and change by appealing to the • The second story – of building local
intrinsic motivation of adults to capacity and expertise – is told by
improve their work rather than relying Jonathan Vander Els, the current
on top-down accountability and executive director of the New
compliance strategies. Hampshire Learning Initiative and
former principal of Memorial
Beginning in 2012, New Hampshire
Elementary School in Sanborn
worked with the Center for Collabora-
Regional School District, one of the
tive Education (CCE) to implement
original PACE districts.
performance assessment literacy
• The last example is presented by
training, using professional develop-
Scott Marion, executive director of
ment and capacity building to lay the
the Center for Assessment and the
groundwork for moving forward. In
lead technical advisor to PACE. He
March 2015, the U.S. Department of
discusses the ways in which the
Education granted permission to New
evaluation of technical quality of the
Hampshire and their advisors from the
PACE assessment system is based on
National Center for Improvement of
the reciprocal notion of supporting
Educational Assessment (Center for
expertise among local educators
Assessment) to pilot PACE, a new
while meeting rigorous psychometric
assessment and accountability system
requirements.
with significantly greater levels of local
design and agency, with an overall goal
to facilitate transformational change in THE VIEW FROM THE STATE:
performance that best supports the goal SHARED LEADERSHIP AND
of significant improvements in college
RECIPROCAL ACCOUNTABILITY
and career readiness.
(PAUL LEATHER)
As part of this shift in orientation, the
Under former Commissioner Virginia
state is supporting a competency-based
Barry’s leadership, the NHDOE has
approach to instruction, learning, and
long practiced reciprocal or “shared
assessment within an internally oriented
leadership” for the major decisions in
accountability model, in which those
our state’s public education. Barry met
being held accountable have responsibil-
with the district superintendents and
ity for co-developing the standards,
Scott F. Marion, Jonathan Vander Els, and Paul Leather VUE 2017, no. 46 21
other educational leadership groups four participating districts, two external
monthly to discuss major issues such as partners (Scott Marion of the Center for
educator effectiveness, educational Assessment and Dan French of CCE),
innovative practices, and the opioid and NHDOE staff (Deputy Commis-
crises. In particular, shared leadership sioner Paul Leather and PACE State
discussions have addressed assessment Director Mariane Gfroerer). Originally,
and accountability for many years, from this group met at least monthly to
the adoption of the Smarter Balanced address all of the issues of design,
Assessment Consortium1 in 2014 to the planning, professional development,
design of state accountability systems implementation, reporting, and techni-
since the onset of No Child Left Behind cal quality. Nothing moved forward
in 2002. It was at just such a discussion, without the full consensus of the group.
held within the confines of the state’s
Now in its third year, the pilot has
Accountability Task Force in 2014,
grown to eight districts and one charter
where the idea for PACE was born.
school, and the makeup of the leader-
The task force, made up of superinten- ship team remains the same, with each
dents, curriculum supervisors, teachers, district or charter school represented at
and association chapter directors, the table. Meanwhile, consistent with
discussed the idea of moving to a new the principles of reciprocal accountabil-
kind of accountability system more in ity, the field leaders and teachers have
keeping with competency-based taken on more and more of the ongoing
education. Chris Rath, then superinten- work of PACE. Eighteen teacher content
dent of the Concord School District, leaders now facilitate the construction
said in no uncertain terms, “We can’t of new common PACE performance
take on something this innovative assessment tasks in English language
without you providing us some space to arts, math, and science for grades 3–7
innovate. With the Common Core, and 9–10.
Smarter Balanced, and other efforts all
With the NHDOE’s support, a new
being implemented this year [2014-
organization has been constructed: the
2015], our educators are overburdened
NH Learning Initiative, which serves as
as it is.” After some discussion, the
an intermediary entity supporting the
group agreed with the idea of advancing
work of both the field and the Depart-
a pilot to include volunteer districts,
ment. Also, the New Hampshire chapter
where Smarter Balanced would be
of the National Education Association
implemented only once each in elemen-
is supporting another group of teacher
tary, middle, and high school, and a
leaders to facilitate PACE implementa-
bank of complex performance tasks
tion with fellow educators within and
would be used in grades and subjects
across districts. All of this work is
where Smarter Balanced was not
overseen by the PACE leadership team,
administered. In this way, the idea of
which continues to meet monthly.
“space to innovate” was integrated into
Members demonstrate their shared
New Hampshire’s accountability system.
ownership and commitment to the
This model of shared decision making success of the pilot in many ways,
became the operational norm for PACE. including through presentations at
A roundtable was created, made up of district, state, and national conferences
field representatives from the original and to state government officials.
1 Smarter Balanced and the Partnership for RECIPROCAL ACCOUNTABILITY
Assessment of Readiness for College and AT THE SCHOOL LEVEL
Careers (PARCC) are assessment systems
(JONATHAN VANDER ELS)
that were developed through collaborations
between groups of states and educators in
When I served as a principal in one of
response to new, more rigorous Common
the original implementing PACE schools,
Core academic standards adopted by most
states in 2010 and 2011. See http://www. reciprocal accountability was at the core
smarterbalanced.org/ and http://www.
parcconline.org/.
22 Annenberg Institute for School Reform
of our vision ensuring that all students accountability effort that was not based
achieve at high levels. I and my teachers on a single, standardized measure to
subscribed to a shared leadership model evaluate students and schools.
in which we were together responsible
Teachers’ capacity and professionalism
for the success of our students, and we
are at the heart of PACE. Relying on
needed to work collaboratively to truly
teacher leadership and autonomy to be
maximize the strength of the whole
“in charge” of the project has put
school.
teachers back into the driver’s seat,
In order for PACE to be effective, the determining students’ competency and
capacity of all educators in each of the utilizing the data from the performance
implementing schools must be devel- assessments to provide support, inter-
oped to the fullest extent possible. vention, and extension, as appropriate,
Teachers must possess deep understand- in a timely manner. For teachers, the
ing of content, discipline-specific essence of reciprocal accountability is a
pedagogy, and well-developed assess- sense of “being heard.” As one of our
ment literacy to teach and assess a lead PACE teachers explained:
rigorous curriculum using complex
I think PACE has been successful so
performance tasks. Teachers must also
far because the people working on the
be willing and able to work collabora-
initiative believe in the work. The
tively in and across schools to develop
people in charge listen to teacher
shared expectations and vision.
feedback and are adaptable. We all
We worked hard to develop a culture in understand the importance of the
which it was safe to innovate. Teachers work and want it to
were used to (and comfortable with) be successful because it’s what is best
working either individually or within for kids.
their school-based team. PACE required
We all have a role to play in the success
teachers across schools and districts to
of PACE, and all clearly understand the
function in a professional learning
need to work with, and for, each other
community, through which they learned
to support our students.
how to work together most effectively,
how to look at student work, under-
stand data, and most importantly, make
A RECIPROCAL ACCOUNTABILITY
changes to their instruction to meet the
APPROACH TO EVALUATING
needs of all learners. Our teachers’ role
was to embrace the uncertainty that TECHNICAL QUALITY
comes with stepping out of their (SCOTT MARION)
comfort zones, committing to working
collaboratively with colleagues, and PACE has been recognized for its
sharing our learning to benefit all. multifaceted approach to the evaluation
of technical quality. (See, for example,
PACE came along at the right time for Lyons & Evans, forthcoming; Rothman
our school and our district. We had & Marion 2016.) In most cases, techni-
transitioned to “competency-based cal quality evaluations are the purview
learning” a few years earlier, but our of highly trained psychometricians like
teachers really began to develop their those of us who work at the Center for
assessment literacy by creating, adminis- Assessment. PACE leadership has always
tering, and refining Quality Performance had a goal of ensuring that only
Assessments, a professional development high-quality assessments were used in
opportunity provided by CCE and participating schools, but we insisted
initially made available over the summer from the beginning of the project that
by the NHDOE. Because we were technical quality had to be a participa-
already engaged in developing high- tory sport. In other words, the
quality performance assessments, PACE evaluations of technical quality had to
was a logical and timely opportunity to both gauge the quality of the assess-
participate in an assessment and ments used and to increase the
Scott F. Marion, Jonathan Vander Els, and Paul Leather VUE 2017, no. 46 23
assessment expertise of participating Reliable and accurate scoring
educators. While there are many aspects
Performance assessments must be
of our shared approach to evaluate
scored accurately and consistently in
assessment system quality, we highlight
order to support their uses to inform
three key components here.
instruction and to serve as accountabil-
High-quality assessment design ity measures. Further, a key tenet of
PACE is that inferences regarding
Assessment quality starts with prin-
student achievement must be compa-
cipled and high-quality assessment
rable across participating districts and
design. The assessment design templates
between pilot and non-pilot districts,
were drafted by staff at the Center for
meaning that given a certain set of
Assessment, but revised based on
student work, a student rated as
feedback and interaction with partici-
“proficient” in one district would be
pating teachers. The Center for
rated similarly by educators in a
Assessment team provides technical
different district.
support and some oversight to the
teacher-led task development teams, but Ensuring scoring quality and compara-
the decisions about which assessments bility starts at the school and district
are used in the project are made levels, where participating PACE
collaboratively among the teacher schools engage in calibration exercises
leaders, project staff, and the technical to develop a shared understanding of
consultants. The teachers lead the student work quality. The PACE
choice of the activity that will anchor calibration protocol was developed and
the performance task, as well as every tested collaboratively among my staff,
step of the task design, including PACE teachers, and PACE district leads.
This process was another example
where more top-down technical quality
“ approaches had to be negotiated with
the practical realities of doing this work
with teachers who have many other
“
responsibilities. For example, we would
Teachers lead the choice of the activity that
have liked to have larger samples of
student work for our calibration work,
will anchor the performance task, as well
but that would have been a burden on
the teachers, so we negotiated a sample
as every step of the task design, including
size that is manageable for the teachers
but still provides enough data for us to
drafting the rubric that will be used to
conduct the necessary technical analyses.
In addition to the internal calibration
score the task.
work, each district collects data on the
degree to which teachers score the
performance tasks consistently with
other teachers in the district. The Center
for Assessment uses these data to
drafting the rubric that will be used to compute inter-rater consistency statis-
score the task. Teachers suggest ways in tics and then reports back to districts so
which the task or tasks will work best they can use the information to improve
within their instructional programs and their scoring quality.
together with the technical advisors
negotiate among district content experts
Comparability of assessment results
and the technical advisors to design
across participating districts
tasks that can serve both instructional
and accountability purposes.
The key activity in evaluating cross-
district comparability involves a massive
collaborative effort led by my psycho-
metric staff and involving hundreds of
24 Annenberg Institute for School Reform
educators and project leaders with the performance assessments for account-
main event taking place over the course ability can be satisfactorily addressed
of two days each summer. Anonymized (Evans & Lyons 2017). PACE provides
student papers are distributed to a vivid example of reciprocal account-
randomly arranged teams of teachers to ability in action, framing the ways in
produce “consensus scores.” These which PACE operates at all levels –
consensus scores serve as benchmarks from the NHDOE, to the approaches
by which local district scoring is for evaluating and improving technical
evaluated. (Out of more than 400 quality of performance assessments, to
papers scored, fewer than five each year the collaboration among teachers, to the
required a third rater to help the interactions between teachers and
original raters come to consensus.) students.
Ideally, there should be only small
differences between the consensus scores REFERENCES
and the scores provided by the original
Elmore, R. 2002. Bridging the Gap
teacher. This alignment would indicate a
between Standards and Achievement: The
high degree of scoring accuracy. The
Imperative for Professional Development
more immediate concern is to ensure
in Education. Washington, DC: Albert
that the average differences between
Shanker Institute.
each district’s local scores and the
consensus scoring are similar across Evans, C., and S. Lyons. 2017. “Compara-
districts. The extent to which a district bility in Innovative Assessment Systems
deviates from other districts is a for State Accountability,” Educational
measure of leniency or stringency in Measurement: Issues and Practice. 36,
local scoring (see Queensland Curricu- no. 1:1–11.
lum & Assessment Authority 2014).
Hargreaves, A., and H. Braun. 2013.
We could have chosen to employ a more Data-Driven Improvement and Account-
ability. Boulder, CO: National Education
typical statistically based approach to
Policy Center. https://goo.gl/gRYXhk.
comparability, but that would have been
more top-down and would have done Marion, S., and P. Leather. 2015. “Assess-
little to build the skills of participating ment and Accountability to Support
teachers. The approach we designed Meaningful Learning,” Education Policy
allows teachers to collaboratively Analysis Archives 23, no. 9. https://goo.gl/
interrogate student work and to have bvASfq.
their consensus judgments play a crucial
New Hampshire Department of Education.
role in the comparability evaluations.
2016. Moving from Good to Great in New
Further, this close examination of
Hampshire: Performance Assessment of
student work allows teachers to build
Competency Education (PACE). Concord:
their assessment literacy and under-
New Hampshire Department of Education.
standing of student learning.
https://goo.gl/fPU3hP.
Queensland Curriculum & Assessment
CONCLUSION
Authority. 2014. School-Based Assessment:
The Queensland System. Queensland,
An innovative assessment and account-
Australia: The State of Queensland
ability project like PACE is unique and
(Queensland Curriculum & Assesment
important for many reasons. The
Authority).
extensive use of performance assess-
ments helps support learning (Shepard Rothman, R., and S. F. Marion. 2016.
2000) and increases teacher assessment “The Next Generation of State Assessment
literacy. The focus on high-quality and Accountability,” Phi Delta Kappan 97,
performance tasks is something we have no. 8:34–7.
not seen on a large-scale since initiatives
Shepard, L. A. 2000. “The Role of
in several states in the 1990s. PACE
Assessment in a Learning Culture,”
seeks to demonstrate that some of the
Educational Researcher 29, no. 7:4–14.
past technical concerns with the use of
Scott F. Marion, Jonathan Vander Els, and Paul Leather VUE 2017, no. 46 25