ERIC ED513537: The Impact of Incentives on Effort: Teacher Bonuses in North Carolina. Program on Education Policy and Governance Working Papers Series. PEPG 10-06

2010·0.27 MB·English

by  ERIC

Program on Education Policy and Governance Working Papers Series

The Impact of Incentives on Effort:
Teacher Bonuses in North Carolina

Tom Ahn Jake Vigdor
University of Kentucky Duke University

PEPG 10-06

Prepared for the PEPG Conference
Merit Pay: Will It Work? Is It Politically Viable?

Harvard Kennedy School
Cambridge, Massachusetts
June 3-4, 2010

The Impact of Incentives on Effort: Teacher Bonuses in North Carolina*
Tom Ahn
University of Kentucky
Jake Vigdor
Duke University

Abstract
Teacher effort, a critical component of education production, has been largely ignored in
the literature due to measurement difficulties. Using a principal-agent model, North
Carolina public school data, and the state's unique accountability system that rewards
teachers for school-level academic growth, we show that we can distill effort from
teacher absence data and capture its effect on student achievement in a structural
framework. We find that:
1. Incentives lead teachers to try harder. The bonus program reduced the number of sick
days taken by about 0.6 days for an average teacher.
2. When teachers try harder, students do better. Increased effort of teachers translates
into improved student performance. Estimates show that standardized reading scores
increased by about 1.3% of a standard deviation and standardized math scores by about
0.9% of a standard deviation.
3. Group-level incentives can actually be more powerful than individual-level incentives.
Policy simulations from the model estimates show that an individual bonus program
would actually produce weaker incentive effects. While free-rider effects are eliminated,
individual incentives push a majority of teachers into one of two categories: those who
would qualify for the bonus even without trying and others would not qualify no matter
how hard they worked.

Performance pay for teachers: are school-level incentives enough?

Over the past ten years, researchers have devoted considerable effort to the measurement
of the output of schools and teachers, using standardized test scores.1 Our ability to infer
the quality of teaching in a school or classroom has developed sufficiently far that school

* We grateful to the Center for Child and Family Policy for access to the NCDPI dataset and to the
American Enterprise Institute for financial support.
1 See Rivkin, Hanushek, and Kain (2005), Clotfelter, Ladd,and Vigdor (2007), Goldhaber and Anthony
(2007), and Rockoff (2004), among many others.
districts across the nation, from Denver to Washington DC and many points in between,
have put incentive programs in place that make student test score performance a major
factor in the evaluation, and in some cases the compensation, of teachers.

There are some sticky issues, however, about how to really make a pay-for-performance
scheme based on test scores operate.2 For one thing, most school systems don’t conduct
standardized tests in every grade, or in every subject. How exactly are we supposed to
measure the performance of a kindergarten teacher? Or a high school Spanish teacher?
Or a middle school physical education teacher? And if we don't evaluate these teachers'
performances, how do we pay them?

A second issue with pay-for-performance schemes is that they may lead teachers to fight
amongst each other for the best students. Teachers might perceive that certain students
hit performance targets more easily; in many cases there is strong evidence to back up
this perception. Principals and other administrators might come under pressure to fiddle
with classroom assignments. In theory, a commitment to randomly assign students to
classrooms would present teachers with a level playing field. But even randomization
makes some people lucky and others unlucky sometimes. To complicate matters, many
schools have instituted different tracks for students with differing academic abilities. Pay-
for-performance would add a financial reason to lobby to teach honors-class students in
addition to the laundry list of non-pecuniary benefits.

2 See Neal (2009) for a review of some of the inefficiencies of the pay-for-performance system.
Another problem is the “noisy" test results issue. The “noise” we are talking about here is
statistical noise, and it is a more severe problem for classroom-sized groups of students
relative to school-sized groups. A “noisier” test result makes it difficult to discern
whether a higher average score for a group of students relative to another group really
means that the former group of students knows more about the subject they were tested
on. The smaller the number of test takers, the higher the possibility that one outlying
result can throw off the entire result. For instance, if one or two students were ill on the
day they took the test, their poor exam scores will significantly drag down the class
average and make the teacher look less competent than she really is. These unfortunate
students would have a smaller impact on the school average, since they make a smaller
proportion of the larger student body.

These three Gordian knots – the presence of untested grades and subjects, non-random
assignment of students to teachers, and the statistical noise problem in small samples –
could be sliced with one modification to pay-for-performance: reward teachers on the
basis of all students in the school, rather than just those in their classroom. With school-
based incentives, we need not worry about what to do with teachers of odd subjects, or in
untested grades. And we need not worry about teachers fighting one another in a zero-
sum game, or about statistical noise leading to good teachers going unrewarded.

The primary theoretical argument against school-based rewards will be familiar to
economics 101 students everywhere: the free-rider problem. Compared with an
individual-level incentive, a group-level incentive should have less impact. When your
own effort determines my compensation, you have a very strong reason to work hard.
When the combined effort of a large group determines your compensation, you may feel
at greater liberty to slack off, since most of your reward depends on the actions of other
people anyway. It’s the tragedy of the commons, the prisoner’s dilemma – whatever you
want to call it. It’s an argument so strong and so intuitive that you don’t hear many
education economists saying that school-level rewards are the way to go.

Until now. New evidence, derived from the experiences of North Carolina public schools,
which have implemented school-based monetary incentives for more than a decade now,
indicates that this conventional wisdom – that individual incentives are more powerful
than group incentives – is in fact wrong. Yes, Virginia, there is a free rider effect. But
what the conventional wisdom fails to incorporate is a powerful countervailing effect,
which we might call the “tortoise and hare” effect, borrowing from Aesop’s fables.
Consider the following scenario. You are an excellent teacher – one of the best in the
business. If the school system sets a bar and promises you rewards if your students
exceed it, you know you can exceed the expectation even without trying. Like the hare in
the fable, your incentive to try your best is undermined by a sense that your success is
inevitable. We may fault the hare for his laziness, but is this really such a surprising
response when victory seems assured?

The teacher next door, on the other hand, is hopelessly incompetent. You know as well
as she does that no matter where the bar is set, her students will almost certainly fall
below it. Like the tortoise in the fable, it is only her personal virtue that implores her to
exert effort: the incentive means very little. Again, we may cheer the tortoise for his
perseverance, but how wise is it to expend effort when there is virtually no chance of
success? So, for both you and your neighbor, the individual-level incentive scheme
provides almost no incentive to exert greater effort. You are bound to be rewarded no
matter what, and your neighbor is destined to fail no matter what. It would be great if
both the tortoise and the hare tried their hardest regardless of the competition, but the
most likely outcome is that the hare would win walking backwards, and the tortoise
would quit before the race even begins.

Now suppose we tie you and your neighbor together: your reward will be based not on
what you do individually, but the sum total of what you accomplish. All of a sudden, you
recognize that the status of your reward is in doubt, and the teacher next door realizes that
she now has a realistic shot at the reward; both you and your neighbor are going to have
to exert some effort to ensure that the average across your two classrooms exceeds the
standard.

While the traditional moral of the fable is that “slow and steady wins the race,” perhaps
we should reconsider the wisdom of such a match-up in the first place. Rather than race
tortoises against hares, we should pair one of each together and judge each pair by their
combined time. In this scenario, each competitor faces a stronger incentive to excel,
because it is their team’s average time that matters, not their rank within the team.
This is a plausible scenario, right? But we just don’t know how common this scenario
might be. How often do very good and very poor teachers share the same school? And
just how powerful is this free rider effect anyway? The answers lie in the research. To
tell you about the research, we first need to spend some time getting to know the setting.

The North Carolina State Accountability System

The North Carolina ABC accountability program (ABC is an acronym for Accountability,
teaching the Basics, and emphasis on local Control) began in the 1996/97 school year. In
its inaugural year, teachers in elementary and middle schools were awarded a cash bonus
of $1,000 if the school’s average year-over-year improvement in reading and math test
scores exceeded the required threshold set by the state. In the following year, the bonus
program was extended to high schools, and the award became two-tiered, with teachers
receiving $750 in schools that cleared a first threshold referred to as “expected” growth in
test scores and $1,500 in schools that cleared a more stringent “exemplary” or “high”
growth threshold. 3

Education authorities face a delicate balancing act in setting criteria for bonus payments.
If teachers perceive that there is no chance of receiving a bonus, or conversely that the
bonus is a sure thing, they have little reason to alter their behavior. This is a basic
statement of the “tortoise and hare” effect described above. Fortunately, in North
Carolina’s case, teachers in most schools face real uncertainty about the amount of their

3 A complete description of the bonus program and the formulas used to set each school’s threshold can be
found in Vigdor (2008).
bonus. Figure 1 shows the proportion of schools in the 1999/00 to 2001/02 school years
qualifying for $750 or $1500 bonuses. Roughly three-quarters of the schools in the state
received bonus payments, but less than half received the full $1,500. The average bonus
paid out is roughly $890 (0.23 X $0 + 0.35 X $750 + 0.42 X $1,500 = $890). Vigdor
(2009) presents additional evidence that among the schools eligible for any bonus at all,
about half receive the full $1,500. There are very few schools that can count on the full
$1,500 as a sure thing, and very few for which the $750 standard is completely
unattainable.

Figure 1: Bonus Receipt by School
No Bonus
23%
$1,500 Bonus
42%
$750 Bonus
35%

Incidentally, North Carolina’s system is made possible because the state has a
longitudinal data system that can link the performance of individual students as they
progress from grade 3 to grade 8. Many other states, unfortunately, have no capacity to
link students across years, implying that they can only judge schools by how the students
perform in a given year, not by how much they improve in a given year. This limitation
forced the federal No Child Left Behind act to focus on proficiency rather than
improvement. Why does this matter? A school that serves very low-performing kids and
manages to improve their performance dramatically might not be rewarded if their
ultimate performance is below the state’s threshold for proficiency.

Figuring out what the bonus program accomplishes

The North Carolina ABC system is not costless. Their state legislature needs to allocate
90 million dollars or more per year for these performance bonuses. And while there’s a
strong economic reason for thinking that performance bonuses improve student
performance, there’s no specific guidance regarding how big the impact should be, let
alone whether the impact is worth the amount of money being spent on the program.4

So is the program worthwhile? How can we tell? The gold-standard method of evaluating
a program such as the ABC initiative would have been to conduct a randomized trial.
Schools in North Carolina would have been randomly assigned into two groups: a
“treatment group” of schools where teachers were awarded the bonus according to the
ABC framework and a “control group” where teachers did not receive the bonus. If the
incentives worked as planned, teachers in the treatment group would have exerted higher
effort to teach students, and this would have translated into higher scores for students in
treatment schools relative to those in control schools.

4 See Figlio and Kenny (2007).