ERIC ED522163: Can Teachers Be Evaluated by Their Students' Test Scores? Should They Be? The Use of Value-Added Measures of Teacher Effectiveness in Policy and Practice. Education Policy for Action Series

2010·0.65 MB·English

by  ERIC

EDUCAT IO N POLI CY FOR A CTION SERIES
E D U C A T I O N C H A L L E N G E S F A C I N G N E W Y O R K C I T Y
Can Teachers be Evaluated
by their Students’ Test Scores?
Should They Be?
The Use of Value-Added
Measures of Teacher Effectiveness
in Policy and Practice
Sean P. Corcoran
in collaboration with
Annenberg Institute research staff

EDUCAT IO N POLICY FOR ACTI ON S ER IES
E D U C A T I O N C H A L L E N G E S F A C I N G N E W Y O R K C I T Y
Can Teachers be Evaluated
by their Students’ Test Scores?
Should They Be?
The Use of Value-Added
Measures of Teacher Effectiveness
in Policy and Practice
Sean P. Corcoran
in collaboration with
Annenberg Institute research staff
About the Annenberg Institute for School Reform
TheAnnenbergInstituteforSchoolReformisanational
policy-researchandreform-supportorganization,affiliated
withBrownUniversity,thatfocusesonimprovingcondi-
tionsandoutcomesforallstudentsinurbanpublic
schools,especiallythoseservingdisadvantagedchildren.
TheInstitute’svisionisthetransformationoftraditional
schoolsystemsinto“smarteducationsystems”thatdevelop
andintegratehigh-qualitylearningopportunitiesinall
areasofstudents’lives–atschool,athome,andinthe
community.
TheInstituteconductsresearch;workswithavarietyof
partnerscommittedtoeducationalimprovementtobuild
capacityinschooldistrictsandcommunities;andshares
itsworkthroughprintandWebpublications.Ratherthan
providingaspecificreformdesignormodeltobeimple-
mented,theInstitute’sapproachistoofferanarrayof
toolsandstrategiestohelpdistrictsandcommunities
strengthentheirlocalcapacitytoprovideandsustain
high-qualityeducationforallstudents.
AgoaloftheInstituteistostimulatedebateinthefieldon
mattersofimportantconsequencefornationaleducation
policy.Thisreportprovidesonesuchperspectivebutit
doesnotnecessarilyreflecttheopinionoftheAnnenberg
InstituteforSchoolReform.
AnnenbergInstituteforSchoolReformatBrownUniversity
Box1985
Providence,RhodeIsland02912
233Broadway,Suite720
NewYork,NewYork10279
www.annenberginstitute.org
©2010BrownUniversity
CONTENTS
Figures iv
About the Author v
About the Series vi
Acknowledgments vi
Introduction ............................................................................................................................1
1 Value-Added Measurement: Motivation and Context ................................................1
2 What Is a Teacher’s Value-Added? ..................................................................................4
3 Value-Added in Practice: New York City and Houston ..............................................6
The New York City Teacher Data Initiative 6
The Houston ASPIRE Program 12
4 Challenges to the Practical Implementation of Value-Added .................................14
What is being measured? 14
Is the measurement tool appropriate? 16
Can a teacher’s unique effect be isolated? 18
Who counts? 19
Are value-added scores precise enough to be useful? 21
Is value-added stable from year to year? 26
5 Discussion ............................................................................................................................28
References ...........................................................................................................................29
Appendix A: Race to the Top Definitions of Teacher Effectiveness
and Student Achievement ......................................................................34
Appendix B: Sample New York City Teacher Data Report, 2010 ........................35
Appendix C: Sample New York City Teacher Data Report, 2009 .......................36
FIGURES
Figure 1 Factorsaffectingaverageachievementintwoclassrooms:hypotheticaldecomposition ...............4
Figure 2 Factorsaffectingyear-to-yeartestscoregainsintwoclassrooms:
hypotheticaldecomposition ......................................................................................................................5
Figure 3 NewYorkCityTeacherDataInitiativetimeline ...................................................................................6
Figure 4 NewYorkCityvalue-addedmodel:predictors ...................................................................................10
Figure 5 Teachervalue-addedontworeadingtests:Houstonfourth-andfifth-gradeteachers ...............17
Figure 6 Percentofstudentswithatestscoreandpercentcontributingtovalue-addedestimates,
gradesfourtosix,Houston,1998–2006 .............................................................................................20
Figure 7 Averageconfidenceintervalwidth,NewYorkCityTeacherDataReports,2008-2009 ............23
Figure 8 Percentofoverlappingconfidenceintervals,ELAandmath ...........................................................24
Figure 9 Year-to-yearstabilityinvalue-addedrankings:HISDreadingtest,2000–2006 ..........................26
Figure 10 Year-to-yearstabilityinELAandmathvalue-addedrankings:NewYorkCityTeacherData
Reports,2007-2008 ..................................................................................................................................27
iv CanTeachersbeEvaluatedbytheirStudents’TestScores?ShouldTheyBe?
About the Author Hisrecentpublicationscanbefoundinthe
JournalofPolicyAnalysisandManagement,the
Sean P. Corcoranisanassistantprofessorof JournalofUrbanEconomics,EducationFinance
educationaleconomicsatNewYorkUniver- andPolicy,andtheAmericanEconomicReview.
sity’sSteinhardtSchoolofCulture,Education,
andHumanDevelopment,anaffiliatedfaculty
oftheRobertF.WagnerGraduateSchoolof
PublicService,andaresearchfellowatthe
InstituteforEducationandSocialPolicy
(IESP).Hehasbeenaresearchassociateofthe
EconomicPolicyInstituteinWashington,
D.C.,since2004andwasselectedasaresident
visitingscholarattheRussellSageFoundation
in2005-2006.Inadditiontobeingamember
oftheboardofdirectorsoftheAssociationfor
EducationFinanceandPolicy(formerlythe
AmericanEducationFinanceAssociation),he
servesontheeditorialboardofthejournal
EducationFinanceandPolicy.
Corcoran’sresearchfocusesonthreeareas:
humancapitalintheteachingprofession,edu-
cationfinance,andschoolchoice.Hisrecent
papershaveexaminedlong-runtrendsinthe
qualityofteachers,theimpactofincome
inequalityandcourt-orderedschoolfinance
reformonthelevelandequityofeducation
fundingintheUnitedStates,andthepolitical
economyofschoolchoicereforms.In2009,
heledthefirstevaluationoftheAspiringPrin-
cipalsPrograminNewYorkCity,andheis
currentlyworkingonaretrospectiveassessment
oftheBloomberg-Kleinreformstoschool
choiceandcompetitioninNewYorkCityfor
theAmericanInstitutesforResearch.Heco-
editsabookseriesonalternativeteachercom-
pensationsystemsfortheEconomicPolicy
Institute,andinrecentyearshehasbeeninter-
estedinvalue-addedmeasuresofevaluating
teachereffectiveness,boththeirstatistical
propertiesandtheirobstaclestopractical
implementation.
AnnenbergInstituteforSchoolReform v
About the Series Acknowledgments
Education Policy for Action: Education IthanktheAnnenbergInstituteforSchool
Challenges Facing New York Cityisaseries Reformfortheinvitationtoconductthis
ofresearchandpolicyanalysesbyscholarsin researchandwritethisreport.DeinyaPhenix
fieldssuchaseducation,economics,public wasanimmensehelpfromstarttofinish.
policy,andchildwelfareincollaborationwith NormFruchter,IvonneGarcia,MeganHester,
stafffromtheAnnenbergInstituteforSchool ChristinaMokhtar,andEricZacharyoffered
Reformandmembersofabroadlydefined thoughtfulandconstructivefeedbackatmulti-
educationcommunity.Papersinthisseries plepointsduringtheprocessofwritingand
aretheproductofresearchbasedontheInsti- preparingformyJanuary27,2010,presenta-
tute’slargelibraryoflocalandnationalpublic tion,whichwaspartoftheEducationPolicy
educationdatabases;workwiththeInstitute’s forActionconversationseries.Manyaudience
dataanalysisteam;andquestionsraisedand membersatthiseventofferedinsightful
conclusionsdrawnduringapublicpresenta- thoughtsandcomments,andIwouldparticu-
tionandconversationwithuniversityand larlyliketoexpressmyappreciationtoLeo
publicschoolstudents,teachers,foundation CaseyfromtheUnitedFederationofTeachers
representatives,policyadvocates,education forservingasadiscussant.
reporters,newsanalysts,parents,youth,and
IwouldalsoliketothankJenniferJennings
communityleaders.
andAndrewBeveridgeforsharingdatafrom
Amongtheissuesthattheseriesaddressesare theHoustonIndependentSchoolDistrict,
severalpressingtopicsthathaveemergedfrom RhondaRosenbergandJackieBennettofthe
theInstitute’sresearchandorganizingefforts. UnitedFederationofTeachersforproviding
Someofthetopicscoveredintheseriesare: assistancewiththeNewYorkCityTeacher
•Confrontingtheimpendinggraduationcrisis DataReportdata,andRachelColeforher
•ThesmallschoolsexperimentinNewYork researchassistance.AmyMcIntoshandJoanna
City CannonoftheNewYorkCityDepartmentof
Educationwereinstrumentalinproviding
•Positivebehaviorandstudentsocialand
backgroundontheTeacherDataInitiativeand
emotionalsupport
assistedmeincorrectingfactualerrors.All
•Modesofnewteacherandprincipalinduc-
remainingerrorsaremyown.
tionandevaluation
ManythankstotheRobertSterlingClark
Foundationforitssupportofthepubliccon-
versationsfromwhichthisreportandtheother
publicationsintheseriesgrew.
Foradownloadableversionofthisreportand
moreinformationabouttheseries,pleasevisit
<www.annenberginstitute.org/WeDo/NYC_
Conversations.php>.
vi CanTeachersbeEvaluatedbytheirStudents’TestScores?ShouldTheyBe?
1
Introduction Value-Added Measurement: Motivation
and Context
Value-addedmeasuresofteachereffectiveness
arethecenterpieceofanationalmovementto
Traditionalmeasuresofteacherqualityhave
evaluate,promote,compensate,anddismiss
alwaysbeencloselylinkedwiththosefoundin
teachersbasedinpartontheirstudents’test
teacherpayschedules:yearsofexperience,pro-
results.Federal,state,andlocalpolicy-makers
fessionalcertification,anddegreeattainment.
haveadoptedthesemethodsenmasseinrecent
Asrecentlyasthe2001NoChildLeftBehind
yearsinanattempttoobjectivelyquantify
Act(NCLB),teacherqualitywascommonly
teachingeffectivenessandpromoteandretain
formalizedasasetofminimumqualifications.
teacherswithademonstratedrecordofsuccess.
UnderNCLB,“highlyqualified”teachersof
Attentiontothequalityoftheteachingforce coresubjectsweredefinedasthosewithatleast
makesagreatdealofsense.Nootherschool abachelor’sdegree,astatelicense,anddemon-
resourceissodirectlyandintenselyfocusedon stratedcompetencyinthesubjectmatter
studentlearning,andresearchhasfoundthat taught(e.g.,througharelevantcollegemajor
teacherscananddovarywidelyintheireffec- ormaster’sdegree).
tiveness(e.g.,Rivkin,Hanushek&Kain2005;
However,theseminimumqualificationshave
Nye,Konstantopoulos&Hedges2004;Kane,
notbeenfoundbyresearcherstobestrongly
Rockoff&Staiger2008).1Furthermore,
predictiveofstudentoutcomesonstandardized
teacherqualityhasbeenfoundtovaryacross
tests(e.g.,Goldhaber2008;Hanushek&
schoolsinawaythatsystematicallydisadvan-
Rivkin2006;Kane,Rockoff&Staiger2008).
tagespoor,low-achieving,andraciallyisolated
Knowingthatateacherpossessesateaching
schools(e.g.,Clotfelter,Ladd&Vigdor2005;
certificate,amaster’sdegree,orarelevantcol-
Lankford,Loeb&Wyckoff2002;Boydetal.
legemajoroftentellsuslittleaboutthat
2008).
teacher’slikelihoodofsuccessintheclassroom.
Butquestionsremainastowhethervalue-added Therearemanyreasonsnottototallydismiss
measuresareavalidandappropriatetoolfor
identifyingandenhancingteachereffectiveness.
1This literature is frequently misinterpreted as stating that teacher
Inthisreport,Iaimtoprovideanaccessible
qualityismoreimportantforstudentachievementthananyotherfac-
introductiontothesenewmeasuresofteaching tor,includingfamilybackground.Statementssuchas“Studiesshow
thatteachersarethesinglemostimportantfactordeterminingstu-
qualityandputthemintothebroadercontext dents’ success in school” have appeared in dozens of press
releases and publications in recent years. For an example, see
ofconcernsoverschoolqualityandachieve-
the May 4, 2010 statement from the U.S. House Committee on
mentgaps.UsingNewYorkCity’sTeacher Education and Labor at <http://edlabor.house.gov/newsroom/
2010/05/congress-needs-to-support-teac.shtml>. I know of no
DataInitiativeandHouston’sASPIRE(Acceler- studythatdemonstratesthis.
atingStudentProgress,IncreasingResults&
Expectations)programascasestudies,Iassess
thepotentialforthesemeasurestoimprove
outcomesinurbanschoolsystems.Indoingso,
Ioutlinesomeofthemostimportantchal-
lengesfacingvalue-addedmeasuresinpractice.
thesequalifications,2butcommonsensesug- teachingeffectiveness.Leadersofbothpolitical
geststhisinformationalonecanonlybea partieshavestronglyendorsedlinkingteacher
crudeindicatorfordifferentiatingteaching evaluationtostudenttestscores,andfounda-
effectiveness. tionssuchastheBill&MelindaGatesFoun-
dation,theMilkenFamilyFoundation,andthe
Overthepastfifteenyears,researchonteacher
BroadFoundationhaveprovidedsignificant
qualityhasadoptedanewparadigm:measur-
financialresourcestosupporttheseefforts.In
ingeffectivenessonthebasisofstudentout-
promotingRacetotheTop–PresidentBarack
comes,asopposedtoteacherinputs(e.g.,
Obama’s$4billioncompetitivegrantprogram
Rivkin,Hanushek&Kain2005;Rockoff
aimedatsystemiceducationreform–President
2004;Sanders&Horn1994).Whileout-
Obama(2009)stated,“Successshouldbe
come-basedmeasuresofteachingeffectiveness
measuredbyresults....That’swhyanystate
arenotanewconcept(Murnane&Cohen
thatmakesitunlawfultolinkstudentprogress
1986;Odden&Kelley2002),severalforces
toteacherevaluationwillhavetochangeits
haveconvergedtoreorientthedefinitionof
ways.”ItiswidelybelievedthatRacetothe
teacherqualityaroundstudentachievement.
Topwillserveasatemplateforthereauthoriza-
First,recentpoliciesofhigh-stakesaccountabil-
tionofthefederalElementaryandSecondary
ityhaveincreasedpressureonschoolstomeas-
EducationActincomingyears.
ureanddemonstrateresults.Giventeachers’
closecontactwithstudents,theextension RacetotheTopisquitespecificinitsconcep-
ofhigh-stakesaccountabilitytoindividual tionofan“effectiveteacher”(seeAppendixA).
teacherswasperhapsinevitable.Second,new “Highlyeffectiveteachers”arethosewhosestu-
longitudinaldatasystemsnowexistthatallow dentsachievehighratesofgrowth,defined–
studentachievementtobetrackedovertime narrowly–bytheprogramasachangeintest
andmatchedtoclassroomteachers.3Arguably, scoresbetweentwoormorepointsintime
nocredibletest-score–basedsystemofteacher (U.S.DepartmentofEducation2010).Supple-
evaluationcouldexistintheabsenceof mentalmeasuresareencouraged,asarealterna-
suchsystems.Third,advancementsindata- tivemetricsforteachersofnon-testedgrades
processingcapacityandstatisticalmodeling andsubjects,buttheprimaryemphasisrests
haveyieldedanarrayofvalue-addedtech-
niqueswiththepotentialforisolatingteachers’
2For example, these qualifications can be viewed as minimum
uniquecontributiontostudentoutcomes.
expectationsforclassroomteachers.Evidencethatsomepracticing
teacherswithoutthesequalificationsareaseffectiveasthosewho
Perhapsmostimportantly,politicalandphil- possess them cannot tell us how the wholesale removal of these
minimum requirements would affect the quality of the teaching
anthropicpreferenceshavealignedtobring
force.Moreover,virtuallyalloftheresearchonteacherqualifica-
aboutseismicshiftsinourconceptionof tions has focused on a narrow measure of success: single-year
gainsonstandardizedtests.Weknowlittleabouthowqualifica-
tionsaffectotheroutcomes,suchasorganizationalstability,student
behavior and motivation, aspirations, engagement, persistence,
andthelike.
3TheDataQualityCampaign,foundedbytheBill&MelindaGates
Foundation, tracks and supports state efforts to create these sys-
tems;see<www.dataqualitycampaign.org>.
2 CanTeachersbeEvaluatedbytheirStudents’TestScores?ShouldTheyBe?