ERIC ED504708: The Accountability Illusion: Vermont

2009·0.26 MB·English

by  ERIC

Vermont
Executive Summary which are fairly rigorous (roughly 87 percent of V
Vermont’s grade 3-8 students are expected to be e
The intent of the No Child Left Behind (NCLB) Act r
proficientinreadingin2008). m
of 2001 is to hold schools accountable for ensuring
o
that all their students achieve mastery in reading and (cid:1) Looking across the 28 state accountability systems n
math,withaparticularfocusongroupsthathavetra- examined in the study, we find Vermont at about t
ditionally been left behind. Under NCLB, states sub- themiddleofthedistributionintermsofthenum-
mit accountability plans to the U.S. Department of ber of elementary sample schools making AYP.
Educationdetailingtherulesandpoliciestobeusedin Specifically,itexceedsfifteenstatesandtieswithfour
trackingtheadequateyearlyprogress(AYP)ofschools others(SouthCarolina,Montana,FloridaandNew
toward these goals. Jersey)(SeeFigure1).
(cid:1) Some of the schools in our sample that failed to
This report examines Vermont’s NCLB accountability
makeAYPinVermontaremeetingexpectedtargets
system—particularly how its various rules, criteria and
for their overall populations but failing because of
practices result in schools either making AYP—or not
theperformanceofindividualsubgroups.2
makingAYP.ItalsogaugeshowtoughVermont’ssystem
iscomparedwithotherstates.Forthisstudy,weselected (cid:1) In Vermont, as in most states, schools with fewer
36schoolsfromaroundthenation,schoolsthatvaryby subgroupsattainAYPmoreeasilythanschoolswith
size, achievement, and diversity, among other factors, more subgroups, even when their average student
and determined whether or not each would make AYP performanceismuchlower.Inotherwords,schools
underVermont’ssystemaswellasunderthesystemsof
27otherstates.Weusedschooldataandproficiencycut
score1estimatesfromacademicyear2005–2006,butap-
plied them against Vermont’s AYP rules for academic
year2007–2008(shortenedto“2008”inthisreport).
Fifteenof18elementaryschoolsand17of18middle
schoolsinoursamplefailtomakeAYPin2008under
Herearesomekeyfindings:
Vermont’saccountabilitysystem.Thisplaces
(cid:1) We estimate that 15 of 18 elementary schools and Vermontataboutthemiddleofthestatedistribution
17of18middleschoolsinoursamplefailtomake intermsofthenumberofschoolsmakingAYP.
adequate yearly progress in 2008 under Vermont’s Vermont’sproficiencystandardsareaboutaverage
accountabilitysystem.Thishighfailurerateispartly
comparedtootherstates,butitsannualtargetsare
explained by our sample, which intentionally in-
fairlyrigorous(roughly87percentofgrade3-8
cludes some schools with a relatively large popula-
studentsareexpectedtobeproficientinreadingin
tionoflow-performingstudents.Butit’salsopartly
explainedbyVermont’sannualproficiencytargets, 2008).Unlikemoststates,Vermontmeasuresits
studentperformancewithaproficiencyindex,which
givespartialcreditforstudentsachieving“partial
1 A cut score is the minimum score a student must receive on
proficiency.”Intheshortterm,theindexmakesit
NWEA’sMeasuresofAcademicProgress(MAP)thatisequivalentto
performingproficientontheNewEnglandCommonAssessment
easierforVermontschoolstomeettheirtargets,but
Program(NECAP).
2It’simportanttonotethatstudentsinsubgroupsnotmeetingthe theeffectoftheindexdiminishesasthetargets
minimumnsizesarestillincludedforaccountabilitypurposesinthe approachthe100percentproficiencyrequirement
overallstudentcalculations;theysimplyarenottreatedastheirown
subgroup. dictatedunderNCLBfor2014.
1 THOMASB.FORDHAMINSTITUTE
18
t 16
n P
Y
o A 14
g
n
m ki
Ma 12
er ools 10
h
V Sc
ple 8
m
a
ofS 6
er
mb 4
u
N
2
0
Massachuse s Nevada Idaho NorthDakota Kansas Washington Wyoming Indiana SouthCarolina Montana Florida Vermont NewJersey NewHampshire Maine NewMexico Delaware Colorado RhodeIsland Georgia Illinois Ohio Minnesota Michigan California Texas Arizona Wisconsin
Elementary Schools Middle Schools
Figure1.NumberofsampleschoolsmakingAYPbystate
Note:MiddleschoolswerenotincludedforTexasandNewJersey;absenceofamiddleschoolbarinthosestatesmeans“notapplicable”asopposedtozero.Stateslike
IdahoandNorthDakota,however,havezeropassingmiddleschools.
withgreaterdiversityandsizefacegreaterchallenges Introduction
inmakingAYP.
TheProficiencyIllusion(Croninetal.2007a)linkedstu-
dent performance on Vermont’s tests and those of 25
(cid:1)MiddleschoolshavegreaterdifficultyreachingAYP
other states to the Northwest Evaluation Association’s
in Vermont than do elementary schools, primarily
(NWEA’s) Measures of Academic Progress (MAP), a
because their student populations are larger and
computerized adaptive test used in schools nationwide.
thereforehavemorequalifyingsubgroups—notbe-
Thissinglecommonscalepermittedcross-statecompar-
causetheirstudentachievementislowerthaninthe
isons of each state’s reading and math proficiency stan-
elementaryschools.
dardstomeasureschoolperformanceundertheNoChild
Left Behind (NCLB) Act of 2001.That study revealed
(cid:1) A strong predictor of whether or not a school will
profounddifferencesinstates’proficiencystandards(i.e.,
make AYP under the Vermont system is whether it
howdifficultitistoachieveproficiencyonthestatetest),
has enough students with disabilities (SWDs)3 or
andevenacrossgradeswithinasinglestate.
Englishlanguagelearnerstoqualifyasaseparatesub-
group.Infact,allschoolswithlimitedEnglishprofi- OurstudyexpandsonTheProficiencyIllusionbyexam-
cient(LEP)4orSWDsubgroupsfailedtomakeAYP. ining other key factors of state NCLB accountability
3SWDsaredefinedasthosestudentsfollowingindividualizededucationplans.WeshouldalsonotethatoursubgroupfindingsforLEP
studentsandSWDsmaybemorenegativethanactualfindings,mostlybecauseofthelikelydifferencesbetweenhowLEPstudentsandSWDs
aretreatedinMAP,theassessmentweusedinthisstudy,andintheNewEnglandCommonAssessmentProgram(NECAP),thestandardized
statetest.Specifically,theU.S.DepartmentofEducationhasissuednewNCLBguidelinesinrecentyearsthatexcludesmallpercentagesof
LEPstudentsandSWDsfromtakingthestatetestorthatallowthemtotakealternativeassessments.Inthisstudy,however,novalidMAP
scoreswereomittedfromconsideration.
4Notethatweuse“LEPstudents”and“Englishlanguagelearners”interchangeablytorefertostudentsinthesamesubgroup.
TheAccountabilityIllusion 2
plansandhowtheyinteractwithstateproficiencystan- tial measurement error within the state test. In some
dards to determine whether the schools in our sample states,thesemarginsarequitewide,whichhastheeffect
made adequate yearly progress (AYP) in 2008. Specifi- ofmakingiteasiertoachieveanannualtarget. V
e
cally, we estimated how a single set of schools, drawn
r
fromaroundthecountry,wouldfareunderthediffering AlloftheseAYPrulesvarybystate,whichmeansthata m
rulesfordeterminingAYPin28states(theoriginal25in schoolthatmakesAYPinWisconsinorOhio,forexam- o
The Proficiency Illusion plus 3 others for which we now ple,mightnotmakeitunderSouthCarolina’sorIdaho’s n
t
have cut score estimates). In other words, if we could rules(U.S.DepartmentofEducation2008).
somehow move these entire schools—with their same
mix of characteristics—from state to state, how would
What We Studied
they fare in terms of making AYP? Will schools with
high-performing students consistently make AYP?Will Wecollectedstudents’MAPtestscoresfromthe2005–
schools with low-performing students consistently fail
2006academicyearfrom18elementaryand18middle
tomakeAYP?IfAYPdeterminationsforschoolsarenot
schoolsaroundthecountry.WealsocollectedtheNCLB
consistentacrossstates,whatleadstotheinconsistencies?
subgroupdesignationsforallstudentsinthoseschools—
inotherwords,whethertheyhadbeenclassifiedasmem-
NCLB requires every state, as a condition of receiving
bersofaminoritygrouporasEnglishlanguagelearners,
Title I funding, to implement an accountability system
amongothersubgroups.
that aims to get 100% of its students to the proficient
level on the state test by academic year 2013–2014. In
Theschoolswerenotselectedasarepresentativesample
theinterveningyears,statessetannualmeasurableobjec-
of the nation’s population. Instead, we selected the
tives(AMOs).Thisisthepercentageofstudentsineach
schoolsbecausetheyexhibitedarangeofcharacteristics
school,andineachsubgroupwithintheschool(suchas
on measures such as academic performance, academic
low income5 or African American among others), that growth, and socioeconomic status (the latter calculated
mustreachtheproficientlevelinorderfortheschoolto bythepercentageofstudentsreceivingfreeorreduced-
makeAYPinagivenyear.TheseAMOsvarybystate(as pricelunches).Appendix1containsacompletediscus-
do,ofcourse,thedifficultyoftheproficiencystandards). sionofthemethodologyforthisprojectalongwiththe
characteristicsoftheschoolsample.6
States also determine the minimum number of students
thatmustconstituteasubgroupinorderforitsscorestobe Proficiency cut score estimates for the New England
analyzedseparately(alsocalledtheminimumn[numberof Common Assessment Program (NECAP) are taken
students in sample] size).The rationale is that reporting from The Proficiency Illusion (as shown in Figure 2),
theresultsofverysmallsubgroups—fewerthantenpupils, whichfoundthatVermont’sproficiencycutscoreswere
for example—could jeopardize students’ confidentiality generallyrankedaboutaveragecomparedwiththestan-
and risk presenting inaccurate results. (With such small dardssetbytheother25statesinthatstudy.Thesecut
groups,randomevents,likeonestudentbeingoutsickon scores were used to estimate whether students would
testday,couldskewtheoutcome.)Becauseofthisflexibil- havescoredasproficientorbetterontheVermonttest,
ity,stateshavesetwidelyvaryingnsizesfortheirsubgroups, giventheirperformanceonMAP.Studenttestdataand
fromasfewas10youngsterstoasmanyas100. subgroupdesignationswerethenusedtodeterminehow
these18elementaryand18middleschoolswouldhave
Manystateshavealsoadoptedconfidenceintervals—ba- fared under Vermont AYP rules for 2008. In other
sicallymarginsofstatisticalerror—toaccountforpoten- words, the school data and proficiency cut score esti-
5Low-incomestudentsarethosewhoreceiveafreeorreduced-pricelunch.
6Wegaveallschoolsinoursamplepseudonymsinthisreport.
3 THOMASB.FORDHAMINSTITUTE
60
t
n 50
o
g
m n 40
ki
n
r aRR
e e 30 Reading
l
V n
e Math
erc 20
P
10
0
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Figure2.Vermontreadingandmathcutscoreestimates,expressedaspercentileranks(2006)
Note:ThisfigureillustratesthedifficultyofVermont’scutscores(orproficiencypassingscores)forthestate’sreadingandmathtests,aspercentilesoftheNWEAnorm,
ingradesthreethrougheight.Higherpercentileranksaremoredifficulttoachieve.Allofthestate’scutscoresarebelowthe55thpercentile.
mates are from academic year 2005–2006, but we are Note that we were unable to examine the impact of
applyingthemagainstVermont’s2008AYPrules. NCLB’s“safeharbor”provision.Thisprovisionpermits
aschooltomakeAYPevenifsomeofitssubgroupsfail,
Table1showsthepertinentVermontAYPrulesthatwere as long as it reduces the number of nonproficient stu-
appliedtoelementaryandmiddleschoolsinthecurrent dentswithinany failingsubgroup by at least 10%rela-
study.Vermont’sminimumsubgroupsizeis40,whichis tivetothepreviousyear’sperformance.Becausewehad
comparabletomostotherstatesweexamined.Moststates accesstoonlyasingleacademicyear’sdata(2005–2006),
examinedalsoapplyconfidenceintervals(ormarginsof wewerenotabletoincludethisinouranalysis.Asare-
error)totheirmeasurementsofstudentproficiencyrates. sult,it’spossiblethatsomeoftheschoolsinoursample
However, Vermont’s 99% confidence interval provides that failed to make AYP according to our estimates
schools with greater leniency than the more commonly wouldhavemadeAYPunderrealconditions.
used 95% confidence interval. This means that while
schools are supposed to get 87% of their grade 3-8 stu- Furthermore,attendanceandtestparticipationratesare
dents to the proficient level on the state reading test, as beyond the scope of the study. (Most states include at-
well as 87% of the students in each subgroup, applying tendanceratesasanadditionalindicatorintheirNCLB
theconfidenceintervalmeansthattherealtargetcanbe accountability system for elementary and middle
lower,particularlywithsmallergroups. schools. Plus, federal law requires 95% of each school’s
students—and95%ofthestudentsineachsubgroup—
Unlike most states, Vermont measures its student per- toparticipateintesting.)
formance with a proficiency index, which gives partial
creditforstudentsachieving“partialproficiency.”Inthe Toreiterate,then,AYPdecisionsinthecurrentstudyare
shortterm,theindexmakesiteasierforVermontschools modeledsolelyontestperformancedataforasingleaca-
tomeettheirtargets,althoughtheeffectoftheindexdi- demic year. For each school, we calculated reading and
minishes as the targets approach the 100% proficiency math proficiency rates (along with any confidence inter-
requirementdictatedunderNCLBfor2014.7 vals) to determine whether the overall school population
7Insixofthestatesstudied(Massachusetts,Minnesota,RhodeIsland,NewHampshire,andWisconsin,aswellasVermont),anindexisused
thatgivesfullcredittostudentswhoachieveproficient(orbetter)andpartialcredittostudentsperformingatlowerlevels.Consequently,the
resultantscoreinstatesusingthis“hybrid”modelisalwayshigherthantheactualproficiencypercentage(givingstudentspartialcreditforachiev-
inglowerproficiencylevelsisobviouslybetterthannocredit,atleastfortheschools’ratings).Theindexprovidesafairamountofhelpwhen
annualtargetsarebelow50%;however,oncetargetsriseabove75%,theindexhasfarlessimpact.
TheAccountabilityIllusion 4
Table1.VermontAYPrulesfor2008
Subgroupminimumn Race/ethnicity:40
V
SWDs:40 e
r
Low-incomestudents:40 m
LEPstudents:40 o
n
CI Appliedtoproficiencyratecalcula-ons? t
Yes;99%CIused
AMOs Baselineproficiencylevelsasof2002(index) 2008targets(index)
READING/LANGUAGEARTS
Grade3 n/a 87.0
Grade4 n/a 87.0
Grade5 n/a 87.0
Grade6 n/a 87.0
Grade7 n/a 87.0
Grade8 n/a 87.0
MATH
Grade3 n/a 85.4
Grade4 n/a 85.4
Grade5 n/a 85.4
Grade6 n/a 85.4
Grade7 n/a 85.4
Grade8 n/a 85.4
Sources:U.S.DepartmentofEducation(2008);CouncilofChiefStateSchoolOfficers(2008).
Abbreviations:SWDs=studentswithdisabilities;LEP=limitedEnglishproficiency;CI=confidenceinterval;AMOs=annualmeasurableobjectives;n/a=notapplicable
and any qualifying subgroups achieved the AMOs. We indicating above-grade-level performance. All schools
deemedthataschoolmadeAYPifitsoverallstudentbody makingAYPareintherighthalfofthefigure,meaning
andallitsqualifyingsubgroupsmetorexceededitsAMOs. thattheyareamongtheschoolswhichcontainthehigh-
Again,Appendix1suppliesfurthermethodologicaldetail. estaverageperformingstudents.
How Did the Sample Schools Yet among these schools with high average performing
students,theonlyschoolsactuallytomakeAYParethose
Fare Under Vermont’s AYP Rules?
with relatively few qualifying subgroups—and thus the
Figure 3 illustrates the AYP performance of the sample fewesttargetstomeet(sinceeachsubgrouphasitsown
elementary schools under Vermont’s 2008 AYP rules. separate targets). For example, Wayne Fine Arts, Win-
OnlythreeelementaryschoolsmadeAYPwhilefifteen chesterandRooseveltmadeit,buthaveonlyfourtargets
failedtomakeit.ThetrianglesinFigure3showtheav- each—two in reading and math for their overall popu-
erage academic performance of students within the lations, and two in reading and math for the only sub-
school,withnegativevaluesindicatingbelow-grade-level groupthatexceedsVermont’sminimum“nsize”:white
performancefortheaveragestudentandpositivevalues students.
5 THOMASB.FORDHAMINSTITUTE
18 15
ol
t 16 ho
n 10 Sc
14 n
mo gets 12 5 ncei
r a
Ver berofTa 180 0 Perform
mm 6 -5 tnt
u e
N d
4 u
-10 St
2 ge
a
r
0 -15 ve
A
1) 1) 1) 7) 5) 3) 7) 7) 5) 9) 1) 2) 3) 7) 8) 0) 8) 4)
( ( ( ( ( ( ( ( ( ( 2 2 ( ( ( 1 2 1
Clarkson Maryweather Few Nemo IslandGrove ohnF.Kennedy Scholls Hissmore WolfCreek Mayberry ayneFineArts( Winchester( Coastal Paramount ForestLake Marigold( Roosevelt( KingRichard(
J W
TargetsPassed TargetsFailed Average Student Performance
Figure3.AYPperformanceoftheelementaryschoolsampleunderVermont’s2008AYPrules
Note:ThisfigureshowshoweachoftheelementaryschoolswithinthesamplefaredundertheVermontAYPrules(asdescribedinTable1).Thebarsshowthenumber
oftargetsthateachschoolhadtomeetinordertomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnotmeetthem(lightblue).The
moresubgroupsinaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailedtomeettheAMOforevenasinglesubgroupdidn’tmakeAYP,
soanylightbluemeanstheschoolfailed.MarigoldElementary,forexample,meetssixofitseighttargets,butbecauseitdidn’tmeetthemall,itdidn’tmakeAYP.Schools
areorderedfromlowesttohighestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbytheaverageMAPperformanceofstudents
withintheschool;itsscaleisshownontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-levelperformanceand
scoresabovezerodenoteabove-grade-levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebettertheaverageperformance
andthelowerthenumber,theworsetheaverageperformance.Thenumberinparenthesesaftereachschoolnameindicatesthenumberofstates(outof28)inwhich
thatschoolwouldhavemadeAYP.
Figure 4 illustrates the AYP performance of the sample all math targets by the confidence intervals. However,
middleschoolsunderthe2008VermontAYPrules.Out JFKstillfailedtomakeAYPduetotheperformanceof
of eighteen in our sample, only one middle school multiplesubgroups(seeFigure3).
made AYP—Walter Jones—a high-performing school
withrelativelyfewqualifyingsubgroups. Theeffect oftheconfidenceintervalsonreadingprofi-
ciencyratesattheelementaryandmiddleschoollevelsis
Figures 5 and 6 indicate the degree to which schools’ similar(notshown).Inreading,twoelementaryschools
math proficiency rates are aided by the confidence in- (Hissmore and Paramount) and two middle schools
terval for elementary and middle schools, respectively. (PogestoandArtemus)wereabletomeettheoveralltar-
On these figures, the darker portions of the bars show getwiththeconfidenceinterval,althoughweknowfrom
theactualproficiencyratesateachschoolandthelighter Figures3and4thattheseschoolsstillfailedtomeettar-
portionsofthebarsshowthedegreetowhichthesepro- gets for their subgroups. In short, applying the confi-
ficiencyrateswereincreasedbyapplyingtheconfidence dence interval (even a generous one like the 99%
interval. The orange lines show the AMOs needed to confidence interval used in Vermont) has little or no
meet AYP.The figures show that one elementary (JFK) effect on whether schools meet their overall reading
andnomiddleschoolsareassistedinmeetingtheirover- andmathtargetsinVermont.8
8Inthecurrentanalyses,confidenceintervalswereappliedtoboththeoverallschoolpopulationandtoalleligiblesubgroupsinoursampleschools.
Thus,theultimateimpactoftheconfidenceintervalislikelylargerthantheimpactdepictedinFigures5and6.However,wechosenottoshow
howtheconfidenceintervalimpactedsubgroupperformancebecauseitwouldhaveaddedgreatlytothereport’slengthandcomplexity.
TheAccountabilityIllusion 6
18 12
ol
16 10 o
h V
c
14 8 nS e
ets 12 6 cei r
g n m
Tar 10 4 ma
of 2 or o
mberm 86 0 ntPerft nt
u -2 e
N d
4 -4 tu
S
2 -6 ge
a
r
0 -8 ve
A
0) 0) 0) 5) 0) 0) 1) 1) 0) 0) 2) 0) 2) 1) 2) 0) 3) 5)
( ( ( 1 ( ( ( ( ( ( ( ( ( ( ( 2 ( (
McBeal arringerCharter MLAndrew Pogesto( McCord Tigerbear Chesterfield Filmore Barban Kekata Hoyt BlackLake LakeJoseph Zeus OceanView WalterJones( Artemus Chaucer
B
Targets Passed Targets Failed Average Student Performance
Figure4.AYPperformanceofthemiddleschoolsampleunderVermont’s2008AYPrules
Note:ThisfigureshowshoweachofthemiddleschoolswithinthesamplefaredundertheAYPrulesinVermont(asdescribedinTable1).Thebarsshowthenumberof
targetsthateachschoolhadtomeetinordertomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnotmeetthem(lightblue).Themore
subgroupsinaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailedtomeettheAMOforevenasinglesubgroupdidn’tmakeAYP,soany
lightbluemeanstheschoolfailed.Chaucer,forexample,meetssevenofitsfourteentargets,butbecauseitdidn’tmeetthemall,itdidn’tmakeAYP.Schoolsareorderedfrom
lowesttohighestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbytheaverageMAPperformanceofstudentswithintheschool;itsscale
isshownontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-levelperformanceandscoresabovezerodenoteabove-grade-
levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebettertheaverageperformanceandthelowerthenumber,theworsethe
averageperformance.Thenumberinparenthesesaftereachschoolnameindicatesthenumberofstates(outof28)inwhichthatschoolwouldhavemadeAYP.
1
0.9
0.8
0.7
e
Rat 0.6
y
nc 0.5
e
cifi 00.44
o
Pr 0.3
0.2
0.1
0
Clarkson Maryweather Few Nemo IslandGrove hnF.Kennedy Scholls Hissmore WolfCreek Mayberry ayneFineArts Winchester Coastal Paramount ForestLake Marigold Roosevelt KingRichard
o W
J
MathProficiency Rate MathProficiency Ratewith CI MathTarget
Figure5.Impactoftheconfidenceintervalonelementaryschoolmathproficiencyrates
Note:Thisfigureshowsthereportedproficiencyrateforthestudentpopulationasawholeandtheimpactoftheconfidenceintervalonmeetingannualtargets.The
darkerportionsofthebarsshowtheactualproficiencyrateachieved,whilethelighter(upper)portionsofthebarsshowthemarginoferrorascomputedbythe
confidenceinterval.Thefigureshowsthatoneofthesampleelementaryschools(JFK)wasassistedbytheconfidenceinterval.Annualtargets(theorangelines)are
consideredtobemetbytheconfidenceintervaliftheyfallwithinthelightblueportion.
7 THOMASB.FORDHAMINSTITUTE
1
0.9
t
n
0.8
o
0.7
m e
Rat 0.6
r y
e nc 0.5
e
V iciof 00.44
r
P 0.3
0.2
0.1
0
McBeal ngerCharter MLAndrew Pogesto McCord Tigerbear Chesterfield Filmore Barban Kekata Hoyt BlackLake LakeJoseph Zeus OceanView WalterJones Artemus Chaucer
arri
B
Math Proficiency Rate MathProficiency Ratewith CI MathTarget
Figure6.Impactoftheconfidenceintervalonmiddleschoolmathproficiencyrates
Note:Thisfigureshowsthereportedproficiencyrateforthestudentpopulationasawholeandtheimpactoftheconfidenceintervalonmeetingannualtargets.The
darkerportionsofthebarsshowtheactualproficiencyrateachieved,whilethelighter(upper)portionsofthebarsshowthemarginoferrorascomputedbythe
confidenceinterval.Thefigureshowsthatnoneofthesamplemiddleschoolswasassistedbytheconfidenceinterval.Annualtargets(theorangelines)areconsidered
tobemetbytheconfidenceintervaliftheyfallwithinthelightblueportion.
Where do schools fail? Theschool-by-schoolfindingsinTables2and3showthat
Figures3and4illustratethatschoolswithlowormid-
(cid:1) Four elementary schools failed to meet both their
dling performance can still make AYP when the school
overallreadingandmathtargets.
has fewer targets to meet, thanks to fewer subgroups.
Thesefiguresdonot,however,indicatewhichsubgroups (cid:1) Thirteen middle schools failed to meet both their
failed in which school. Information on individual sub- reading and math targets for their overall popula-
groupperformanceappearsinTables2and3forelemen- tions.
taryandmiddleschools,respectively.
(cid:1)Threeelementaryschools(Scholls,ForestLake,and
KingRichard)failedfortheirSWDsubgrouponly.
Tables2and3showwhichsubgroupsqualifiedforeval-
uation at each school (i.e., whether the number of stu-
(cid:1)Oneelementaryschool(AliceMayberry)mettargets
dents within that subgroup exceeded the state’s
foreverysubgroupexceptforitslowincomestudents.
minimum n), and whether that subgroup passed or
failed. Although all schools are evaluated on the profi- Tables4and5summarizesubgroupperformanceforel-
ciency rate of their overall population, potential sub- ementaryandmiddleschools,respectively.First,theper-
groups that are separately evaluated for AYP include formanceofSWDsandLEPstudentswereparticularly
SWDs,studentswithLEP,low-incomestudents,andthe challenging for Vermont schools. Every single school
following race/ethnic categories: African American, withenoughstudentstocompriseaSWDorLEPsub-
Asian/Pacific Islander, Hispanic/Latino, American In- group failed to make AYP, in part due to these groups'
dian/AlaskaNative,andwhite.Tables2and3alsoshow performances.Traditionallyacademicallydisadvantaged
whether a school met AYP under the 2008 Vermont subgroups, such as low income and Hispanic students,
rules,andthetotalnumberofstateswithinthestudyin also had difficulty under Vermont’s accountability sys-
whichthatschoolmetAYP. tem,especiallyatthemiddleschoollevel.
TheAccountabilityIllusion 8
Table2.Elementaryschoolsubgroupperformanceofsampleschoolsunderthe2008VermontAYPrules
SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP? Vermont
Clarkson 65.8% 62.0% N N N N N N N N 8 0 0% N 1
Maryweather 68.2% 66.4% N N N N N N N N Y Y 10 2 20% N 1
Few 73.2% 69.7% N N N N N N N N N N 10 0 0% N 1
Nemo 76.3% 81.5% N Y N N Y Y 6 3 50% N 7
IslandGrove 78.3% 79.9% N N N N N N Y Y 8 2 25% N 4
JFK 82.2% 78.0% Y N N N N N N N Y Y 10 3 30% N 3
Scholls 86.3% 81.7% Y Y N N Y Y Y Y Y Y 10 8 80% N 7
Hissmore 85.7% 84.3% Y Y N N Y Y Y N Y Y 10 7 70% N 7
WolfCreek 78.7% 81.2% N Y N N N N Y Y 8 3 38% N 5
AliceMayberry 85.5% 86.2% Y Y Y N Y Y Y Y 8 7 88% N 9
WayneFineArts 85.8% 92.4% Y Y Y Y 4 4 100% Y 21
Winchester 84.7% 88.3% Y Y Y Y 4 4 100% Y 22
Coastal 87.9% 83.4% Y Y N N N N Y N Y N N N Y Y 14 6 43% N 3
Paramount 85.3% 84.6% Y Y N N N N Y Y 8 4 50% N 7
ForestLake 92.4% 92.0% Y Y N N Y Y Y Y 8 6 75% N 8
Marigold 94.7% 91.2% Y Y Y N Y N Y Y 8 6 75% N 10
Roosevelt 96.2% 96.2% Y Y Y Y 4 4 100% Y 28
KingRichard 93.8% 93.5% Y Y Y N Y Y Y 7 6 86% N 14
Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino=
Hispanic;AmericanIndian/AlaskaNative=AI/AN.
Note:Schoolsareorderedfromlowest(Clarkson)tohighest(KingRichard)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading
performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof
studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs.
Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber
ofstatesinthestudyforwhichthatschoolmetAYP.
Characteristics of Schools in Vermont. A striking difference between schools that
consistentlymakeanddon’tmakeAYP,appearstobethe
that Did and Didn’t Make AYP
numberofsubgroupsforwhicheachisheldaccountable
AcloselookatFigures3and4indicatesthatVermont’s —andhence,thenumberofacademictargetsforwhich
NCLB accountability system is, in many respects, be- eachmustdemonstrateproficiency.
having similarly to those in other states. For example,
amongtheelementaryschoolsinoursample,Roosevelt, This is consistent with the patterns shown in Table 6,
Winchester, and Wayne Fine Arts all made AYP in the whichcomparestheschoolsthatdidanddidn’tmakeAYP
greatestnumberofstates—28,22,and21,respectively. onseveralacademicanddemographicdimensions.Within
AndtheseschoolsallmadeAYPinVermont,too.Like- thesample,elementaryschoolsthatmakeAYPdoindeed
wise, the elementary and middle schools that fail to show higher average student performance, but they also
makeAYPinthegreatestnumberofstatesalsofailAYP differinthefollowingways:theyhavemuchsmallerstu-
9 THOMASB.FORDHAMINSTITUTE
Table3.Middleschoolsubgroupperformanceofsampleschoolsunderthe2008VermontAYPrules
Vermont SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP?
McBeal 58.0% 65.0% N N N N N N N N N N N N N N N Y 16 1 6% N 0
BarringerCharter 65.3% 73.1% N N N N N N N N N N 10 0 0% N 0
MLAndrew 58.9% 71.2% N N N N N N N N N N N N 12 0 0% N 0
Pogesto 61.6% 74.5% N Y N Y 4 2 50% N 15
McCordCharter 61.1% 73.9% N N N N N N N N N N N Y 12 1 8% N 0
Tigerbear 68.0% 68.8% N N N N N N N N N N 10 0 0% N 0
Chesterfield 72.1% 72.9% N N N N N N N N Y N 10 1 10% N 1
Filmore 71.4% 78.9% N N N N N N N N N Y 10 1 10% N 1
Barban- 67.3% 74.1% N N N N N N N N N N Y Y 12 2 17% N 0
Kekata 75.3% 76.5% N N N N N N N N N N N N Y Y 14 2 14% N 0
Hoyt 77.5% 79.3% N N N N N N N N Y Y 10 2 20% N 2
BlackLake 80.0% 78.7% N N N N N N N N N N Y N 12 1 8% N 0
LakeJoseph 76.7% 82.4% N N N N N N N N N N Y Y 12 2 17% N 2
Zeus 79.9% 80.5% N N N N N N N N N N N N Y Y 14 2 14% N 1
OceanView 81.1% 87.3% N Y N N N N N N N N Y Y 12 3 25% N 2
WalterJones 86.6% 88.9% Y Y Y Y Y Y 6 6 100% Y 20
Artemus 86.2% 85.9% Y Y N N N N N N Y Y 10 4 40% N 3
Chaucer 87.7% 91.0% Y Y N N N N N N Y Y N Y Y Y 14 7 50% N 5
Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino=
Hispanic;AmericanIndian/AlaskaNative=AI/AN.
Note:Schoolsareorderedfromlowest(McBeal)tohighest(Chaucer)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading
performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof
studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs.
Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber
ofstatesinthestudyforwhichthatschoolmetAYP.
dentpopulations,fewersubgroups(andthusfewertargets the country to see how these schools would have fared
tomeet),andlowerpercentagesoflow-incomestudents. underVermont’s AYP rules and annual measurable ob-
Similarly, middle schools that make AYP have slightly jectives for 2008. We found that only 3 elementary
higher performing students, on average, than middle schoolsand1middleschool—4inallfromasampleof
schools that failed to make it, but have dramatically 36—wouldhavemadeAYPinVermont.Lookingacross
smallertotalenrollments,smallernonwhitepopulations, the 28 state accountability systems examined in this
andfewersubgroups(andthustargetstomeet). study,thisputsVermontataboutthemiddleofthedis-
tributionintermsofthenumberofelementarysample
schoolsmakingAYP(asshowninFigure1).
Concluding Observations
This study examined the test performance data of stu- BecausetheoverridinggoalofNCLBistoeliminateed-
dentsfrom18elementaryand18middleschoolsacross ucationaldisparitieswithinandacrossstates,it’simpor-
TheAccountabilityIllusion 10