ERIC ED504686: The Accountability Illusion: Idaho

2009·0.21 MB·English

by  ERIC

Idaho
Executive Summary countable for more subgroups than would similar I
d
schoolsinotherstateswithlargersubgroupsizes.
a
TheintentoftheNoChildLeftBehind(NCLB)Actof
h
2001istoholdschoolsaccountableforensuringthatall (cid:1) Looking across the 28 state accountability systems o
of their students achieve mastery in reading and math, examined in the study, we find the number of ele-
with a particular focus on groups that have traditionally mentary schools that made AYP in Idaho was ex-
beenleftbehind.UnderNCLB,statessubmitaccounta- ceeded in 20 other states (Idaho ties 5 other states
bilityplanstotheU.S.DepartmentofEducationdetailing inhavingjust2elementaryschoolsthatmadeAYP).
therulesandpoliciestobeusedintrackingtheadequate Idaho joins Massachusetts, Montana, South Car-
yearlyprogress(AYP)ofschoolstowardthesegoals. olina, and North Dakota in having no middle
schoolsthatmakeAYPinoursample(seeFigure1).
ThisreportexaminesIdaho’sNCLBaccountabilitysys-
tem—particularly how its various rules, criteria, and (cid:1) Many of the schools in our sample that failed to
makeAYPinIdahoaremeetingexpectedtargetsfor
practices result in schools either making AYP or not
their overall populations but failed because of the
makingAYP.ItalsogaugeshowtoughIdaho’ssystemis
compared with other states. For this study, we selected performance of individual subgroups, particularly
36schoolsfromvariousstatesaroundthenation,schools students with disabilities (SWD) and English lan-
that vary by size, achievement, and diversity, among guagelearners.2
otherfactors,anddeterminedwhethereachwouldmake
(cid:1) Schools with fewer subgroups attained AYP more
AYPunderIdaho’ssystemaswellasunderthesystemsof
easily in Idaho than schools with more subgroups,
27otherstates.Weusedschooldataandproficiencycut
even when their average student performance is
score1estimatesfromacademicyear2005–2006,butap-
much lower. In other words, schools with greater
plied them against Idaho’s AYP rules for academic year
2007–2008(shortenedto“2008”inthisreport).
Herearesomekeyfindings:
(cid:1) We estimate that 16 of 18 elementary schools and Onlytwoelementaryschoolsandnoneofthemiddle
all of the middle schools in our sample failed to schoolsinoursamplemadeAYPin2008under
makeAYPin2008underIdaho’saccountabilitysys- Idaho’saccountabilitysystem.Anumberoffactors
tem.Thehighfailurerateispartlyexplainedbyour
likelycontributetothislownumber.First,Idaho’s
sample, which intentionally includes some schools
minimumsubgroupsizeis34,whichisrelatively
witharelativelylargepopulationoflow-performing
smallincomparisontomostotherstatesexaminedin
students. It’s also partly explained by Idaho’s mini-
thestudy.ThismeansthatschoolsinIdahowillbe
mumsubgroupsize(34),whichisrelativelysmallin
comparison to most other states examined in the accountabletomoresubgroupsthanwouldsimilar
study.This means that schools in Idaho will be ac- schoolsinotherstateswithhighersubgroupsizes.
Notonlydidmanydisadvantagedsubgroupsfailtheir
1 A cut score is the minimum score a student must receive on annualtargetsinIdaho,quiteafewwhitesubgroups
NWEA’sMeasuresofAcademicProgress(MAP)thatisequivalentto
failedaswell,especiallyinreading.Anotherfactor
performingproficientontheIdahoStandardsAchievementTests.
2It’simportanttonotethatstudentsinsubgroupsnotmeetingthe whichmakesitdifficultforschoolstomakeAYPin
minimumnsizesarestillincludedforaccountabilitypurposesinthe Idahoisthatnoconfidenceinterval(marginoferror)
overallstudentcalculations;theysimplyarenottreatedastheirown
subgroup. isappliedtoproficiencyratecalculations.
1 THOMASB.FORDHAMINSTITUTE
18
o 16
h YP
A 14
a g
n
d Maki 12
I ols
o 10
h
Sc
ple 8
m
a
ofS 6
er
mb 4
u
N
2
0
Massachuse s Nevada Idaho NorthDakota Kansas Washington Wyoming Indiana SouthCarolina Montana Florida Vermont NewJersey NewHampshire Maine NewMexico Delaware Colorado RhodeIsland Georgia Illinois Ohio Minnesota Michigan California Texas Arizona Wisconsin
Elementary Schools Middle Schools
Figure1.NumberofsampleschoolsmakingAYPbystate
Note:MiddleschoolswerenotincludedforTexasandNewJersey;absenceofamiddleschoolbarinthosestatesmeans“notapplicable”asopposedtozero.Stateslike
IdahoandNorthDakota,however,havezeropassingmiddleschools.
diversityandsizefacegreaterchallengesinmaking Introduction
AYP.Thisistrueinotherstatesaswell.
TheProficiencyIllusion(Croninetal.2007a)linkedstu-
(cid:1) Middle schools have somewhat greater difficulty dentperformanceonIdaho’stestsandthoseof25other
reachingAYPinIdahothandoelementaryschools, states to the Northwest Evaluation Association’s
primarily because their student populations are (NWEA’s) Measures of Academic Progress (MAP), a
larger and therefore have more qualifying sub- computerized adaptive test used in schools nationwide.
groups—not because their student achievement is Thissinglecommonscalepermittedcross-statecompar-
anylowerthanintheelementaryschools. isons of each state’s reading and math proficiency stan-
dardstomeasureschoolperformanceundertheNoChild
(cid:1)Astrongpredictorofwhetherornotaschoolwillmake Left Behind (NCLB) Act of 2001.That study revealed
AYP under Idaho’s system is whether it has enough profounddifferencesinstates’proficiencystandards(i.e.,
SWDsandenoughEnglishlanguagelearnerstoqualify howdifficultitistoachieveproficiencyonthestatetest),
asaseparatesubgroup.EveryschoolwithanSWDor andevenacrossgradeswithinasinglestate.
limited English proficient (LEP)3 subgroup failed to
makeAYP,inpartbecausethesestudentsdidnotmeet OurstudyexpandsonTheProficiencyIllusionbyexam-
thestate’sproficiencytargetsinreadingormath.4 ining other key factors of state NCLB accountability
3Notethatweuse“LEPstudents”and“Englishlanguagelearners”interchangeablytorefertostudentsinthesamesubgroup.
4SWDsaredefinedasthosestudentsfollowingindividualizededucationplans.WeshouldalsonotethatoursubgroupfindingsforLEP
studentsandSWDsmaybemorenegativethanactualfindings,mostlybecauseofthelikelydifferencesbetweenhowLEPstudentsandSWDs
aretreatedinMAP,theassessmentweusedinthisstudy,andintheIdahoStandardsAchievementTests,thestandardizedstatetests.Specifically,
theU.S.DepartmentofEducationhasissuednewNCLBguidelinesinrecentyearsthatexcludesmallpercentagesofLEPstudentsandSWDs
fromtakingthestatetestsorthatallowthemtotakealternativeassessments.Inthisstudy,however,novalidMAPscoreswereomittedfrom
consideration.
TheAccountabilityIllusion 2
plansandhowtheyinteractwithstateproficiencystan- Manystateshavealsoadoptedconfidenceintervals—ba-
dards to determine whether the schools in our sample sicallymarginsofstatisticalerror—toaccountforpoten-
made adequate yearly progress (AYP) in 2008. Specifi- tial measurement error within the state test. In some I
d
cally, we estimated how a single set of schools, drawn states,thesemarginsarequitewide,whichhastheeffect a
fromaroundthecountry,wouldfareunderthediffering ofmakingiteasiertoachieveanannualtarget. h
o
rulesfordeterminingAYPin28states(theoriginal25in
The Proficiency Illusion plus 3 others for which we now AlloftheseAYPrulesvarybystate,whichmeansthata
have cut score estimates). In other words, if we could schoolthatmakesAYPinWisconsinorOhio,forexam-
somehow move these entire schools—with their same ple,mightnotmakeitunderSouthCarolina’sorIdaho’s
mix of characteristics—from state to state, how would rules(U.S.DepartmentofEducation2008).
they fare in terms of making AYP? Will schools with
high-performing students consistently make AYP?Will
What We Studied
schools with low-performing students consistently fail
tomakeAYP?IfAYPdeterminationsforschoolsarenot Wecollectedstudents’MAPtestscoresfromthe2005–
consistentacrossstates,whatleadstotheinconsistencies? 2006academicyearfrom18elementaryand18middle
schoolsaroundthecountry.WealsocollectedtheNCLB
NCLB requires every state, as a condition of receiving subgroupdesignationsforallstudentsinthoseschools—
TitleIfunding,toimplementanaccountabilitysystem inotherwords,whethertheyhadbeenclassifiedasmem-
that aims to get 100% of its students to the proficient bersofaminoritygrouporasEnglishlanguagelearners,
level on the state test by academic year 2013–2014. In amongothersubgroups.
the intervening years, states set annual measurable ob-
jectives (AMOs).This is the percentage of students in Theschoolswerenotselectedasarepresentativesample
each school, and in each subgroup within the school of the nation’s population. Instead, we selected the
(suchaslow-income5orAfricanAmerican,amongoth- schoolsbecausetheyexhibitedarangeofcharacteristics
ers),thatmustreachtheproficientlevelinorderforthe on measures such as academic performance, academic
schooltomakeAYPinagivenyear.TheAMOsvaryby growth, and socioeconomic status (the latter calculated
state (as do, of course, the difficulty of the proficiency bythepercentageofstudentsreceivingfreeorreduced-
standards). pricelunches).Appendix1containsacompletediscus-
sionofthemethodologyforthisprojectalongwiththe
Statesalsodeterminetheminimumnumberofstudents characteristicsoftheschoolsample.6
thatmustconstituteasubgroupinorderforitsscoresto
beanalyzedseparately(alsocalledtheminimumn[num- Proficiency cut score estimates for the Idaho Standards
berofstudentsinsample]size).Therationaleisthatre- AchievementTests(ISAT)aretakenfromTheProficiency
portingtheresultsofverysmallsubgroups—fewerthan Illusion(asshowninFigure2),whichfoundthatIdaho’s
tenpupils,forexample—couldjeopardizestudents’con- definitionsofproficiencygenerallyrankedaboutaverage
fidentialityandriskpresentinginaccurateresults.(With comparedwiththestandardssetbytheother25statesin
suchsmallgroups,randomevents,likeonestudentbeing that study. These cut scores were used to estimate
out sickontest day, could skew theoutcome.) Because whetherstudentswouldhavescoredasproficientorbet-
of this flexibility, states have set widely varying n sizes terontheIdahotest,giventheirperformanceonMAP.
for their subgroups, from as few as 10 youngsters to as Student test data and subgroup designations were then
manyas100. usedtodeterminehowthese18elementaryand18mid-
5Low-incomestudentsarethosewhoreceiveafreeorreduced-pricelunch.
6Wegaveallschoolsinoursamplepseudonymsinthisreport.
3 THOMASB.FORDHAMINSTITUTE
50
45
o
h 40
a g 35
n
d ki
n 30
I aRR
e 25 Reading
l
en 20 Math
c
er 15
P
10
5
0
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Figure2.Idahoreadingandmathcutscoreestimates,expressedaspercentileranks(2006)
Note:ThisfigureillustratesthedifficultyofIdaho’scutscores(orproficiencypassingscores)foritsreadingandmathtests,aspercentilesoftheNWEAnorm,ingrades
threethrougheight.Higherpercentileranksaremoredifficulttoachieve.AllofIdaho’scutscoresarebelowthe50thpercentile.
Table1.IdahoAYPrulesfor2008
Subgroupminimumn Race/ethnicity:34
SWDs:34
Low-incomestudents:34
LEPstudents:34
CI Appliedtoproficiencyratecalcula-ons?
CInotused
AMOs Baselineproficiencylevelsasof2002(%) 2008targets(%)
READING/LANGUAGEARTS
Grade3 66 78
Grade4 66 78
Grade5 66 78
Grade6 66 78
Grade7 66 78
Grade8 66 78
MATH
Grade3 51 70
Grade4 51 70
Grade5 51 70
Grade6 51 70
Grade7 51 70
Grade8 51 70
Sources:U.S.DepartmentofEducation(2008);CouncilofChiefStateSchoolOfficers(2008).
Abbreviations:SWDs=studentswithdisabilities;LEP=limitedEnglishproficiency;CI=confidenceinterval;AMOs=annualmeasurableobjectives
TheAccountabilityIllusion 4
18 15
ol
16 ho I
10 Sc d
14 n
gets 12 5 ncei ah
r a
Ta 10 m o
erof 8 0 erfor
b P
mm 6 -5 tnt
u e
N d
4 u
-10 St
2 ge
a
r
0 -15 ve
A
1) 1) 1) 7) 5) 3) 7) 7) 5) 9) 1) 2) 3) 7) 8) 0) 8) 4)
( ( ( ( ( ( ( ( ( ( 2 2 ( ( ( 1 2 1
Clarkson Maryweather Few Nemo IslandGrove ohnF.Kennedy Scholls Hissmore WolfCreek Mayberry ayneFineArts( Winchester( Coastal Paramount ForestLake Marigold( Roosevelt( KingRichard(
J W
TargetsPassed TargetsFailed Average Student Performance
Figure3.AYPperformanceoftheelementaryschoolsampleunderIdaho’s2008AYPrules
Note:ThisfigureindicateshoweachoftheelementaryschoolswithinthesamplefaredunderIdaho’sAYPrules(asdescribedinTable1).Thebarsshowthenumberof
targetsthateachschoolhastomeettomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnotmeetthem(lightblue).Themore
subgroupsinaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailedtomeettheAMOsforevenasinglesubgroupdidn’tmakeAYP,so
anylightbluemeansthattheschoolfailed.WayneFineArts,forexample,metfourofitssixtargets,butbecauseitdidn’tmeetthemall,itdidn’tmakeAYP.Schoolsare
orderedfromlowesttohighestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbyaverageMAPperformanceofstudentswithinthe
school;itsscaleisshownontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-levelperformanceandscoresabove
zerodenoteabove-grade-levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebettertheaverageperformanceandthelower
thenumber,theworsetheaverageperformance.Thenumberinparenthesesaftereachschoolnameindicatesthenumberofstates(outof28)inwhichthatschool
wouldhavemadeAYPinthestudy.
dleschoolswouldhavefaredunderIdahoAYPrulesfor port a confidence interval around their school profi-
2008. (In other words, the school data are from 2005– ciencyrates.
2006,asareourproficiencycutscoreestimates,butwe
areapplyingthemagainstIdaho’s2008AYPrules.) Note that we were unable to examine the impact of
NCLB’s“safeharbor”provision.Thisprovisionpermits
Table1showsthepertinentIdahoAYPrulesthatweap- aschooltomakeAYPevenifsomeofitssubgroupsfail,
plied to elementary and middle schools in the current as long as it reduces the number of nonproficient stu-
study. Idaho’s minimum subgroup size is 34, which is dentswithinany failingsubgroup by at least 10%rela-
relatively small in comparison to most other states ex- tivetothepreviousyear’sperformance.Becausewehad
aminedinthestudy. ThismeansthatschoolsinIdaho accesstoonlyasingleacademicyear’sdata(2005–2006),
willbeaccountableformoresubgroupsthanwouldsim- wewerenotabletoincludethisinouranalysis.Asare-
ilarschoolsinotherstateswithlargersubgroupsizes.7 sult,it’spossiblethatsomeoftheschoolsinoursample
that failed to make AYP according to our estimates
Furthermore, although the majority of states examined wouldhavemadeAYPunderrealconditions.
in the study apply confidence intervals to their student
proficiencyrates,Idahodoesnot.ThismeansthatIdaho Furthermore,attendanceandtestparticipationratesare
schools will have greater difficulty achieving their beyondthescopeofthestudy.Notethatmoststatesin-
AMOs than equivalent schools in other states that re- cludeattendanceratesasanadditionalindicatorintheir
7Keepinmind,however,thatschoolsizeandnsizearerelated(e.g.,smallnsizesmakesenseforsmallschools).
5 THOMASB.FORDHAMINSTITUTE
20 12
ol
18 10 o
o h
c
h 16 8 S
n
a ets 14 6 cei
Id ofTarg 1102 24 orman
ber 8 0 Perf
mm ntt
u 6 -2 e
N d
u
4 -4 t
S
e
2 -6 g
a
r
0 -8 ve
A
0) 0) 0) 5) 0) 0) 1) 1) 0) 0) 2) 0) 2) 1) 2) 0) 3) 5)
( ( ( 1 ( ( ( ( ( ( ( ( ( ( ( 2 ( (
McBeal arringerCharter MLAndrew Pogesto( McCord Tigerbear Chesterfield Filmore Barban Kekata Hoyt BlackLake LakeJoseph Zeus OceanView WalterJones( Artemus Chaucer
B
Targets Passed Targets Failed Average Student Performance
Figure4.AYPperformanceofthemiddleschoolsampleunderIdaho’s2008AYPrules
Note:ThisfigureshowshoweachofthemiddleschoolswithinthesamplefaredunderIdaho’sAYPrules(asdescribedinTable1).Thebarsshowthenumberoftargetsthat
eachschoolhadtomeetinordertomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnotmeetthem(lightblue).Themoresubgroups
inaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailstomeettheAMOsforevenasinglesubgroupdidn’tmakeAYP,soanylightblue
meansthattheschoolfailed.WalterJones,forexample,metfourofitseighttargets,butbecauseitdidn’tmeetthemall,itdidn’tmakeAYP.Schoolsareorderedfromlowest
tohighestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbyaverageMAPperformanceofstudentswithintheschool;itsscaleisshown
ontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-levelperformanceandscoresabovezerodenoteabove-grade-
levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebettertheaverageperformanceandthelowerthenumber,theworsethe
averageperformance.Thenumberinparenthesesaftereachschoolnameindicatesthenumberofstates(outof28)inwhichthatschoolwouldhavemadeAYP
NCLBaccountabilitysystemforelementaryandmiddle 2 elementary schools made AYP while 16 failed to
schools. In addition, federal law requires 95% of each makeit.Thetrianglesinthefigureshowtheaverageac-
school’sstudents—and95%ofthestudentsineachsub- ademicperformanceofstudentswithintheschool,with
group—toparticipateintesting. negative values indicating below-grade-level perform-
ancefortheaveragestudent,andpositivevaluesindicat-
Toreiterate,then,AYPdecisionsinthecurrentstudyare ingabove-grade-levelperformance.Thepassingschools
modeledsolelyontestperformancedataforasingleac-
areintherighthalfofthefigure,meaningthatthehigh-
ademicyear.Foreachschool,wecalculatedreadingand
estperformingstudentswerefoundattheseschools.
mathproficiencyrates(alongwithanyconfidenceinter-
vals)todeterminewhethertheoverallschoolpopulation
Yet,almostwithoutregardtoaveragestudentperform-
and any qualifying subgroups achieved the AMOs.We
ance, the only schools that actually made AYP were
deemed that a school made AYP if its overall student
those with relatively few qualifying subgroups—and
body and all its qualifying subgroups met or exceeded
thusthefewesttargetstomeet(becauseeachsubgroup
itsAMOs.Again,Appendix1suppliesfurthermethod-
ologicaldetail. has its own separate targets). Only Winchester and
Rooseveltpassed,andtheyhadjustfourandsixtargets,
respectively. Each had to make AYP for its overall stu-
How Did the Sample Schools
dentpopulationinreadingandmath(twotargets)and
Fare under Idaho’s AYP Rules?
for its white population (two more targets); Roosevelt
Figure3illustratestheAYPperformanceofthesample also had to make AYP for its low-income population
elementaryschoolsunderIdaho’s2008AYPrules.Only (twotargets).
TheAccountabilityIllusion 6
Table2.Elementaryschoolsubgroupperformanceofsampleschoolsunderthe2008IdahoAYPrules
SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP? Idaho
Clarkson 51.1% 40.8% N N N N N N N N 8 0 0% N 1
Maryweather 57.1% 50.7% N N N N N N N N N N Y N 12 1 8% N 1
Few 64.6% 51.9% N N N N N N N N N N 10 0 0% N 1
Nemo 66.0% 67.0% N N N N Y N 6 1 17% N 7
IslandGrove 69.3% 66.7% N N N N N N Y N 8 1 13% N 4
JFK 72.9% 60.4% Y N N N N N N N Y N 10 2 20% N 3
Scholls 81.7% 68.8% Y N N N Y N N N Y N 10 3 30% N 7
Hissmore 80.6% 72.1% Y N N N Y N Y N Y N 10 4 40% N 7
WolfCreek 72.5% 67.6% Y N N N N N N Y N 9 2 22% N 5
AliceMayberry 77.2% 75.1% Y N N N Y N Y N Y Y 10 5 50% N 9
WayneFineArts 79.3% 81.0% Y Y N N Y Y 6 4 67% N 21
Winchester 78.8% 79.1% Y Y Y Y 4 4 100% Y 22
Coastal 82.2% 74.8% Y N N N N N Y N N N N N Y Y 14 4 29% N 3
Paramount 81.0% 76.1% Y N N N N N Y Y 8 3 38% N 7
ForestLake 88.5% 84.4% Y Y N N Y N Y Y 8 5 63% N 8
Marigold 91.0% 85.6% Y Y N N N N Y Y 8 4 50% N 10
Roosevelt 93.6% 91.5% Y Y Y Y Y Y 6 6 100% Y 28
KingRichard 89.9% 88.8% Y Y N N N N Y Y 8 4 50% N 14
Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino=
Hispanic;AmericanIndian/AlaskaNative=AI/AN.
Note:Schoolsareorderedfromlowest(Clarkson)tohighest(KingRichard)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading
performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof
studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs.
Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber
ofstatesinthestudyforwhichthatschoolmetAYP.
Figure4illustratestheAYPperformanceofthesample schoolhasfewertargetstomeetbecauseithasfewersub-
middle schools under the 2008 Idaho AYP rules. groups.Figures3and4donot,however,indicatewhich
None of the 18 schools in our sample passed—even subgroupsfailedinwhichschool.Informationonindi-
WalterJones,themiddleschoolthatmakesAYPinthe vidualsubgroupperformanceappearsinTables2and3
greatest number of states (20) or the school with the forelementaryandmiddleschools,respectively.
highest performing students (Chaucer) didn’t make
AYP in Idaho. Tables2and3showwhichsubgroupsqualifiedforeval-
uation at each school (i.e., whether the number of stu-
dents within that subgroup exceeded the state’s
Where Do Schools Fail?
minimum n), and whether that subgroup passed or
Figure 3 illustrates how some elementary schools with failed. Although all schools are evaluated on the profi-
middling performance can still make AYP when the ciency rate of their overall population, potential sub-
7 THOMASB.FORDHAMINSTITUTE
Table3.Middleschoolsubgroupperformanceofsampleschoolsunderthe2008IdahoAYPrules
Idaho SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP?
McBeal 48.3% 52.8% N N N N N N N N N N N N N N N N Y N 18 1 6% N 0
BarringerCharter 53.3% 57.0% N N N N N N N N N N 10 0 0% N 0
MLAndrew 48.5% 56.8% N N N N N N N N N N N N 12 0 0% N 0
Pogesto 44.4% 61.1% N N N N 4 0 0% N 15
McCordCharter 50.8% 60.3% N N N N N N N N N N N N 12 0 0% N 0
Tigerbear 60.9% 55.7% N N N N N N N N Y N 10 1 10% N 0
Chesterfield 62.5% 57.8% N N N N N N N N Y N 10 1 10% N 1
Filmore 62.0% 66.4% N N N N N N N N Y N 10 1 10% N 1
Barban- 59.0% 61.7% N N N N N N N N N N Y Y 12 2 17% N 0
Kekata 68.4% 66.1% N N N N N N N N N N N N Y Y 14 2 14% N 0
Hoyt 69.2% 68.6% N N N N N N N N Y Y 10 2 20% N 2
BlackLake 73.2% 68.8% Y N N N N N N N Y Y N N Y N 14 4 29% N 0
LakeJoseph 70.1% 72.5% Y N N N N N N N N N Y Y 12 3 25% N 2
Zeus 72.2% 71.6% Y N N N N N N N N N N N Y N 14 2 14% N 1
OceanView 74.3% 81.3% Y Y N N N N N N N N Y Y 12 4 33% N 2
WalterJones 82.0% 82.9% Y Y N N N N Y Y 8 4 50% N 20
Artemus 81.0% 78.1% Y Y N N N N Y N N N Y Y 12 5 42% N 3
Chaucer 83.2% 86.6% Y Y N N N N N N Y Y N N Y Y 14 6 43% N 5
Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino=
Hispanic;AmericanIndian/AlaskaNative=AI/AN.
Note:Schoolsareorderedfromlowest(McBeal)tohighest(Chaucer)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading
performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof
studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs.
Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber
ofstatesinthestudyforwhichthatschoolmetAYP.
groups that are separately evaluated for AYP include schoollevelmetmathorreadingtargets,exceptfor
SWDs,studentswithLEP,low-incomestudents,andthe whiteyoungstersinmath.
following race/ethnic categories: African American,
Asian/Pacific Islander, Hispanic/Latino, American In- Tables4and5summarizesubgroupperformanceforele-
dian/AlaskaNative,andwhite. mentaryandmiddleschools,respectively.Weseethatevery
schoolwithlargeenoughpopulationsofstudentswithdis-
Theschool-by-schoolfindingsinTables2and3showthat: abilities, LEP, Hispanic, or American Indian/Alaska Na-
tives to qualify as separate subgroups failed to meet its
(cid:1) Most elementary schools met targets in math, but reading and math targets for these students. In fact, the
notinreading,fortheiroverallstudentpopulations. onlysubgroupswhereanyschoolsinthesamplemettheir
targets in both reading and math were Asian/Pacific Is-
(cid:1) Almost no subgroups at the elementary or middle lander,andwhite.
TheAccountabilityIllusion 8
Table4.Summaryofsubgroupperformanceofsampleelementaryschoolsunderthe2008APYIdahorules
Numberofschoolswith Numberofschoolswhere Numberofschoolswhere
SUBGROUP qualifyingsubgroups subgroupfailedtomeetmath subgroupfailedtomeetreading I
d
target target
a
Studentswithdisabili"es 11 10 11 h
o
StudentswithlimitedEnglish
4 4 4
proficiency
Low-incomestudents 17 11 16
African-Americanstudents 5 3 5
Asian/PacificIslanderstudents 0 0 0
Hispanicstudents 7 7 7
AmericanIndian/AlaskaNa"ve
0 0 0
students
Whitestudents 16 0 7
Table5.Summaryofsubgroupperformanceofsamplemiddleschoolsunder2008APYIdahorules
Numberofschoolswith Numberofschoolswhere Numberofschoolswhere
SUBGROUP qualifyingsubgroups subgroupfailedtomeetmath subgroupfailedtomeetreading
target target
Studentswithdisabili"es 16 16 16
StudentswithlimitedEnglish
7 7 7
proficiency
Low-incomestudents 17 17 17
African-Americanstudents 10 10 10
Asian/PacificIslanderstudents 4 1 2
Hispanicstudents 14 14 14
AmericanIndian/AlaskaNa"ve
1 1 1
students
Whitestudents 17 3 9
Characteristics of Schools withMassachusetts,SouthDakota,Montana,andNorth
Dakota) that have zero passing middle schools in our
that Did and Didn’t Make AYP
sample(seeFigure1).
A close look at Figures 3 and 4 indicates that Idaho’s
NCLBaccountabilitysystem(atleastintermsoftheel- ThemostlikelyexplanationforthedifferenceinIdaho,
ements studied here) failed more schools than do most relative to the other states examined, has to do with
statesinoursample.Infact,onlytwostates(Massachu- Idaho’smorestringentAYPrules.Definingsubgroupsat
setts and Nevada) failed more elementary schools than 34 means that a school in Idaho will have more sub-
Idaho. Similarly, Idaho is one of only five states (along groups and consequently more chances to fail to make
9 THOMASB.FORDHAMINSTITUTE
Table6.Comparisonsbetweenschoolsthatdidanddidn’tmakeAYPinIdaho,2008
o ElementarySchools MiddleSchools
h
a MadeAYP FailedtomakeAYP MadeAYP FailedtomakeAYP
d
Numberofschoolsinsample 2 16 0 18
I
Averagestudentbodysize 225 315 n/a 859
Average%lowincome 13 50 n/a 45
Average%nonwhite 25 43 n/a 44
Averageperformance† 6.16 0.61 n/a -0.05
Average%growth‡ 121 114 n/a 98
Averagenumberoftargetstomeet 5 9 n/a 12
†StudentperformanceismeasuredbyNWEA’sMAPassessmentandisexpressedasanindexofgradelevelnormativeperformance.Scoresbelowzero(whichisthegrade
levelmedian)denotebelow-grade-levelperformanceandscoresabovezerodenoteabove-grade-levelperformance.Oneunitdoesnotequalagradelevel;however,
thehigherthenumber,thebettertheaverageperformanceandthelowerthenumber,theworsetheaverageperformance.
‡AveragegrowthreferstoimprovementfromfalltospringontheNWEAMAPassessments,averagedacrossallstudentswithintheschool.Growthisexpressedasan
indexvaluerelativetoNWEAnormsandisscaledasapercentage.Thus,100%meansthatstudentsattheschoolareachievingnormativelevelsofgrowthfortheirage
andgrade.Lessthan100%growthmeansthattheaveragestudentisincreasingbylessthannormativeamounts,whilepercentagesover100meanthattheaverage
studentisexceedingnormativegrowthexpectations.
AYP.8Additionally,Idahodoesnotuseaconfidenceinter- Idaho. Looking across the 28 state accountability sys-
valasdomostoftheotherstatesexamined.Thismeans tems, this puts Idaho near the lower end of the sample
thatitsschoolshaveamoredifficulttimemeetingtheir distribution in terms of the number of schools making
targetscomparedtostatesthatuseconfidenceintervals. AYP(seeFigure1).
This is consistent with the patterns shown inTable 6, Several other factors are important to note for Idaho.
which compares schools making and not making AYP First,Idaho’sminimumsubgroupsizeisrelativelysmall
on several academic and demographic dimensions. in comparison to most other states examined in the
Within the sample, schools that make AYP do indeed study,meaningthatschoolsinIdahowillbeaccountable
showhigheraveragestudentperformance,buttheyalso formoresubgroupsthanwouldsimilarschoolsinother
differ in the following ways: they have smaller student stateswithhighersubgroupsizes.Finally,Idaho,unlike
populations,fewersubgroups(andthusfewertargetsto mostotherstates,doesnotapplyconfidenceintervalsso
meet),andlowerpercentagesoflow-incomestudents. schoolswillhavegreaterdifficultyachievingtheirannual
targetsthanequivalentschoolsinotherstatesthatreport
aconfidenceinterval.
Concluding Observations
This study examined the test performance data of stu- TheoverridinggoalofthefederalNCLBistoeliminate
dentsfrom18elementaryand18middleschoolsacross educationaldisparitieswithinandacrossstates;itisim-
the country to see how these schools would fare under portant to consider whether states’ annual decisions
Idaho’sAYPrules(andAMOs)for2008.Wefoundthat about the progress of individual schools are consistent
only2elementaryschoolsandnomiddleschools—2in withthisaim.Insomerespects,Idaho’sNCLBaccount-
all from a sample of 36—would have made AYP in ability system is working exactly as Congress intended:
8Itisworthnoting,however,thatschoolsinIdahoarelikelytobesmallandannsizeof34probablymakessense.
TheAccountabilityIllusion 10