Table Of Content

Kansas Executive Summary (cid:1)Lookingacrossthe28stateaccountabilitysystems K examinedinthestudy,onlytwostatespassedfewer a TheintentoftheNoChildLeftBehind(NCLB)Actof n of the sample elementary schools than Kansas s 2001istoholdschoolsaccountableforensuringthatall (Kansasties5otherstateswithonly2elementary a of their students achieve mastery in reading and math, s schools making AYP). In addition, Kansas is one with a particular focus on groups that have traditionally of 6 states with a single passing middle school in beenleftbehind.UnderNCLB,statessubmitaccounta- thesample(seeFigure1). bilityplanstotheU.S.DepartmentofEducationdetailing therulesandpoliciestobeusedintrackingtheadequate (cid:1) Many of the schools in our sample that failed to yearlyprogress(AYP)ofschoolstowardthesegoals. make AYP in Kansas are meeting expected targets fortheiroverallpopulationsbutfailedbecauseofthe ThisreportexaminesKansas’sNCLBaccountabilitysys- performance of individual subgroups, particularly tem—particularly how its various rules, criteria, and studentswithdisabilities(SWDs)andstudentswith practices result in schools either making AYP or not limitedEnglishproficiency(LEP).2 makingAYP.ItalsogaugeshowtoughKansas’ssystemis compared with other states. For this study, we selected 36schoolsfromvariousstatesaroundthenation,schools that vary by size, achievement, and diversity, among otherfactors,anddeterminedwhethereachwouldmake AYPunderKansas’ssystemaswellasunderthesystems UnderKansas’saccountabilitysystem,16of18 of27otherstates.Weusedschooldataandproficiency elementaryschoolsand17of18middleschoolsin cutscore1estimatesfromacademicyear2005–2006,but oursamplefailtomakeAYPin2008.Thisplaces applied them against Kansas’s AYP rules for academic Kansasnearthelowendofthestatedistributionin year2007–2008(shortenedto“2008”inthisreport). termsofthenumberofschoolsmakingAYP. Kansas’sdefinitionsofproficiencygenerallyranked Herearesomekeyfindings: aboutaveragecomparedwiththestandardssetby (cid:1) We estimate that 16 of 18 elementary schools and theotherstates.However,Kansas’sannualtargets 17of18middleschoolsinoursamplewouldfailto inreading(thepercentageofstudentsinvarious make adequate yearly progress in 2008 under subgroupsthathavetomeetproficiency)are Kansas’saccountabilitysystem.Thishighfailurerate relativelydifficulttoachieve.Specifically,75.6 ispartlyexplainedbyoursample,whichintention- percentofagivenpopulationinanyschoolwould allyincludessomeschoolswithrelativelylargepop- havetobeproficientonthestatereadingexamfor ulations of low-performing students. But it’s also partially explained by Kansas’s demanding annual theschooltomakeAYPin2008.Inaddition, targets for students (roughly 75% of students were Kansas’sminimumsubgroupsize(30)isslightly expectedtomeetproficiencytargetsin2008). lowerthaninmanyoftheotherstateswe examined.Thismeansthatmoregroupsofstudents 1 A cut score is the minimum score a student must receive on areheldseparatelyaccountablethanwouldbein NWEA’sMeasuresofAcademicProgress(MAP)thatisequivalentto manyotherstates.Infact,everysingleschoolwitha performingproficientontheKansasAssessmentProgram. 2It’simportanttonotethatstudentsinsubgroupsnotmeetingthe limitedEnglishproficient(LEP)orstudentswith minimumnsizesarestillincludedforaccountabilitypurposesinthe disabilities(SWD)subgroupfailedtomakeAYPin overallstudentcalculations;theysimplyarenottreatedastheirown subgroup. Kansas. 1 THOMASB.FORDHAMINSTITUTE 18 s 16 a P Y s gA 14 n n ki a Ma 12 K ols o 10 h Sc ple 8 m aa ofS 6 er mb 4 u N 2 0 Massachuse s Nevada Idaho NorthDakota Kansas Washington Wyoming Indiana SouthCarolina Montana Florida Vermont NewJersey NewHampshire Maine NewMexico Delaware Colorado RhodeIsland Georgia Illinois Ohio Minnesota Michigan California Texas Arizona Wisconsin Elementary Schools Middle Schools Figure1.NumberofsampleschoolsmakingAYPbystate Note:MiddleschoolswerenotincludedforTexasandNewJersey;absenceofamiddleschoolbarinthosestatesmeans“notapplicable”asopposedtozero.Stateslike IdahoandNorthDakota,however,havezeropassingmiddleschools. (cid:1) In Kansas, as in most states, schools with fewer themorecommonlyused95%confidenceinterval. subgroups attained AYP more easily than schools Althoughtheconfidenceintervaldidhelpahandful withmoresubgroups,evenwhentheiraveragestu- ofschoolsinKansasmeetoverallreadingandmath dentperformancewasmuchlower.Inotherwords, targets,ithadlittleornoimpactonfinalAYPout- schools with greater diversity and size face greater comes because individual subgroups still failed to challengesinmakingAYP. meettheirtargets(allofaschool’ssubgroupsmust havemettheirtargetsfortheschooltomakeAYP). (cid:1) As in other states, middle schools had greater diffi- culty reaching AYP in Kansas than did elementary (cid:1) A strong predictor of whether or not a school will schools,primarilybecausetheirstudentpopulations make AYP under Kansas’s system is whether it has are larger and therefore have more qualifying sub- enoughSWDsorEnglishlanguagelearners3toqual- groups—not because their student achievement is ifyasaseparatesubgroup.Everysingleschoolwith lowerthanintheelementaryschools. evenonesuchsubgroupfailedtomakeAYP.4 (cid:1) Most states examined apply confidence intervals (or Introduction marginsofstatisticalerror)totheirmeasurementsof student proficiency rates. However, Kansas’s 99% TheProficiencyIllusion(Croninetal.2007a)linkedstu- confidenceintervalgiveschoolsgreaterleniencythan dentperformanceonKansas’stestsandthoseof25other 3Notethatweuse“LEPstudents”and“Englishlanguagelearners”interchangeablytorefertostudentsinthesamesubgroup. 4SWDsaredefinedasthosestudentsfollowingindividualizededucationplans.WeshouldalsonotethatoursubgroupfindingsforLEP studentsandSWDsmaybeslightlymorenegativethanactualfindings,mostlybecauseofthedifferencesintestingpracticesbetweenthe MeasuresofAcademicProgress(MAP),theassessmentweusedinthisstudy,andintheKansasAssessmentProgram,thestandardizedstate test.Specifically,theU.S.DepartmentofEducationhasissuedNCLBguidelinespermittingschoolstoexcludesmallpercentagesofLEP studentsandSWDsfromtakingstatetests,orprovidingthemalternateassessments.Inthisstudy,however,novalidMAPscoreswereomitted fromconsideration. TheAccountabilityIllusion 2 states to the Northwest Evaluation Association’s beanalyzedseparately(alsocalledtheminimumn[num- (NWEA’s) Measures of Academic Progress (MAP), a berofstudentsinsample]size).Therationaleisthatre- computerized adaptive test used in schools nationwide. portingtheresultsofverysmallsubgroups—fewerthan K a Thissinglecommonscalepermittedcross-statecompar- tenpupils,forexample—couldjeopardizestudents’con- n isons of each state’s reading and math proficiency stan- fidentialityandriskpresentinginaccurateresults.(With s dardstomeasureschoolperformanceundertheNoChild suchsmallgroups,randomevents,likeonestudentbeing a s Left Behind (NCLB) Act of 2001.That study revealed out sickon test day, could skew theoutcome.) Because profounddifferencesinstates’proficiencystandards(i.e., of this flexibility, states have set widely varying n sizes howdifficultitistoachieveproficiencyonthestatetest), for their subgroups, from as few as 10 youngsters to as andevenacrossgradeswithinasinglestate. manyas100. Our study expandsonTheProficiencyIllusion by exam- Manystateshavealsoadoptedconfidenceintervals—ba- ining other key factors of state NCLB accountability sicallymarginsofstatisticalerror—totrytoaccountfor plansandhowtheyinteractwithstateproficiencystan- potential measurement error within the state test. In dards to determine whether the schools in our sample somestates,thesemarginsarequitewide,whichhasthe made adequate yearly progress (AYP) in 2008. Specifi- effectofmakingiteasiertoachieveanannualtarget. cally, we estimated how a single set of schools, drawn fromaroundthecountry,wouldfareunderthediffering AlloftheseAYPrulesvarybystate,whichmeansthata rulesfordeterminingAYPin28states(theoriginal25in schoolthatmakesAYPinWisconsinorOhio,forexam- The Proficiency Illusion plus 3 others for which we now ple,mightnotmakeitunderSouthCarolina’sorIdaho’s have cut score estimates). In other words, if we could rules(U.S.DepartmentofEducation2008). somehow move these entire schools—with their same mix of characteristics—from state to state, how would What We Studied they fare in terms of making AYP? Will schools with high-performing students consistently make AYP? Will Wecollectedstudents’MAPtestscoresfromthe2005– schoolswithlow-performingstudentsconsistentlyfailto 2006academicyearfrom18elementaryand18middle make AYP? If AYP determinations for schools are not schoolsaroundthecountry.WealsocollectedtheNCLB consistentacrossstates,whatleadstotheinconsistencies? subgroupdesignationsforallstudentsinthoseschools— NCLB requires every state, as a condition of receiving inotherwords,whethertheyhadbeenclassifiedasmem- Title I funding, to implement an accountability system bersofaminoritygrouporasEnglishlanguagelearners, that aims to get 100% of its students to the proficient amongothersubgroups. level on the state test by academic year 2013–2014. In theinterveningyears,statessetannualmeasurableobjec- Theschoolswerenotselectedasarepresentativesample tives(AMOs).Thisisthepercentageofstudentsineach of the nation’s population. Instead, we selected the school,andineachsubgroupwithintheschool(suchas low income5 or African-American, among others), that schoolsbecausetheyexhibitedarangeofcharacteristics mustreachtheproficientlevelinorderfortheschoolto on measures such as academic performance, academic make AYP in a given year.The AMOs vary by state (as growth, and socioeconomic status (the latter calculated do,ofcourse,thedifficultyoftheproficiencystandards). bythepercentageofstudentsreceivingfreeorreduced- pricelunches).Appendix1containsacompletediscus- Statesalsodeterminetheminimumnumberofstudents sionofthemethodologyforthisprojectalongwiththe thatmustconstituteasubgroupinorderforitsscoresto characteristicsoftheschoolsample.6 5Low-incomestudentsarethosewhoreceiveafreeorreduced-pricelunch. 6Wegaveallschoolsinoursamplepseudonymsinthisreport. 3 THOMASB.FORDHAMINSTITUTE 50 45 s a 40 s g 35 n n ki a an 30 RR K e 25 Reading l en 20 Math c er 15 P 10 5 0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Figure2.Kansasreadingandmathcutscoreestimates,expressedaspercentileranks(2006) Note:ThisfigureillustratesthedifficultyofKansas’scutscores(orproficiencypassingscores)foritsreadingandmathtests,aspercentilesoftheNWEAnorm,ingrades threethrougheight.Higherpercentileranksaremoredifficulttoachieve.AllofKansas’scutscoresareatorbelowthe45thpercentile. ProficiencycutscoreestimatesfortheKansasAssessment whileschoolsaresupposedtoget75.6%oftheirgrade3- SystemaretakenfromTheProficiencyIllusion(asshown 8 students to the “proficient” level on the state reading in Figure 2), which found that Kansas’s definitions of test, and 75.6% of the grade 3-8 students in each sub- proficiency generally ranked about average compared group, applying the confidence interval means that the with the standards set by the other 25 states in that realtargetcanbelower(particularlywithsmallergroups).7 study. These cut scores were used to estimate whether studentswouldhavescoredasproficientorbetteronthe Note that we were unable to examine the impact of Kansas test, given their performance on MAP. Student NCLB’s“safeharbor”provision.Thisprovisionpermits test data and subgroup designations were then used to aschooltomakeAYPevenifsomeofitssubgroupsfail, determine how these 18 elementary and 18 middle as long as it reduces the number of nonproficient stu- schools would have fared under Kansas AYP rules for dentswithinany failingsubgroup by at least 10%rela- 2008. In other words, the school data and our profi- ciencycutscoreestimatesarefromacademicyear2005– tivetothepreviousyear’sperformance.Becausewehad 2006,butweareapplyingthemagainstKansas’s2008 accesstoonlyasingleacademicyear’sdata(2005–2006), wewerenotabletoincludethisinouranalysis.Asare- Table 1 shows the pertinent Kansas AYP rules that were sult,it’spossiblethatsomeoftheschoolsinoursample applied to elementary and middle schools in this study. that failed to make AYP according to our estimates Kansas’sminimumsubgroupsizeis30,whichisslightly wouldhavemadeAYPunderrealconditions. lowerthaninmanyoftheotherstatesweexamined.This means that Kansas’s schools would have to account for Furthermore,attendanceandtestparticipationratesare moresubgroupsthanwouldsimilarschoolsinotherstates. beyondthescopeofthestudy.Notethatmoststatesin- Furthermore,althoughmoststatesalsoapplyconfidence cludeattendanceratesasanadditionalindicatorintheir intervals(ormarginsofstatisticalerror)totheirmeasure- NCLBaccountabilitysystemforelementaryandmiddle ments of student proficiency rates, Kansas’s 99% confi- schools. In addition, federal law requires 95% of each denceintervalgivesschoolsgreaterleniencythanthemore school’sstudents—and95%ofthestudentsineachsub- commonlyused95%confidenceinterval.Soforinstance, group—toparticipateintesting. 7Wealsoconductedananalysistoshowtheeffectofconfidenceintervalsonthereadingandmathproficiencyratesforelementaryandmiddle schools.Wedescribethoseresultslaterinthereport. TheAccountabilityIllusion 4 Table1.KansasAYPrulesfor2008 Subgroupminimumn Race/ethnicity:30 K SWDs:30 a n Low-incomestudents:30 s a LEPstudents:30 s CI Appliedtoproficiencyratecalcula-ons? Yes;99%CI AMOs Baselineproficiencylevelsasof2002(%) 2008targets(%) READING/LANGUAGEARTS Grade3 67.7 75.6 Grade4 67.7 75.6 Grade5 67.7 75.6 Grade6 67.7 75.6 Grade7 67.7 75.6 Grade8 67.7 75.6 MATH Grade3 62.5 73.4 Grade4 62.5 73.4 Grade5 62.5 73.4 Grade6 62.5 73.4 Grade7 62.5 73.4 Grade8 62.5 73.4 Sources:U.S.DepartmentofEducation(2008);CouncilofChiefStateSchoolOfficers(2008). Abbreviations:SWDs=studentswithdisabilities;LEP=limitedEnglishproficiency;CI=confidenceinterval;AMOs=annualmeasurableobjectives Toreiterate,then,AYPdecisionsinthecurrentstudyare in Figure 3 show the average academic performance of modeledsolelyontestperformancedataforasingleaca- studentswithintheschool,withnegativevaluesindicat- demic year. For each school, we calculated reading and ing below-grade-level performance for the average stu- math proficiency rates (along with any confidence inter- dent, and positive values indicating above-grade-level vals) to determine whether the overall school population performance.The schools making AYP are in the right and any qualifying subgroups achieved the AMOs. We halfofthefigure,meaningthatthehighestperforming deemedthataschoolmadeAYPifitsoverallstudentbody studentswerefoundattheseschools. andallitsqualifyingsubgroupsmetorexceededitsAMOs. Again,Appendix1suppliesfurthermethodologicaldetail. Yetalmostwithoutregardtoaveragestudentperformance, theonlyschoolsactuallytomakeAYPwerethosewithrel- How Did the Sample Schools ativelyfewqualifyingsubgroups—andthusthefewesttar- getstomeet.Forexample,WayneFineArtspassed,buthad Fare under Kansas’s AYP Rules? onlyeighttargets–twoinreadingandmathfortheoverall Figure 3 illustrates the AYP performance of the sample population, two in reading and math for its low-income elementaryschoolsunderKansas’s2008AYPrules.Only population,twoinreadingandmathforitsAsian/PacificIs- 2elementaryschoolsoutof16madeAYP.Thetriangles landerpopulation,andtwoforitswhitepopulation. 5 THOMASB.FORDHAMINSTITUTE 18 15 ol 16 o s h 10 c a 14 nS s ets 12 5 cei n g n r a a Ta 10 m K erof 8 0 erfor b P mm 6 -5 tnt u e N d 4 u -10 St 2 ge a r 0 -15 ve A 1) 1) 1) 7) 5) 3) 7) 7) 5) 9) 1) 2) 3) 7) 8) 0) 8) 4) ( ( ( ( ( ( ( ( ( ( 2 2 ( ( ( 1 2 1 Clarkson Maryweather Few Nemo IslandGrove ohnF.Kennedy Scholls Hissmore WolfCreek Mayberry ayneFineArts( Winchester( Coastal Paramount ForestLake Marigold( Roosevelt( KingRichard( J W TargetsPassed TargetsFailed Average Student Performance Figure3.AYPperformanceoftheelementaryschoolsampleunderKansas’s2008AYPrules Note:ThisfigureindicateshoweachoftheelementaryschoolswithinthesamplefaredunderKansas’sAYPrules(asdescribedinTable1).Thebarsshowthenumber oftargetsthateachschoolhastomeetinordertomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnotmeetthem(lightblue).The moresubgroupsinaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailedtomeettheAMOsforevenasinglesubgroupdidn’tmake AYP,soanylightbluemeansthattheschoolfailed.WinchesterElementary,forexample,metsevenofitseighttargets,butbecauseitdidn’tmeetthemall,itdidn’t makeAYP.Schoolsareorderedfromlowesttohighestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbytheaverageMAPperformance ofstudentswithintheschool;itsscaleisshownontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-level performanceandscoresabovezerodenoteabove-grade-levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebetterthe averageperformanceandthelowerthenumber,theworsetheaverageperformance. Figure 4 illustrates the AYP performance of the sample areassistedbytheconfidenceinterval(notehowtheor- middleschoolsunderthe2008KansasAYPrules.Of18 ange line falls within the light blue band). We know middleschoolsinoursample,onlyonemadeAYP(Wal- fromFigures3and4,however,thatalloftheseschools terJones),whichhasrelativelyfewqualifyingsubgroups. failedtomakeAYPbecauseofsubgroupperformance. Figures 5 and 6 indicate the degree to which schools’ The effect of confidence intervals on the reading profi- overallmathproficiencyratesareaidedbyKansas’scon- ciency rates for elementary and middle schools is much fidence interval for elementary and middle schools, re- thesame(notshown).Inreading,fourelementaryschools spectively.Onthesefigures,thedarkbluebarsshowthe (Hissmore,Mayberry,Coastal,andParamount)andthree actualproficiencyratesateachschool,andthelightblue middleschools(Pogesto,Hoyt,andZeus)mettheoverall barsshowthedegreetowhichtheseproficiencyratesare targetwiththeconfidenceinterval,althoughtheseschools increased by the application of the confidence interval. stillfailedtomeetalltheirsubgrouptargets(seeFigures3 The orange lines show the AMO needed to meet AYP. and 4). So, though the confidence interval does help Figures5and6showthatfourofthesampleelementary someschoolstomeetoverallreadingandmathtargets, schools(Nemo,IslandGrove,JFK,andWolfCreek)and it has little or no impact on final AYP outcomes since three middle schools (Kekata, Hoyt, and Lake Joseph) individualsubgroupsfailedtomeettargets.8 8Inthecurrentanalyses,confidenceintervalswereappliedtoboththeoverallschoolpopulationandtoalleligiblesubgroupsinoursampleschools. Thus,theultimateimpactoftheconfidenceintervalislikelylargerthantheimpactdepictedinFigures5and6.However,wechosenottoshow howtheconfidenceintervalimpactedsubgroupperformancebecauseitwouldhaveaddedgreatlytothereport’slengthandcomplexity. TheAccountabilityIllusion 6 20 12 ol 18 10 o h K c 16 8 S n a ets 14 6 cei n g n Tar 12 4 ma s of 10 2 or a ber 8 0 Perf s mm ntt u 6 -2 e N d u 4 -4 St e 2 -6 g a r 0 -8 ve A 0) 0) 0) 5) 0) 0) 1) 1) 0) 0) 2) 0) 2) 1) 2) 0) 3) 5) ( ( ( 1 ( ( ( ( ( ( ( ( ( ( ( 2 ( ( McBeal arringerCharter MLAndrew Pogesto( McCord Tigerbear Chesterfield Filmore Barban Kekata Hoyt BlackLake LakeJoseph Zeus OceanView WalterJones( Artemus Chaucer B Targets Passed Targets Failed Average Student Performance Figure4.AYPperformanceofthemiddleschoolsampleunderKansas’s2008AYPrules Note:ThisfigureshowshoweachofthemiddleschoolswithinthesamplefaredunderKansas’sAYPrules(asdescribedinTable1).Thebarsshowthenumberoftargets thateachschoolhadtomeetinordertomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnotmeetthem(lightblue).Themoresubgroups inaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailedtomeettheAMOsforevenasinglesubgroupdidnotmakeAYP,soanylightblue meansthattheschoolfailed.Chaucer,forexample,met11ofits15targets,butbecauseitdidn’tmeetthemall,itdidn’tmakeAYP.Schoolsareorderedfromlowestto highestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbytheaverageMAPperformanceofstudentswithintheschool;itsscaleisshown ontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-levelperformanceandscoresabovezerodenoteabove-grade- levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebettertheaverageperformanceandthelowerthenumber,theworsethe averageperformance.Thenumberinparenthesesaftereachschoolnameindicatesthenumberofstates(outof28)inwhichthatschoolwouldhavemadeAYP. 1 0.9 0.8 0.7 e Rat 0.6 y nc 0.5 e cifi 00.44 o Pr 0.3 0.2 0.1 0 Clarkson Maryweather Few Nemo IslandGrove hnF.Kennedy Scholls Hissmore WolfCreek Mayberry ayneFineArts Winchester Coastal Paramount ForestLake Marigold Roosevelt KingRichard o W J MathProficiency Rate Math Proficiency Ratewith CI MathTarget Figure5.Impactoftheconfidenceintervalonelementaryschoolmathproficiencyrates Note:Thisfigureshowsthereportedproficiencyrateforthestudentpopulationasawholeandtheimpactoftheconfidenceintervalonmeetingannualtargets.The darkerportionsofthebarsshowtheactualproficiencyrateachieved,whilethelighter(upper)portionsofthebarsshowthemarginoferrorascomputedbythe confidenceinterval.Thefigureshowsthatfourschools(Nemo,IslandGrove,JFK,andWolfCreek)wereassistedbytheconfidenceinterval.Annualtargets(theorange lines)areconsideredtobemetbytheconfidenceintervaliftheyfallwithinthelightblueportion. 7 THOMASB.FORDHAMINSTITUTE 1 0.9 s a 0.8 s 0.7 n e a Rat 0.6 y K nc 0.5 e ciiofi 00.44 Pr 0.3 0.2 0.1 0 McBeal ngerCharter MLAndrew Pogesto McCord Tigerbear Chesterfield Filmore Barban Kekata Hoyt BlackLake LakeJoseph Zeus OceanView WalterJones Artemus Chaucer arri B Math Proficiency Rate MathProficiency RatewithCI MathTarget Figure6.Impactoftheconfidenceintervalonmiddleschoolmathproficiencyratesunderthe2008KansasAYPrules Note:Thisfigureshowsthereportedproficiencyrateforthestudentpopulationasawholeandtheimpactoftheconfidenceintervalonmeetingannualtargets.The darkerportionsofthebarsshowtheactualproficiencyrateachieved,whilethelighter(upper)portionsofthebarsshowthemarginoferrorascomputedbythe confidenceinterval.Thefigureshowsthatthreeschools(Kekata,Hoyt,andLakeJoseph)wereassistedbytheconfidenceinterval.Annualtargets(theorangelines)are consideredtobemetbytheconfidenceintervaliftheyfallwithinthelightblueportion. Where Do Schools Fail? Theschool-by-schoolfindingsinTables2and3showthat: Figures3and4illustratethatschoolswithlowormid- (cid:1)Threeelementary schools(Clarkson, Maryweather, dling performance can still make AYP when the school andFew)failedtomeetbothmathandreadingtar- hasfewertargetstomeetbecauseithasfewersubgroups. getsfortheiroverallschoolpopulation. Thesefiguresdonot,however,indicatewhichsubgroups failedorpassedinwhichschool.Informationonindivid- (cid:1)Fiveotherelementaryschools(Nemo,IslandGrove, ualsubgroupperformanceappearsinTables2and3for JFK, Scholls, andWolf Creek) in the sample failed elementaryandmiddleschools,respectively. tomeettheirreadingtargetsfortheiroverallpopu- lations. Tables2and3showwhichsubgroupsqualifiedforeval- (cid:1)Eightofthe17failingmiddleschoolsinthesample uation at each school (i.e., whether the number of stu- (McBeal, Barringer, ML Andrew, McCord, Tiger- dentswithinthatsubgroupexceededthestate’sminimum bear,Chesterfield,Filmore,andBarbanti)failedfor n),andwhetherthatsubgrouppassedorfailed.Although bothreadingandmathfortheiroverallpopulations. all schools are evaluated on the proficiency rate of their overallpopulation,potentialsubgroupsthatareseparately (cid:1) Most schools did not make AYP because of more evaluated for AYP purposes include SWDs, LEP stu- thanonesubgroup. dents, and the following race/ethnic categories: African American,Asian/PacificIslander,Hispanic/Latino,Amer- Tables4and5summarizetheperformanceofthevari- icanIndian/AlaskaNative,andwhite.Tables2and3also ous subgroups for elementary and middle schools, re- showwhetheraschoolmetAYPundertheKansasrules, spectively. We see that the performance of SWDs is andthetotalnumberofstateswithinthestudyinwhich provingespeciallychallengingundertheKansasaccount- thatschoolmetAYP. abilitysystem.Infact,everySWDgroupatthemiddle TheAccountabilityIllusion 8 Table2.Elementaryschoolsubgroupperformanceofsampleschoolsunderthe2008KansasAYPrules SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP? Kansas Clarkson 51.1% 36.1% N N N N N N N N N N 10 0 0% N 1 Maryweather 57.1% 47.0% N N N N N N N N N N Y Y 12 2 17% N 1 Few 64.6% 48.5% N N N N N N N N Y N Y N 12 2 17% N 1 Nemo 66.0% 63.7% Y N N N Y Y 6 3 50% N 7 IslandGrove 69.3% 65.4% Y N N N N N N Y Y 9 3 33% N 4 JFK 72.9% 57.5% Y N N N Y N N N Y N 10 3 30% N 3 Scholls 81.7% 66.5% Y N N N Y N Y N Y Y 10 5 50% N 7 Hissmore 80.6% 69.8% Y Y N N Y N Y N Y Y 10 6 60% N 7 WolfCreek 72.5% 66.7% Y N N N N N N N N Y Y 11 3 27% N 5 AliceMayberry 77.2% 71.3% Y Y N N Y N Y N Y Y 10 6 60% N 9 WayneFineArts 79.3% 77.6% Y Y Y Y Y Y Y Y 8 8 100% Y 21 Winchester 78.8% 77.3% Y Y Y N Y Y Y Y 8 7 88% N 22 Coastal 82.2% 72.9% Y Y N N N N Y N Y N Y N Y Y 14 7 50% N 3 Paramount 81.0% 73.9% Y Y Y N Y N Y Y 8 6 75% N 7 ForestLake 88.5% 83.3% Y Y N N Y Y Y Y 8 6 75% N 8 Marigold 91.0% 84.1% Y Y Y N Y N Y Y 8 6 75% N 10 Roosevelt 93.6% 90.5% Y Y Y Y Y Y 6 6 100% Y 28 KingRichard 89.9% 86.4% Y Y N N Y N Y N Y N Y Y 12 7 58% N 14 Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino= Hispanic;AmericanIndian/AlaskaNative=AI/AN. Note:Schoolsareorderedfromlowest(Clarkson)tohighest(KingRichard)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs. Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber ofstatesinthestudyforwhichthatschoolmetAYP. school level failed to meet targets in both reading and elementaryschoolsinoursample,RooseveltandWayne math.Asimilarproblemexistsforstudentswithlimited FineArts,madeAYPinthegreatestnumberofstates— Englishproficiency.Allofthosesubgroupsfailedtomeet 28and21,respectively.AndtheseschoolsmadeAYPin their targets, save for one passing (in math) at the ele- Kansas, too. Likewise, the elementary and middle mentarylevel(KingRichard). schoolsthatfailedtomakeAYPinthegreatestnumber ofstatesalsofailedtomakeAYPinKansas. Characteristics of Schools ButKansasisalsohometoananomaly.WinchesterEl- that Did and Didn’t Make AYP ementary(seeFigure3)madeAYPin22ofthe28states A close look at Figures 3 and 4 indicates that Kansas’s inoursample,butnotinKansas.InexaminingTable2, NCLBaccountabilitysystemis,insomerespects,behav- we can see that Winchester missed only one target in ing like those in other states. For example, among the reading for its SWD subgroup. This may be because 9 THOMASB.FORDHAMINSTITUTE Table3.Middleschoolsubgroupperformanceofsampleschoolsunderthe2008KansasAYPrules Kansas SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP? McBeal 52.3% 55.9% N N N N N N N N N N Y Y N N N N Y Y 18 4 22% N 0 BarringerCharter 55.3% 57.2% N N N N N N N N Y N 10 1 10% N 0 MLAndrew 50.1% 59.8% N N N N N N N N N N N N 12 0 0% N 0 Pogesto 50.0% 68.5% N Y N Y 4 2 50% N 15 McCordCharter 53.2% 63.0% N N N N N N N N N N N Y 12 1 8% N 0 Tigerbear 63.2% 61.0% N N N N N N N N Y Y 10 2 20% N 0 Chesterfield 66.3% 63.0% N N N N N N N N Y Y 10 2 20% N 1 Filmore 64.6% 71.1% N N N N N N N N N N Y Y 12 2 17% N 1 Barban- 61.8% 66.2% N N N N N N N N N N Y Y 12 2 17% N 0 Kekata 69.9% 69.0% Y N N N N N N N N N N N Y Y 14 3 21% N 0 Hoyt 71.6% 72.4% Y Y N N N N N N Y Y 10 4 40% N 2 BlackLake 75.4% 72.4% Y N N N N N N N N Y Y Y Y Y Y 15 7 47% N 0 LakeJoseph 71.5% 76.9% Y Y N N N N N Y N N Y Y 12 5 42% N 2 Zeus 74.4% 74.4% Y Y N N N N N N N N N N Y Y 14 4 29% N 1 OceanView 77.2% 83.4% Y Y N N N N N N N N Y Y 12 4 33% N 2 WalterJones 81.4% 81.1% Y Y Y Y Y Y Y Y 8 8 100% Y 20 Artemus 81.0% 81.8% Y Y N N N N Y Y N N Y Y 12 6 50% N 3 Chaucer 85.2% 88.1% Y Y N N N N Y Y Y Y Y Y Y Y Y Y 16 12 75% N 5 Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino= Hispanic;AmericanIndian/AlaskaNative=AI/AN. Note:Schoolsareorderedfromlowest(McBeal)tohighest(Chaucer)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs. Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber ofstatesinthestudyforwhichthatschoolmetAYP. Kansas’s minimum subgroup size is somewhat smaller fewer subgroups (and thus fewer targets to meet), and thaninmostotherstatesexamined,meaningthatschool lowerpercentagesoflow-incomeandnonwhitestudents. may have more accountable subgroups under Kansas rulesthanitwouldinotherstates. Concluding Observations This is consistent with the patterns shown inTable 6, This study examined the test performance data of stu- which compares schools that did and didn’t make AYP dentsfrom18elementaryand18middleschoolsacross onanumberofacademicanddemographicdimensions. the country to see how these schools would fare under Within the sample, schools that make AYP do indeed Kansas’sAYPrules(andAMOs)for2008.Wefoundthat showhigheraveragestudentperformance,buttheyalso only2elementaryschoolsand1middleschool—3out differinthefollowingways:theyhavemuchsmallerstu- ofasampleof36—makeAYPinKansas.Lookingacross dentpopulations(especiallyatthemiddleschoollevel), the 28 state accountability systems examined in the TheAccountabilityIllusion 10