ERIC ED504719: The Accountability Illusion: North Dakota

2009·0.26 MB·English

by  ERIC

North Dakota
Executive Summary tionallyincludessomeschoolswitharelativelylarge N
populationoflow-performingstudents.) o
TheintentoftheNoChildLeftBehind(NCLB)Actof r
t
2001istoholdschoolsaccountableforensuringthatall (cid:1)Lookingacrossthe28stateaccountabilitysystemsex- h
theirstudentsachievemasteryinreadingandmath,with aminedinthestudy,wefindthatthenumberofschools
D
aparticularfocusongroupsthathavetraditionallybeen makingAYPinNorthDakotaisexceededin20other
a
leftbehind.UnderNCLB,statessubmitaccountability samplestates(fivestatestiewithNorthDakota,each k
o
planstotheU.S.DepartmentofEducationdetailingthe withtwoelementaryschoolsmakingAYP).Inaddition,
t
rules and policies to be used in tracking the adequate North Dakota is one of five states with zero passing a
yearlyprogress(AYP)ofschoolstowardsthesegoals. middleschoolsinthesample(seeFigure1).
ThisreportexaminesNorthDakota’sNCLBaccounta- (cid:1) Many of the schools in our sample that failed to
make AYP in North Dakota are meeting expected
bilitysystem—particularlyhowitsvariousrules,criteria
targets for their overall populations but failing be-
and practices result in schools either making AYP—or
cause of the performance of individual subgroups,
not making AYP. It also gauges how tough North
particularly students with disabilities and English
Dakota’s system is compared with other states. For this
languagelearners.2
study, we selected 36 schools from around the nation,
schools that vary by size, achievement, and diversity,
(cid:1) Two sample schools failed to make AYP in North
among other factors, and determined whether or not
DakotathatmadeAYPinmostotherstates.Thisis
eachwouldmakeAYPunderNorthDakota’ssystemas
likelyduetothefactthatNorthDakota’sminimum
well as under the systems of 27 other states. We used
subgroup size of 10 is small, compared to other
schooldataandproficiencycutscore1estimatesfromac-
statesinthestudy.3Inaddition,NorthDakota’san-
ademic year 2005–2006, but applied them against
nualtargetsforproficiencyarerelativelyambitious.
NorthDakota’sAYPrulesforacademicyear2007–2008
(shortenedto“2008”inthisreport).
Herearesomekeyfindings:
Onlytwoofthe36schoolsinoursamplemakeAYPin
(cid:1) We estimate that 16 of 18 elementary schools and
2008underNorthDakota’saccountabilitysystem.
allofthe18middleschoolsinoursamplefailedto
makeadequateyearlyprogressin2008underNorth Thegreatestcontributingfactortothehighfailure
Dakota’s accountability system. (This high failure rateisthatNorthDakota’sminimumsubgroupsizeis
rateispartlyexplainedbyoursample,whichinten- 10,whichisconsiderablysmallerthanmostother
statesweexamined.ThismeansthatschoolsinNorth
1 A cut score is the minimum score a student must receive on
Dakotawillhavemoreaccountablesubgroupsthan
NWEA’sMeasuresofAcademicProgress(MAP)thatisequivalentto
performing proficient on the North Dakota State Assessment wouldsimilarschoolsinotherstates.Ontheother
(NDSA).
hand,NorthDakota’sproficiencystandardsareabout
2It’simportanttonotethatstudentsinsubgroupsnotmeetingthe
minimumnsizesarestillincludedforaccountabilitypurposesinthe averagewhencomparedtotheotherstatesinthe
overallstudentcalculations;theyaresimplynottreatedastheirown
subgroup study.Thestatealsousesa99percentconfidence
3ThestateofNorthDakotadoesnothaveaminimumschoolsize, intervalwhichprovidesschoolswithgreaterleniency
soithasalargenumberofverysmallschools.Inaddition,thestate’s
thanthemorecommonlyused95percentconfidence
populationhasbeendeclininginrecentyears.TheU.S.CensusBu-
reau(2002)listsNorthDakota’spopulationatalittleover642,000,
interval.Thelatterlikelyexplainswhytwosample
47thintheUnitedStates.Therefore,smallersubgroupsizesarelikely
warranted. schoolswereabletomakeAYPinNorthDakota.
1 THOMASB.FORDHAMINSTITUTE
18
a 16
t P
Y
o A 14
g
k kin
a Ma 12
D ols
o 10
h
h Sc
t mple 8
r a
o ofS 6
N er
mb 4
u
N
2
0
Massachuse s Nevada Idaho NorthDakota Kansas Washington Wyoming Indiana SouthCarolina Montana Florida Vermont NewJersey NewHampshire Maine NewMexico Delaware Colorado RhodeIsland Georgia Illinois Ohio Minnesota Michigan California Texas Arizona Wisconsin
Elementary Schools Middle Schools
Figure1.NumberofsampleschoolsmakingAYPbystate
Note:MiddleschoolswerenotincludedforTexasandNewJersey;absenceofamiddleschoolbarinthosestatesmeans“notapplicable”asopposedtozero.Stateslike
IdahoandNorthDakota,however,havezeropassingmiddleschools.
(cid:1)Asinotherstates,middleschoolshavegreaterdiffi- Introduction
culty reaching AYP in North Dakota than do ele-
TheProficiencyIllusion(Croninetal.2007a)linkedstu-
mentary schools, primarily because their student
dentperformanceNorthDakota’stestsandthoseof25
populationsarelargerandthereforehavemorequal-
other states to the Northwest Evaluation Association’s
ifying subgroups—not because their student
MeasuresofAcademicProgress(MAP),acomputerized
achievementislower.
adaptive test used in schools nationwide. This single
commonscalepermittedcross-statecomparisonsofeach
(cid:1)Partofthereasonallmiddleschoolsfailedtomake
state’sreadingandmathproficiencystandardstomeas-
AYPinNorthDakotaisthatitsschoolshaveenough
ureschoolperformanceundertheNoChildLeftBehind
low-income,disabled,orlimitedEnglishproficiency
(NCLB)Actof2001.Thatstudyrevealedprofounddif-
(LEP)4 students to qualify as separate subgroups.
ferences in states’ proficiency standards (i.e., how diffi-
EachofoursamplemiddleschoolsinNorthDakota
cult it is to achieve proficiency on the state test), and
hasoneormoreofthesesubgroupsandeachfailed evenacrossgradeswithinasinglestate.
to make AYP. Likewise, many elementary schools
withenoughstudentsqualifyingforthesesubgroups OurstudyexpandsonTheProficiencyIllusionbyexam-
also failed, though they tended to reach their math ining other key factors of state NCLB accountability
targetsmoreoftenthantheirreadingtargets.5 plansandhowtheyinteractwithstateproficiencystan-
4Notethatweuse“LEPstudents”and“Englishlanguagelearners”interchangeablytorefertostudentsinthesamesubgroup.
5SWDsaredefinedasthosestudentsfollowingindividualizededucationplans.WeshouldalsonotethatoursubgroupfindingsforLEP
studentsandSWDsmaybemorenegativethanactualfindings,mostlybecauseofthelikelydifferencesbetweenhowLEPstudentsandSWDs
aretreatedinMAP,theassessmentweusedinthisstudy,andintheNorthDakotaStateAssessment(NDSA),thestandardizedstatetest.
Specifically,theU.S.DepartmentofEducationhasissuednewNCLBguidelinesinrecentyearsthatexcludesmallpercentagesofLEPstudents
andSWDsfromtakingthestatetestorthatallowthemtotakealternativeassessments.Our2005–2006MAPdatadonotcapturethesesub-
groupnuances.Inthisstudy,however,novalidMAPscoreswereomittedfromconsideration.
TheAccountabilityIllusion 2
dards to determine whether the schools in our sample states,thesemarginsarequitewide,whichhastheeffect
made adequate yearly progress (AYP) in 2008. Specifi- ofmakingiteasiertoachieveanannualtarget.
cally, we estimated how a single set of schools, drawn N
o
fromaroundthecountry,wouldfareunderthediffering All of these AYP rules vary by state.This means that a
r
rulesfordeterminingAYPin28states(theoriginal25in schoolmakingAYPinWisconsinorOhio,forexample, t
h
The Proficiency Illusion plus 3 others for which we now
might not make it under South Carolina’s or Idaho’s
have cut score estimates). In other words, if we could D
rules(U.S.DepartmentofEducation2008).
somehow move these entire schools—with their same a
k
mix of characteristics—from state to state, how would
o
What We Studied
they fare in terms of making AYP? Will schools with t
a
high-performing students consistently make AYP?Will
Wecollectedstudents’MAPtestscoresfromthe2005-
schools with low-performing students consistently fail
06 academic year from 18 elementary and 18 middle
tomakeAYP?IfAYPdeterminationsforschoolsarenot
schoolsaroundthecountry.WealsocollectedtheNCLB
consistentacrossstates,whatleadstotheinconsistencies?
subgroupdesignationsforallstudentsinthoseschools—
inotherwords,whethertheyhadbeenclassifiedasmem-
NCLB requires every state, as a condition of receiving
bers of a minority group, such as English language
Title I funding, to implement an accountability system
learners,amongothersubgroups.
that aims to get 100% of its students to the proficient
level on the state test by academic year 2013–2014. In
Theschoolswerenotselectedasarepresentativesample
theinterveningyears,statessetannualmeasurableobjec-
of the nation’s population. Instead, we selected the
tives(AMOs).Thisisthepercentageofstudentsineach
schoolsbecausetheyexhibitedarangeofcharacteristics
school,andineachsubgroupwithintheschool(suchas
on measures such as academic performance, academic
low income6 or African American, among others), that
growth, and socioeconomic status (the latter calculated
mustreachtheproficientlevelinorderfortheschoolto
bythepercentageofstudentsreceivingfreeorreduced-
make AYP in a given year.The AMOs vary by state (as
pricelunches).Appendix1containsacompletediscus-
do,ofcourse,thedifficultyoftheproficiencystandards).
sionofthemethodologyforthisprojectalongwiththe
characteristicsoftheschoolsample.7
States also determine the minimum number of students
thatmustconstituteasubgroupinorderforitsscorestobe
Proficiency cut score estimates for the North Dakota
analyzedseparately(alsocalledtheminimumn[numberof
StateAssessment(NDSA)aretakenfromTheProficiency
students in sample] size).The rationale is that reporting
theresultsofverysmallsubgroups—fewerthantenpupils, Illusion(asshowninFigure2),whichfoundthatNorth
for example—could jeopardize students’ confidentiality Dakota’s definitions of proficiency generally ranked
and risk presenting inaccurate results. (With such small about average compared with the standards set by the
groups,randomevents,likeonestudentbeingoutsickon other25statesinthatstudy.Thesecutscoreswereused
testday,couldskewtheoutcome.)Becauseofthisflexibil- toestimatewhetherstudentswouldhavescoredaspro-
ity,stateshavesetwidelyvaryingnsizesfortheirsubgroups, ficient or better on the North Dakota test, given their
fromasfewas10youngsterstoasmanyas100. performance on MAP. Student test data and subgroup
designationswerethenusedtodeterminehowthese18
Manystateshavealsoadoptedconfidenceintervals—ba- elementary and 18 middle schools would have fared
sicallymarginsofstatisticalerror—toaccountforpoten- under North Dakota AYP rules for 2008. In other
tial measurement error within the state test. In some words,theschooldataandourproficiencycutscorees-
6Low-incomestudentsarethosewhoreceiveafreeorreduced-pricelunch.
7Wegaveallschoolsinoursamplepseudonymsinthisreport.
3 THOMASB.FORDHAMINSTITUTE
45
a 40
t
35
o
g
k n 30
ki
a n
a 2255
D RR
e Reading
l 20
h en Math
t erc 15
r P
10
o
N 5
0
Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8
Figure2.NorthDakotareadingandmathcutscoreestimates,expressedaspercentileranks(2006)
Note:ThisfigureillustratesthedifficultyofNorthDakota’scutscores(orproficiencypassingscores)foritsreadingandmathtests,aspercentilesoftheNWEAnorm,
ingradesthreethrougheight.Higherpercentileranksaremoredifficulttoachieve.AllofNorthDakota’scutscoresarebelowthe45thpercentile.
timatesarefromacademicyear2005–2006,butweare Note that we were unable to examine the effect of
applyingthemagainstNorthDakota’s2008AYPrules. NCLB’s“safeharbor”provision.Thisprovisionpermits
aschooltomakeAYPevenifsomeofitssubgroupsfail,
Table 1 shows the pertinent North Dakota AYP rules as long as it reduces the number of nonproficient stu-
that were applied to elementary and middle schools in dentswithinany failingsubgroup by at least 10%rela-
this study. North Dakota’s minimum subgroup size is tivetothepreviousyear’sperformance.Becausewehad
10,whichisconsiderablysmallerthanmostotherstates accesstoonlyasingleacademicyear’sdata(2005–2006),
we examined.8 This means that schools in North wewerenotabletoincludethisinouranalysis.Asare-
Dakota will have more accountable subgroups than sult,itispossiblethatsomeoftheschoolsinoursample
would similar schools in other states. North Dakota’s that failed to make AYP according to our estimates
annual targets also differ by grade and subject. For ex- wouldhavemadeAYPunderrealconditions.
ample,66.7%ofgrade8mathstudentsareexpectedto
beproficientin2008;thepercentageforgrade3reading Furthermore,attendanceandtestparticipationratesare
studentsis82.6%. beyondthescopeofthestudy.Notethatmoststatesin-
cludeattendanceratesasanadditionalindicatorintheir
Moststatesexaminedalsoapplyconfidenceintervals(or NCLBaccountabilitysystemforelementaryandmiddle
marginsofstatisticalerror)totheirmeasurementsofstu- schools. In addition, federal law requires 95% of each
dent proficiency rates. However, North Dakota’s 99% school’s students—and 95% of the students in each
confidence interval provides schools with greater le- school’ssubgroup—toparticipateintesting.
niencythanthemorecommonlyused95%confidence
interval.So,forinstance,whileschoolsaresupposedto Toreiterate,then,AYPdecisionsinthecurrentstudyare
get82.6%oftheirstudentsingrade3tothe“proficient” modeledsolelyontestperformancedataforasingleac-
levelonthestatereadingtest,and82.6%ofthestudents ademicyear.Foreachschool,wecalculatedreadingand
in each subgroup, applying the confidence interval mathproficiencyrates(alongwithanyconfidenceinter-
meansthattherealtargetcanbelower. vals)todeterminewhethertheoverallschoolpopulation
8ThestateofNorthDakotadoesnothaveaminimumschoolsize,soithasalargenumberofverysmallschools.Inaddition,thestate’spop-
ulationhasbeendeclininginrecentyears.TheU.S.CensusBureau(2002)listsNorthDakota’spopulationatalittleover642,000,47thin
theUnitedStates.Therefore,smallersubgroupsizesarelikelywarranted.
TheAccountabilityIllusion 4
Table1.NorthDakotaAYPrulesfor2008
Subgroupminimumn Race/ethnicity:10
N
SWDs:10 o
r
Low-incomestudents:10 t
h
LEPstudents:10
D
CI Appliedtoproficiencyratecalcula-ons? a
k
Yes;99%CIused
o
AMOs Baselineproficiencylevelsasof2002(%) 2008targets(%) t
a
READING/LANGUAGEARTS
Grade3 n/a 82.6
Grade4 65.1 82.6
Grade5 n/a 82.6
Grade6 n/a 80.7
Grade7 n/a 80.7
Grade8 61.4 80.7
MATH
Grade3 n/a 72.9
Grade4 45.7 72.9
Grade5 n/a 72.9
Grade6 n/a 66.7
Grade7 n/a 66.7
Grade8 33.3 66.7
Sources:U.S.DepartmentofEducation(2008);CouncilofChiefStateSchoolOfficers(2008).
Abbreviations:SWDs=studentswithdisabilities;LEP=limitedEnglishproficiency;CI=confidenceinterval;AMOs=annualmeasurableobjectives;n/a=notapplicable
and any qualifying subgroups achieved the AMOs.We indicatingbelow-grade-levelperformancefortheaverage
deemed that a school made AYP if its overall student student,andpositivevaluesindicatingabove-grade-level
body and all its qualifying subgroups met or exceeded performance. The two schools making AYP are in the
itsAMOs.Again,Appendix1suppliesfurthermethod- righthalfofthefigure,meaningthattheyareamongthe
ologicaldetail. schoolsthatcontainthehigheraverageperformingstu-
dents. Figure 4 illustrates the AYP performance of the
sample middle schools under the 2008 North Dakota
How Did the Sample Schools Fare
AYP rules. Not a single middle school in the sample
Under North Dakota’s AYP Rules?
makesAYPundertheNorthDakotarules.
Figure 3 illustrates the AYP performance of the sample
elementary schools under North Dakota’s 2008 AYP Figures5and6indicatethedegreetowhichschools’math
rules.Only2elementaryschools(WayneFineArtsand proficiencyratesareaidedbytheconfidenceintervalfor
Roosevelt) made AYP while 16 failed to make it.The elementaryandmiddleschools,respectively.Onthesefig-
trianglesinFigure3showtheaverageacademicperform- ures,thedarkbluebarsshowtheactualproficiencyrates
anceofstudentswithintheschool,withnegativevalues ateachschool,andthelightbluebarsshowthedegreeto
5 THOMASB.FORDHAMINSTITUTE
18 15
ol
a 16 ho
10 c
t 14 nS
ko gets 12 5 ncei
r a
a Ta 10 m
D erof 8 0 erfor
h mbm 6 -5 tPnt
t Nu de
r 4 u
o -10 St
N 2 age
r
0 -15 ve
A
1) 1) 1) 7) 5) 3) 7) 7) 5) 9) 1) 2) 3) 7) 8) 0) 8) 4)
( ( ( ( ( ( ( ( ( ( 2 2 ( ( ( 1 2 1
Clarkson Maryweather Few Nemo IslandGrove ohnF.Kennedy Scholls Hissmore WolfCreek Mayberry ayneFineArts( Winchester( Coastal Paramount ForestLake Marigold( Roosevelt( KingRichard(
J W
TargetsPassed TargetsFailed Average Student Performance
Figure3.AYPperformanceoftheelementaryschoolsampleunderNorthDakota2008AYPrules
Note:ThisfigureindicateshoweachoftheelementaryschoolswithinthesamplefaredunderNorthDakota’sAYPrules(asdescribedinTable1).Thebarsshowthe
numberoftargetsthateachschoolhastomeetinordertomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnot(lightblue).The
moresubgroupsinaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailedtomeettheAMOforevenasinglesubgroupdidn’tmakeAYP,
soanylightbluemeanstheschoolfailed.MarigoldElementary,forexample,mettenofitsfourteentargets,butbecauseitdidn’tmeetthemall,itdidn’tmakeAYP.
Schoolsareorderedfromlowesttohighestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbytheaverageMAPperformanceof
studentswithintheschool;itsscaleisshownontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-levelperformance
andscoresabovezerodenoteabove-grade-levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebettertheaverage
performanceandthelowerthenumber,theworsetheaverageperformance.Thenumberinparenthesesaftereachschoolnameindicatesthenumberofstates,out
of28,inwhichthatschoolmakesAYPinthestudy.
whichtheseproficiencyratesareincreasedbytheapplica- middleschool(Pogesto)mettheoveralltargetwiththe
tionoftheconfidenceinterval.Theorangelinesshowthe confidenceinterval,butweknowfromFigures3and4
annualmeasurableobjective(orannualtarget)neededto thattheseschoolsstillfailtomeettargetsforsubgroups.
meetAYP.Thefiguresshowthatonlyoneofthesample Inshort,theapplicationoftheconfidenceintervalhad
elementaryschools(Maryweather)andthreeofthemiddle little or no impact on whether schools achieved their
overall math and reading targets in North Dakota (or
schools (Tigerbear, Chesterfield, and Filmore) were as-
whethertheymadeAYP).9
sisted by the confidence intervals (note how the orange
linesfallwithinthelightbluebands).However,weknow
thatalloftheseschoolsstillfailedtomakeAYPbecauseof Where Do Schools fail?
low subgroup performance (see Figures 3 and 4).Tiger-
Figures3and4illustratethenumberofsubgrouptargets
bear,forinstance,didn’tmeetnineofitstwelvetargets.
ateachofthesampleschools,butthesefiguresdonotin-
Theeffect ofconfidenceintervalsonthereadingprofi- dicatewhichsubgroupsfailedorpassedinwhichschool.
ciency rates for elementary and middle schools shows Information on individual subgroup performance ap-
largelythesamepattern(notshown).Inreading,twoel- pears in Tables 2 and 3 for elementary and middle
ementary schools (Mayberry and Paramount) and one schools,respectively.
9Inthecurrentanalyses,confidenceintervalswereappliedtoboththeoverallschoolpopulationandtoalleligiblesubgroupsinoursampleschools.
Thus,theultimateimpactoftheconfidenceintervalislikelylargerthantheimpactdepictedinFigures5and6.However,wechosenottoshow
howtheconfidenceintervalimpactedsubgroupperformancebecauseitwouldhaveaddedgreatlytothereport’scomplexityandlength.
TheAccountabilityIllusion 6
20 12
ol
18 10 o
ch N
16 8 S
n o
ets 14 6 cei r
g n
Tar 12 4 ma th
of 10 2 or
ber 8 0 Perf D
mm ntt a
u 6 -2 e
N d k
u
4 -4 St o
2 -6 ge t
ra a
0 -8 ve
A
0) 0) 0) 5) 0) 0) 1) 1) 0) 0) 2) 0) 2) 1) 2) 0) 3) 5)
( ( ( 1 ( ( ( ( ( ( ( ( ( ( ( 2 ( (
McBeal arringerCharter MLAndrew Pogesto( McCord Tigerbear Chesterfield Filmore Barban Kekata Hoyt BlackLake LakeJoseph Zeus OceanView WalterJones( Artemus Chaucer
B
Targets Passed Targets Failed Average Student Performance
Figure4.AYPperformanceofthemiddleschoolsampleunderNorthDakota2008AYPrules
Note:ThisfigureindicateshoweachofthemiddleschoolswithinthesamplefaredunderNorthDakota’sAYPrules(asdescribedinTable1).Thebarsshowthenumberof
targetsthateachschoolhastomeetinordertomakeAYPunderthestate’sNCLBrules,andwhethertheymetthem(darkblue)ordidnot(lightblue).Themoresubgroups
inaschool,themoretargetsitmustmeet.Underthestudyconditions,aschoolthatfailedtomeettheAMOforevenasinglesubgroupdidn’tmakeAYP,soanylightblue
meanstheschoolfailed.Pogesto,forexample,metsevenofitseighttargets,butbecauseitdidn’tmeetthemall,itdidn’tmakeAYP.Schoolsareorderedfromlowestto
highestaveragestudentperformance(shownbytheorangetriangles).ThisismeasuredbytheaverageMAPperformanceofstudentswithintheschool;itsscaleisshown
ontherightsideofthefigure.Scoresbelowzero(whichisthegradelevelmedian)denotebelow-grade-levelperformanceandscoresabovezerodenoteabove-grade-
levelperformance.Oneunitdoesnotequalagradelevel;however,thehigherthenumber,thebettertheaverageperformanceandthelowerthenumber,theworsethe
averageperformance.Thenumberinparenthesesaftereachschoolnameindicatesthenumberofstates,outof28,inwhichthatschoolwouldmakeAYPinthestudy.
1
0.9
0.8
0.7
e
Rat 0.6
y
nc 0.5
e
cifi 00.44
o
r
P 0.3
0.2
0.1
0
Clarkson Maryweather Few Nemo IslandGrove hnF.Kennedy Scholls Hissmore WolfCreek Mayberry ayneFineArts Winchester Coastal Paramount ForestLake Marigold Roosevelt KingRichard
o W
J
MathProficiency Rate Math Proficiency Ratewith CI MathTarget
Figure5.ImpactoftheconfidenceIntervalonelementaryschoolmathproficiencyratesunderNorthDakota2008AYPrules
Note:Thisfigureshowsthereportedproficiencyrateforthestudentpopulationasawholeandtheimpactoftheconfidenceintervalonmeetingannualtargets.The
darkerportionsofthebarsshowtheactualproficiencyrateachieved,whilethelighter(upper)portionsofthebarsshowthemarginoferrorascomputedbythe
confidenceinterval.Thefigureshowsthatoneofthesampleelementaryschools(Maryweather)wasassistedbytheconfidenceinterval.Annualtargets(theorange
lines)areconsideredtobemetbytheconfidenceintervaliftheyfallwithinthelightblueportion.
7 THOMASB.FORDHAMINSTITUTE
1
a 0.9
t 0.8
o
0.7
k e
a Rat 0.6
D ncy 0.5
e
h ciiof 00.44
t Pr 0.3
r
o 0.2
N
0.1
0
McBeal ngerCharter MLAndrew Pogesto McCord Tigerbear Chesterfield Filmore Barban Kekata Hoyt BlackLake LakeJoseph Zeus OceanView WalterJones Artemus Chaucer
arri
B
MathProficiency Rate MathProficiency Ratewith CI MathTarget
Figure6.ImpactoftheconfidenceintervalonmiddleschoolmathproficiencyratesunderNorthDakota2008AYPrules
Note:Thisfigureshowsthereportedproficiencyrateforthestudentpopulationasawholeandtheimpactoftheconfidenceintervalonmeetingannualtargets.The
darkerportionsofthebarsshowtheactualproficiencyrateachieved,whilethelighter(upper)portionsofthebarsshowthemarginoferrorascomputedbythe
confidenceinterval.Thefigureshowsthatthreeofthesamplemiddleschools(Tigerbear,Chesterfield,andFilmore)wereassistedbytheconfidenceinterval.Annual
targets(theorangelines)areconsideredtobemetbytheconfidenceintervaliftheyfallwithinthelightblueportion.
Tables2and3showwhichsubgroupsqualifiedforeval- Few, Nemo, Island Grove, JFK, Scholls, Hissmore,
uation at each school (i.e., whether the number of stu- WolfCreek,andCoastal)failedtomeettheiroverall
dentswithinthatsubgroupexceededthestate’sminimum readingtargets.
n),andwhetherthatsubgrouppassedorfailed.Whileall
schoolsareevaluatedontheproficiencyrateoftheirover- (cid:1)Twelvemiddleschools(McBeal,Barringer,MLAn-
all population, potential subgroups that are separately drew, McCord, Tigerbear, Chesterfield, Filmore,
evaluatedforAYPpurposesincludeSWDs,studentswith Barbanti,Kekata,Hoyt,BlackLake,andZeus)failed
LEP,low-incomestudents,andthefollowingrace/ethnic to meet their overall reading targets, and six
categories:AfricanAmerican(AA),Asian/PacificIslander (McBeal,Barringer,MLAndrew,Pogesto,McCord,
(Asian), Hispanic/Latino (Hispanic), American andBarbanti)failedinmath.
Indian/AlaskaNative(AI/AN),andWhite.Tables2and
(cid:1)Oneelementaryschool(ForestLake)meteverytar-
3alsoshowwhetheraschoolmadeAYPundertheNorth
getexceptforitsreadingtargetforstudentswithdis-
Dakota rules, and the total number of states within the
abilities.
studyinwhichthatschoolmakesAYP.
(cid:1)Onemiddleschool(WalterJones)metalltargetsfor
The school-by-school findings in Tables 2 and 3 are
everysubgroupexceptforitsAfricanAmericanpop-
summarizedasshown:
ulation.
(cid:1)Oneofthe18elementaryschools(Clarkson)failed Tables4and5summarizetheperformanceofthevarious
tomeetbothreadingandmathtargetsforitsoverall subgroups for elementary and middle schools, respec-
studentpopulation,whilenineothers(Maryweather, tively.10TheperformanceofSWDsisprovingchallenging
10RecallthatelementarystudentsgenerallydobetteronNorthDakota’smathtestthanmiddleschoolstudents,partlybecauseNorthDakota’s
cutscoresarelowerinmaththaninreadingattheelementarylevel(seeFigure2).
TheAccountabilityIllusion 8
Table2.Elementarysubgroupperformanceofsampleschoolsunderthe2008NorthDakotaAYPrules
SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP? NorthDa
Clarkson 62.9% 43.1% N N N N N N N N N N Y Y 12 2 17% N 1 k
o
Maryweather 65.8% 52.5% Y N N N N N N N Y N N N Y N Y Y 16 5 31% N 1
t
Few 73.2% 55.3% Y N N N N N Y N Y N Y N Y N Y N 16 6 38% N 1 a
Nemo 74.9% 71.6% Y N N N Y N N N Y N Y Y 12 5 42% N 7
IslandGrove 76.9% 68.3% Y N N N N N Y N Y N Y Y 12 5 42% N 4
JFK 81.4% 66.0% Y N Y N Y N Y N Y N 10 5 50% N 3
Scholls 86.6% 71.7% Y N N N Y N Y N Y N Y N Y Y 14 7 50% N 7
Hissmore 85.2% 74.8% Y N N N Y N Y N Y Y 10 5 50% N 7
WolfCreek 77.1% 73.1% Y N N N Y N Y N Y Y Y N Y Y 14 8 57% N 5
AliceMayberry 86.2% 78.9% Y Y N N Y N Y N Y Y 10 6 60% N 9
WayneFineArts 86.8% 83.3% Y Y Y Y Y Y Y Y Y Y Y Y Y Y 14 14 100% Y 21
Winchester 82.5% 82.0% Y Y Y N Y N Y Y Y Y Y Y Y Y Y Y 16 14 88% N 22
Coastal 85.9% 75.2% Y N N N Y N Y N Y N Y N Y Y 14 7 50% N 3
Paramount 85.9% 78.7% Y Y Y N N N Y N Y Y Y Y Y N Y Y Y Y 18 13 72% N 7
ForestLake 92.8% 86.6% Y Y Y N Y Y Y Y Y Y Y Y Y Y 14 13 93% N 8
Marigold 93.5% 84.8% Y Y Y N Y N Y N Y Y Y N Y Y 14 10 71% N 10
Roosevelt 97.0% 92.9% Y Y Y Y Y Y Y Y Y Y Y Y Y Y 14 14 100% Y 28
KingRichard 92.9% 87.8% Y Y Y N Y N Y N Y Y Y N Y Y 14 10 71% N 14
Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino=
Hispanic;AmericanIndian/AlaskaNative=AI/AN.
Note:Schoolsareorderedfromlowest(Clarkson)tohighest(KingRichard)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading
performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof
studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs.
Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber
ofstatesinthestudyforwhichthatschoolmetAYP.
forschoolsunderNorthDakota’ssystemwherethissub- Otherstatereportscontainasectioncomparingsomeof
grouptendstohaveenoughstudentstomeetthestate’s thecharacteristicsofthesampleschoolsthatmadeAYP
minimumnof10.Infact,allbuttwoelementaryschools versusthosethatdidnot.InNorthDakota,noneofthe
andallbutonemiddleschoolinthestudywithqualifying sample middle schools made AYP, and among elemen-
SWDs reading subgroups failed to make AYP. Students tary schools, there were no striking differences among
withLEPandlow-incomestudentsarealsostrugglingto schoolsthatdidanddidn’tmakeAYP.Theoneexception
meetthestate’stargets;almosteverysingleschoolwitha (rather expected) was that schools that made AYP had
largeenoughpopulationtoqualifyasaseparatesubgroup students with higher average performance than did
failedtomeetitstargetsforthesestudents(thoughthey schoolsthatdidn’tmakeit,asmeasuredbyNWEAread-
tendtodobetterinmathattheelementarylevel). ingandmathtests.
9 THOMASB.FORDHAMINSTITUTE
Table3.Middleschoolsubgroupperformanceofsampleschoolsunderthe2008NorthDakotaAYPrules
hDakota SPCSHEUODOOLNYM MatOverallh ProficiencyRRateeading MOverallR MSWDsR MLEPStudentsR MLow-incomeStudentsR MAAR MAsianR MHispanicR MAI/ANR MWhiteR AYPTargetsRequired TargetsMET %ofTargetsMet SchoolMetAYP? NumberofstatesinwhichschoolmetAYP?
t McBeal 52.4% 54.2% N N N N N N N N N N Y Y N N N N Y N 18 3 17% N 0
r
o BarringerCharter 60.9% 60.2% N N N N Y Y N N N N Y N Y Y 14 5 36% N 0
N
MLAndrew 50.8% 59.8% N N N N N N N N N N N N Y N 14 1 7% N 0
Pogesto 50.0% 68.5% N Y Y Y Y Y Y Y 8 7 88% N 15
McCordCharter 53.7% 61.4% N N N N N N N N N N Y Y N N Y N 16 3 19% N 0
Tigerbear 63.6% 59.6% Y N N N N N N N Y N Y N 12 3 25% N 0
Chesterfield 66.0% 60.9% Y N N N N N N N Y Y Y N 12 4 33% N 1
Filmore 64.7% 69.9% Y N N N N N N N Y Y N N Y Y 14 5 36% N 1
Barban- 62.4% 64.8% N N N N N N N N Y Y Y Y N N Y Y 16 6 38% N 0
Kekata 70.4% 68.5% Y N N N N N N N N N Y Y N N Y Y 16 5 31% N 0
Hoyt 71.6% 71.7% Y N N N N N N N N N N N Y Y 14 3 21% N 2
BlackLake 75.6% 71.2% Y N N N N N Y N N N Y Y Y N Y Y Y N 18 8 44% N 0
LakeJoseph 71.7% 77.2% Y Y N N N N Y N Y Y N N Y Y 14 7 50% N 2
Zeus 74.2% 72.7% Y N N N N N N N Y N Y Y N N Y N 16 5 31% N 1
OceanView 76.7% 83.2% Y Y N N N N N N Y Y Y Y N N Y Y 16 8 50% N 2
WalterJones 86.0% 83.4% Y Y Y Y Y Y Y N Y Y Y Y 12 11 92% N 20
Artemus 81.7% 80.5% Y Y N N N N Y Y Y Y N N Y Y 14 8 57% N 3
Chaucer 85.1% 86.9% Y Y N N N N Y N Y Y Y Y Y Y Y Y 16 11 69% N 5
Abbreviations:M=math;R=reading;N=no;Y=yes;SWDs=studentswithdisabilities;AA=AfricanAmerican;Asian/PacificIslander=Asian;Hispanic/Latino=
Hispanic;AmericanIndian/AlaskaNative=AI/AN.
Note:Schoolsareorderedfromlowest(McBeal)tohighest(Chaucer)averagestudentperformanceasmeasuredbycombinedandweightedmathandreading
performanceontheMAPassessment(notshownintable).Ablankspaceunderneathasubgroupmeansthatsubgroupcontainedfewerthantheminimumnumberof
studentsrequiredforevaluation,soitwasn’tcounted.A“Y”inbluemeansthatthegroupmettheAMOsandan“N”inpeachmeansthatthegroupdidnotmeettheAMOs.
Thetworightmostcolumnsshow(1)whetherthatschoolmetAYP(i.e.,itmetthetargetsforitsoverallpopulationandallrequiredsubgroups);and(2)thetotalnumber
ofstatesinthestudyforwhichthatschoolmetAYP.
Characteristics of Schools ButNorthDakotaisalsohometoafewanomalies.First,
that Did and Didn’t Make AYP considerWinchesterElementary(seeFigure3).Itmade
AYP in 22 of the 28 states in our sample, yet not in
A close look at Figures 3 and 4 indicates that North North Dakota. Examining Table 2, one can see that
Dakota’s NCLB accountability system is, in some re- Winchester didn’t meet reading targets for its LEP or
spects,behavinglikethoseinotherstates.Forexample, SWD subgroups, although the school’s overall reading
amongtheelementaryschoolsinoursample,Roosevelt proficiencyratewas82%.Second,lookatWalterJones
andWayne Fine Arts each made AYP in many states— Middle School (Figure 4). Even with its relatively high
28and21,respectively.AndtheseschoolsmadeAYPin average performance it didn’t make AYP in North
NorthDakota,too.Likewise,theelementaryandmiddle Dakota,butmadeAYPin20of28states.LikeWinches-
schoolsthatfailedtomakeAYPinthegreatestnumber ter, it missed the AYP mark in North Dakota probably
ofstatesalsofailedAYPinNorthDakota. becauseofNorthDakota’srelativelysmallminimumn.
TheAccountabilityIllusion 10