ERIC ED499540: Impact of Nebraska Reading First on Reading Comprehension across Three Years of Implementation, 2005-2007

2007·0.16 MB·English

by  ERIC

G R E A T P L A I N S I N S T I T U T E

O F R E A D I N G A N D W R I T I N G

C O L L E G E O F E D U C A T I O N A N D H U M A N
S C I E N C E S

U N I V E R S I T Y O F N E B R A S K A L I N C O L N

IMPACT OF NEBRASKA READING FIRST ON

RREEAADDIINNGG CCOOMMPPRREEHHEENNSSIIOONN

ACROSS THREE YEARS OF IMPLEMENTATION
2005-2007

Prepared by

Malinda Murphy Ph.D.
Guy Trainin Ph.D.
GROWTH IN READING COMPREHENSION—2005-2007

OVERVIEW

The current report focuses on the impact of Reading First on reading comprehension across the first three years
of implementation in Nebraska. Grade level performance is defined as performance at or above the 40th
percentile using grade level standards. Within each grade level section reading comprehension achievement is
shown for all students and then by cluster membership. As stated in the 2006-2007 annual report, clusters are
used to group similar schools together and enable Reading First leadership and school administration to make
valid comparisons. Given the significant differences between schools in size, proportion of minority,
disadvantaged and special education students, it was clear that analyzing student achievement based on cluster
membership would better allow educators to make meaningful comparisons to school facing similar challenges
and thus make instructional adjustments that meet the unique needs of students in their schools. Cluster
membership presented in this report is consistent with those reported in the annual report. Achievement gaps
between English language learners (ELL), economically disadvantaged students (FRL), minority students, and
student with disabilities (special education) and their peers are presented within each grade level section as results
vary between grades and by category.

FIRST GRADE

Each spring a random sample of first grade students throughout Reading First schools in Nebraska are selected
to take the Gray Oral Reading Test (GORT). The GORT is an individually administrated norm referenced test.
Nebraska law does not allow group administered tests before second grade. Four students (and two alternates)
were randomly selected to complete this assessment from each first grade classroom in Reading First schools.
This sampling procedure increases the relative impact of smaller classrooms as compared to measures in second
and third grade.

The GORT comprehension subtest was selected for administration due to its high level of reliability. Reliability
estimates indicates “the extent to which individual differences in test scores are attributable to ‘true’ differences
in the characteristics under consideration and the extent to which they are attributable to chance errors”
(Anastasi & Urbina, 1997, p. 84). The reliability coefficients for the GORT comprehension subtest for the age
range of this sample ranges between .95-.96. Reliability estimates above .90 are the most desirable as they indicate
an assessment is highly reliable.

Figure 1 shows the growth in reading comprehension scores by the random sample of first grade students in
Nebraska Reading First schools. The numbers represent the percentage of first graders in the sample who scored
at or above grade level on the GORT comprehension subtest. In the spring of the first year of Reading First
implementation only 27% of
these students were performing First Grade Comprehension-All Students
at or above grade level on this
measure. By the end of the el100%
v
third year of Reading First e
L 75%
implementation the percentage e
d
a
of first grade students r 50%
G
pleevreflo irnmcrineags aetd o bry a 2b0o%ve. gMraodset nt at 25% 27% 44% 47%
of the increase, however, was e
rc 0%
accomplished between year one e
P
and two, indicating that need 2005 2006 2007
for further attention to
Figure 1: Percent of first grade students at grade level 2005-2007.
comprehension.
Number of students tested: 325 (2005), 367 (2006), 385 (2007).
2
GROWTH IN READING COMPREHENSION—2005-2007

A comparison of first grade performance in reading comprehension by cluster membership is presented in figure
2. Cluster Two (low poverty, low minority, smaller schools) has shown the largest and most consistent growth
since the first year of Reading First implementation (36%). This clearly indicates that teachers in this cluster are
building on their instructional strengths in this
First Grade Comprehension by Cluster
area year after year. Cluster Three (larger
100%
schools, high poverty and ethnic minority
student population) shows a fairly strong el
v
growth trajectory with a 21% increase in e 75%
L 63%
percentage of first grade students at or above e
grade level in comprehension. Given the very ad 50%
r
G 50%
low starting point (18% of students at or at 35% 47%
above grade level) and the high mobility rate nt 39%
e
of students in this cluster this represents c 25% 27% 34%
remarkable instructional improvements by er
P 18%
first grade teachers in cluster three. Cluster
0%
One (high poverty, low minority) has shown
the greatest difficulty in sustaining the growth 2005 2006 2007
in performance of first grade students on this Cluster One Cluster Two Cluster Three
measure after a gain of 12% in year two the
Figure 2: First grade comprehension growth by cluster 2005-2007.
numbers for this cluster actually declined and
resulted in only a 4% increase over the three years. The results for this cluster explain in part the limited overall
growth between years two and three.

Achievement gaps in reading comprehension across the first
First Grade ELL Gaps
100% three years of Reading First implementation were analyzed
between first grade students who are classified as English
el
v language learners, minorities, economically disadvantaged
Le 75%
e (FRL) and special education, and their peers (see figures 3-
d
a 6). The achievement gaps in first grade must be interpreted
r
at G 50% English Only with caution given the random sampling procedures related
nt to administration of this measure. The achievement gaps,
ce 25% however, do seem consistent with growing discrepancies
Per ELL across categories. With the exception of the slight
narrowing of the gap between white and minority students
0%
in the spring of 2007, the gaps are wider now than the first
2005 2006 2007
year of Reading First implementation.
Figure 3: First grade ELL gaps 2005-2007. The gaps for English language learners, shown in figure 3,
reveals a drop in percent of students at grade level. This is a
result of a uniquely successful ELL group of students
that were first grade students in spring 2005. Thus the First Grade SpEd Gaps
100%
2006-2007 trend is a better indicator of overall progress.
This trend shows a small reduction in the gap between el
v
English learners and their English only peers. Figure 4 Le 75%
e
shows that very little progress in reading comprehension d
a
was achieved with students with disabilities. This result is Gr 50%
indicative of a stronger focus on basic phonological at General Ed
awareness and decoding skills with these students. Most nt
ce 25%
likely low decoding ability constrains their ability to be Per SpEd
successful in this assessment.
0%
2005 2006 2007
Figure 4: First grade SpEd gaps 2005-2007.
3
GROWTH IN READING COMPREHENSION—2005-2007

The gaps for ethnic minorities and economically disadvantaged students in figures 5 and 6 show that both groups
improved but that mainstream students are currently benefitting more from Reading First instruction. First grade
results clearly highlight the need to focus more on comprehension skills in first grade. The focus is even more
crucial for populations at-risk as their growth is not as strong as that of mainstream students.
First Grade FRL Gaps
First Grade Ethnicity Gaps 100%
100%
el
el v
v e 75%
e Le 75% de L
d a
a r
r G 50% Non-FRL
nt at G 50% White ent at
Perce 25% Minority Perc 25% FRL
0% 0%
2005 2006 2007 2005 2006 2007
Figure 5: First grade ethnic minority gaps Figure 6: First grade FRL gaps 2005-2007.
2005-2007.

SECOND GRADE

Each spring all second and third grade students are administered the Gates-MacGinitie comprehension
assessment. Reliability estimates for students in this age range are .91 for second grade and .93 for third grade.
This is indicative of a highly reliable assessment for students at these grade levels.

Figure 7 shows the increase in the percentage of all second grade students in Nebraska Reading First schools
performing at or above grade level in comprehension based on this assessment. Sixty-one percent of these
students were at or above grade level during the spring of 2005. Growth between the first and second year of
Reading First implementation was stagnant with only a 1% increase in the percent of students performing at or
above grade level in reading comprehension. There was a sizable increase between years two and three with a
13% increase.
Second Grade Comprehension-All Students
100%
el
v
e
L 75%
e
d 75%
a
Gr 50% 61% 62%

at
nt 25%
e
c
r
e 0%
P
2005 2006 2007
Figure 7: Percent of second grade students at grade level 2005-2007.
Number of students tested: 711 (2005), 823 (2006), 845 (2007).
4
GROWTH IN READING COMPREHENSION—2005-2007

A comparison of second grade performance in reading comprehension was also conducted by cluster
membership (see figure 8). The growth trajectories between all three clusters were relatively consistent across all
three years of implementation. There was very little change in reading comprehension performance across all
three clusters between the first and second year of implementation. The most notable increase in percentage of
students at or above grade level on this measure occurred between years two and three. Cluster Two had the
largest percentage increase (16%) compared to Cluster One and Cluster Three, with 12% and 13% increases
respectively. Cluster One and Cluster Two had only 15% or less of second grade students performing below
grade level in reading comprehension in the spring of 2007.

Second Grade Comprehension by Cluster

100%
el 85%
v
Le 73% 74%
de 75% 71% 84%
a 68%
r
G 50%
at 56%
nt 43% 41%
e 25%
c
er
P
0%

2005 2006 2007

Cluster One Cluster Two Cluster Three

Figure 8: Second grade comprehension growth by cluster 2005-2007.

Achievement gaps in reading comprehension across the first three years of Reading First implementation were
analyzed between second grade students who are classified as English language learners, minorities, economically
disadvantaged (FRL) and special education, and their peers. The achievement gap between ELL and English
only students narrowed slightly during the second year of implementation, but was more widened in the spring
of 2007 (see figure 9). As a result, the achievement gap between these two groups of students has only decreased
by 14% since 2005. The gap in reading comprehension between special education students and their general
education peers has increased since the first year of Reading First implementation (see figure 10). The initial gap
between these two groups was 18% which has risen to nearly 30% in the last school year.

Second Grade ELL Gaps Second Grade SpEd Gaps
100% 100%
el el
v v
Le 75% English Only Le 75% General
e e
d d
a a
Gr 50% Gr 50%
at at Special
nt nt
ce 25% ELL ce 25%
er er
P P
0% 0%
2005 2006 2007 2005 2006 2007
Figure 9: Second grade ELL gaps 2007-2008. Figure 10: Second grade SpEd gaps 2007-2008.
5
GROWTH IN READING COMPREHENSION—2005-2007

The gaps between white and minority students remained unchanged between the first and second years of
Reading First implementation but narrowed by approximately 12% between years two and three (figure 11). The
gap in reading comprehension between economically disadvantaged (FRL) students and their peers has actually
increased by 5% since the first year of implementation (see figure 12).
Second Grade Ethnicity Gaps Second Grade FRL Gaps
100% 100%
el White el Non-FRL
v v
Le 75% Le 75%
e e
d d
a a
Gr 50% Gr 50%
nt at Minority nt at FRL
ce 25% ce 25%
er er
P P
0% 0%
2005 2006 2007 2005 2006 2007
Figure 11: Second grade ethnic minority gaps Figure 12: Second grade FRL gaps 2007-2008.
2007-2008.

THIRD GRADE

Each spring all third grade students are administered the Gates-MacGinitie comprehension assessment. Figure 13
shows the increase in percentage of third grade students performing at or above grade level on this measure
across the first three years of Reading First implementation. As shown, the growth in reading comprehension for
this grade level has not improved tremendously since the inception of Reading First. There was a 7% increase in
the number of third grade students performing at or above grade level between the first and second year of
implementation and only a 5% increase between years two and three. Although there has been a 12% increase in
the number of third grade students performing at or above grade level in reading achievement since the spring of
2005, it is clear that more attention to this area of instruction is necessary in order to bolster the reading
comprehension skills of students at this grade level.

Third Grade Comprehension-All Students

100%
l
e

v
e
L 75%

e
d
a
r 50% 64%
G 59%
t 52%
a
t 25%
n
e
c
er 0%
P
2005 2006 2007

Figure 13: Percent of third grade students at grade level 2005-2007.
Number of students tested: 849 (2005), 789 (2006), 838 (2007).
6
GROWTH IN READING COMPREHENSION—2005-2007

A comparison of third grade performance in reading comprehension was also conducted by cluster membership
(see figure 14). Third grade students in Cluster Two have shown the greatest increase (16%) in the percentage of
students performing at or above grade level in reading comprehension. There was an 11% increase in Cluster
One and a 10% in Cluster Three.

Although the difference between the clusters is significant, the growth trajectories between Cluster One and
Cluster three are commensurate. The cluster analysis clearly shows the need for greater instructional focus on
reading comprehension strategies in Cluster One and Cluster Three.

Third Grade Comprehension by Cluster

100%

83%
el
v 73%
e 75% 67%
e L
ad 68% 67%
Gr 50% 56%
at
nt 42%
ce 25% 32% 36%
r
e
P

0%

2005 2006 2007

Cluster One Cluster Two Cluster Three

Figure 14: Third grade comprehension by cluster 2005-2007.

Achievement gaps in reading comprehension across the first three years of Reading First implementation were
analyzed between third grade students who are classified as English language learners, minorities, economically
disadvantaged (FRL) and special education, and their peers (see figure 16). Analysis of achievement gaps in
reading comprehension between ELL and English only students have narrowed by 17% since 2005. The gap
between special education students and their general education peers has only decreased by 2 % since 2005 with
a slight widening of the gap in 2006.

Third Grade ELL Gaps Third Grade SpEd Gaps
100% 100%
el el
v v
Le 75% Le 75% General
e English Only e
d d
a a
Gr 50% Gr 50%
at at
Percent 25% Percent 25% Special
ELL
0% 0%
2005 2006 2007 2005 2006 2007
Figure 15: Third grade ELL gaps 2005-2007.
Figure 16: Third grade SpEd gaps 2005-2007.
7
GROWTH IN READING COMPREHENSION—2005-2007

The gap between minority and economically disadvantaged (FRL) students and their peers have slightly widened
since the inception of Reading First in Nebraska, 4% and 5% respectively.

Third Grade Ethnicity Gaps Third Grade FRL Gaps
100% 100%

el el Non-FRL
Lev 75% White Lev 75%
e e
d d
a a
Gr 50% Gr 50%
at at
cent 25% Minority cent 25% FRL
er er
P P

0% 0%
2005 2006 2007 2005 2006 2007
Figure 17: Third grade ethnic minority gaps 2005-2007. Figure 18: Third grade FRL gaps 2005-2007.

SUMMARY
Reading comprehension scores in first through third grades have increased fairly steadily since Reading First
implementation began in 2005. Figure 19 shows the percentage of students performing at or above grade level at
each grade level. It is important to reiterate that the measure and procedures used in first grade are different from
those in second and third grades. Since first grade comprehension assessments involve random selection of a
small number of students per grade level the standard error of this assessment may be higher than in second and
third grades.

The percentage of students performing at or above grade level is the highest for second grade (75%) followed by
third grade at 64 percent. Although first grade has the lowest percentage of students performing at or above
grade level in comprehension, they have shown the highest percentage increase (20%) since the first year of
Reading First implementation.

Comprehension Growth 2005-2007
100%
el
ev 75%
L75%
e 61% 62%
d
a
r 64%
G50% 59%
t 52%
t a 44% 47%
n
e25%
rc 27%
e
P
0%
2005 2006 2007
First Second Third
Figure 19: Comprehension growth in first, second, and third grades
2005-2007.
8
GROWTH IN READING COMPREHENSION—2005-2007

Figure 20 shows the within cohort comparison in comprehension growth by cluster. This figure represents the
growth in student comprehension performance of the cohort of students who were in first grade during the first
year of Reading First implementation. This figure shows the greatest percentage increase in reading
comprehension between first and second grades with the growth leveling and even dropping off in third grade.
Differences in starting points and strength of growth between the clusters are apparent. Students in Cluster Two
have shown the most solid and consistent growth in reading comprehension since 2005 with a 48% increase in
the percentage of students at or above grade level. This level of performance is followed by Cluster One with a
40% increase. Cluster Three has shown the least amount of growth with only a 24% increase in the percent of
students performing at or above grade level in reading comprehension.

Within Cohort Comprehension Growth by Cluster
100%

el 83%
v
e75%
L
e 67%
d
a
Gr50%
t 42%
a
nt 35%
e25% 27%
c
r 18%
e
P
0%

2005 2006 2007

First Grade Second Grade Third Grade

Cluster One Cluster Two Cluster Three

Figure 20: Comprehension growth in first, second, and third

grades by cluster 2005-2007.

RECOMMENDATIONS

Although there have been increases in the percentage of students performing at or above grade level in reading
comprehension since the inception of Reading First in Nebraska it is recommended that teachers at all three
grade levels increase the level of instructional focus allocated to reading comprehension instruction. The aim of
instructional improvements in Nebraska Reading First schools should focus on reading comprehension throughout
the reading process. As stated by the National Reading Panel (2000), “Comprehension is the reason for reading.
If readers can read the words but do not understand what they are reading, they are not really reading” (p. 48).
Effective comprehension instruction supports the student by aiding the development of effective strategies.
These strategies include linking to prior knowledge before reading, monitor comprehension during reading, and
organize information through the use of graphic organizers and semantic mapping throughout the reading
process. Comprehension is also supported by teaching students how to recognize story structure, summarization,
and question generation.
9