ERIC ED469747: A Deeper Look at NAEP Science Results.

11 Pages·2002·0.21 MB·English

by  ERIC

DOCUMENT RESUME
ED 469 747
TM 034 538
Barton, Paul
AUTHOR
A Deeper Look at NAEP Science Results.
TITLE
Educational Testing Service, Princeton, NJ. Policy
INSTITUTION
Information Center.
PUB DATE 2002-00-00
10p.; Theme issue.
NOTE
AVAILABLE FROM
ETS Policy Information Center, Educational Testing Center,
Rosedale Road, 04-R, Princeton, NJ 08541-0001. Tel: 609-734-
5694; e-mail: [email protected]; Web site:
http://www.ets.org/research/ pic.
Collected Works
PUB TYPE
Serials (022)
ETS Policy Notes; 11 nl Fall 2002
JOURNAL CIT
EDRS Price MF01/PC01 Plus Postage.
EDRS PRICE
DESCRIPTORS
*Educational Trends; Elementary Education; Grade 4; Grade 8;
Longitudinal Studies; *Low Achievement; National Surveys;
*Science Achievement; Science Tests; Standards; Trend
Analysis
*National Assessment of Educational Progress
IDENTIFIERS
ABSTRACT
This issue presents the results of a special analysis of NAEP
state-by-state science data from 1996 to 2000 carried out by the Educational
Testing Service under contract with the National Center for Education
Statistics. The focus is on changes in the gap in scores between different
groups of students. There were a few bright spots, but overall the results
over this 4-year period of national concentration on standards-based reform
were disappointing. There were no improvements on three indicators: the gap
between the top and bottom quartile, the gap between white and minority
students, and the gap between poor and nonpoor students. The gap actually
grew in 11 states between the top and bottom quartiles and in 7 states
between poor and nonpoor students. Only one state showed an improvement for
the bottom quartile of students, but 17 states registered an improvement for
the top quartile of the student population, and 7 states increased the
percentage of students reaching the "proficient" level. Only three states,
however, raised the statewide average scores. One state improved on four of
seven indicators, and just two states improved on three indicators. Results
underscore the importance of examining multiple indicators of state
performance when analyzing NAEP data.
(SLD)
Reproductions supplied by EDRS are the best that can be made
from the original document.
ETS POLICY NOTES
News from the ETS Policy Information Center
VOLUME 11, Number 1
Fall 2002
A Deeper Look at NAEP
Science Results
U.S. DEPARTMENT OF EDUCATION
Office of Educational Research and Improvement
PERMISSION TO REPRODUCE AND
EDUCATIONAL RESOURCES INFORMATION
DISSEMINATE THIS MATERIAL HAS
CENTER (ERIC)
BEEN GRANTED BY
his document has been reproduced as
received from the person or organization
R. J. Coley
originating it.
Minor changes have been made to
improve reproduction quality.
TO THE EDUCATIONAL RESOURCES
Points of view or opinions stated in this
document do not necessarily represent
INFORMATION CENTER (ERIC)
official OERI position or policy.
Policy Information Center
Educational Testing Service (ETS)
co
HAM
l;
181Es"
iDo
Errs POLICY NOTES
News from the ETS Policy Information
Center
Volume 11, Number 1
Educational Testing Service
Fall 2002
A Deeper Look at NAEP Science
This IssueA Deeper Look
Results
at NAEP Science Results
Introduction
This issue of ETS Policy Notes
presents the results of a special
Persistent and disturbing differences in academic
analysis of NAEP state-by-state
achievement among U.S. students, usually
apparent
science data from 1996 and 2000
when students are grouped by race/ethnicity and/or
carried out by ETS under
con-
some measure of socioeconomic status, have received
tract with the National Center
renewed attention from education policymakers. This
for Education Statistics. The
focus on the achievement
gap in the U.S. education
focus is on changes in the
gap
system carries forward Goal 3 of the national education
in scores between different
goals that students should demonstrate knowledge in
groups of students.
"challenging subject matter," with the objective that
"the academic performance of all students... will
Readers interested in
more gen-
increase significantly in every quartile, and the distri-
eral results from the 2000 NAEP
bution of minority students in
every quartile will more
science assessment should
see
closely reflect the student population
as a whole." This
Science Highlights 2000, available
goal is renewed in the No Child Left Behind Act of
from the National Center for
2001, which focuses on raising the academic achieve-
Education Statistics (http://
ment of all students, setting a minimum threshold of
nces.ed.gov/nationsreportcard/
performance toward which the lowest achieving sub-
science/results/index.asp).
groups of students must show progress on assessments
in use in each state.
Editor's note: This issue of ETS
Policy Notes was written by Paul
The major tool available to
measure the overall
Barton, with data analysis
pro-
achievement of U.S. students is the National Assess-
vided by David Freund and
ment of Educational Progress (NAEP), which regularly
Mei-jung Lin.
reports average scale scores and the percentage of stu-
dents reaching specific "achievement levels"
set by
the National Assessment Governing Board.' National
1 For more information
on NAEP, go to http:/ /nces.ed.gov /nationsreportcard.
3
and state results in science were reported in
Changes in the gap between White and
November 2001.2
minority students'
This issue of ETS Policy Notes provides a more
Changes in the gap between
poor and
in-depth examination of the gaps in science
nonpoor students'
achievement than is provided in regular NAEP
reports. It is based on a special analysis of state-
A Deeper Look at the Data
by-state NAEP public school data from the
1996 and 2000 science assessments for Grade 8
While there were a few bright spots, the
conducted by ETS under contract with the
results over this four-year period of national
National Center for Education Statistics.
concentration on standards-based reform
Thirty-three states participated in both assess-
were disappointing.
ments. Examining these state data can reveal
patterns of performance that can be masked in
Figure 1 shows the change in performance for
national analyses. In addition to looking at
each state, on each of the seven indicators.
changes in overall average scores and in the
Table 1 summarizes these results.6
percentage of students reaching the "proficient
lever' this issue of ETS Policy Notes describes:
Table 1 reveals that:
Changes in achievement among top scorers
There were no improvements on three indi-
(fourth quartile) between 1996 and 2000
the gap between the top and bot-
cators
tom quartile, the gap between White and
Changes in achievement among low scorers
minority students, and the gap between
(first quartile)
poor and nonpoor students.
Changes in the gap between the top and
The gap grew in 11 states between the top
bottom quartiles
and bottom quartile and in seven states
between poor and nonpoor students.
2 http://nces.ed.govinationsreportcard/pdf/main2000/2002452.pdf.
3 NAEP defines this level as representing solid academic performance for each grade assessed.
Students reaching this level have demonstrated competency
over challenging subject matter, including subject-matter knowledge, application of such knowledge to real-world situations, and analytical skills appropriate
to the subject matter.
4 Minority students in this analysis are Black and Hispanic.
5 While NAEP does not include a poverty
measure, it does report the percentage of students who are eligible for a free or reduced-price lunch. This measure
is used as a proxy for "poor" students.
6 All differences noted are statistically significant based
on r-tests.
4
2
Figure 1: Changes in
NAEP
Eighth-Grade Science Scores,
1996 to 2000
t Improved
Change in:
M Unchanged
4
White/
Worse
Poor/
Percent
Top
Bottom
Quartile
Minority
Nonpoor
t
Means
Proficient
Quartile
Quartile
Gap
Gap
Gap
4
Nation
401
ir
1
44
4mk
Alabama
11
44
i
w
4*
44
40
Arizona
44
j
44
40
44
4
44
44
Arkansas
44
40
1
4
44
44
California
44
44
44
40
Connecticut
40.
401
401
4wo
44
44
Georgia
44
40
40
401
401
I
1
44
4
44
Hawaii
44
4
t
t
t
44
Indiana
41
40
44
4=1
401
t
Kentucky
44
44
4w1
44
1
4
Louisiana
44
11
1
4
4
44
Maine
40
44
44
44
I
t
01
Maryland
4=1
44
401
4
401
t
44
44
Massachusetts
44
40
H
44
4
Michigan
40
44
4110
44
40
Minnesota
44
401
I
10.
4m1
t
4
t
t
40
Mississippi
401
44
401
4
lr
Missouri
01
44
4
44
lr
44
Montana
44
110
4=1
1
44
44
Nebraska 40
44
0
4=1
4=1
44
00
4..
40
New Mexico
401
4=1
01
l
01
New York
40
01
4=1
01
410
5
4
44
40
North Carolina
44
-1.
4
4
40
4
01
40
North Dakota
4.
44
01
4
Oregon
40
44
I
4=1
44
44
im
Rhode Island
44
40
H
t
44
00
South Carolina
44
401
44
4=1
4
01
44
Tennessee
44
40
4=1
4
4
1
44
t
4*
44
Texas
4=1
I
I
I
4 1
01
4
Utah
44
4=1
I
1°
Vermont
44
44
44
t
I
Virgina
44
401
44
400
401
4.
4
5
44
t
West Virginia
44
4
Wyoming
44
401
44
5
3
Table 1: Change Between 1996 and 2000 in NAEP
Eighth-Grade Science Achievement
on Seven Indicators
Change,
States
States
States
1996 to 2000 in:
Improving
Unchanged
Worse
Average Score
3
28
2
Percent "Proficient"
7
26
0
Top Quartile
17
14
2
Bottom Quartile
25
7
1
Gap - Top and Bottom Quartile
0
22
11
Gap - White and Minority
0
33
0
Gap - Poor and Nonpoor
0
26
7
Only one state showed
an improvement for
no state did worse on four or more of the six
the bottom quartile of students.
indicators, six states did
worse on one indicator,
four states did
worse on two indicators, and five
However, 17 states registered
an improve-
states did worse on three indicators.
ment for the top quartile of the student
population, and
These results underscore the
importance of
examining multiple indicators of
state perfor-
Seven states increased the
percentage of
mance when analyzing NAEP data. A look just
students reaching the
"proficient"
level,
at the percentage reaching the "proficient" level
although only three
states raised the state-
of performance would fail
to disclose that no
wide average scones.
progress was made among those students in
the bottom quartile of the
score distribution.
Table 2 provides another
summary of how the
Without looking
at the quartiles separately, we
states fared on the seven indicators of change
would not know the significant
improvements
that were used in this analysis.
in average scores for the
top quartile of the
students. And
as we can see from Figure 1,
As shown, none of the
states improved on all
the gap between the
top and bottom quartiles
seven or six or five of the indicators. Just
one
increased in 11
states. In two of these states,
state improved on four of the seven indicators,
scores dropped for the bottom quartile and
rose
and just two
states improved on three indica-
for the top quartile, and in three
states, scores
tors. Four states improved on two of the
declined in the bottom quartile
without chang-
indicators and 10
on a single indicator. While
ing in the top quartile.
6
4
Table 2: State Performance on Seven Indicators of
NAEP Science Achievement, 1996
- 2000
Number of
States
States
States
Indicators
Improving
Unchanged
Worse
10
0
1
6
2
4
0
4
3
2
4
5
4
8
0
1
5
0
5
0
6
0
7
0
7
0
9
0
The overriding
concern is that there was little
the score gap between the
top and the bottom
improvement in science achievement; there
was
quartiles. In an earlier
report on fourth grade
no change in the nation as a whole in average
reading (1992-1998), there
was a similar result;
scores or the percent reaching proficient
scores declined in the bottom quartile in 18
although a few states managed
to pull average
states, while they rose in the top quartile in
scores up.' There was an increase in scores in
12 states, with the gap between the
top and
the top quartile for the
country as a whole and
bottom enlarging in 16
states. This was not the
in the top quartile in 17 states.
case for mathematics, where there were gener-
ally gains at both the
top and bottom.'
What is also noteworthy is that something in
the dynamics of our educational
system during
There is
no
basis for speculation from these
this period was resulting in improvement in the
data as to what caused these growing
gaps.
scores of the best students, and deterioration
However, the tracking of state-by-state
progress
or no changein the scores of the lowest scor-
points out the need to delve
more deeply into
ing students, resulting in an enlargement of
the gaps than just examining statewide
averages
7 When nonpublic schools
are included, there was a small but statistically significant increase in the percentage of eighth-graders reaching the "proficient"
level or higher between 1996 and 2000.
8 Paul E. Barton, Raising Achievement and Reducing Gaps: Reporting
Progress Toward Goals for Academic Achievement, A Report
to the National Education
Goals Panel, March 2001. Available from http:// www. ets .org/research/pic/raising.pdf
7
5
or the percentage reaching a particular score
NAEP provides
a rich source of information
cut-point.
about elementary and secondary
education
for the nation and for the
participating states.
Measuring what school students know
and
Persistent and Large Gaps
are able to do is a little like wanting, from
outside, to know what is inside
The previous section has summarized the
a very large
situa-
house. You look in
tion in regard to whether the
one window and see a
gaps are narrow-
little, and
you look in another window and
ing or widening over this four-year period.
see a little more. But to know what is inside
Figure 2 shows the size of these
gaps for
the house,
you have to look in all the win-
the participating
states, for three indicators,
dows. This
report looks in several windows,
in 2000. Typically, the
gap between poor and
but by no
means all of them.
nonpoor students is the smallest, and the gap
between White and minority students is
a little
Another important dimension
larger. However, the
is the measure-
gap between the bottom
ment of growth in achievement from the
and top quartile of all students is much
larger,
fourth to the eighth grade; in
about 2 1/2 times the size of the
other words;
gap between
how much students raised their
White and minority students. For the
scores over
2000
the four-year period. This is
national results, these
possible in math
gaps were 32, 37, and 88
and reading where NAEP has been
points, respectively. These
operating
are large gaps, and
longer at the state level. This approach
they have proven to be unyielding.
is
used in Growth in School: Achievement
Gains
from the Fourth
to the Eighth Grade, by Paul
These gaps vary, state-by-state,
as can be seen
Barton and Richard Coley,
in Figure 2. However,
a publication
no significance should
of the ETS Policy Information
be given to relatively small differences
Center
among
(ftp://ftp.ets.org/pub/res/growsch.pdf). In
states, given the errors due to the sampling
terms of state accountability testing, only
approach in the NAEP
assessment.
Tennessee
measures growth in knowledge
during a year of schooling; the other
states
compare students at the end of a school
year
with their
counterparts in previous years.
6
s
a
a
t
t
n
a
n
e
d
t
a
i
o
t
i
s
u
l
n
l
o
o
i
i
c
o
u
a
p
c
n
e
a
a
a
s
r
k
a
y
i
d
n
a
r
i
t
h
p
i
n
e
x
k
a
l
t
n
a
i
g
a a
o
s
m
k a
g
i
a
n
c
n
k
a
i
c
r
e
s
r
a
o
C
a
t
a
s
s
r
I
s
C D
c
o
n
n
0
u
e
i
i
s
r
n
n
a
n
a
s

M
a
i
g
g
i
a
n
e
n
s
i
o
e
u
V
e
i
a

Y
a
i
5
a
n
o
s
o
o

e
l
o
s
m
h
i
t
i
r
b
i
a
n
y
a
h h
s
n
t
f
n
d
h
w
t
s
n
r
i s

o
a
g
m
n
s
n
z i
w
t
u
n
w
r
a
t
i
i
a
h
b
k l
t t
n
i
o
o
c s
s
u
d
a
o
e
s
a
a a
n
a
e
i
g
r r
e
o
N
o
x
l
r r
a
e
e r
e
i i i
i
o o
y
h
o
n
C M
e
A
G
C
e
H
M M
r
M M M
M
A A
r
K e
e
L
M
t
N
N
N
W
O
i
N N
R
I
S
W
T
U
V
V
T
8
3
0
4
4
4
2
e
3
3
2
3
3
g
1
0
r
0
3
9
9
3
a
o
3
9
2
9
:
9
2
7
2
0
2
r
o
2
2 2
2 7
7
0
e
2
2
3
P
4
4
3
v
2
2
2 3

2
2
A
2 2
n
1
2
2
0
1
0
e

2
2
2
r
e
o
7
0
6
1
w
2
1
o
4
3
4
1
p
t
e
1
n
1
B

o
1

0
N
p
)
1
1
4
a

4
d
G
-
-
n
-
-
a
-
-
0
0
5
1
1
0
4
4
9
4
9
9
3
7
3
3
7
6
3
0
6
e
6
3
3
4
5
e
1
4
g
4
i
3
3
3
a
3
3
2
t
s
a
2
2
3
3
i
3
t
1
3
3
h
0
n
3
r
9
9
8
9 9
e
i
W
2
0
o
2
7
2
7
2 2
0
v
2
2
P
0
3

A

n
4
2
e
4
2
3
2

e
l
2
2

a
y
2
,
e
c
0
t
s
9
S
1
i
w
1
r
e
0

o
P
2
t
r
e
n
E
o
B
i
A
c
M

N
S
p

0
d

a
1
e
n
G
c
a
n
e
i
0
c
S

P
0
0
E
1
A
0
9
N
8
9
8
8
8
7
8
8
0
6
5
1
e
9
5
4
4
8
4
4 4
0
4
3
9
3
9
d
e
8
9 9
8
8
2
1
1
l
8
a
8
1
8
i
p
8
t
9
r
8
r
7
7
7
o
6
G
7
6
a
6
6
0
7
7
6
7
7
T
7
7
8
7
u
-

3
h
Q
n
7
0
1
e
t
0

7
h
7
7
m
e
1
6
g
0
6
w
6
7
6
o
i
-
t
E
t
e
t
-
o

B
n
-
B

i
p
0

e
6
s
a
d
p
G
n
n
a
a
i
G
a
0
a
s
a
a
n
t
a
a
d
5
a
i a
s

n
a
y
i
i
u
a
o
a
a
i
d
n
a
a
n
p
m
a
t
r
k
e
i
a
s
h
i
n
g
:
n
t
a
M
n
t
k
a
o
n
n
g
n
u
c
k c n
t
n
t
o
n
s o
r
o
p
i
a
e
w
2
a
o
a
c n
e
a
i
a
n
g
a
o
a
o
r
i
s
i
r
o
i
n
i
n
a
g
i
s
t
z
i
x x
s
l
t
o
u
k
o
i
i
l
s
t
i l
s
a
i
i
Y
a
c
b
t o m
s
U
d
m
o
i
s
y
a e
h
l
e g
e

i
e
s
a
n
a
g
t
e
f
s
u
s
r
H
e
r
a
s r

n
n i
r
r
i
e
c
T
k n
r
r
M
A
e
r
o
b
u
D
w
l
I
a
r
i
G
o
h
a
a
N
n
i
O
i
l
a
s
e
I
e
i
r
i
M
n
V
A
n
e
M
o
M
V
C
e
y
c
M
C
e
A

n
r
s
C
V
K
i
w
n
h
N
L
W
a
d
M
N

i
o
u

M
t
e
h
t
s
h
o
e
s
C
r
T
t
s
o
g
h
t
N
e
r
u
a
o
N
R
W
o
i
M
N
F
S
Recently Published by the ETS Policy Information
Center...
www.ets.orglresearchlpic
The Closing of the Education Frontier?
Meeting the Need for Scientists, Engineers, and
an Educated Citizenry in a
11
Technological Society
An Uneven Start: Indicators of Inequality in School Readiness
Staying on Course in Education Reform
Facing the Hard Facts in Education Reform
The Twin Challenges of Mediocrity and Inequality:
an
Literacy in the U.S. from
International Perspective
ETS Policy Notes is published by the
NON-PROFIT
ORGANIZATION
ETS Policy Information Center
US POSTAGE
Educational Testing Service
PAID
Rosedale Road, 04-R
EDUCATIONAL
Princeton, NJ 08541-0001
TESTING SERVICE
[phone]
e-mail: [email protected]
www.ets.org/research/pic
Editor: Richard J. Coley
Copyright © 2002 by Educational
Testing Service. All rights reserved.
Educational Testing Service is an Affirmative
Action/Equal Opportunity Employer.
EDUCATIONAL TESTING SERVICE, ETS,
and the ETS logo are registered trademarks of
Educational Testing Service.
The modernized ETS logo is a trademark of
Educational Testing Service.
88503-02536 Y102E6 Printed in the U.SA
I.N. 995824