ERIC ED416221: NAEP's Constituents: What Do They Want? Report of the National Assessment of Educational Progress Constituents' Survey and Focus Groups. Analytic Report.

120 Pages·1.4 MB·English

by  ERIC

DOCUMENT RESUME
TM 028 073
ED 416 221
Levine, Roger; Rathbun, Amy; Selden, Ramsay; Davis, Andrew
AUTHOR
NAEP's Constituents: What Do They Want? Report of the
TITLE
National Assessment of Educational Progress Constituents'
Survey and Focus Groups. Analytic Report.
National Center for Education Statistics (ED), Washington,
INSTITUTION
DC
NCES-98-521
REPORT NO
1998-01-00
PUB DATE
NOTE
119p.
National Library of Education, Office of Educational
AVAILABLE FROM
Research and Improvement, U.S. Department of Education, 555
New Jersey Avenue, N.W., Washington, DC 20208-5641; phone:
800-424-1616; World Wide Web: http://nces.ed.gov/naep
-- Reports - Evaluative
Numerical/Quantitative Data (110)
PUB TYPE
(142)
MF01/PC05 Plus Postage.
EDRS PRICE
*Academic Achievement; Educational Assessment; Educational
DESCRIPTORS
Research; Educational Trends; Elementary Secondary
Education; Evaluation Utilization; Focus Groups; Information
Dissemination; Institutional Characteristics; National
Surveys; *Research Reports; Response Rates (Questionnaires);
State Programs; *Student Characteristics; Tables (Data);
*Test Use; *Trend Analysis; *User Needs (Information)
Educational Indicators; *National Assessment of Educational
IDENTIFIERS
Progress
ABSTRACT
The National Assessment of Educational Progress (NAEP), the
only accurate and credible indicator of educational performance capable of
informing about national trends and state differences in student achievement,
serves many different constituencies whose opinions must figure heavily in
determining the future directions of the NAEP. To identify, analyze, and
compare opinions of some key NAEP constituencies, a multi-stage process was
undertaken that began with the identification of some key components through
consultation with the National Center for Education Statistics and the
National Assessment Governing Board. A survey was developed and administered
to representatives of the eight key constituent groups, generally
associations of educators and government representatives. With telephone
follow-up, the overall response rate was 83W (352) completed surveys. Focus
groups were conducted to assess opinions about these issues with
representatives of seven other NAEP constituencies, including educational
administrators, teachers, and the public. Although the constituent groups did
not have identical opinions on the issues discussed in the survey and focus
groups, there were many cases in which similar opinions were expressed.
Attitudes common to most respondents include the recognition that
instructional practices, student characteristics, and school characteristics
are important background variables to measure, although they might be
measured and reported in a way that would make the process more rapid. A
multistep approach could be used to allow the immediate reporting of
achievement results with the detailed reporting of background material as
available. Most respondents thought that reading and writing should be
assessed as individual subjects, while other subjects could be assessed in
+++++ ED416221 Has Multi-page SFR--- Level =l +++++
clusters. Subscale reports were seen as most important for mathematics. Most
respondents also thought that NAEP assessments should be administered
annually, and that the government should support efforts to link the NAEP to
international assessments. Three appendixes present the survey, a summary of
(Contains 9
responses, and the focus groups protocol for the media group.
tables and 25 figures.)
(SLD)
********************************************************************************
Reproductions supplied by EDRS are the best that can be made
from the original document.
********************************************************************************
NATIONAL CENTER FOR EDUCATION STATISTICS
Analytic Report
January 1998
NAEP's Constituents:
What Do They want?
Report of the National Assessment of
Educational Progress Constituents'
Survey and Focus Groups
THE NATION'S
REPORT
CARD
U.S. DEPARTMENT OF EDUCATION
Ohm of Educational Research and Improvement
ED
TIONAL RESOURCES INFORMATION
CENTER (ERIC)
This document has been reproduced as
received from the person or organization
originating it.
Minor changes have been made to
improve reproduction quality.
Points of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.
MMQ
Yv
U.S. Department of Education
0
Office of Educational Research and Improvement
NCES 98-521
lam CORNY AVEILA
II;
What is The Nation's Report Card?
THE NATION'S REPORT CARD, the National Assessment
of Educational Progress (NAEP), is the only nationally
representative and continuing assessment of what America's students
know and can do in various subject areas. Since 1969,
assessments have been conducted periodically in reading, mathematics, science, writing,
history/geography, and other fields.
By making objective information on student performance
available to policymakers at the national, state, and local levels,
NAEP is an integral part of our nation's evaluation of the
condition and progress of education. Only information related
to
academic achievement is collected under this
program. NAEP guarantees the privacy of individual students and their families.
NAEP is a congressionally mandated project of the National
Center for Education Statistics, the U.S. Department of
Education. The Commissioner of Education Statistics is responsible,
by law, for carrying out the NAEP project through
competitive awards to qualified organizations. NAEP
reports directly to the Commissioner, who is also responsible
for providing continuing reviews, including validation studies
and solicitation of public comment, on NAEP's conduct
and usefulness.
In 1988, Congress established the National Assessment
Governing Board (NAGB) to formulate policy guidelines for
NAEP. The Board is responsible for selecting the subject
areas to be assessed from among those included in the National
Education Goals; for setting appropriate student performance levels;
for developing assessment objectives and test specifications
through a national consensus approach; for designing the
assessment methodology; for developing guidelines for reporting
and disseminating NAEP results; for developing standards and
procedures for interstate, regional, and national comparisons;
for determining the appropriateness of test items and ensuring
they are free from bias; and for taking actions to improve the
form and use of the National Assessment.
The National Assessment Governing
Board
Mark D. Musick, Chair
Edward H. Haertel
Fannie L. Simmons
President
Professor of Education
Mathematics Coordinator
Southern Regional Education Board
Stanford University
District 5 of Lexington/Richland County
Atlanta, Georgia
Stanford, California
Ballentine, South Carolina
Mary R. Blanton, Vice Chair
William J. Moloney
Adam Urbanski
Attorney
Commissioner of Education
President
Salisbury, North Carolina
State of Colorado
Rochester Teachers Association
Denver, Colorado
Patsy Cavazos
Rochester, New York
Principal
Honorable Annette Morgan
Deborah Voltz
W.G. Love Accelerated Elementary School
Former Member
Houston, Texas
Assistant Professor
Missouri House of Representatives
Department of Special Education
Jefferson City, Missouri
Catherine A. Davidson
University of Louisville
Mitsugi Nakashima
Secondary Education Director
Louisville. Kentucky
Central Kitsap School District
First Vice-Chairperson
Silverdale. Washington
Hawaii State Board of Education
Marilyn A. Whirry
Honolulu, Hawaii
Twelfth-Grade English Teacher
Edward Donley
Mira Costa High School
Michael T. Nettles
Former Chairman
Manhattan Beach, California
Air Products & Chemicals, Inc.
Professor of Education & Public Policy
University of Michigan
Allentown. Pennsylvania
Dennie Palmer Wolf
Ann Arbor. Michigan
Honorable James Edgar
Senior Research Associate
and Director
Member Designate
Harvard Graduate School of Education
Frederick D. Patterson Research Institute
Governor of Illinois
Cambridge, Massachusetts
United Negro College Fund
Springfield, Illinois
Ricky T. Takai (Ex- Officio)
Honorable Norma Paulus
James E. Ellingson
Acting Assistant Secretary of Education
Superintendent of Public Instruction
Fourth-Grade Classroom Teacher
Office of Educational Research
Oregon State Department of Education
Probstfield Elementary School
and Improvement
Salem. Oregon
Moorhead, Minnesota
U.S. Department of Education
Honorable William T. Randall
Washington, DC
Thomas H. Fisher
Former Commissioner of Education
Director, Student Assessment Services
State of Colorado
Florida Department of Education
Roy 'flyby
Denver, Colorado
Tallahassee, Florida
Executive Director, NAGB
Honorable Roy Romer
Washington, DC
Michael J. Guerra
Governor of Colorado
Executive Director
Denver, Colorado
Secondary Schools Department
National Catholic Educational Association
Washington, DC
NATIONAL CENTER FOR EDUCATION STATISTICS
Analytic Report
January 1998
NAEP's Constituents:
What Do They want?
Report of the National Assessment of
Educational Progress Constituents'
Survey and Focus Groups
THE NATION'S
REPORT
CARD
Roger Levine
Amy Rathbun
Ramsay Selden
Andrew Davis
U.S. Department of Education
Office of Educational Research and Improvement
NCES 98-521
U.S. Department of Education
Richard W. Riley
Secretary
Office of Educational Research and Improvement
Ricky T. Takai
Acting Assistant Secretary
National Center for Education Statistics
Pascal D. Forgione, Jr.
Commissioner
Education Assessment Group
Gary W. Phillips
Associate Commissioner
January 1998
SUGGESTED CITATION
They Want? Report of
Levine, R., Rathbun, A., Selden, R., and Davis, A. NAEP's Constituents: What Do
for Education
the NAEP Constituents' Survey and Focus Groups. Washington, DC: National Center
Statistics, 1998.
FOR MORE INFORMATION
Contact:
Arnold A. Goldstein
202-219-1741
For ordering information on this report, write:
National Library of Education
Office of Educational Research and Improvement
U.S. Department of Education
555 New Jersey Avenue, NW
Washington, DC 20208-5641
metropolitan area call 202-219-1651).
or call 1-800-424-1616 (in the Washington, DC,
This report also is available on the World Wide Web: http://nces.ed.gov/naep.
for Education
The work upon which this publication is based was performed for the National Center
Services Institute.
Statistics, Office of Educational Research and Improvement, by the Education Statistics
Foreword
The National Assessment of Educational Progress (NAEP) has
many important
constituents: state and local education managers and policy-makers, researchers, the
public,
practitioners. But, how do these various groups feel about the
current features of NAEP and the
different directions NAEP could go in the future? Unless
we canvas them directly, we have no
way of knowing what these important consumers of NCES information want and need.
The NAEP Constituents' Survey was done in the first
part of 1997 by the
Education Statistics Services Institute as part of
an extensive effort to redesign NAEP by the
National Center for Education Statistics and the National Assessment Governing
Board.
Through a comprehensive survey of managers and policy-makers
at the local, state, and national
levels and through focus groups of other
users, project staff asked these groups what directions
NAEP should take. What kinds of background questions should it include? How
should results
be released? How useful is subscale information within the various subjects? Is it important
to
link NAEP to the results of international studies?
The findings of this effort already have been extremely useful
to NCES in guiding future
plans for NAEP. We hope you will find them useful and interesting
as well.
Pascal D. Forgione, Jr., Ph.D.
Commissioner
Acknowledgments
NAEP's Constituents: What Do They Want? Report of the NAEP Constituents'
Survey and Focus Groups was authored under a contract with the Education Statistics
Services Institute (ESSI), which provides direct support to the National Center for
Education Statistics (NCES). The authors, Roger Levine, Amy Rathbun, Ramsay Selden,
and Andrew Davis of ESSI/American Institutes for Research (AIR), worked as a team
under the general direction of Sharif Shakrani, current National Assessment Governing
Board (NAGB) staff member and former project officer at NCES, and Arnold Goldstein,
current project officer at NCES.
The authors wish to thank all of those who contributed to the production of this
report. Special thanks go to: NCES and NAGB for their evaluation and feedback on draft
versions of the NAEP Constituents' Survey; Jo Ann Blue, Sarah Benson, Sara Davison,
John Konstant, and Freddy Mendez of ESSI, who contacted constituents by telephone to
increase survey response rates and assisted in careful reviews of the report; Iby Heller and
Kari Wolman of AIR, who also assisted with the follow up phone calling and helped to
prepare the summary of responses to open-ended comments; and Fran Stancavage of AIR,
who provided valuable input into the development of the survey items and focus group
protocol.
We would also like to thank several individuals who served as reviewers and
provided many insightful and helpful comments. The reviewers were: Peggy Carr,
Patricia Dabbs, Arnold Goldstein, Steven Gorman, Andrew Kolstad, and Kelly Weddel of
NCES; Ray Fields of NAGB; Doris Redfield, a consultant; and Jeff Nelhaus of the
Massachusetts Department of Education.
Lastly, thanks are due to the chief state school officers, state education agency
assessment directors, state education agency curriculum directors, state board of
education chairs, Governor's education policy advisors, state legislature education
committee staff, superintendents of large urban and suburban school districts, staff of
education associations, and education researchers who responded to the survey; and to the
principals, teachers, members of the media, representatives of national business
organizations which conduct efforts to support and improve schools, and members of the
public who participated in the focus groups. Their input to this study was vital to the
redesign of NAEP.
ii
Table of Contents
EXECUTIVE SUMMARY
v
1.0 INTRODUCTION
1
1.1 Identification of Constituencies and Issues
3
1.2 NAEP Constituents' Survey
4
1.2.1 Survey Instrument Development
4
1.2.2 Survey Mai lout
5
1.2.3 Follow-up Operations
7
1.2.4 Data File Preparation
10
1.2.5 Data Analysis: Response Frequencies
11
1.2.6 Data Analysis: Cross-tabulations and Tests of
Statistical Significance
12
1.3 Focus Groups: Protocol Development
13
1.3.1 Focus Groups: Participant Recruitment and Operations
14
.
. .
1.3.2 Demographics of Focus Group Participants
17
1.3.3 Analysis of Focus Group Responses
21
2.0 SURVEY RESULTS
22
2.1 Background Information
Emphasis
22
2.2 Background Questions
- Impacts on Release of Results
23
..
2.3 Technical Documentation of NAEP vs. Timeliness of Reports
24
2.4 Including a Parent Survey
25
2.5 Reporting: Schedule for Release of Results
26
2.6 State Mandates for Participation in NAEP
28
2.7 Subjects Assessed at the State Level
30
2.8 Assessing Subject Areas in Combinations or Individually
32
2.9 Desire for Information on Skill Areas within Subjects
35
2.10 Frequency of Data Collections
39
2.11 Linking NAEP to International Assessments
40
iii
42
2.12 Obtaining State-Level Results
47
2.13 States Paying for Some Services
48
3.0 FOCUS GROUP RESULTS
48
Emphasis
3.1 Background Information
50
3.2 Background Questions - Impacts on Release of Results
3.3 Technical Documentation of NAEP vs. Timeliness of Reports .
51
52
3.4 Including a Parent Survey
54
3.5 Reporting: Schedule for Release of Results
55
3.6 Assessing Subject Areas in Combinations or Individually
56
3.7 Desire for Information on Skill Areas within Subjects
58
3.8 Frequency of Data Collections
59
3.9 Linking NAEP to International Assessments
60
3.10 Impact of Results (General)
63
3.11 General Public Focus Group Findings
APPENDIX A: NAEP Constituents' Survey (Version I: Long Form)
APPENDIX B: Summary of Responses to Open-Ended Comments
APPENDIX C: Focus Group Protocol (Media Group)
iv