Measuring Literacy: Performance Levels for Adults

353 Pages·2005·1.43 MB·English

by  Committee on Performance Levels for Adult Literacy

MEASURING
LITERACY
PERFORMANCE LEVELS FOR ADULTS
Committee on Performance Levels for Adult Literacy
Robert M. Hauser, Christopher F. Edley, Jr., Judith Anderson Koenig, and
Stuart W. Elliott, editors
Board on Testing and Assessment
Center for Education
Division of Behavioral and Social Sciences and Education
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Govern-
ing Board of the National Research Council, whose members are drawn from the
councils of the National Academy of Sciences, the National Academy of Engineer-
ing, and the Institute of Medicine. The members of the committee responsible for
the report were chosen for their special competences and with regard for appropri-
ate balance.
This study/publication was supported by Cooperative Agreement No.
R215U990016 between the National Academy of Sciences and the U.S. Department
of Education. Additional funding was provided by an award from the Presidents’
Committee of The National Academies. Any opinions, findings, conclusions or
recommendations expressed in this publication are those of the authors and do not
necessarily reflect the views of the organizations or agencies that provided support
for the project.
Library of Congress Cataloging-in-Publication Data
Measuring literacy : performance levels for adults / Committee on Performance
Levels for Adult Literacy ; Robert M. Hauser ... [et al.], editors.
p. cm.
Includes bibliographical references.
ISBN 0-309-09652-9 (pbk.) — ISBN 0-309-55015-7 (pdf) 1. Functional
literacy—United States—Evaluation. I. Hauser, Robert Mason. II. National
Research Council (U.S.). Committee on Performance Levels for Adult Literacy.
LC149.7.M4 2005
302.2′244—dc22
2005021251
Additional copies of this report are available from the National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; [phone] or
(202) 334-3313 (in the Washington metropolitan area); Internet, http://
www.nap.edu.
Printed in the United States of America
Copyright 2005 by the National Academy of Sciences. All rights reserved.
Suggested citation: National Research Council. (2005). Measuring literacy: Perfor-
mance levels for adults. Committee on Performance Levels for Adult Literacy, R.M.
Hauser, C.F. Edley, Jr., J.A Koenig, and S.W. Elliott, editors. Board on Testing and
Assessment, Center for Education. Division of Behavioral and Social Sciences and
Education. Washington, DC: The National Academies Press.
The National Academy of Sciences is a private, nonprofit, self-perpetuating society
of distinguished scholars engaged in scientific and engineering research, dedicated
to the furtherance of science and technology and to their use for the general welfare.
Upon the authority of the charter granted to it by the Congress in 1863, the Acad-
emy has a mandate that requires it to advise the federal government on scientific and
technical matters. Dr. Ralph J. Cicerone is president of the National Academy of
Sciences.
The National Academy of Engineering was established in 1964, under the charter of
the National Academy of Sciences, as a parallel organization of outstanding engi-
neers. It is autonomous in its administration and in the selection of its members,
sharing with the National Academy of Sciences the responsibility for advising the
federal government. The National Academy of Engineering also sponsors engineer-
ing programs aimed at meeting national needs, encourages education and research,
and recognizes the superior achievements of engineers. Dr. Wm. A. Wulf is presi-
dent of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of
Sciences to secure the services of eminent members of appropriate professions in the
examination of policy matters pertaining to the health of the public. The Institute
acts under the responsibility given to the National Academy of Sciences by its
congressional charter to be an adviser to the federal government and, upon its own
initiative, to identify issues of medical care, research, and education. Dr. Harvey V.
Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences
in 1916 to associate the broad community of science and technology with the
Academy’s purposes of furthering knowledge and advising the federal government.
Functioning in accordance with general policies determined by the Academy, the
Council has become the principal operating agency of both the National Academy
of Sciences and the National Academy of Engineering in providing services to the
government, the public, and the scientific and engineering communities. The Coun-
cil is administered jointly by both Academies and the Institute of Medicine. Dr.
Ralph J. Cicerone and Dr. Wm. A. Wulf are chair and vice chair, respectively, of the
National Research Council.
www.national-academies.org

COMMITTEE ON PERFORMANCE LEVELS
FOR ADULT LITERACY
CHRISTOPHER F. EDLEY, Jr. (Co-Chair), School of Law, University of
California, Berkeley
ROBERT M. HAUSER (Co-Chair), Center for Demography of Health
and Aging, University of Wisconsin–Madison
JUDITH A. ALAMPRESE, Abt Associates Inc., Bethesda, MD
MICHAEL X. DELLI CARPINI, Annenberg School for Communication,
University of Pennsylvania
VIVIAN L. GADSDEN, National Center on Fathers and Families,
University of Pennsylvania
ANDREW J. HARTMAN, Independent Consultant, Denver, CO
GLYNDA A. HULL, Graduate School of Education, University of
California, Berkeley
REBECCA A. MAYNARD, University Trustee Professor of Education
and Social Policy, University of Pennsylvania
LORRAINE M. McDONNELL, Department of Political Science,
University of California, Santa Barbara
LARRY MIKULECKY, Language Education Department, Indiana
University
ROBERT J. MISLEVY, Department of Measurement, Statistics, and
Evaluation, University of Maryland
NORMAN G. PETERSON, Satisfaction Performance Research (SPR)
Center, Inc., Minneapolis, MN
JOHN P. POGGIO, Department of Psychology and Research in
Education, University of Kansas
RIMA E. RUDD, Department of Health and Social Behavior, Harvard
School of Public Health
MARY JANE SCHMITT, TERC, Cambridge, MA
DAVID M. THISSEN, Department of Psychology, University of North
Carolina, Chapel Hill
HEIDE SPRUCK WRIGLEY, LiteracyWork Associates, San Mateo, CA
JUDITH A. KOENIG, Study Director
STUART W. ELLIOTT, Senior Program Officer
CONSTANCE F. CITRO, Senior Program Officer
LORI HOUGHTON WRIGHT, Program Officer
ANDREW E. TOMPKINS, Research Assistant
LYNNE STEUERLE SCHOFIELD, Research Assistant
MICHAEL DECARMINE, Senior Project Assistant
v
BOARD ON TESTING AND ASSESSMENT
2003-2004
EVA L. BAKER (Chair), The Center for the Study of Evaluation,
University of California, Los Angeles
LORRAINE MCDONNELL (Vice-Chair), Department of Political
Sciences, University of California, Santa Barbara, CA
LAURESS L. WISE (Vice-Chair), Human Resources Research
Organization, Alexandria, VA
CHRISTOPHER F. EDLEY, JR., Harvard Law School
EMERSON J. ELLIOTT, Consultant, Arlington, VA
MILTON D. HAKEL, Department of Psychology, Bowling Green State
University
ROBERT M. HAUSER, NAS, Institute for Research on Poverty, Center
for Demography, University of Wisconsin, Madison
PAUL W. HOLLAND, Educational Testing Service, Princeton, NJ
DANIEL M. KORETZ, Graduate School of Education, Harvard
University
EDWARD P. LAZEAR, Stanford University, Stanford, CA
RICHARD J. LIGHT, Graduate School of Education and John F.
Kennedy School of Government, Harvard University
ROBERT J. MISLEVY, Department of Measurement and Statistics,
University of Maryland
JAMES W. PELLEGRINO, University of Illinois, Chicago, IL
LORETTA A. SHEPARD, School of Education, University of Colorado,
Boulder
KENNETH I. WOLPIN, Department of Economics, University of
Pennsylvania
STUART W. ELLIOTT, Director
LISA D. ALSTON, Administrative Coordinator
vi
Foreword
I
n the summer of 2002, the National Center for Education Statistics
approached the Board on Testing and Assessment (BOTA) of the Na-
tional Academies with a request for assistance in setting performance
standards for their upcoming assessment of adults’ literacy skills. This was
a unique request for BOTA. Over the years, BOTA had explored and
provided advice on a variety of issues related to setting performance stan-
dards on educational achievement tests—from discussions of the standards-
based reform movement in education and its effects on various groups of
student to recommendations for best practice in setting performance stan-
dards for the National Assessment of Educational Progress. Undertaking
the process of actually setting performance standards, however, was a new
endeavor for BOTA.
Setting performance standards is an inherently judgmental task. The
process involves determining the number and nature of the performance
levels used for reporting the test results (such as “proficient” or “below
basic”), the descriptions of the levels, and the test scores used to demark
the range of scores associated with each performance level. A variety of
standard-setting procedures are documented in the measurement literature,
procedures that lay out the methodologies and best practices, but all ulti-
mately rely on the judgments of testing experts, policy makers, and other
stakeholders and users of the test results. The answers to questions such as
“How much literacy is enough?” or “What constitutes a literacy prob-
lem?”—either for an individual or for society as a whole—are not exclu-
sively technical. Bringing scientific principles to the process of collecting
and summarizing judgmental information was a daunting challenge.
vii
viii FOREWORD
Consistent with its mission, BOTA accepted this request. Formed to
provide scientific advice to policy makers and the public about critical
issues related to testing and assessment, BOTA draws on the interdiscipli-
nary expertise of its members to bring a wide variety of perspectives to bear
on such complex problems. Members of BOTA welcomed this opportunity
to demonstrate a scientific approach to the problem and offer advice about
two critical and timely issues: how to help describe and understand the
literacy skill levels of adults in this country and how to set performance
standards in a meaningful and technically valid way.
Under the auspices of BOTA, the Committee on Performance Levels for
Adult Literacy was formed as an interdisciplinary panel of 17 members
with expertise in the areas of adult education and adult literacy, economics,
educational measurement and standard setting, law, political science, and
sociology. BOTA remained actively involved with this work, with four
members of BOTA serving on the committee, two of whom served as
committee co-chairs. Members of BOTA provided ongoing oversight for
the work in formulating the committee’s charge and overall approach to its
tasks, identifying individuals to serve on the committee, offering feedback
to the committee, and reviewing the draft report and recommendations.
The committee was convened in December 2002 and held six meetings.
During the course of its work, the committee solicited feedback from stake-
holders using a variety of mechanisms, including a public forum held in
February 2004. The committee also convened two standard-setting ses-
sions, in July and September 2004, which involved experts in adult literacy,
adult education, teaching, and other relevant fields. This report presents the
findings and recommendations that resulted from these activities and the
committee’s deliberations. It is BOTA’s hope that this report will be of use
to a variety of audiences: the U.S. Department of Education in its final
decision making about performance standards for its adult literacy assess-
ments and plans for future assessments; policy makers and practitioners in
the adult literacy field as they make programmatic decisions; and the psy-
chological measurement community as they grapple with the complex tech-
nical and judgmental issues involved in the task of setting valid perfor-
mance standards in similar situations.
BOTA extends its sincere appreciation to the committee for its hard
work on this challenging project, and particularly to Christopher Edley, Jr.,
and Robert Hauser, who served as co-chairs.
Lauress L. Wise, Chair
Board on Testing and Assessment
Acknowledgments
T
he work of the Committee on Performance Levels for Adult Literacy
benefited tremendously from the contributions of many people, and
the committee is grateful for their assistance and support.
First, we wish to acknowledge the National Center for Education Sta-
tistics (NCES), which sponsored this project. We think that the leadership
of the NCES was wise, both to pursue improvements in standard setting
and—as a statistical agency—to choose an impartial external body to estab-
lish its reporting standards. The committee thanks Gary Phillips for his
willingness to initiate the study and extends its heartfelt thanks to Peggy
Carr for her interest in this important topic and her constant support
throughout the project. During the course of this project other NCES staff
members, including Sheida White and Andrew Kolstad, gave generously of
their time. We thank each of them for the wealth of information they
provided and their prompt answers to all of the committee’s questions.
The American Institutes for Research (AIR) served as contractor to
NCES for work on National Assessment of Adult Literacy (NAAL), and
many of its staff were generous with both advice and assistance. Mark
Kutner was an invaluable resource to the committee, and we are grateful
for his responsiveness to all of the committee’s requests. We also thank AIR
staff members Stephan Baldi, Elizabeth Greenburg, and Eugene Johnson for
their ongoing assistance and readiness to answer the committee’s questions.
Special thanks are also due to Irwin Kirsch, who led the work on the
1992 National Adult Literacy Survey (NALS) at the Educational Testing
Service. Irwin was a tremendous resource to the committee as they worked
to reconstruct and understand procedures that had been used to determine
ix