Using the Results of a National Assessment of Educational Achievement (National Assessments of Educational Achievement)

188 Pages·2009·1.34 MB·English

by  Thomas Kellaghan

National Assessments of Educational Achievement
VOLUME 5
Using the Results of a
National Assessment of
Educational Achievement
Thomas Kellaghan
Vincent Greaney
T. Scott Murray

Using the Results
of a National
Assessment of
Educational
Achievement

National Assessments of Educational Achievement
VOLUME 5
Using the Results of a
National Assessment
of Educational
Achievement
Thomas Kellaghan
Vincent Greaney
T. Scott Murray
© 2009 The International Bank for Reconstruction and Development / The
World Bank
1818 H Street, NW
Washington, DC 20433
Telephone: [phone]
Internet: www.worldbank.org
E-mail: [email protected]
All rights reserved
1 2 3 4 12 11 10 09
This volume is a product of the staff of the International Bank for Reconstruction
and Development / The World Bank. The fi ndings, interpretations, and conclu-
sions expressed in this volume do not necessarily refl ect the views of the Executive
Directors of The World Bank or the governments they represent.
The World Bank does not guarantee the accuracy of the data included in this
work. The boundaries, colors, denominations, and other information shown on
any map in this work do not imply any judgement on the part of The World Bank
concerning the legal status of any territory or the endorsement or acceptance of
such boundaries.
Rights and Permissions
The material in this publication is copyrighted. Copying and/or transmitting por-
tions or all of this work without permission may be a violation of applicable law.
The International Bank for Reconstruction and Development / The World Bank
encourages dissemination of its work and will normally grant permission to repro-
duce portions of the work promptly.
For permission to photocopy or reprint any part of this work, please send a
request with complete information to the Copyright Clearance Center Inc., 222
Rosewood Drive, Danvers, MA 01923, USA; telephone: [phone]; fax: 978-
750-4470; Internet: www.copyright.com.
All other queries on rights and licenses, including subsidiary rights, should be
addressed to the Offi ce of the Publisher, The World Bank, 1818 H Street, NW,
Washington, DC 20433, USA; fax: [phone]; e-mail: pubrights@worldbank
.org.
Cover design: Naylor Design, Washington, DC
ISBN: 978-0-8213-7929-5
eISBN: 978-0-8213-7966-0
DOI: 10.1596/978-0-8213-7929-5
Library of Congress Cataloging-in-Publication Data
Kellaghan, Thomas.
Using the results of a national assessment of educational achievement / Thomas
Kellaghan, Vincent Greaney, T. Scott Murray.
p. cm. — (National assessments of educational achievement ; v. 5)
Includes bibliographical references and index.
ISBN 978-0-8213-7929-5 (alk. paper) — ISBN 978-0-8213-7966-0 (electronic)
1. Educational tests and measurements. 2. Achievement tests. 3. Education and
state. I. Greaney, Vincent. II. Murray, T. Scott. III. Title.
LB3051.K189 2009
371.26′2—dc22
2009009954
CONTENTS
FOREWORD xi
ACKNOWLEDGMENTS xiii
ABBREVIATIONS xv
1. FACTORS AFFECTING THE USE AND NONUSE OF
NATIONAL ASSESSMENT FINDINGS 1
The Political Context of a National Assessment 3
Accountability 5
The Quality of the Assessment Instrument 15
Type of Assessment 17
Underuse of National Assessment Findings 20
Conclusion 24
2. REPORTING A NATIONAL ASSESSMENT:
THE MAIN REPORT 29
Context of the National Assessment 31
Objectives of the National Assessment 31
Framework for the National Assessment 32
Procedures in Administration of the National Assessment 32
Description of Achievement in the National Assessment 33
Correlates of Achievement 37
Change in Achievement Over Time 42
Conclusion 45
v
vi | CONTENTS
3. REPORTING A NATIONAL ASSESSMENT: OTHER
INSTRUMENTS TO COMMUNICATE FINDINGS 47
Product Specifi cation Sheet 48
Briefi ng Ministers and Senior Policy Personnel 49
Publishing Summary Reports 50
Publishing Technical Reports 53
Publishing Thematic Reports 56
Securing Media Reports 56
Issuing Press Releases 58
Holding Press Conferences 62
Conducting Individual Briefi ngs 63
Posting Web Site Reports 63
Making Assessment Data Available 64
Other Dissemination Instruments 66
Conclusion 67
4. TRANSLATING ASSESSMENT FINDINGS INTO POLICY
AND ACTION 69
Institutional Capacity to Absorb and Use Information 70
Trustworthiness and Relevance of Information Provided
by an Assessment 71
Procedures to Identify Appropriate Policy or Action
Following an Assessment 71
Determination of a Systemwide or Targeted Intervention 75
Complexity of Policy Formation and Decision Making 76
Conclusion 77
5. NATIONAL ASSESSMENT FINDINGS, POLICY, AND
EDUCATIONAL MANAGEMENT 79
Describing Achievement 80
Describing Resources 81
Monitoring Achievement 82
Reviewing the Education System 84
Formulating General Policy and Assisting in
Decision Making 84
Setting Standards 88
Providing Resources to Schools 90
Supporting Curriculum Revision 92
Revising Textbooks 94
Conclusion 95
CONTENTS | vii
6. NATIONAL ASSESSMENT FINDINGS AND TEACHING 97
Teachers’ Professional Development 98
Focus on Schools and Classroom Teaching 108
Conclusion 121
7. NATIONAL ASSESSMENT FINDINGS AND PUBLIC
AWARENESS 125
Examples of the Use of Assessment Results to
Inform the Public 127
The Role of the Media 129
Developing a Communication Strategy 130
Conclusion 132
8. TOWARD OPTIMIZING THE USE AND VALUE OF
NATIONAL ASSESSMENTS 135
Optimizing the Use of National Assessment Findings 137
Developing National Assessments to Increase Their Value 139
Conclusion 146
REFERENCES 147
INDEX 163
BOXES
1.1 Issues on Which a National Assessment Can Throw Light 2
1.2 Evaluating Achievement on Limited Evidence 8
1.3 The Negative Effect of High-Stakes Accountability Uses 10
1.4 Assigning Accountability for Student Achievement 13
1.5 Shifting the Blame for School Failure 13
1.6 Two Uses of Data from an International Assessment 20
1.7 Extent of Use of National Assessment Results, Honduras 21
3.1 Summary Report of Illinois State Grade 4 Students’ Reading
Achievement Levels on the 2007 NAEP: United States 51
3.2 Suggestions for Applying the PISA Approach to Teaching
and Learning Mathematics: Ireland 52
3.3 Press Release Excerpt, NAEP: United States 60
3.4 Sample News Release: United States 61
3.5 Web Site, Mathematics Achievement in Primary Schools:
Ireland 64
3.6 Site Map, NAEP Web Site: United States 65
viii | CONTENTS
5.1 Using Assessment Results to Promote Reforms:
Dominican Republic 86
5.2 Myths about Education in the United States 87
6.1 Place-Value Matching Item: Pakistan 102
6.2 Use of National Assessment Findings to Improve
Teacher Education: Minas Gerais, Brazil 106
6.3 Main Features of a Teacher In-Service Program Based on
National Assessment Results: Uruguay 107
6.4 Poster Extract Published after National Assessment:
Uganda 109
6.5 Mathematics Item 111
6.6 Recommendations Following a National Assessment in
Mathematics, Grade 5: Ireland 114
6.7 Discussion Topics: National Assessment Results and
School Effectiveness Variables 116
6.8 Discussion Topics: National Assessment Results and
Teacher Effectiveness Variables 116
6.9 Discussion Topics: National Assessment Results and
Individual Student and Family Factors Associated with
Student Learning 117
7.1 Freedom of Information Laws 126
7.2 Agreements on the Release of National Assessment
Results: Uruguay 127
7.3 Procedures to Maximize the Impact of National
Assessment Findings 130
7.4 Brochure Cover for National Assessment, Ethiopia 133
FIGURES
2.1 Grade 4 Reading Skills and Strategies and Cut-Point Scores,
by Benchmark Points, for the Combined Reading Literacy
Scale, PIRLS, 2001 35
2.2 Percentages of Male and Female Students Achieving
Mastery in National Assessment, Grade 4, by Curriculum
Area: Sri Lanka 38
2.3 Percentages of Students Watching Television for Varying
Amounts of Time, by Gender and Average Reading
Score: Kuwait 38
2.4 Regional Differences in Achievement, Grade 4: Ethiopia 40
2.5 Mean Reading Test Scores of Students Plotted against
Number of Books in the Home, Grades 1 and 5: Ireland 42