Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis

623 Pages·2013·8.211 MB·English

by  Leslie Rutkowski, Matthias von Davier, David Rutkowski

Chapman & Hall/CRC
Statistics in the Social and Behavioral Sciences Series
Handbook
of International
Large-Scale
Assessment
Background, Technical Issues,
and Methods of Data Analysis
Edited by
Leslie Rutkowski
Indiana University
Bloomington, USA
Matthias von Davier
Educational Testing Service
Princeton, New Jersey, USA
David Rutkowski
Indiana University
Bloomington, USA
Chapman & Hall/CRC
Statistics in the Social and Behavioral Sciences Series
Series Editors
Jeff Gill Steven Heeringa
Washington University, USA University of Michigan, USA
Wim van der Linden J. Scott Long
CTB/McGraw-Hill, USA Indiana University, USA
Tom Snijders
Oxford University, UK
University of Groningen, UK
Aims and scope
Large and complex datasets are becoming prevalent in the social and behavioral
sciences and statistical methods are crucial for the analysis and interpretation of such
data. This series aims to capture new developments in statistical methodology with
particular relevance to applications in the social and behavioral sciences. It seeks to
promote appropriate use of statistical, econometric and psychometric methods in
these applied sciences by publishing a broad range of reference works, textbooks and
handbooks.
The scope of the series is wide, including applications of statistical methodology in
sociology, psychology, economics, education, marketing research, political science,
criminology, public policy, demography, survey methodology and official statistics. The
titles included in the series are designed to appeal to applied statisticians, as well as
students, researchers and practitioners from the above disciplines. The inclusion of real
examples and case studies is therefore essential.
Published Titles
Analysis of Multivariate Social Science Data, Second Edition
David J. Bartholomew, Fiona Steele, Irini Moustaki, and Jane I. Galbraith
Latent Markov Models for Longitudinal Data
Francesco Bartolucci, Alessio Farcomeni, and Fulvia Pennoni
Statistical Test Theory for the Behavioral Sciences
Dato N. M. de Gruijter and Leo J. Th. van der Kamp
Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences
Brian S. Everitt
Bayesian Methods: A Social and Behavioral Sciences Approach,
Second Edition
Jeff Gill
Multiple Correspondence Analysis and Related Methods
Michael Greenacre and Jorg Blasius
Applied Survey Data Analysis
Steven G. Heeringa, Brady T. West, and Patricia A. Berglund
Informative Hypotheses: Theory and Practice for Behavioral and
Social Scientists
Herbert Hoijtink
Foundations of Factor Analysis, Second Edition
Stanley A. Mulaik
Linear Causal Modeling with Structural Equations
Stanley A. Mulaik
Handbook of International Large-Scale Assessment: Background, Technical
Issues, and Methods of Data Analysis
Leslie Rutkowski, Matthias von Davier, and David Rutkowski
Generalized Linear Models for Categorical and Continuous Limited
Dependent Variables
Michael Smithson and Edgar C. Merkle
Incomplete Categorical Data Design: Non-Randomized Response Techniques
for Sensitive Questions in Surveys
Guo-Liang Tian and Man-Lai Tang
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2014 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20130923
International Standard Book Number-13: 978-1-4398-9514-6 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been obtained. If any copyright material has
not been acknowledged please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit-
ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system,
without written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.
com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the CCC,
a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Editors ......................................................................................................................ix
Contributors ............................................................................................................xi
Section I Policy and Research Relevance of
International Large-Scale Assessment Data
1. A Brief Introduction to Modern International Large-Scale
Assessment .......................................................................................................3
David Rutkowski, Leslie Rutkowski, and Matthias von Davier
2. International Large-Scale Assessments: From Research to Policy .....11
Hans Wagemaker
3. The Impact of International Studies of Academic Achievement
on Policy and Research ................................................................................37
Stephen P. Heyneman and Bommi Lee
Section II Analytic Processes and Technical
Issues around International Large-Scale
Assessment Data
4. Assessment Design for International Large-Scale Assessments ........75
Leslie Rutkowski, Eugene Gonzalez, Matthias von Davier, and Yan Zhou
5. Modeling Country-Specific Differential Item Functioning ................97
Cees Glas and Khurrem Jehangir
6. Sampling, Weighting, and Variance Estimation in International
Large-Scale Assessments ..........................................................................117
Keith Rust
7. Analytics in International Large-Scale Assessments: Item
Response Theory and Population Models ............................................155
Matthias von Davier and Sandip Sinharay
8. Imputing Proficiency Data under Planned Missingness
in Population Models ................................................................................175
Matthias von Davier
v
vi Contents
9. Population Model Size, Bias, and Variance in Educational
Survey Assessments ...................................................................................203
Andreas Oranje and Lei Ye
10. Linking Scales in International Large-Scale Assessments................229
John Mazzeo and Matthias von Davier
11. Design Considerations for the Program for International
Student Assessment ...................................................................................259
Jonathan P. Weeks, Matthias von Davier, and Kentaro Yamamoto
12. Innovative Questionnaire Assessment Methods to Increase
Cross-Country Comparability .................................................................277
Patrick C. Kyllonen and Jonas P. Bertling
13. Relationship between Computer Use and Educational
Achievement ................................................................................................287
Martin Senkbeil and Jörg Wittwer
14. Context Questionnaire Scales in TIMSS and PIRLS 2011 .................299
Michael O. Martin, Ina V. S. Mullis, Alka Arora, and Corinna Preuschoff
15. Motivation and Engagement in Science around the Globe:
Testing Measurement Invariance with Multigroup Structural
Equation Models across 57 Countries Using PISA 2006.....................317
Benjamin Nagengast and Herbert W. Marsh
16. Contextual Indicators in Adult Literacy Studies: The Case
of PIAAC ......................................................................................................345
Jim Allen and Rolf van der Velden
Section III Advanced Analytic Methods for
Analyzing International Large-Scale
Assessment Data
17. Incorporating Sampling Weights into Single-
and Multilevel Analyses ...........................................................................363
Laura M. Stapleton
18. Multilevel Analysis of Assessment Data ...............................................389
Jee-Seon Kim, Carolyn J. Anderson, and Bryan Keller
Contents vii
19. Using Structural Equation Models to Analyze ILSA Data ................425
Leslie Rutkowski and Yan Zhou
20. Efficient Handling of Predictors and Outcomes Having
Missing Values ............................................................................................451
Yongyun Shin
21. Multilevel Modeling of Categorical Response Variables ..................481
Carolyn J. Anderson, Jee-Seon Kim, and Bryan Keller
22. Causal Inference and Comparative Analysis with Large-Scale
Assessment Data .........................................................................................521
Joseph P. Robinson
23. Analyzing International Large-Scale Assessment Data within
a Bayesian Framework ...............................................................................547
David Kaplan and Soojin Park
24. A General Psychometric Approach for Educational Survey
Assessments: Flexible Statistical Models and Efficient
Estimation Methods ...................................................................................583
Frank Rijmen, Minjeong Jeon, Matthias von Davier, and Sophia
Rabe-Hesketh
Index .....................................................................................................................607
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
Editors
Leslie Rutkowski, PhD, is an assistant professor of inquiry methodology
in the Department of Counseling and Educational Psychology at Indiana
University, Bloomington, Indiana, USA. Leslie joined the inquiry method-
ology program at Indiana University in January of 2010. She earned her
PhD in educational psychology with a specialization in statistics and edu-
cational measurement from the University of Illinois at Urbana-Champaign.
Leslie’s research is focused in the area of international large-scale assess-
ment from both methodological and applied perspectives. Her interests
include the impact of background questionnaire quality on achievement
results, latent variable models for achievement estimation, and examining
methods for comparing heterogeneous populations in international surveys.
Leslie teaches courses in quantitative methods, including covariance struc-
ture analysis, multi level modeling, and categorical data analysis. As part of
her work at the university, Leslie frequently collaborates with or consults
for international organizations, including the International Association for
the Evaluation of Educational Achievement (IEA) and the Organization
for Economic Co-Operation and Development (OECD). Leslie is cur-
rently involved with the 2013 cycle of the OECD’s Teaching and Learning
International Survey (TALIS). She also conducts seminars and workshops in
quantitative methods for researchers from around the world.
Matthias von Davier, PhD, is a research director in the Research &
Development Division at the Educational Testing Service (ETS). He joined
ETS, which is located in Princeton, New Jersey, USA, in 2000. He earned
his PhD in psychology from the University of Kiel, Germany, in 1996, spe-
cializing in psychometrics. At ETS, Dr. von Davier manages a group of
researchers concerned with methodological questions arising in large-scale
international comparative studies in education. He is one of the editors of the
periodical Issues and Methodologies in Large-Scale Assessments, which is jointly
published by the International Association for the Evaluation of Educational
Achievement (IEA) and ETS through the IEA-ETS Research Institute (IERI).
His current work at ETS involves the psychometric methodologies used in
analyzing cognitive skills data and background data from large-scale edu-
cational surveys, such as the Organization for Economic Co-Operation and
Development’s upcoming Program for the International Assessment of
Adult Competencies (PIAAC) and the ongoing Programme for International
Student Assessment (PISA), as well as the IEA’s Trends in International
Mathematics and Science Study (TIMSS) and Progress in International
Reading Literacy Study (PIRLS). His work at ETS also involves the develop-
ment of software for multidimensional models for item response data, and
ix