ERIC ED610699: IEA International Computer and Information Literacy Study 2018. Technical Report

2020·2.9 MB·English

by  ERIC

IEA International Computer and
Information Literacy Study 2018
TECHNICAL REPORT
Edited by
Julian Fraillon
John Ainley
Wolfram Schulz
Tim Friedman
Daniel Duckworth
IEA International Computer and
Information Literacy Study 2018
Technical Report

Editors
Julian Fraillon John Ainley
Wolfram Schulz Tim Friedman
Daniel Duckworth
IEA International Computer and
Information Literacy Study 2018
Technical Report
Contributors
John Ainley Sebastian Meyer
Ralph Carstens Ekaterina Mikheeva
Alex Daraganov Lauren Musu
Sandra Dohr Louise Ockwell
David Ebbs Wolfram Schulz
Julian Fraillon Sabine Tieck
Tim Friedman
Julian Fraillon Tim Friedman
The Australian Council for Educational Research The Australian Council for Educational Research
Camberwell, Victoria Camberwell, Victoria
Australia Australia
John Ainley Daniel Duckworth
The Australian Council for Educational Research The Australian Council for Educational Research
Camberwell, Victoria Camberwell, Victoria
Australia Australia
Wolfram Schulz
The Australian Council for Educational Research
Camberwell, Victoria
Australia
IEA
Keizersgracht 311
1016 EE Amsterdam
The Netherlands
Telephone: [phone]
Fax: + 31 20 420 7136
Email: [email protected]
Website: www.iea.nl
ISBN/EAN: 9789079549351
© International Association for the Evaluation of Educational Achievement (IEA) 2020
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted
in any form or by any means, electronic, electrostatic, magnetic tape, mechanical, photocopying, recording or
otherwise without permission in writing from the copyright holder.
The International Association for the Evaluation of Educational Achievement (IEA), with
headquarters in Amsterdam, is an independent, international cooperative of national
research institutions and governmental research agencies. It conducts large-scale
comparative studies of educational achievement and other aspects of education, with
the aim of gaining in-depth understanding of the effects of policies and practices within
and across systems of education.
Design by Becky Bliss Design and Production, Wellington, New Zealand
Cover design by Studio Lakmoes, Arnhem, The Netherlands
v
Contents
List of tables and figures ix
Chapter 1: Overview of the IEA International Computer and Information Literacy 1
Study 2018
Introduction 1
Instruments 2
Computer-based test delivery 4
Measures 4
Data 7
Outline of the technical report 9
References 9
Chapter 2: ICILS 2018 test development 11
Introduction 11
Test scope and format 11
Test-development process 15
Field trial test design and content 18
Main survey test design and content 22
Released CIL test module and CT tasks 28
References 29
Chapter 3: Computer-based assessment systems 31
Introduction 31
ICILS 2018 computer-based components and architecture 31
Developing the computer-based delivery platform for ICILS 2018 32
Challenges with computer-based delivery in ICILS 2018 36
Chapter 4: ICILS 2018 questionnaire development 39
Introduction 39
The conceptual framework used to guide questionnaire development 39
Development of the ICILS context questionnaires 41
Development of the student questionnaire 42
Development of the teacher questionnaire 43
Development of the school principal and ICT coordinator questionnaires 44
Development and implementation of the national contexts survey 45
References 47
Chapter 5: Instrument preparation and verification of the ICILS 2018 study instruments 49
Introduction 49
Adaptation and translation of the ICILS study instruments 50
International verification processes 54
Summary 57
Chapter 6: Sampling design and implementation 59
Introduction 59
Target population definitions 59
Coverage and exclusions 61
School sampling design 65
Within-school sampling design 69
Sample size requirements 70
Efficiency of the ICILS 2018 sample design 72
References 78
vi ICILS 2018 TECHNICAL REPORT
Chapter 7: Sampling weights, non-response adjustments, and participation rates 79
Introduction 79
Types of sampling weights 79
Calculating student weights 80
Calculating teacher weights 82
Calculating school weights 83
Calculating participation rates 84
ICILS 2018 standards for sampling participation 89
References 93
Chapter 8: ICILS 2018 field operations 95
Introduction 95
Field operations personnel 95
Field operations resources 96
Field operations processes 98
Field trial procedures 106
Summary 107
Chapter 9: Quality assurance procedures for ICILS 2018 109
Introduction 109
International quality control program 109
Survey activities questionnaire 116
Summary 120
Chapter 10: Data management and creation of the ICILS 2018 database 121
Introduction 121
Data sources 121
Confirming the integrity of the national databases 124
The ICILS 2018 international database 131
Summary 132
References 132
Chapter 11: Scaling procedures for ICILS 2018 test items 133
Introduction 133
The scaling model 133
Test coverage and item dimensionality 134
Assessment of item fit 136
Dimensionality and local dependence 140
Assessment of scorer reliabilities 141
Differential item functioning by gender 144
National reports with item statistics 146
Cross-national measurement equivalence 146
Evaluating the impact of missing responses 148
International item calibration and test reliability 150
CIL and CT estimates 152
Equating CIL scores from ICILS 2013 and 2018 154
The development of proficiency levels for CIL 156
References 158
CONTENTS vii
Chapter 12: Scaling procedures for ICILS 2018 questionnaire items 159
Introduction 159
Simple indices 159
Scaling procedures 164
Item response modeling 166
Describing questionnaire scale indices 168
Scaled indices 170
References 218
Chapter 13: The reporting of ICILS 2018 results 221
Overview 221
Estimation of sampling variance 221
Estimation of imputation variance for CIL and CT scores 224
Reporting of differences 225
Multiple regression modeling of teacher data 228
Hierarchical linear modeling to explain variation in students’ CIL and CT 230
References 233
Appendices
Appendix A: Organizations and individuals involved in ICILS 2018 235
Appendix B: Characteristics of national samples 239
Appendix C: National items excluded from scaling 257
Appendix D: Student background variables used for conditioning 259

List of tables and figures
Tables
Table 2.1: Test development processes and timeline 13
Table 2.2: Summary of ICILS 2018 CIL test modules and large tasks 14
Table 2.3: Field trial CIL test module composition by items derived from tasks 19
Table 2.4: Field trial CT test module composition by items derived from tasks 19
Table 2.5: Field trial test form design and contents 20
Table 2.6: Field trial CIL item mapping to the CIL framework 21
Table 2.7: Field trial CT item mapping to the CT framework 21
Table 2.8: Main survey CIL test module composition by items derived from tasks 23
Table 2.9: Main survey CT test module composition by items derived from tasks 23
Table 2.10: Main survey test form design and contents 24
Table 2.11: Main survey CIL item mapping to the CIL framework 24
Table 2.12: Main survey CT item mapping to the CT framework 25
Table 2.13: Example scoring of the correctness of student coding responses 27
Table 2.14: Example scoring of the efficiency of student coding responses 28
Table 2.15: Example scoring of correctness and efficiency combined 29
Table 3.1: ICILS computer-based or computer-supported systems and operations 33
Table 3.2: Unweighted participation status percentages across participants based on full
database and original coding by test administrators (reference: all sampled
students) 37
Table 4.1: Mapping of variables to contextual framework with examples 41
Table 5.1: Languages used for the ICILS 2018 study instruments 53
Table 6.1: Percentages of schools excluded from the ICILS target population 62
Table 6.2: Percentages of students excluded from the ICILS target population 63
Table 6.4: School, student, and teacher sample sizes 71
Table 6.5: Design effects of main outcome variables —Student survey 75
Table 6.6: Design effects and effective samples sizes of mean of scale scores and 76
plausible values—Student survey
Table 6.7: Design effects of main outcome variables—Teacher survey 77
Table 7.1: Unweighted school and student participation rates—Student survey 86
Table 7.2: Weighted school and student participation rates—Student survey 86
Table 7.3: Unweighted school and teacher participation rates—Teacher survey 88
Table 7.4: Weighted school and teacher participation rates – Teacher survey 89
Table 7.5: Achieved participation categories by country 92
Table 8.1: Hierarchical identification codes 98
Table 8.2: Timing of the ICILS assessment 103
Table 9.1: Preparation of the testing room 113
Table 9.2: Test administrator adherence to the administration script 113
Table 9.3: Technical problems experienced with procedures or devices 114
Table 9.4: Teacher and student listing forms and teacher tracking forms used for the 115
assessment
Table 9.5: School coordinator information 118