TALIS 2008 Technical Report

278 Pages·2010·2.873 MB·English

by  OECD

TALIS 2008 Technical Report
TALIS 2008
The OECD’s new Teaching and Learning International Survey (TALIS) has been designed to provide
data and analyses on the conditions needed for effective teaching and learning in schools. As the
first international survey with this focus, it seeks to fill important information gaps that have been Technical Report
identified at the national and international levels of education systems.
This TALIS Technical Report describes the development of the TALIS instruments and methods
used in sampling, data collection, scaling and data analysis phases of the first round of the survey.
It also explains the rigorous quality control programme that operated during the survey process,
which included numerous partners and external experts from around the world.
The information in this report complements the first international report from TALIS, Creating
Effective Teaching and Learning Environments: First Results from TALIS (OECD, 2009) and the User
Guide for the TALIS International Database (available for download from www.oecd.org/edu/talis/).
Further reading:
Education at a Glance 2009: OECD Indicators (OECD, 2009)
Teachers Matter: Attracting, Developing and Retaining Effective Teachers (OECD, 2008)
Improving School Leadership (OECD, 2008)
PISA 2006: Science Competencies for Tomorrow’s World (OECD, 2007)
T
A
L
I
S
2
0
0
8

T
e
c
h
n
ic
a
The full text of this book is available on line via this link: l R
e
http://www.sourceoecd.org/education/9789264079854 p
o
Those with access to all OECD books on line should use this link: r
t
http://www.sourceoecd.org/9789264079854
SourceOECD is the OECD’s online library of books, periodicals and statistical databases.
For more information about this award-winning service and free trials ask your librarian, or write to us T e a c h i n g A n d L e a r n i n g I n t e r n a t i o n a l S u r v e y
at [email protected].
ISBN 978-92-64-07985-4
87 2010 01 1P
www.oecd.org/publishing

TALIS 2008
Technical Report
Teaching And Learning International Survey
ORGANISATION FOR ECONOMIC CO-OPERATION
AND DEVELOPMENT
The OECD is a unique forum where the governments of 30 democracies work together to
address the economic, social and environmental challenges of globalisation. The OECD is also at
the forefront of efforts to understand and to help governments respond to new developments
and concerns, such as corporate governance, the information economy and the challenges of an
ageing population. The Organisation provides a setting where governments can compare policy
experiences, seek answers to common problems, identify good practice and work to co-ordinate
domestic and international policies.
The OECD member countries are: Australia, Austria, Belgium, Canada, the Czech Republic,
Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea,
Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic,
Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States. The Commission of
the European Communities takes part in the work of the OECD.
OECD Publishing disseminates widely the results of the Organisation’s statistics gathering and
research on economic, social and environmental issues, as well as the conventions, guidelines and
standards agreed by its members.
This work is published on the responsibility of the Secretary-General of the OECD. The
opinions expressed and arguments employed herein do not necessarily reflect the official
views of the Organisation or of the governments of its member countries.
Cover illustration:
© Mike Kemp/Rubberball Productions/Getty Images
© Laurence Mouton/PhotoAlto Agency RF Collections/Inmagine ltb.
Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda.
© OECD 2010
You can copy, download or print OECD content for your own use, and you can include excerpts from OECD publications, databases and multimedia
products in your own documents, presentations, blogs, websites and teaching materials, provided that suitable acknowledgment of OECD as source
and copyright owner is given. All requests for public or commercial use and translation rights should be submitted to [email protected]. Requests for
permission to photocopy portions of this material for public or commercial use shall be addressed directly to the Copyright Clearance Center (CCC)
at [email protected] or the Centre français d’exploitation du droit de copie (CFC) at [email protected].
3
Foreword
The oecD’s new Teaching and Learning International Survey (TALIS) has been designed to provide data and
analyses on the conditions needed for effective teaching and learning in schools. As the first international
survey with this focus, it seeks to fill important information gaps that have been identified at the national and
international levels.
TALIS is conceptualised as a programme of surveys, with successive rounds designed to address policy-relevant
issues chosen by countries. This Technical Report relates to the first round of TALIS conducted in 2007-08,
which studied lower secondary education in both the public and private sectors. It examined important aspects
of teachers’ professional development; teacher beliefs, attitudes and practices; teacher appraisal and feedback;
and school leadership.
This report describes the development of the TALIS instruments and the methods used in sampling, data
collection, scaling and data analysis, for those wishing to review and replicate the procedures used. Furthermore,
it provides insight into the rigorous quality control programme that operated during all phases of the survey,
involving numerous partners and external experts.
The information in this report complements the User Guide for the TALIS International Database (available for
download from www.oecd.org/edu/talis/). The Guide provides a basic yet thorough introduction to the TALIS
international database and to the results included in the first international report from TALIS, Creating Effective
Teaching and Learning Environments: First Results from TALIS (oecD, 2009).
The database contains the survey responses from more than 70 000 teachers and more than 4 000 school
principals in the 24 countries who participated in the first round of TALIS. The database was used to generate
information and to act as a base for the production of the initial international report from the first round of TALIS
published in June 2009 (oecD, 2009).
TALIS is a collaborative effort by the participating countries, guided by their governments on the basis of shared
policy-driven interests. Representatives of each country form the TALIS Board of participating countries, which
determines the policy orientations of TALIS as well as the analysis and results produced from it.
The oecD recognises the significant contributions of Dirk hastedt, Steffen Knoll, Ralph carstens and
Friederike Westphal of the IeA Data processing and Research center in hamburg, Germany, who co-
edited this report. The principal authors of specific chapters were for chapter 1: Dirk hastedt, Steffen Knoll
and Friederike Westphal; chapter 4: Barbara Malak-Minklewiez and Suzanne Morony; chapters 5 and 10:
Jean Dumais and Sylvie LaRoche; chapter 6: Friederike Westphal and Steffen Knoll; chapter 7: Ralph carstens;
chapter 8: Barbara Malak-Minklewiez, Suzanne Morony and Friederike Westphal; chapter 9: Alena Becker
and Ralph carstens; chapter 11: Svenja Vieluf, Juan Leon and Ralph carstens. chapters 2 and 3 and parts of
chapter 11 were authored by the oecD Secretariat.
The editorial work at the oecD Secretariat was carried out by eric charbonnier, Michael Davidson, Ben Jensen,
niccolina clements, Soojin park and elisabeth Villoutreix. A full list of contributors to the TALIS project is
included in Annex A of this report. This report is published under the responsibility of the Secretary-General of
the oecD.
Barbara Ischinger
Director for Education, OECD
TALIS 2008 TechnIcAL RepoRT © OECD 2010

5
Table of Contents
Foreword ..................................................................................................................................................................................................3
List oF abbreviations and acronyms ........................................................................................................................15
chapter 1 IntroductIon and Summary..................................................................................................................17
Abstract .........................................................................................................................................................................................................18
overview of TALIS .................................................................................................................................................................................18
Management of the study .................................................................................................................................................................18
Standardised procedures to ensure high-quality data ...................................................................................................19
national centres and national project Managers ............................................................................................................19
Main tasks of the national project Managers .............................................................................................................19
phases of the TALIS study .................................................................................................................................................................20
pilot study ...........................................................................................................................................................................................20
Field trial...............................................................................................................................................................................................20
Main survey .......................................................................................................................................................................................21
chapter 2 overvIew of taLIS 2008 and framework deveLopment ................................................23
Abstract .........................................................................................................................................................................................................24
The rationale for and aims of TALIS ...........................................................................................................................................24
The TALIS survey programme design ........................................................................................................................................25
TALIS linked to pISA ....................................................................................................................................................................25
The conceptual and analytic framework of the programme .....................................................................................25
choosing the policy focus for TALIS .........................................................................................................................................26
References ..................................................................................................................................................................................................28
chapter 3 deveLopment of teacher and prIncIpaL QueStIonnaIreS .........................................29
Abstract .........................................................................................................................................................................................................30
The role and membership of the oecD TALIS Instrument Development expert Group .......................30
Timeline and phases of questionnaire development ......................................................................................................31
Frameworks guiding questionnaire development ............................................................................................................31
Teacher background characteristics and professional development ...........................................................33
principal and school background characteristics ....................................................................................................34
School leadership and management ................................................................................................................................34
Appraisal of and feedback to teachers ............................................................................................................................35
Teaching practices, attitudes and beliefs .......................................................................................................................37
pilot test ........................................................................................................................................................................................................39
Field trial ......................................................................................................................................................................................................39
References ..................................................................................................................................................................................................42
chapter 4 tranSLatIon and cuLturaL adaptatIon .....................................................................................45
Abstract .........................................................................................................................................................................................................46
overview .....................................................................................................................................................................................................46
Instruments to be translated ............................................................................................................................................................46
cultural and national adaptations ..............................................................................................................................................47
Translation procedures .......................................................................................................................................................................47
Identifying the target language .............................................................................................................................................48
engaging translators .....................................................................................................................................................................49
producing translations ................................................................................................................................................................49
Submitting materials for external verification ............................................................................................................50
TALIS 2008 TechnIcAL RepoRT © OECD 2010
6
TablE Of COnTEnTs
International translation verification .........................................................................................................................................51
processes of translation verification ..................................................................................................................................51
Translation verification report ...............................................................................................................................................51
Translation verification summary........................................................................................................................................51
Feedback from national project Managers on translation and translation verification .........................52
International Quality control Monitor review of translation ...................................................................................52
Layout verification: paper and on line data collection .................................................................................................52
References ..................................................................................................................................................................................................54
chapter 5 SampLe deSIgn.........................................................................................................................................................55
Abstract .........................................................................................................................................................................................................56
overview .....................................................................................................................................................................................................56
International sampling plan ............................................................................................................................................................56
Target population and survey population: International requirements and national
implementations ............................................................................................................................................................................56
Sample size requirements ........................................................................................................................................................58
national sampling strategies ..........................................................................................................................................................58
Sampling frames .............................................................................................................................................................................59
Stratification........................................................................................................................................................................................59
Sample selection ............................................................................................................................................................................59
Sampling for the field trial .......................................................................................................................................................60
national sampling plans ...........................................................................................................................................................60
References ..................................................................................................................................................................................................64
chapter 6 Survey operatIonS procedureS ...........................................................................................................65
Abstract .........................................................................................................................................................................................................66
Manuals and software .........................................................................................................................................................................66
contacting schools and within-school sampling procedures ..................................................................................67
Identification numbers, Teacher Listing Forms and Teacher Tracking Forms .........................................67
Assigning materials to teachers and school principals .........................................................................................68
Administering the questionnaires and national quality control .....................................................................69
Monitoring the on line questionnaires ............................................................................................................................69
Material receipt and preparing for data entry .....................................................................................................................69
Survey Activities Questionnaire ...................................................................................................................................................69
References ..................................................................................................................................................................................................71
chapter 7 on LIne data coLLectIon ............................................................................................................................73
Abstract .........................................................................................................................................................................................................74
overview .....................................................................................................................................................................................................74
Design and mixed-mode considerations ...............................................................................................................................74
Technical implementation ...............................................................................................................................................................76
operations ..................................................................................................................................................................................................78
Field trial ....................................................................................................................................................................................................79
Main study participation, mode distribution and evaluation ...................................................................................80
References ..................................................................................................................................................................................................88
chapter 8 QuaLIty aSSurance ............................................................................................................................................89
Abstract .........................................................................................................................................................................................................90
overview .....................................................................................................................................................................................................90
Quality control in TALIS survey administration .................................................................................................................90
International quality control monitoring programme ...........................................................................................91
Survey anonymity ..................................................................................................................................................................................91
School co-ordinator interviews ....................................................................................................................................................92
Initial preparations ........................................................................................................................................................................92
Survey administration activities ...............................................................................................................................................93
General observations ..................................................................................................................................................................93
© OECD 2010 TALIS 2008 TechnIcAL RepoRT
7
TablE Of COnTEnTs
Survey Activities Questionnaire ...................................................................................................................................................94
contacting schools .......................................................................................................................................................................94
preparing survey materials ......................................................................................................................................................95
conversion to on line data collection .............................................................................................................................96
Survey administration .................................................................................................................................................................96
Data entry and verification .....................................................................................................................................................97
national quality control monitoring programme .....................................................................................................97
References ...............................................................................................................................................................................................101
chapter 9 creatIng and checkIng the InternatIonaL databaSe ..............................................103
Abstract ......................................................................................................................................................................................................104
overview ..................................................................................................................................................................................................104
Data entry and verification at national centres ..............................................................................................................104
Data checking, editing and quality control at the IeA Data processing and Researcher center .......106
Import, documentation and structure check ............................................................................................................107
Identification variable and linkage cleaning ............................................................................................................108
Resolving inconsistencies in questionnaire data ...................................................................................................108
handling missing data ............................................................................................................................................................110
Interim data products .......................................................................................................................................................................111
Building the international database ........................................................................................................................................111
References ...............................................................................................................................................................................................113
chapter 10 eStImatIon weIghtS, partIcIpatIon rateS and SampLIng error ....................115
Abstract ......................................................................................................................................................................................................116
overview ..................................................................................................................................................................................................116
elements of the estimation weight (final weight) ...........................................................................................................116
School base weight (school design weight) ..............................................................................................................117
School non-response adjustment factor ......................................................................................................................117
Final school weight ...................................................................................................................................................................118
Teacher base weight (teacher design weight) ..........................................................................................................118
Teacher non-response adjustment factor ....................................................................................................................119
Teacher adjustment factor for incidental exclusions ..........................................................................................119
Teacher multiplicity adjustment factor .........................................................................................................................119
Final teacher weight .................................................................................................................................................................120
participation rates ...............................................................................................................................................................................120
participation rates for schools ............................................................................................................................................121
participation rate for teachers .............................................................................................................................................121
overall participation rates ....................................................................................................................................................122
Reporting participation rates ...............................................................................................................................................122
Meeting participation rates standard for TALIS .......................................................................................................122
Sampling error with Balanced Repeated Replication .................................................................................................123
creating replicates for Balanced Repeated Replication ...................................................................................123
estimating the sampling error .............................................................................................................................................125
Design effect and effective sample size ......................................................................................................................125
References ...............................................................................................................................................................................................129
chapter 11 conStructIon and vaLIdatIon of ScaLeS and IndIceS ............................................131
Abstract ......................................................................................................................................................................................................132
overview ..................................................................................................................................................................................................132
Simple questionnaire indices, ratios and indicators ....................................................................................................132
Student-teacher ratio ................................................................................................................................................................132
Validation of complex questionnaire scale indices .....................................................................................................135
Indices derived through principal components Analysis ................................................................................135
Description of complex scale indices and their parameters ..................................................................................145
School leadership indices .....................................................................................................................................................145
References ...............................................................................................................................................................................................205
TALIS 2008 TechnIcAL RepoRT © OECD 2010
8
TablE Of COnTEnTs
annex a taLIS conSortIum, expertS and conSuLtantS .........................................................................207
annex b characterIStIcS of natIonaL SampLeS ............................................................................................211
Australia ....................................................................................................................................................................................................212
Austria ........................................................................................................................................................................................................213
Belgium (Flanders) .............................................................................................................................................................................214
Brazil ...........................................................................................................................................................................................................215
Bulgaria .....................................................................................................................................................................................................216
Denmark ...................................................................................................................................................................................................217
estonia ........................................................................................................................................................................................................218
hungary ....................................................................................................................................................................................................219
Iceland .......................................................................................................................................................................................................220
Ireland ........................................................................................................................................................................................................221
Italy ...............................................................................................................................................................................................................222
Korea ...........................................................................................................................................................................................................223
Lithuania ...................................................................................................................................................................................................224
Malaysia ....................................................................................................................................................................................................225
Malta ...........................................................................................................................................................................................................226
Mexico .......................................................................................................................................................................................................227
norway ......................................................................................................................................................................................................228
poland .........................................................................................................................................................................................................229
portugal ......................................................................................................................................................................................................230
Slovak Republic ...................................................................................................................................................................................231
Slovenia .....................................................................................................................................................................................................232
Spain (excluding La Rioja, canarias) .....................................................................................................................................233
The netherlands ..................................................................................................................................................................................234
Turkey .........................................................................................................................................................................................................235
annex c SampLIng formS......................................................................................................................................................237
Sampling Form 1 – participation ...............................................................................................................................................238
Sampling Form 2 – national Target population ..............................................................................................................239
Sampling Form 3 – Stratification...............................................................................................................................................240
Sampling Form 4 – Sampling Frame Description ..........................................................................................................241
Sampling Form 5 – excluded schools ...................................................................................................................................242
Sampling Form 6 – population counts by strata .............................................................................................................243
Sampling Form 7 – Sample allocation ..................................................................................................................................244
annex d taLIS prIncIpaL and teacher QueStIonnaIre .............................................................................245
principal Questionnaire .................................................................................................................................................................246
Background information ........................................................................................................................................................247
School background information .......................................................................................................................................248
School management .................................................................................................................................................................250
Teacher appraisal ........................................................................................................................................................................253
School resources .........................................................................................................................................................................255
Teacher Questionnaire ....................................................................................................................................................................259
Background information ........................................................................................................................................................260
professional development.....................................................................................................................................................262
Teacher appraisal and feedback .......................................................................................................................................264
Teaching practices, beliefs and attitudes ....................................................................................................................267
Your teaching in a particular <class> at this school............................................................................................271
© OECD 2010 TALIS 2008 TechnIcAL RepoRT