Learning for tomorrow’s world : first reseults from PISA 2003 : Programme for International Student Assessment.

474 Pages·2004·5.731 MB·English

by  OECD

Programme for International Student Assessment
Learning for
Tomorrow’s World
First Results from PISA 2003
OECD
ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT
ORGANISATION FOR ECONOMIC CO-OPERATION
AND DEVELOPMENT
Pursuant to Article 1 of the Convention signed in Paris on 14th December 1960, and which came into
force on 30th September 1961, the Organisation for Economic Co-operation and Development (OECD)
shall promote policies designed:
– to achieve the highest sustainable economic growth and employment and a rising standard of living in
member countries, while maintaining financial stability, and thus to contribute to the development
of the world economy;
– to contribute to sound economic expansion in member as well as non-member countries in the
process of economic development; and
– to contribute to the expansion of world trade on a multilateral, non-discriminatory basis in
accordance with international obligations.
The original member countries of the OECD are Austria, Belgium, Canada, Denmark, France,
Germany, Greece, Iceland, Ireland, Italy, Luxembourg, the Netherlands, Norway, Portugal, Spain,
Sweden, Switzerland, Turkey, the United Kingdom and the United States. The following countries became
members subsequently through accession at the dates indicated hereafter: Japan (28th April 1964), Finland
(28th January 1969), Australia (7th June 1971), New Zealand (29th May 1973), Mexico (18th May 1994),
the Czech Republic (21st December 1995), Hungary (7th May 1996), Poland (22nd November 1996),
Korea (12th December 1996) and the Slovak Republic (14th December 2000). The Commission of the
European Communities takes part in the work of the OECD (Article 13 of the OECD Convention).
Publié en français sous le titre :
Apprendre aujourd’hui, réussir demain – Premiers résultats de PISA 2003
Originalfassungen veröffentlicht unter dem Titel:
Lernen für die Welt von morgen – Erste Ergebnisse von PISA 2003
PISATM, OECD/PISATM and the PISA logo are trademarks of the Organisation for Economic Co-operation and
Development (OECD). All use of OECD trademarks is prohibited without written permission from the OECD.
© OECD 2004
Permission to reproduce a portion of this work for non-commercial purposes or classroom use should be obtained through the Centre français
d’exploitation du droit de copie (CFC), 20, rue des Grands-Augustins, 75006 Paris, France, tel. [phone], fax [phone], for every country
except the United States. In the United States permission should be obtained through the Copyright Clearance Center, Customer Service, [phone],
222 Rosewood Drive, Danvers, MA 01923 USA, or CCC Online: www.copyright.com. All other applications for permission to reproduce or translate all or part
of this book should be made to OECD Publications, 2, rue André-Pascal, 75775 Paris Cedex 16, France.
Foreword d
r
o
w
e
r
o
F
Compelling incentives for individuals, economies and societies to raise levels of
education have been the driving force for governments to improve the quality of
educational services. The prosperity of countries now derives to a large extent
from their human capital, and to succeed in a rapidly changing world, individuals
need to advance their knowledge and skills throughout their lives. Education
systems need to lay strong foundations for this, by fostering knowledge and
skills and strengthening the capacity and motivation of young adults to continue
learning beyond school.
All stakeholders – parents, students, those who teach and run education systems
as well as the general public – need to be informed on how well their education
systems prepare students for life. Many countries monitor students’ learning in
order to provide answers to this question. Assessment and evaluation – coupled
with appropriate incentives – can motivate students to learn better, teachers to
teach more effectively and schools to become more supportive and productive
environments. Comparative international analyses can extend and enrich the
national picture by providing a larger context within which to interpret national
results. They can provide countries with information to judge their areas of relative
strength and weakness and to monitor progress. They can also stimulate countries
to raise aspirations. And they can provide evidence to direct national policy, for
schools’ curricula and instructional efforts and for students’ learning.
In response to the need for cross-nationally comparable evidence on student
performance, the Organisation for Economic Co-operation and Develoment
(OECD) launched the Programme for International Student Assessment (PISA) in
1997. PISA represents a commitment by governments to monitor the outcomes
of education systems in terms of student achievement on a regular basis and within
an internationally accepted common framework. It aims to provide a new basis for
policy dialogue and for collaboration in defining and implementing educational
goals, in innovative ways that reflect judgements about the skills that are relevant
to adult life. The first PISA assessment was conducted in 2000. Focusing on reading
literacy, PISA 2000 revealed wide differences in the extent to which countries
succeed in enabling young adults to access, manage, integrate, evaluate and reflect
on written information in order to develop their potential and further expand their
horizon. For some countries, the results were disappointing, showing that their
15-year-olds’ performance lagged considerably behind that of other countries,
sometimes by the equivalent of several years of schooling and sometimes despite
high investments in education. PISA 2000 also highlighted significant variation in
the performance of schools and raised concerns about equity in the distribution of
learning opportunities.
3
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003
d How have things changed since 2000? This report presents first results from the PISA
r
o 2003 assessment, which focused on mathematics. It shows that average performance
w
e in the group of the 25 OECD countries for which data can be compared has
r
o
F increased in one of the two content areas of mathematics that was measured in both
2000 and 2003,1 while performance in science, reading and the other comparable
area of mathematics has essentially remained unchanged. However, performance
changes have been uneven across OECD countries. Finland, the top performing
country in the PISA 2000 reading assessment, has maintained its high level of
reading performance while further improving its performance in mathematics and
science, placing it now on a par with the East Asian countries, whose performance
in mathematics and science had been previously unmatched. By contrast, in Mexico,
the lowest performing OECD country in the 2000 assessment, the pressure to
expand the still limited access to secondary education (OECD, 2004a) may have
been one of the factors contributing to lower performance in 2003 in all three
assessment areas.
However, the report goes well beyond an examination of the relative standing
of countries in mathematics, science and reading. It also looks at a wider range
of educational outcomes that include students’ motivation to learn, their beliefs
about themselves and their learning strategies. Furthermore, it examines how
performance varies between the genders and between socio-economic groups.
It also provides insights into some of the factors that are associated with the
development of knowledge and skills at home and at school, and into how these
factors interact and what the implications are for policy development. Most
importantly, the report sheds light on countries that succeed in achieving high
performance standards while at the same time providing an equitable distribution
of learning opportunities. Results in these countries pose challenges for other
countries by showing what it is possible to achieve.
The report is the product of a collaborative effort between the countries
participating in PISA, the experts and institutions working within the framework
of the PISA Consortium, and the OECD. The report was drafted by the OECD
Directorate for Education, principally by Andreas Schleicher, Claudia Tamassia and
Miyako Ikeda, with advice and analytic support from Raymond Adams, Cordula
Artelt (who developed the model underlying Chapter 3), Alla Berezner, Jude
Cosgrove, John Cresswell, Donald Hirsch, Yuko Nonoyama, Christian Monseur,
Claudia Reiter, Wolfram Schulz, Ross Turner and Sophie Vayssettes. Chapters 4
and 5 also draw on analytic work undertaken in the context of PISA 2000 by
Jaap Scheerens and Douglas Willms. The PISA assessment instruments and the
1. In 2003, mathematics was assessed in detail and results are reported on four content scales.
In 2000, a minor assessment of mathematics was reported on only one scale, but the assessment
covered two content areas of the PISA mathematics framework, namely space and shape and change
and relationships (see OECD, 2001a). To allow for comparisons with results from PISA 2003,
separate reporting scales were retrospectively constructed for the 2000 results in these two
content areas.
4
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003
data underlying the report were prepared by the PISA Consortium, under the d
r
direction of Raymond Adams at the Australian Council for Educational Research. o
w
e
r
o
The development of the report was steered by the PISA Governing Board that F
is chaired by Ryo Watanabe (Japan). Annex C of the report lists the members of
the various PISA bodies as well as the individual experts and consultants who
have contributed to this report and to PISA in general.
The report is published on the responsibility of the Secretary-General of the
OECD.
Ryo Watanabe Barry McGaw
Chair of the PISA Governing Board Director for Education, OECD
5
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003
Table of Contents ts
n
e
t
n
o
C

f
o

e
CHAPTER 1 bl
a
INTRODUCTION ...........................................................................................................19 T
PISA – An overview ......................................................................................................20
What PISA measures and how ....................................................................................23
• Literacy in PISA: what is measured ..................................................................25
• The PISA instruments: how measurement takes place ................................25
• The PISA student population .............................................................................27
What is different about the PISA 2003 survey? ......................................................28
• It establishes a detailed understanding of student performance in
mathematics ...........................................................................................................28
• It deepens exploration of cross-curricular competencies ...........................29
• It introduces new background information about students
and schools .............................................................................................................29
• It allows for comparison of change over time ...............................................29
Organisation of the report ...........................................................................................30
READERS’ GUIDE ...........................................................................................................33
CHAPTER 2
A PROFILE OF STUDENT PERFORMANCE IN MATHEMATICS ........................35
Introduction ....................................................................................................................36
The PISA approach to assessing mathematics performance .................................37
• How mathematics is defined ..............................................................................37
• How mathematics is measured ..........................................................................38
• How the PISA tests were constructed .............................................................42
• How the PISA tests were designed, analysed and scaled .............................44
• How results are reported ...................................................................................46
What students can do in four areas of mathematics ..............................................51
• Student performance on the mathematics/space and shape scale ............51
• Student performance on the mathematics/change and
relationships scale .................................................................................................64
• Student performance on the mathematics/quantity scale ..........................74
• Student performance on the mathematics/uncertainty scale ....................85
Overall performance in mathematics ........................................................................89
• The relative strengths and weaknesses of countries in different areas
of mathematical content .....................................................................................89
• A summary picture of mathematics performance ........................................90
• Gender differences in mathematics .................................................................95
The socio-economic context of country performance ........................................99
Implications for policy ................................................................................................103
7
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003
s CHAPTER 3
t
n STUDENT LEARNING: ATTITUDES, ENGAGEMENT AND STRATEGIES ..........109
e
t
on Introduction ..................................................................................................................110
C • Existing evidence on student approaches to learning and

f
o how it frames PISA’s approach ........................................................................113

le • Measuring whether students are likely to adopt effective
b
Ta approaches to learning ......................................................................................114
Students’ engagement with learning in mathematics and school
more generally ..............................................................................................................116
• Interest in and enjoyment of mathematics ....................................................116
• Instrumental motivation ...................................................................................121
• Students’ perception of how well school has prepared them for life ........125
• Students’ sense of belonging at school ..........................................................127
Students’ beliefs about themselves ..........................................................................132
• Students’ self-concept in mathematics ..........................................................132
• Students’ confidence in overcoming difficulties in mathematics .............136
Students’ anxiety in mathematics .............................................................................138
Students’ learning strategies .....................................................................................141
• Controlling the learning process ....................................................................141
• Memorisation and elaboration strategies ......................................................145
How learner characteristics relate to each other
and influence performance ........................................................................................145
How learner characteristics vary across schools ...................................................150
A summary picture of gender differences in learner characteristics ...............151
Implications for policy ................................................................................................156
CHAPTER 4
HOW STUDENT PERFORMANCE VARIES BETWEEN SCHOOLS AND
THE ROLE THAT SOCIO-ECONOMIC BACKGROUND PLAYS IN THIS .........159
Introduction ..................................................................................................................160
Securing consistent standards for schools: a profile of between-
and within-school differences in student performance .......................................160
The quality of learning outcomes and equity in the distribution
of learning opportunities ...........................................................................................164
Socio-economic difference, school difference and the role that education
policy can play in moderating the impact of socio-economic
disadvantage ..................................................................................................................186
Implications for policy ................................................................................................191
CHAPTER 5
THE LEARNING ENVIRONMENT AND THE
ORGANISATION OF SCHOOLING ..........................................................................207
Introduction ..................................................................................................................208
The learning environment and school climate ......................................................211
8
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003
• Students’ perceptions of individual support from their teachers ............211 ts
• Student-related factors affecting the school climate for mathematics ......214 en
• Teacher-related factors affecting the general school climate ...................219 nt
o
• The combined effect of school climate factors ............................................225 C

f
School policies and practices .....................................................................................228 o

e
• School admittance policies ...............................................................................228 bl
• Assessment policies and practices ...................................................................229 Ta
• Approaches to school management ................................................................233
• The combined effect of school policies and practices ................................238
Resources invested in education ...............................................................................240
• Student time invested in learning ...................................................................240
• Availability and quality of human resources .................................................245
• The quality of schools’ physical infrastructure and
educational resources ........................................................................................248
• Public and private stakeholders .......................................................................250
• The combined effect of school resources .....................................................254
What makes a difference for school performance ...............................................255
Institutional differentiation ........................................................................................261
Implications for policy ................................................................................................265
CHAPTER 6
A PROFILE OF STUDENT PERFORMANCE IN READING AND SCIENCE ........271
Introduction ..................................................................................................................272
How reading literacy is measured in PISA .............................................................272
Student performance in reading ...............................................................................273
• The mean performances of countries in reading ........................................280
• Differences in reading performance between PISA 2000
and PISA 2003 .....................................................................................................282
• Gender differences in reading literacy ..........................................................284
How science performance is measured in PISA ...................................................286
Student performance in science ...............................................................................293
• The mean performances of countries in science .........................................293
• Differences in science performance between PISA 2000
and PISA 2003 .....................................................................................................295
• Gender differences in science ..........................................................................296
Implications for policy ................................................................................................298
• Reading .................................................................................................................298
• Science ..................................................................................................................299
REFERENCES .................................................................................................................301
ANNEX A ...................................................................................................................305
Annex A1 Construction of indices and other derived measures
from the student and school context questionnaires .....................306
9
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003
ts Annex A2 Issues relating to the reporting of mathematics performance ........317
n
te Annex A3 The PISA target population, the PISA samples and
n
o the definition of schools .......................................................................320
C

f Annex A4 Standard errors, significance tests and subgroup
o
e comparisons ............................................................................................329
l
b
Ta Annex A5 Quality assurance ...................................................................................332
Annex A6 Development of the PISA assessment instruments ........................333
Annex A7 Reliability of the marking of open-ended items .............................337
Annex A8 Comparison of results from the PISA 2000 and PISA 2003
assessments ..............................................................................................338
ANNEX B ...................................................................................................................339
Annex B1 Data tables for the chapters .................................................................340
Annex B2 Performance differences between regions within countries ..........451
ANNEX C ...................................................................................................................473
The development and implementation of PISA – a collaborative effort ............474
10
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003
LIST OF BOXES s
t
n
Box 1.1 Key features of the PISA 2003 assessment ...................................................................................................................24 e
t
n
Box 2.1 Interpreting sample statistics .........................................................................................................................................58 o
C
Box 2.2 Interpreting differences in PISA scores: how large a gap? .........................................................................................60 f
o
Box 2.3 Changes in gender differences in mathematics and science performance between lower and
e
upper levels of educational systems ..............................................................................................................................96 bl
a
T
Box 3.1 Students who regulate their learning perform better ..............................................................................................113
Box 3.2 Interpreting the PISA indices .......................................................................................................................................117
Box 3.3 Comparing the magnitude of differences across countries .....................................................................................117
Box 3.4 Do students’ beliefs about their abilities simply mirror their performance? .......................................................135
Box 4.1 How to read Figure 4.8 ................................................................................................................................................177
Box 5.1 Interpreting the data from schools and their relationship to student performance ............................................210
LIST OF FIGURES
Figure 1.1 A map of PISA countries ................................................................................................................................................21
Figure 1.2 Summary of the assessment areas in PISA 2003 covered in this volume ................................................................26
Figure 2.1 The relationship between items and students on a proficiency scale .......................................................................45
Figure 2.2 Summary descriptions for the six levels of proficiency in mathematics ..................................................................47
Figure 2.3 A map of selected mathematics items ..........................................................................................................................48
Figure 2.4a A sample of mathematics items used in PISA for the space and shape scale:
Unit CARPENTER .........................................................................................................................................................52
Figure 2.4b A sample of mathematics items used in PISA for the space and shape scale:
Unit STAIRCASE ............................................................................................................................................................53
Figure 2.4c A sample of mathematics items used in PISA for the space and shape scale:
Unit NUMBER CUBES ..................................................................................................................................................54
Figure 2.5 Summary descriptions of six levels of proficiency on the mathematics/space and shape scale ...........................55
Figure 2.6a Percentage of students at each level of proficiency on the mathematics/space and shape scale .........................57
Figure 2.6b Multiple comparisons of mean performance on the mathematics/space and shape scale ....................................59
Figure 2.6c Comparisons between PISA 2003 and PISA 2000 on the mathematics/space and shape scale ..........................62
Figure 2.6d Differences in mean scores between PISA 2003 and PISA 2000 on the mathematics/space and shape scale ..........63
Figure 2.7a A sample of mathematics items used in PISA for the change and relationships scale:
Unit WALKING ...............................................................................................................................................................64
Figure 2.7b A sample of mathematics items used in PISA for the change and relationships scale:
Unit GROWING UP ......................................................................................................................................................66
Figure 2.8 Summary descriptions of six levels of proficiency on the mathematics/change and relationships scale ............68
Figure 2.9a Percentage of students at each level of proficiency on the mathematics/change and relationships scale ..........70
Figure 2.9b Multiple comparisons of mean performance on the mathematics/change and relationships scale ....................71
Figure 2.9c Comparisons between PISA 2003 and PISA 2000 on the mathematics/change and relationships scale ...........73
Figure 2.9d Differences in mean scores between PISA 2003 and PISA 2000 on the mathematics/change
and relationships scale .....................................................................................................................................................74
Figure 2.10a A sample of mathematics items used in PISA for the quantity scale:
Unit EXCHANGE RATE ...............................................................................................................................................75
Figure 2.10b A sample of mathematics items used in PISA for the quantity scale:
Unit SKATEBOARD .......................................................................................................................................................76
Figure 2.11 Summary descriptions of six levels of proficiency on the mathematics/quantity scale ........................................78
Figure 2.12a Percentage of students at each level of proficiency on the mathematics/quantity scale .......................................80
Figure 2.12b Multiple comparisons of mean performance on the mathematics/quantity scale .................................................81
11
© OECD 2004 Learning for Tomorrow’s World – First Results from PISA 2003