ERIC ED480801: The PISA 2003 Assessment Framework: Mathematics, Reading, Science and Problem Solving Knowledge and Skills.

200 Pages·2003·2.9 MB·English

by  ERIC

DOCUMENT RESUME
ED 480 801
TM 035 222
The PISA 2003 Assessment Framework: Mathematics, Reading,
TITLE
Science and Problem Solving Knowledge and Skills.
Organisation for Economic Cooperation and Development, Paris
INSTITUTION
(France).
ISBN-92-64-10172-1
ISBN
2003-00-00
PUB DATE
NOTE
199p.
AVAILABLE FROM
OECD Publications, 2, rue Andre-Pascal, Paris CEDEX 16,
France. Web site: http://www.SourceOECD.org.
PUB TYPE
Books (010)
Reports
Descriptive (141)
EDRS PRICE
EDRS Price MF01/PC08 Plus Postage.
DESCRIPTORS
*Academic Achievement; *Educational Assessment; Elementary
Secondary Education; Foreign Countries; *International
Education; International Studies; Mathematics Achievement;
Problem Solving; Reading Achievement; Scientific Literacy;
*Student Evaluation
ABSTRACT
The Programme for International Student Assessment (PISA) of
the Organisation for Economic Cooperation and Development (OECD) represents a
commitment by OECD member countries to monitor the outcomes of educational
systems in terms of student achievement within a common international
framework. This publication presents the guiding principles of the PISA 2003
assessment, described in terms of content students need to acquire, processes
that need to be performed, and the contexts in which the knowledge and skills
are to be applied. It also illustrates the assessment domains with a range of
sample tasks. The second cycle of PISA in 2003 has been implemented in 42
countries, with tests typically administered to between 4,500 and 10,000
students in each country. Following an introduction that describes PISA, the
chapters are:
"Mathematical Literacy";
"Reading Literacy";
(1)
(2)
(3)
"Scientific Literacy"; and (4)
"Problem Solving." Each chapter contains a
definition of the domain, a discussion of the assessment, and examples of
assessment items. (Contains 83 references.)
(SLD)
Reproductions supplied by EDRS are the best that can be made
from the original document.
PERMISSION TO REPRODUCE AND
TrA TR.
DISSEMINATE THIS MATERIAL HAS
0213
BEEN GRANTED BY
TIY1
The PISA 2003
TO THE EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
Assessment
Framework
MATHEMATICS, READING,
SCIENCE AND PROBLEM SOLVING
KNOWLEDGE AND SKILLS
if
/
es
I
U S DEPARTMENT OF EDUCATION
CN
Office ot Educational Research and Improvement
EDUCATIONAL RESOURCES INFORMATION
In
CENTER (ERIC)
(i)
ihis document has been reproduced as
0
received from the person or organization
originating it
E
C Minor changes have been made to
improve reproduction quality
Points of view or oomions stated in this
document do not necessarily represent
official OERI position or policy
Programme for Onternational Student Assessment
BEST COPY
AVAILABLE
4.
Programme for International Student Assessment
The PISA 2003
Assessment Framework
Mathematics,
Reading,
Science and
Problem Solving
Knowledge and Skills
OECD
ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT
ORGANISATION FOR ECONOMIC CO-OPERATION
AND DEVELOPMENT
Pursuant to Article 1 of the Convention signed in Paris on 14th December 1960, and which came
into force on 30th September 1961, the Organisation for Economic Co-operation and Development
(OECD) shall promote policies designed:
to achieve the highest sustainable economic growth and employment and a rising standard of
living in member countries, while maintaining financial stability, and thus to contribute to the
development of the world economy;
- to contribute to sound economic expansion in member as well as non-member countries in the
process of economic development; and
to contribute to the expansion of world trade on a multilateral, non-discriminatory basis in
accordance with international obligations.
The original member countries of the OECD are Austria, Belgium, Canada, Denmark, France,
Germany, Greece, Iceland, Ireland, Italy, Luxembourg, the Netherlands, Norway, Portugal, Spain,
Sweden, Switzerland, Turkey, the United Kingdom and the United States. The following countries
became members subsequently through accession
the dates indicated
at
hereafter: Japan
(28th April 1964), Finland (28th January 1969), Australia (7th June 1971), New Zealand (29th May 1973),
Mexico (18th May 1994), the Czech Republic (21st December 1995), Hungary (7th May 1996), Poland
(22nd November 1996), Korea (12th December 1996) and the Slovak Republic (14th December 2000). The
Commission of the European Communities takes part in the work of the OECD (Article 13 of the OECD
Convention).
Publié en frangais sous le titre :
Cadre d'évaluation de PISA 2003
Connaissances et compétences en mathématiques, lecture, science et resolution de problèmes
© OECD 2003
Permission to reproduce a portion of this work for non-commercial purposes or classroom use should be obtained through the Centre francais
d'exploitation du droit de copie (CFC), 20, rue des Grands-Augustins, 75006 Paris, France, tel. [phone], fax [phone],
for every country
except the United States. In the United States permission should be obtained through the Copyright Clearance Center, Customer Service, [phone],
222 Rosewood Drive, Danvers, MA 01923 USA, or CCC Online: www.copyright.com. All other applications for permission to reproduce or translate all or part
of this book should be made to OECD Publications, 2, rue André-Pascal, 75775 Paris Cedex 16, France.
4
BEST-COPY AVAILABLE
Table of contents
7
FOREWORD
INTRODUCTION TO THE OECD/PISA 2003 ASSESSMENT
9
Overview
9
Basic features of PISA 2003
10
What makes PISA unique
13
An overview of what is being assessed in each domain
15
How the assessment in 2003 will take place
and how results will be reported
17
The context questionnaires and their use
18
Collaborative development of OECD/PISA
and its assessment frameworks
19
Chapter 1: MATHEMATICAL LITERACY
23
Definition of the domain
24
Theoretical basis for the OECD/PISA mathematics framework
26
Organisation of the domain
30
Situations or contexts
32
Mathematical content The four "overarching ideas"
34-
. Mathematical processes
38
49
Assessing mathematical literacy
49
Task characteristics
Assessment structure
53
Reporting mathematical proficiency
53
Aids and tools
55
Summary
55
Additional examples
56
Mathematics Unit 1: LIGHTHOUSE
57
o Mathematics Unit 2: POSTAL CHARGES
61
Mathematics Unit 3: HEARTBEAT
64
67
Mathematics Unit 4: PAYMENTS BY AREA
Mathematics Unit 5: STUDENT HEIGHTS
69
Mathematics Unit 6: SWING
71
Mathematics Unit 7: WATER TANK
73
Mathematics Unit 8: REACTION TIME
75
Mathematics Unit 9: BUILDING BLOCKS
78
Mathematics Unit 10: DRUG CONCENTRATIONS
82
Mathematics Unit 11: TWISTED BUILDING
85
Mathematics Unit 12: ROCK CONCERT
90
Mathematics Unit 13: MOVING WALKWAYS
91
0 OECD 2003 The PISA 2003 Assessment Framework - Mathematics, Reading, Science and Problem Solving Knowledge and Skills
5
BEST COPY AVAILABLE
Elaboration of the overarching ideas
93
Quantity
93
Space and shape
96
Change and relationships
99
Uncertainty
102
Chapter 2: READING LITERACY
107
Definition of the domain
108
Text format
109
Continuous texts
109
Non-continuous texts
110
Characteristics of the items
112
Five processes (aspects)
112
Item types
117
Marking
118
Situations
118
Reporting of outcomes
119
Scaling the reading literacy tasks
119
o
Reporting
120
Building an item map
122
- Levels of reading literacy proficiency
125
Chapter 3: SC I E NTI FIC LITERACY
131
Definition of the domain
132
Organisation of the domain
135
Scientific knowledge or concepts
135
Scientific processes
136
Situations or context: the areas of application
138
Test characteristics and examples
140
Science Unit 1: STOP THAT GERM!
141
Science Unit 2: PETER CAIRNEY
143
Science Unit 3: CORN
146
Assessment structure
148
Reporting scales
151
Other issues
152
Chapter 4: PROBLEM SOLVING
153
Introduction
154
Background
154
Definition of the domain
156
Organisation of the domain
157
Problem types
159
Problem Solving Unit 1: SAY NO TO PAIN
161
Problem Solving Unit 2: MANAGING CD SALES
165
Problem Solving Unit 3: BICYCLE PUMP
169
4
The PISA 2003 Assessment Framework
Mathematics, Reading, Science and Problem SoMng Knowledge and Skills
0 OECD 2003
170
. Problem solving processes
. Summary of problem types
171
. Situations
171
171
Locating problem solving within PISA 2003
. Key competencies
173
... Problem solving in employment trends and skills demand
173
177
Assessment characteristics
177
. Accessibility and equity
. Calculators
177
177
Item types
177
o Multiple-choice items
. Closed constructed-response items
178
o Open constructed-response items
178
. Groups or units of items
179
179
. Marking guides
. Double-digit scoring
180
. General structure of the assessment
180
Analyses and reporting
181
Potential extensions of the framework for future OECD/PISA cycles
181
- Collaborative problem solving
182
. Computer-based delivery
182
182
Additional Examples
184
o Problem Solving Unit 4: BATTERIES
187
. Problem Solving Unit 5: ROLLERS
. Problem Solving Unit 6: BOOK SALES
192
195
REFERENCES
199
APPENDIX
0 OECD 2003 The PISA 2003 Assessment Framework Mathematics, Reading, Science and Problem Solving Knowledge and Skills
17
Foreword
0
0
The OECD Programme for International Student Assessment (PISA), created in
1997, represents a commitment by the governments of OECD Member countries to
monitor the outcomes of education systems in terms of student achievement, within
a common international framework. OECD/PISA is, above all, a collaborative
effort, bringing together scientific expertise from the participating countries and
steered jointly by their governments on the basis of shared, policy-driven interests.
Participating countries take responsibility for the project at the policy level. Experts
from participating countries also serve on working groups that are charged with
linking the OECD/PISA policy objectives with the best available substantive and
technical expertise in the field of internationally comparative assessment. Through
participating in these expert groups, countries ensure that the OECD/PISA
assessment instruments are internationally valid and take into account the cultural
and curricular context of OECD Member countries; have strong measurement
properties; and, place an emphasis on authenticity and educational validity.
PISA 2003 represents a continuation of the data strategy adopted in 1997 by
OECD countries and the assessed domains build on those used in PISA 2000.
However, while the emphasis in PISA 2000 was on the assessment of reading
literacy, the focus of PISA 2003 is now on mathematical literacy, defined as t.he
capacity of students to identify, understand and engage in mathematics and to
make well-founded judgements about the role that mathematics plays in life. In
addition, an assessment of problem-solving skills has been integrated as a new
element into PISA, defined as the ability of students to use cognitive processes to
solve real cross-disciplinary problems where the solution path is not obvious.
This publication presents the guiding principles of the PISA 2003 assessment,
described in terms of the content t.hat students need to acquire, the processes
that need to be performed, and the contexts in which knowledge and skills are
applied. It also illustrates the assessment domains with a range of sample tasks.
These have been developed by expert panels under the direction of Raymond
Adams, Barry McCrae, Ross Turner and Margaret Wu of the Australian Council
for Educational Research (ACER). The mathematics panel was chaired by Jan
de Lange of the University of Utrecht in the Netherlands, the reading expert
panel was chaired by Irwin Kirsch of the Educational Testing Service in the
United States, the science expert panel was chaired by Wynne Harlen from the
United Kingdom, and the problem solving panel was chaired by John Dossey
of Illinois State University from the United States. The members of the expert
groups are listed in the Appendix of this publication. The frameworks have also
been reviewed by expert panels in each of t.he participating countries.
This publication was prepared by the OECD Directorate for Education, under
the responsibility of Andreas Schleicher and Claudia Tamassia. This report is
published on the responsibility of the Secretary-General of the OECD. J
7
OECD 2003 The PISA 2003 Assessment Framework
Mathematics, Reading, Science and Problem Solving Knowledge and Skills
8
Introdudion to
the OECP/PISA 2003
Assessment
COVERVOIEW
The OECD Programme for International Student Assessment (PISA) is a
collaborative effort on the part of the Member countries of the OECD to measure
how well students at age 15, and therefore approaching the end of compulsory
schooling, are prepared to meet the challenges of today's societies. The OECD/
PISA assessment takes a broad approach to assessing knowledge and skills that
reflect the current changes in curricula, moving beyond the school based approach
towards the use of knowledge in everyday tasks and challenges. These skills
reflect the ability of students to continue learning throughout their lives by
applying what they learn in school to non-school environments, evaluating their
choices and making decisions. The assessment, jointly steered by the participating
governments, brings together the policy interest of countries with scientific
expertise at the national and international levels.
OECD/PISA combines the assessment of domain-specific areas such as reading,
mathematical and scientific literacy with important cross-curricular areas, also a
priority among OECD countries. These areas are covered through an assessment
of self-regulated learning and information technologies, complemented in 2003
by an assessment of problem solving skills. The outcomes are then associated
with contextual information on students, families and institutions collected
through the questionnaires. OECD/ PISA is based on: i) strong quality assurance
mechanisms for translation, sampling and data collection; ii) measures to achieve
cultural and linguistic breadth in the assessment materials, particularly through
countries' participation in the development and revision processes and the
cultural review panels; and iii) the latest methodology for data analysis. The
combination of these measures produces high quality instruments and outcomes
wit.h superior levels of validity and reliability to improve the understanding of
education systems and students' characteristics.
OECD/PISA is based on a dynamic model of lifelong learning in which new
knowledge and skills necessary for successful adaptation to a changing world
are continuously acquired throughout life. OECD/ PISA focuses on things that
15-year-olds will need in the future and seeks to assess what they can do with
but not constrained
by
what they have learned. The assessment is informed
the common denominator of national curricula. Thus, while OECD /PISA does
assess students' knowledge, it also examines their ability to reflect, and to apply
their knowledge and experience to real-world issues. For example, in order to
0 OECD 2003 The PISA 2003 Assessment Framework - Mathematics, Reading. Science and Problem Solving Knowledge and Skills
9
understand and evaluate scientific advice on food safety, an adult would need
not only to know some basic facts about the composition of nutrients, but also
to be able to apply that information. The term "literacy" is used to encapsulate
this broader conception of knowledge and skills.
OECD/PISA was designed to collect information promptly and efficiently
through three-yearly cycles. It presents data on the reading, mathematical and
scientific literacy of students, schools and countries, provides insights into the
factors that influence the development of the skills at home and at school, and
examines how these factors interact and what the implications are for policy
development.
This publication presents the conceptual framework underlying the PISA 2003
assessments: the framework for the assessment of reading and scientific literacy
from PISA 2000, an expanded framework for the in-depth assessment of
mathematical literacy, as well as the framework for the new assessment of
problem solving as a cross-curricular competency. Within each domain, the
framework defines the content that students need to acquire, the processes
that need to be performed and the contexts in which knowledge and skills are
applied. Finally, it illustrates the domains and their aspects with sample tasks.
/UDC FEATURES OF PUSA 2003
PISA 2003 is the second cycle of a data strategy defined in 1997 by participating
countries. The publication Measuring Student Knowledge and Skills A New Framework
for Assessment (OECD, 1999) presented the conceptual framework underlying the
first cycle, known as PISA 2000. The results from that first cycle, presented
in December 2001 in t.he publication Knowledge and Skills for Life
First Results
from NSA 2000 (OECD, 2001)1, allows national policy makers to compare the
performance of their education systems with those of other countries. Similar
to PISA 2000, the 2003 assessment covers the domains of reading, mathematical
and scientific literacy, with the major focus shifting from reading literacy towards
mathematical literacy. Furthermore, the ability of students to solve problems in
real-life settings is examined through an assessment of problem solving. Students
also respond to a background questionnaire, and additional supporting information
is gathered from the school authorities. Forty-two countries, including all 30
OECD Member countries, are taking part in the PISA 2003 assessment.
Since the aim of OECD /PISA is to assess the cumulative yield of education
systems at an age where schooling is compulsory, testing will focus on 15-year-
olds enrolled in both school-based and work-based educational programmes.
Between 5 000 and 10 000 students from at least 150 schools will typically be
tested in each country, providing a good sampling base from which to break
down the results according to a range of student characteristics.
1.This publication is also available through the Web address www.pisa.oecd.org.
0
The PISA 2003 Assessment Framework
Mathematics, Reading. Science and Problem Solving Knowledge and Skills
0 OECD 2003
1 0