The Development of a Common Framework Scale of Language Proficiency

480 Pages·2000·7.595 MB·

by  Brian North

THEORETICAL STUDIES
IN SECOND
LANGUAGE ACQUISITION
The Development of
a Common Framework Scale
of Language Proficiency
Theoretical Studies in
Second Language Acquisition
Simon Belasco
General Editor
Vol. 8
PETER LANG
New York • Washington, D.C./Baltimore • Boston • Bern
Frankfurt am Main • Berlin • Brussels • Vienna • Oxford
Brian North
The Development of
a Common Framework Scale
of Language Proficiency
PETER LANG
New York • Washington, D.C./Baltimore • Boston • Bern
Frankfurt am Main • Berlin • Brussels • Vienna • Oxford
Library of Congress Cataloging-In-Publication Data
North, Brian.
The development of a common framework scale
of language proficiency I Brian North.
p. em.-(Theoretical studies in second language acquisition; vol. 8)
Includes bibliographical references.
1. Language and languages--Ability testing. 2. Scaling (Social sciences).
3. Communicative competence. I. Title. II. Series.
P53.4.N67 418'.0076-dc21 99-045893
ISBN 0-8204-4852-4
ISSN 10 51-66 70
Die Deutsche Blbllothek-CIP-Einheltsaufnahme
North, Brian:
The development of a common framework scale
of language proficiency I Brian North.
-New York; Washington, D.C.IBaltimore; Boston; Bern;
Frankfurt am Main; Berlin; Brussels; Vienna; Oxford: Lang.
(Theoretical studies in second language acquisition; Vol. 8)
ISBN 0-8204-4852-4
The paper in this book meets the guidelines for permanence and durability
of the Committee on Production Guidelines for Book Longevity
of the Council of Library Resources.
© 2000 Peter Lang Publishing, Inc., New York
All rights reserved.
Reprint or reproduction, even partially, in all forms such as microfilm,
xerography, microfiche, microcard, and offset strictly prohibited.
Printed in the United States of America
Acknowledgments
I would like to express my thanks ftrst of all to Neil Jones for introducing
me to the delights and frustrations of Rasch Modelling. At that time Neil,
now working for the University of Cambridge Local Examinations Syn
dicate (UCLES), was developing Itembanker for Eurocentres whilst doing a
PhD at Edinburgh with Alistair Pollitt. Thanks also to Darry Kenyon at the
Center for Applied Linguistics in Washington for pointing me in the direc
tion of Mike Linacre and FACETS analysis when I came knocking on his
door in 1992 asking: "Isn't there some connection between scales of profi
ciency and Rasch scaling?" Very many thanks to Mike Linacre (Chicago:
MESA Laboratory) and to Alistair Pollitt (now also UCLES) for all the help
and advice they themselves provided with what turned out to be a rather
complicated analysis.
Thanks to Peter Skehan, then running Thames Valley University ELT
Department for his support, guidance and consideration as my PhD super
visor, and to Gunther Schneider of the University of Fribourg, who
"bought" the idea of this methodology and was an ideal friend and
colleague. I would also like to thank the 100 English teachers who let them
selves be cajoled into taking part in the 1994 survey which is reported
most of all my wife Leslie for not only taking part but for continuing to talk
to me during the following few years when I was, shall we say, distracted.
Finally, I would also like to take this opportunity to thank Dick
Lambert (formerly National Foreign Language Center) and the NFLC for
offering me the Mellon visiting fellowship in 1992 which enabled me to
broaden my horizons, ftnd out what had been happening in terms of scales
and scaling outside the language teaching world, and made it feasible for me
to give an academic basis to my involvement in the Council of Europe
Framework development. Thanks also to the Eurocentres Foundation for
their sponsorship of my involvement in the development of the Frame
work, and for allowing me to go part-time in order to undertake the study
reported in this book.
This page intentionally left blank
Contents
List of Figures Xl
List of Tables Xlll
Preface XV
List of Abbreviations XVll
Introduction 1
The Common European Framework 1
Scaling Proficiency Descriptors 3
The Context of the Study 4
1. Language Proficiency Scales 9
Definitions 11
Attractions 11
Origins 13
Purposes 17
Types 21
Metaphors for Scales 23
Descriptions of Behaviour and Behavioural Objectives 25
Difficulties with Common Framework Scales 28
2. Description 41
Definitions of Language Proficiency 41
Interpretations of Communicative Competence 43
Competence and Proficiency 47
The Native Speaker as Criterion 54
User Perspectives 60
Categories for Communicative Language Proficiency 62
Categories for Communicative Activity 98
A Unitary Competence: Holistic Approaches 115
Towards Balanced Scale Categories 123
v111 The Development of a Common Framework Scale ojL:mguage Prr!fidemy
Summary on Description Issues 128
3. Measurement 131
Criterion-referenced Assessment 131
The Development of Behaviourally-based Assessment Scales 139
Uniformity of Scales and Grids 146
Dimensionality 149
Types of Measurement Scales 153
Essentials of a Valid Measurement Scale 156
Common Methods of Scale Construction 161
The Rasch Measurement Model 162
Developing a Framework Scale with the Rasch Model 166
The Many-faceted Rasch Model 175
Summary on Measurement Issues 179
4. Developing a Descriptor Pool 181
Provisional Categories 182
Provisional Levels 183
Editing 184
Pre-testing \Vorkshops with Teachers 185
5. Data Collection and Correction 193
Connecting Questionnaires 194
Rating Conference 198
Rating Scale 206
Subjects 207
Problems with the Analysis 208
6. Constructing the Scale 223
Analysis Specifications and Data Organisation 223
FACETS Output 225
Investigating Use of the Rating Scale 230
Dimensionality: Identifying Problematic Content Strands 233
Refining the Dimension: Quality Control of Anchor Items 246
Investigating Variation across Sectors and Regions 255
Refining the Bank: Quality Control on Individual Descriptors 260
Establishing an Item Quality Hierarchy 268
Contents lX
7. Interpreting the Scale 271
Setting Cut-offs between Levels 271
Scale Shrinkage 283
Content Coherence 285
Progression in Proficiency 290
8. Learner Achievement 311
Decisions on Data Inclusion 311
Analysis 316
Achievement in Educational Sectors 322
Learner Calibrations from Two Data Sets 329
Concurrent Validity 331
9. Conclusions 335
Project Results 335
Replication in Year 2 338
Conclusions in Relation to the Rasch Model 341
Conclusions in Relation to Descriptors 343
Areas for Follow up and Further research 346
Appendices 351
Appendix 1: Sample Data Collection Questionnaire 351
Appendix 2: Sample Rating Conference Mini-questionnaire 357
Appendix 3: Vertical Scale of Descriptors with Fit and SEM 358
Appendix 4: Classified Scales of Descriptors with Sources 386
References 417