Language Testing and Validation - An Evid.-based Appr. [linguistics]

316 Pages·2005·1.97 MB·English

by  C. Weir

Language Testing and
Validation
An Evidence-based Approach
Cyril J. Weir
Language Testing and Validation
Research and Practice in Applied Linguistics
General Editors: Christopher N. Candlin and David R. Hall.
All books in this series are written by leading researchers and teachers in Applied
Linguistics, with broad international experience. They are designed for the MA or
PhD student in Applied Linguistics, TESOL or similar subject areas and for the
language professional keen to extend their research experience
Titles include:
Cyril J. Weir
LANGUAGE TESTING AND VALIDATION
Forthcoming titles:
Martin Bygate and Virginia Samuda
TASKS IN LANGUAGE LEARNING
Francesca Bargiela, Catherine Nickerson and Brigitte Planken
BUSINESS COMMUNICATION
Sandra Gollin and David R. Hall
LANGUAGE FOR SPECIFIC PURPOSES
Sandra Hale
COMMUNITY INTERPRETING
Geoff Hall
LITERATURE IN LANGUAGE EDUCATION
Richard Kiely and Pauline Rea-Dickins
PROGRAM EVALUATION IN LANGUAGE EDUCATION
Martha Pennington
PRONUNCIATION
Devon Woods and Emese Bukor
INSTRUCTIONAL STRATEGIES IN LANGUAGE EDUCATION
Tony Wright
LANGUAGE EDUCATION AND CLASSROOM MANAGEMENT
Research and Practice in Applied Linguistics
Series Standing Order ISBN 1–4039–1184–3 hardcover
Series Standing Order ISBN 1–4039–1185–1 paperback
(outside North America only)
You can receive future titles in this series as they are published by placing a standing
order. Please contact your bookseller or, in case of difficulty, write to us at the address
below with your name and address, the title of the series and one of the ISBNs quoted
above.
Customer Services Department, Macmillan Distribution Ltd, Houndmills, Basingstoke,
Hampshire RG21 6XS, England
Language Testing and
Validation
An Evidence-based Approach
Cyril J. Weir
Centre for Research in Testing, Evaluation and Curriculum (CRTEC)
Roehampton University
© Cyril J. Weir 2005
All rights reserved. No reproduction, copy or transmission of this
publication may be made without written permission.
No paragraph of this publication may be reproduced, copied or transmitted
save with written permission or in accordance with the provisions of the
Copyright, Designs and Patents Act 1988, or under the terms of any licence
permitting limited copying issued by the Copyright Licensing Agency,
90 Tottenham Court Road, London W1T 4LP.
Any person who does any unauthorised act in relation to this publication
may be liable to criminal prosecution and civil claims for damages.
The author has asserted his right to be identified
as the author of this work in accordance with the Copyright,
Designs and Patents Act 1988.
First published 2005 by
PALGRAVE MACMILLAN
Houndmills, Basingstoke, Hampshire RG21 6XS and
175 Fifth Avenue, New York, N.Y. 10010
Companies and representatives throughout the world
PALGRAVE MACMILLAN is the global academic imprint of the Palgrave
Macmillan division of St. Martin’s Press, LLC and of Palgrave Macmillan Ltd.
Macmillan® is a registered trademark in the United States, United Kingdom
and other countries. Palgrave is a registered trademark in the European
Union and other countries.
ISBN 1–4039–1188–6 hardback
ISBN 1–4039–1189–4 paperback
This book is printed on paper suitable for recycling and made from fully
managed and sustained forest sources.
A catalogue record for this book is available from the British Library.
A catalog record for this book is available from the Library of Congress.
10 9 8 7 6 5 4 3 2 1
14 13 12 11 10 09 08 07 06 05
Printed and bound in Great Britain by
Antony Rowe Ltd, Chippenham and Eastbourne
To Shigeko and Jamie, and to my friends
This page intentionally left blank
Contents
General Editors’ Preface x
Acknowledgements xi
Abbreviations xii
Introduction 1
Part 1 Testing as Validity 3
1 Language Testing Past and Present 5
1.1 The Cambridge Proficiency Examination 1913–1945:
‘The Garden of Eden’, ‘the pre-scientific era’ 5
1.2 Developments in the 1960s: the move towards a
language-based examination 7
1.3 The 1975 and 1984 revisions: ‘The Promised Land’? 8
2 The Nature of Test Validity 11
3 Before the Test Event: A Priori Validity Evidence 17
3.1 Theory-based validity 17
3.2 Context validity 19
4 After the Test Event: A Posteriori Validity Evidence 22
4.1 Scoring validity 22
4.2 Criterion-related validity 35
4.3 Consequential validity 37
Part 2 New Frameworks for Developing and Validating
Tests of Reading, Listening, Speaking and Writing 41
Introduction 43
5 Test Takers 51
5.1 Physical/physiological characteristics:
making accommodations 52
5.2 Psychological characteristics: affective schemata 53
5.3 Experiential characteristics: familiarity 54
6 Context Validity in Action 56
6.1 Task setting 57
6.2 Task demands 68
6.3 Setting and test administration 82
vii
viii Contents
7 Theory-based Validity in Action 85
7.1 Reading 87
7.2 Listening 95
7.3 Speaking 102
7.4 Writing 108
8 Response Formats 119
8.1 Techniques for testing reading comprehension 119
8.2 Techniques for testing listening comprehension 132
8.3 Techniques for testing speaking 143
8.4 Techniques for testing written production 161
9 Scoring Validity in Action 177
9.1 Scoring written production 179
9.2 Scoring speaking tests 191
9.3 Internal reliability of receptive tests 201
9.4 Scores, grading and post-exam validation
procedures 205
10 External Validities in Action 207
10.1 Criterion-related validity 207
10.2 Consequential validity 210
Part 3 Generating Validity Evidence 217
Introduction 219
11 Research Methodologies for Exploring the
Validity of a Test 221
11.1 An introductory note on research 221
11.2 A priori validation: investigating the
specification of the construct and the
operationalization of the test 222
11.3 Establishing context validity 224
11.4 Establishing theory-based validity evidence 233
11.5 Establishing scoring validity evidence 247
11.6 Establishing evidence on a posteriori validities 259
Part 4 Further Resources in Language Testing 271
12 Key Sources 273
12.1 Books 273
12.2 Journals 274
12.3 Professional associations 276
12.4 Principal testing conferences 276
12.5 Email lists and bulletin boards 277
12.6 Internet sites 277
Contents ix
12.7 Databases 280
12.8 Statistical packages 280
Postscript 283
References 285
Index 299