Building a Validity Argument for the Test of English as a Foreign Language

385 Pages·2007·1.31 MB·English

by  Carol A. Chapelle

BUILDING A VALIDITY ARGUMENT
FOR THE TEST OF ENGLISH AS A
FOREIGN LANGUAGE™
“… makes a signifi cant contribution to the fi eld of language assessment by
bringing together in one volume all of the excellent work that has gone into
the revision of one of the world’s most infl uential tests. A welcome addition
to the library of anyone interested in assessment issues, both in the U.S. and
internationally.”
Sarah Cushing Weigle, Georgia State University, United States
“… a unique and important contribution to the fi elds of educational measure-
ment, applied linguistics, and language testing. Most of what one fi nds in the
professional literature in terms of validation work involves research and analy-
ses of data gathered from the use of an instrument that is already ‘live.’ What
makes this volume so interesting and unique is that it offers measurement ex-
perts and language testers insights into how construct theory — in this case
foreign/second language profi ciency — and validity arguments come together
to drive the text development process.”
Micheline Chalhoub-Deville, University of North Carolina, United States
The Test of English as a Foreign Language™ (TOEFL®) is among the most im-
portant tests used around the world, to assess English language profi ciency for
academic admissions and placement decisions, and for guiding English language
instruction. This landmark volume provides a detailed description and analysis
of Educational Testing Service’s research and development efforts to develop a
major revision of the TOEFL. The result is a book that serves as a case study
of test design drawing upon theory in the complex domain of English language
profi ciency while attempting to meet standards of educational measurement.
Building a Validity Argument for the Test of English as a Foreign Language™
is distinctive in its attempt to develop a coherent story of the rationale for a test
or its revision, explain the research and development process, and provide the
results of the validation process. Through its treatment of one test, it expands
on and tests principles and approaches to educational measurement, providing
an in-depth, integrated perspective on the overall process of test revision. More-
over, because the conceptual foundation and history are presented alongside
the empirical studies and validity argument, these sometimes disparate areas
are presented in a way that demonstrates their connections — an approach that
represents a departure from, or extension of, conventional materials on test
revision.
This volume is particularly relevant for professionals and graduate students in
educational measurement, applied linguistics, and second language acquisition
as well as anyone interested in assessment issues.
ESL & APPLIED LINGUISTICS PROFESSIONAL SERIES
Eli Hinkel, Series Editor
Liu Idioms: Description, Comprehension, Acquisition, and Pedagogy
Chapelle/Enright/Jamieson, Eds. Building a Validity Argument for the Test of
English as a Foreign Language™
Kondo-Brown/Brown, Eds. Teaching Chinese, Japanese, and Korean Heritage
Students: Curriculum Needs, Materials, and Assessments
Youmans Chicano-Anglo Conversations: Truth, Honesty, and Politeness
Birch English L2 Reading: Getting to the Bottom, Second Edition
Luk/Lin Classroom Interactions as Cross-cultural Encounters: Native Speakers in
EFL Lessons
Levy/Stockwell CALL Dimensions: Issues and Options in Computer Assisted
Language Learning
Nero, Ed. Dialects, Englishes, Creoles, and Education
Basturkmen Ideas and Options in English for Specifi c Purposes
Kumaravadivelu Understanding Language Teaching: From Method to Postmethod
McKay Researching Second Language Classrooms
Egbert/Petrie, Eds. CALL Research Perspectives
Canagarajah, Ed. Reclaiming the Local in Language Policy and Practice
Adamson Language Minority Students in American Schools: An Education in
English
Fotos/Browne, Eds. New Perspectives on CALL for Second Language Classrooms
Hinkel Teaching Academic ESL Writing: Practical Techniques in Vocabulary and
Grammar
Hinkel/Fotos, Eds. New Perspectives on Grammar Teaching in Second Language
Classrooms
Hinkel Second Language Writers’ Text: Linguistic and Rhetorical Features
Visit www.routledge.com for additional information on titles in the ESL & Applied
Linguistics Professional Series
BUILDING A VALIDITY
ARGUMENT FOR
THE TEST OF ENGLISH
AS A FOREIGN LANGUAGE™
Edited by
Carol A. Chapelle
Iowa State University, USA
Mary K. Enright
Educational Testing Service, Princeton, USA
Joan M. Jamieson
Northern Arizona University, USA
First published 2008
by Routledge
270 Madison Ave, New York, NY 10016
Simultaneously published in the UK
by Routledge
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
Routledge is an imprint of the Taylor & Francis Group, an informa business
This edition published in the Taylor & Francis e-Library, 2011.
To purchase your own copy of this or any of Taylor & Francis or Routledge’s
collection of thousands of eBooks please go to www.eBookstore.tandf.co.uk.
© 2008 by the Taylor & Francis Group.
All rights reserved. No part of this book may be reprinted or reproduced or utilized in any
form or by any electronic, mechanical, or other means, now known or hereafter invented, in-
cluding photocopying and recording, or in any information storage or retrieval system, without
permission in writing from the publishers.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks,
and are used only for identifi cation and explanation without intent to infringe.
e-rater, ETS, Graduate Record Examinations, TOEFL, TSE, and TWE are registered trademarks
of Educational Testing Service (ETS). SpeechRater, Test of English as a Foreign Language, Test
of Spoken English, and Test of Written English are trademarks of ETS.
Library of Congress Cataloging-in-Publication Data
Building a Validity argument for the Test of English as a Foreign Language™ / edited by Carol
A. Chapelle, Mary K. Enright, Joan M. Jamieson.
p. cm. -- (ESL & applied linguistics professional series)
Includes bibliographical references and index.
ISBN 978-0-8058-5455-8 (alk. paper) -- ISBN 978-0-8058-5456-5 (pbk. : alk. paper)
1. Test of English as a Foreign Language--Validity. 2. English language--Study and teaching--
Foreign speakers. 3. English language--Ability testing. I. Chapelle, Carol A., 1955- II. Enright,
Mary. III. Jamieson, Joan.
PE1128.A2.B768 2008
428.0076--dc22
2007023704
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library
ISBN 0-203-93789-9 Master e-book ISBN
ISBN 10: 0-8058-5455-X (hbk)
ISBN 10: 0-8058-5456-8 (pbk)
ISBN 10: 0-203-93789-9 (ebk)
ISBN 13: 978-0-8058-5455-8 (hbk)
ISBN 13: 978-0-8058-5456-5 (pbk)
ISBN 13: 978-0-203-93789-1 (ebk)
For
Jim and Annie
David, Chris, and Kate
Rita, George, and Nora

Contents
Preface ix
Acknowledgments xi
Contributors xiii
1 Test Score Interpretation and Use 1
Carol A. Chapelle, Mary K. Enright, and Joan M. Jamieson
2 The Evolution of the TOEFL 27
Carol A. Taylor and Paul Angelis
3 Frameworks for a New TOEFL 55
Joan M. Jamieson, Daniel Eignor, William Grabe, and
Antony John Kunnan
4 Prototyping New Assessment Tasks 97
Mary K. Enright, Brent Bridgeman, Daniel Eignor, Robert N. Kantor,
Pamela Mollaun, Susan Nissan, Donald E. Powers, and Mary Schedl
5 Prototyping Measures of Listening, Reading, Speaking,
and Writing 145
Mary K. Enright, Brent Bridgeman, Daniel Eignor, Yong-Won Lee,
and Donald E. Powers
6 Prototyping a New Test 187
Kristen Huff, Donald E. Powers, Robert N. Kantor, Pamela Mollaun,
Susan Nissan, and Mary Schedl
7 Finalizing the Test Blueprint 227
Mari Pearlman
vii
viii CONTENTS
8 A Final Analysis 259
Lin Wang, Daniel Eignor, and Mary K. Enright
9 The TOEFL Validity Argument 319
Carol A. Chapelle
Appendix A 1995 Working Assumptions that Underlie
an Initial TOEFL 2000 Design Framework 353
Appendix B Summary of 1995 Research Recommendations 357
Appendix C Timeline of TOEFL Origins and the New TOEFL
Project—Key Efforts and Decisions 359
Index 363
Preface
The three of us met sometime during the 1990s at the many meetings that
were held as part of what was then called the TOEFL 2000 project at Edu-
cational Testing Service in the United States. The project entailed discussion
and research studies aimed at the ultimate development of a new Test of
English as a Foreign Language™ (TOEFL®) and validation of its use for ad-
missions decisions at English-medium universities. Because of the high-stakes
decisions involved, development and validation of the new TOEFL proceeded
according to accepted professional practices, as outlined in the Standards
for Educational and Psychological Testing (American Educational Research
Association, American Psychological Association, & National Council on
Measurement in Education, 1999) and the ETS Standards for Quality and
Fairness (Educational Testing Service, 2002). These sources provide a basis
for conceptualizing test design and validation in view of the intended inter-
pretation and use of the test. On the surface, the two anchors—interpreta-
tion and use—appear to provide guidance for specifying the necessary test
tasks and planning validation research. However, the project team found it
extremely diffi cult to design tasks and plan research for the complex domain
of academic English language ability. Moreover, specifying the interpretation
and use in the terms available to test developers throughout the 1990s did not
point to a clear means of synthesizing research into a validity argument.
Synthesizing the products from the TOEFL research and development ef-
forts into a validity argument was the challenge we faced when we embarked
on this book project. By 2002, when the TOEFL 2000 teams had completed
their work, a wealth of knowledge and research results pertaining to the new
TOEFL had been produced. But how could this work be represented in a way
that argued for the validity of TOEFL interpretation and use? The guidelines
all advocated gathering a lot of evidence pertaining to validity, but they did
not clearly express what should be done with it once gathered. We spent con-
siderable time over the past 5 years attempting to sort out what to do with
the validity evidence.
ix