Statistical Analyses for Language Testers

329 Pages·2013·9.68 MB·English

by  Rita Green

Statistical Analyses for
Language Testers
Rita Green
Statistical Analyses for Language Testers

Statistical Analyses for
Language Testers
Rita Green
© Rita Green 2013
Foreword © J. Charles Alderson 2013
All rights reserved. No reproduction, copy or transmission of this
publication may be made without written permission.
No portion of this publication may be reproduced, copied or transmitted
save with written permission or in accordance with the provisions of the
Copyright, Designs and Patents Act 1988, or under the terms of any licence
permitting limited copying issued by the Copyright Licensing Agency,
Saffron House, 6–10 Kirby Street, London EC1N 8TS.
Any person who does any unauthorized act in relation to this publication
may be liable to criminal prosecution and civil claims for damages.
The author has asserted her right to be identified as the author of this work
in accordance with the Copyright, Designs and Patents Act 1988.
First published 2013 by
PALGRAVE MACMILLAN
Palgrave Macmillan in the UK is an imprint of Macmillan Publishers Limited,
registered in England, company number 785998, of Houndmills, Basingstoke,
Hampshire RG21 6XS.
Palgrave Macmillan in the US is a division of St Martin’s Press LLC,
175 Fifth Avenue, New York, NY 10010.
Palgrave Macmillan is the global academic imprint of the above companies
and has companies and representatives throughout the world.
Palgrave® and Macmillan® are registered trademarks in the United States,
the United Kingdom, Europe and other countries.
ISBN: 978–1–137–01827–4 hardback
ISBN: 978–1–137–01828–1 paperback
This book is printed on paper suitable for recycling and made from fully
managed and sustained forest sources. Logging, pulping and manufacturing
processes are expected to conform to the environmental regulations of the
country of origin.
A catalogue record for this book is available from the British Library.
A catalog record for this book is available from the Library of Congress.
10 9 8 7 6 5 4 3 2 1
22 21 20 19 18 17 16 15 14 13
Printed and bound in Great Britain by
CPI Antony Rowe, Chippenham and Eastbourne
Contents
Foreword by J. Charles Alderson vii
Introduction and Overview ix
Classical Test Theory versus Modern Test Theory xii
Acknowledgements xiv
Symbols and Acronyms xv
1 Data Entry 1
2 Checking and Correcting Data Files 12
3 Item Analysis 25
4 Descriptive Statistics 41
5 Analysing Test Taker Feedback 56
6 Comparing Performance I: Means, Scatterplots and Correlations 69
7 C omparing Performance II: Parametric and
Non-Parametric Analyses 88
8 Comparing Performance III: ANOVA 107
9 Factor Analysis 122
10 Creating a Control File and Convergence Table 140
11 Analysing the Convergence Table and Creating a Variable Map 150
12 Item and Person Statistics 167
13 Distracter Analysis 185
14 Creating and Running a Specifications File 194
15 Analysing the Iteration Report and Vertical Ruler 206
16 Rater and Item Measurement Reports 215
Appendix 1 Data Files 229
Appendix 2 Data Spreadsheet 231
Appendix 3 Item Analysis 232
Appendix 4 Descriptive Statistics 235
Appendix 5 Comparing Performance I: Means, Scatterplots and Correlations 239
v
vi Contents
Appendix 6 C omparing Performance II: Parametric
and Non-Parametric Analyses 248
Appendix 7 Comparing Performance III: ANOVA 255
Appendix 8 Factor Analysis 260
Appendix 9 Creating a Control File, Convergence Table and Variable Map 271
Appendix 10 Item and Person Statistics 276
Appendix 11 Distracter Analysis 289
Appendix 12 Creating a Specifications File, Iteration Report
and Vertical Ruler 294
Appendix 13 Rater and Assessment Criteria Measurement Reports 299
References and Further Reading 305
Index 307
Foreword
Language testing is both an art and a science, and language test developers
need a range of skills and interests. Obviously an interest in and knowledge of
language and languages, how languages ‘work’, how they are learnt and taught
and how the various aspects of language use – the four skills of reading, writ-
ing, listening and speaking – can be described and developed, all these are
essential for anybody with a professional involvement in language. Language
learning and teaching usually appeal to those who have studied the humani-
ties or the social sciences, rather than the ‘hard’ sciences, and therefore many
language teachers approach the necessary task of assessing their learners either
with trepidation or reluctance.
However, even such teachers will benefit from reading this book. It will pro-
vide them with an eye-opening experience which those who have to develop
tests for their learners, for their institution or for their careers will surely also
benefit from. The title of this volume, Statistical Analyses for Language Testers,
may appear as dry as dust and unattractive to those who hated maths at school.
However, those who open the pages of this book out of curiosity or from a
sense of professional responsibility will certainly benefit. They will rapidly be
drawn into the fascinating world of using figures to understand much better
how good their tests are. They will learn how to improve their tests on the basis
of statistical analyses, and they will explore how ‘numbers’ can throw light on
the art of test design, development and administration.
I have often thought that good test developers and insightful, creative item
writers are probably born, rather than trained. However, this book shows very
clearly how one can become a better test developer, with a more professional
attitude to and understanding of what contributes to the quality, validity and
reliability of a language test. With the help of earlier drafts of this book, par-
ticipants on the summer course Language Testing at Lancaster have experi-
enced how useful it can be to examine the results of a test. They have explored
what makes a test item difficult or easy, why a learner might unexpectedly get
an item wrong, and how to improve the reliability and meaningfulness of test
results – through simple statistical analyses.
I used to tell my incoming Masters students at Lancaster that if they can
do simple arithmetical operations like add, subtract, multiply and divide then
they can easily use, understand and explore statistical procedures that can
reveal even the deepest secrets of the tests they have constructed or used.
When the author of the textbook they are using or the teacher of the course
they are taking is as experienced a teacher and as clear an explainer of even
vii
viii Foreword
the most complex concepts as Dr Rita Green is, then they are certainly in for a
treat. People of all ages, both students and teachers, from the UK, Europe, Asia
and elsewhere – indeed from the four corners of the globe – have had the good
luck to have attended Rita’s courses in language testing and especially in the
use of statistics for language test development. Such learners have discovered
how statistics can reveal all sorts of interesting things about test items, test
tasks and test scores. And those who have not had or will not have the oppor-
tunity of being taught by Rita in person can now experience her clear exposi-
tions, her amusing exemplifications and her sheer good pedagogic sensitivity
to the needs of her audience by working through this wonderfully clear and
readable, practical, sensible and thoroughly enjoyable approach to statistical
analyses for the language test developer.
Enjoy!
J. Charles Alderson
Professor of Linguistics and English Language Education
Lancaster University
Introduction and Overview
Who is this book for?
I have dabbled with data analysis since the early 1980s when I first became
interested in the field of language testing and since that time have used the
Statistical Package for Social Sciences (SPSS) to help me investigate how items
and tasks are performing. Since the 1990s I have taught SPSS to numerous stu-
dents from countries around the world, many of them working on national and
international projects including a number of high-stakes tests; others engaged
in MA or PhD studies. Over the last ten years, I have added Winsteps and Facets
to the programmes I teach my students. It is these language testers for whom I
have written this book and I have done this for a number of reasons.
First, because I want to encourage the ‘everyday’ language test developer or
item writer to embrace the insights that data analysis can offer them in their
work. Second, although there are many books available which deal with both
classical test theory (CTT) and modern test theory (MTT), few of them focus
on readers from the field of language testing. Third, many of the current books
are, I suspect, somewhat intimidating to the type of reader I have in mind.
I feel there is a stepping stone which is missing as far as test developers are
concerned; I see this book as filling that gap, providing a ‘taster’ of what is
out there – something to work through and then decide whether you want to
delve further into the mysteries of statistical analyses. And, of course, I hope
you do.
You may already be thinking ... hmm, statistics ... not for me; or perhaps you
have visions of school maths already swimming in front of your eyes. Before
you take these thoughts further, read on. This book is not about mathemat-
ics; it is not about theoretical statistics per se – although of course I do discuss
some of the concepts which are the foundation of the applied statistics on
which this book is based. The word ‘applied’ is crucial; it is the application of
these analyses to the field of language testing which makes this book hopefully
more accessible than others which have been written with sociologists, psy-
chologists, economists and other types of scientists in mind. To this end the
analyses carried out in this book are based on data which come from real tests,
developed for real purposes, and the data are real data. The tests and question-
naires, from which the data come, are not perfect – this is not their purpose;
they were chosen as vehicles to show you the reader how to apply and interpret
the relevant statistical methods which will in turn provide insights into your
own test development work.
ix