The IELTS Writing Assessment Revision Project

295 Pages·2008·1 MB·English

Issue 01 January 2008

The IELTS
Writing Assessment
Revision Project:
Towards a revised rating scale

Stuart Shaw and Peter Falvey
For further information contact:
University of Cambridge ESOL Examinations,
1 Hills Road, Cambridge, CB1 2EU, United Kingdom
Tel. [phone] Fax. [phone]
1
Email [email protected]
www.CambridgeESOL.org/research
Contents

Preface
Acknowledgements
1. Introduction

2. Phase 1 Consultation, initial planning and design

3. Phase 2 Development

4. Phase 3 Validation: background

5. Phase 3 Validation: the quantitative study

6. Phase 3 Validation: the qualitative study

7. Phase 3 Validation: the Writing task rubric

8. Phase 4 Implementation

9. Phase 5 Operation

10. Conclusion

Bibliography
Appendices
Appendix A: Final Version of the IELTS Examiners’ Questionnaire – Phase 1
Appendix B: Assessment Criteria for Rating Scale – Phase 2
Appendix C: The Trial Methodology – Phase 3
Appendix D: Trial Script Selection and Commentary – Phase 3
Appendix E: Trial Examiner Questionnaire – Phase 3
Appendix F: Revised Method of Assessment for IELTS Writing – Phase 4
Appendix G: Training Event Timetable – Phase 4
Appendix H: Training Examiner Questionnaire – Phase 4
Appendix I: Band Level Descriptors – Phase 5
Appendix J: Research Notes Articles

2
Preface
This report has arisen from the need to make available to the reader a detailed description and
discussion of the processes involved in the revision of a component of a high-stakes
international examination. In this case, the revision of the Writing component of the
International English Language Testing System (IELTS) is reported. The need to make the
revision processes available fits well with growing demands to de-mystify examinations, to
make the examination as transparent as possible (within the constraints of examination
security) and to respond to the growing acceptance in the language testing community of the
need for ethical approaches to testing (Shohamy 1997). In addition to these reasons, the
description and the discussion of the revision process help to make the Writing component of
the examination that much more accessible to the stakeholders in IELTS (the test developers,
the clients, the test takers, the Writing component raters, researchers, the testing community
and all those who come into contact with either the examination or the results of the
examination) than it might otherwise have been.
The process of making the report accessible to its readers has benefited enormously by the
availability of the documentation detailing the meticulous accumulation of notes of the
meetings of the IELTS Writing Assessment Revision Working Group and the reports and
research studies that were generated both internally (by Cambridge ESOL members) and
externally (by researchers). Readers will note how Chapters 5, 6 and 7 provide very detailed
accounts of the studies embodied in the quantitative and qualitative research on validation
carried out during the revision process and by the revision process-inspired study of the
ethical issues involved in investigating proposals for rubric change. Changes to the rubric of
the IELTS Writing component were not part of the original brief but emerged later as a
necessary and integral part of the process.
This change to the original brief for the revision, proposed by the IELTS Policy Group (a
committee composed of the three IELTS partners, viz. Cambridge ESOL, British Council and
IDP Education Australia), indicates clearly the way in which a thorough and rigorous revision
process can develop. It began with consultation and planning and the study of research reports
commissioned before the revision project began as part of the British Council/IDP-funded
research studies into various aspects of IELTS (see Taylor and Falvey, 2007). It then required
further studies as the process was underway, e.g. the need for an investigation into proposed
rubric changes that emerged from the process of revision development.
Another interesting aspect of the revision process that is detailed here is the iterative process
employed by Cambridge ESOL in its revision projects. This means that no one solution to a
problem or a decision about the revisions is accepted as a one-off event. The changes to the
3
assessment criteria and the descriptive bands in this project underwent constant change and
redefinition throughout the years 2001–2005 as the Working Group became aware of new
information, emerging concerns and ongoing research findings which impinged on the
revision project. The writing assessment bands and descriptors, for example, went through
sixteen formal drafts before being finalised for operation in 2005.
Although the Acknowledgements page thanks those who took part in the revision project, it
is also worth noting how large was the number of people who were involved in the revision
processes – processes that took place during the five phases of revising the Writing
component. Most classroom and even institutional examinations simply do not and cannot
have available to them large numbers of human resources when new developments and
examination revisions are initiated. High stakes examinations, however, because of their very
nature, demand that every effort is made to ensure that the developments and/or revisions are
as rigorous and integrity-based as possible because of the implications of the results for the
lives and careers of hundreds of thousands every year. This approach necessarily entails the
involvement of large numbers of participants in the revision process. The Cambridge ESOL
approach to the development and revision of examinations is encapsulated by the four
essential examination elements of the following paradigm: Validity, Reliability, Impact and
Practicality (VRIP). In order to ensure that the requirements for these four elements are met,
substantial resources are provided and used. This, as the reader will see, has been done for the
IELTS Writing Assessment Revision Project.
Stuart Shaw Peter Falvey
Validation Officer Consultant
Research and Validation Cambridge ESOL
Cambridge ESOL

January 2008
4
Acknowledgements
As in any major revision of a public, high-stakes examination, a large number of stakeholders
and other people contributed to the thinking involved in the revision process, in the various
phases through which the revisions went, in the research studies that formed part of the
process of revision, in the processes of testing and re-testing the use of the revised bands and
descriptors, and in responding to questionnaires and participating in trials. It is impossible to
mention all these people by name. To all of them, we are extremely grateful.

Those named below were key contributors to the project.

Graeme Bridges
Nick Charge
Liz Dixon
Patricia Dooey
Sheila Hennessey
Mary Jane Hogan
Vanessa Jakeman
Helen Johnston
Julie King
Shakeh Manassian
Caroline Malthus
Mary Martina
Kate McPherson
Don Oliver
Denley Pike
Lynda Taylor
Morgan Terry
Andrew Thomas
Paul Thompson
Steve Walsh

5
CHAPTER 1
Introduction

Chapter overview
The first section of this chapter provides a short history of IELTS. This is followed by a
discussion of the revisions to the Writing component of IELTS.

IELTS – the history
The English Language Testing Service (ELTS), as it was then known, made its first
appearance in 1980 when it replaced the English Proficiency Test Battery (EPTB), a
traditional largely multiple–choice test battery that had been used by the British Council in its
overseas student recruitment operation since the mid 1960s for the purpose of screening
international applicants to universities and colleges in the UK. The 1980 test had an
innovative format that reflected changes in language learning and teaching theory and
developments in language testing. In particular, the ELTS was influenced by the growth in
‘communicative’ language learning and ‘English for specific purposes’. Test tasks were based
on an analysis of the ways in which language was used in academic contexts and were
intended to reflect the use of language in the ‘real world’.

The format of the ELTS test
The ELTS test offered a choice of six modules covering five broad areas of study of UK
tertiary education, plus one non-specific area. The six modules were:

• Life Sciences
• Social Studies
• Physical Sciences
• Technology
• Medicine
• General Academic.

There was also a Non-Academic test for vocational candidates.
Each candidate was required to take three sections in their subject area or module and two
common tests in the General section:
6
M1 Study Skills G1 General Reading
M2 Writing G2 General Listening
M3 Individual Interview

A further feature of the test was that the three subject area modules were thematically
linked: candidates were required to write on a topic connected to one of the texts in the Study
Skills paper. Similarly, in the Interview the candidate would be asked to discuss a topic
already covered in M1.
ELTS continued in the form outlined above until 1989. During the 1980s the test numbers
were quite low (4,000 in 1981 rising to 10,000 by 1985), and it was clear that there were
practical difficulties with the administration of the test, relating to the number of test items
and the time taken to complete the test; there were also powerful reasons for change on the
grounds of test redundancy.
In 1987 British Council and UCLES EFL (now known as Cambridge ESOL) commissioned
Edinburgh University to conduct a validation study (see Criper and Davies 1988; Hughes,
Porter and Weir 1988). Following this report the ELTS Revision Project, under the academic
direction of Professor Charles Alderson of Lancaster University, was set up to oversee the
design and construction of the revised test (Alderson and Clapham 1993).
There was consensus to broaden the international participation in the revision project and in
response to this the International Development Program of Australian Universities and
Colleges (IDP), now known as IDP Education Australia, joined the British Council and
Cambridge ESOL to form an international partnership, reflected in the new name for the test:
the International English Language Testing System (IELTS). The recommendations of the
revision team to simplify and shorten ELTS were accepted and a compromise was sought
‘between practicality and maximum predictive power’. The number of subject-specific
modules was reduced from six to three and the Non-Academic test was replaced by the
General Module. IELTS first became operational in 1989 (Clapham and Alderson 1997).

The format of the 1989 IELTS
From 1989 IELTS candidates took two non-specialised modules, Listening and Speaking, and
two specialised modules, Reading and Writing. The non-specialised modules tested general
English while the specialised modules were intended to test skills in particular areas suited to
a candidate’s chosen course of study. Specialised reading and writing modules (incorporating
a direct link between the reading and writing activities) were available in three discipline
fields which linked together related fields that had previously been separate modules in the
ELTS battery, as shown below:
7
Module A – Physical Science and Technology
Module B – Life and Medical Sciences
Module C – Arts (later Business Studies) and Social Sciences

Module Module Module
Reading General
A B C
Module Module Module
Writing General
A B C
Listening Non-specialised module
Speaking Non-specialised module

Over the next five years the number of people taking the test rose by around 15% each year so
that by 1995 there were over 43,000 candidates in 210 test centres around the world.

The 1995 revision of IELTS
In keeping with the commitment of the IELTS partners to respond to developments in applied
linguistics, measurement theory and teaching practice, further modifications to the test were
implemented in April 1995. In addition to a number of modifications to improve security and
administration, there were three areas of significant change:

1. The field-specific Reading and Writing Modules A, B and C were replaced with ONE
Academic Reading Module and ONE Academic Writing Module. Details of the
research behind this change to the test design can be found in Clapham (1996). She
concluded that the different subject modules did not appear justified in terms of
accessibility to specialists.
2. The thematic link between the reading and writing activities was also removed to
avoid confusing the assessment of reading ability with that of writing ability.
3. General Training Reading and Writing Modules were brought into line with the
Academic Reading and Writing Modules in terms of timing allocation, length of
written responses and reporting of scores. The difference between the Academic and
General Training Modules is in terms of the content, context and purpose for
testing rather than the scales of ability.

In addition, measures were introduced to gather data on test performance and candidate
background so that issues of fairness relating to test use and users could be more effectively
8
monitored. A brief summary of the 1995 revision of IELTS can be found in Charge and
Taylor (1997).

Continuing development
In keeping with this history of innovation, the IELTS partners continue to be committed to the
ongoing development of the test. A revision project for the Speaking test was launched in
1998 and the revised IELTS Speaking test was introduced in July 2001. New assessment
criteria for the Writing test, the focus of this report, became operational in January 2005. The
computerised version of IELTS (CBIELTS) went online in May 2005 following successful
trialling and validation.
For a fuller discussion of the assessment of Academic English and its history over the past
half century, see Davies (forthcoming).

Introduction to the report
This report describes in detail the five phases of the 2001–2005 Writing Revision Project
which began with the formation of an IELTS Writing Assessment Revision Working Group,
first proposed at a meeting of the IELTS Joint Policy Group (the overseeing policy body for
IELTS) in June 2001 when approaches to writing assessment, more particularly methods of
analytic and holistic scoring, were discussed in depth.
The content and format of the IELTS Writing Module had been subject to considerable
revision as part of the 1995 IELTS Revision Project thus a full-scale review of the IELTS
Writing component was deemed unnecessary. However, in the 1995 revision process,
approaches to the assessment of writing had received far less attention than the test content
and format. It was decided that the approach to assessment should be investigated further.
Table 1.1 sets out the format and content of the IELTS Writing Module current until 2004.
9
Table 1.1 Format and content of the IELTS Writing Module
Academic Writing Module General Training Writing Module
Appropriate responses in the Academic The General Training Writing Module requires
Writing Module consist of short essays or candidates to write personal semi-formal or
general reports addressed to tutors or to an formal correspondence, or to write on a given
educated non-specialist audience. topic as part of a simulated class assignment.
There are two compulsory tasks. There are two compulsory tasks.
Task 1 requires at least 150 words. Task 1 requires at least 150 words.
Candidates are asked to look at a diagram or Candidates are asked to respond to a given
some data (graph, chart or table) and to problem with a letter.
present the information in their own words.
Task 2, the more heavily weighted task, Task 2, the more heavily weighted task,
requires at least 250 words. requires at least 250 words.

All candidates are presented with a point of All candidates are presented with a point of
view, argument or problem and asked to view, argument or problem and asked to
provide general factual information, outline provide general factual information, outline
and/or present a solution, justify an opinion, and/or present a solution, justify an opinion,
and evaluate ideas and evidence. and evaluate ideas and evidence.

Pre-2005 writing assessment approach
Each of the two IELTS Writing tasks was assessed independently and the assessment of Task
2 carried more weight in marking than Task 1. Detailed band descriptors were developed to
describe written performance at each of the nine IELTS bands. These existed in two formats:
as three ‘profile’ or analytical scales for each task: Task 1 – Task Fulfilment (TF), Coherence
and Cohesion (CC) and Vocabulary and Sentence Structure (VSS) and Task 2 – Arguments,
Ideas and Evidence (AIE), Communicative Quality (CQ) and Vocabulary and Sentence
Structure (VSS), and also as a global or holistic scale (i.e. the descriptors for each task are
conflated into a single set of band descriptors). Until July 2003, examiners were able to select
the global or profile approach according to whether a script had a ‘flat’ or ‘jagged’ profile.
The jagged profile system was introduced together with sample monitoring in 1995 in order to
maintain a check on the worldwide reliability of IELTS performance assessment. According
to Taylor and Falvey (2007): ‘The jagged profile system involves routine targeted double
marking of candidates identified as being at risk of misclassification, based on the level of
divergence between their Writing and/or Speaking scores and their Reading and Listening
scores. (It has been estimated that approximately 8% of candidates have their Writing or
Speaking performances re-marked because their profiles are identified as jagged.)’ Table 1.2
lists the task criteria that were the focus of the revision process.
10