The CLOZENTROPY test was also administered to 200 native speakers of Engl

74 Pages·2012·2.28 MB·English

RESUME
DOCUMENT
AL 001 598
ED 024 039
24
By-Darnell, Donald K.
Using a Clozentropy
The Development of an English Language Proficiency Test of Foreign Students,
Procedure. Final Report
Colorado Univ., Boulder. Dept. of Speech and Drama.
of Research.
Spons Agency-Office of Education (DREW), Washington, D.C. Bureau
Bureau No- BR- 7-H- 010
Pub Date Oct 68
Grant- OEG-8- 8- 070010-2000-057
Note- 73p.
EDRS Price MF-$0.50 HC-$3.75
Information
(Second Language), *Foreign Students,
Descriptors- *Cloze Procedure, College Students, *English
Validity
Test Interpretation, Test Reliability, Test
Theory. *Language Tests, Statistical Analysis, Testing,
TOEFL
Identifiers-*Clozentropy, Test Of English As A Foreign Language,
combining doze procedure and
This final report presents a description of a test
compatibity of a
(CLOZENTROPY), designed to measure the
an entropy analysis
who are native speakers of
English.
foreign student's English with that of his peers
administered
Foreign Language (TOEFL) were
This test, and the Test of English as a
CLOZENTROPY test was
University of Colorado. (The
to 48 foreign students at the
of English at
the same university.)
also administered to 200 native speakers
the two
approximately .86 were obtained for
Comparable reliability coefficients of
of
the two tests was .833. Analysis
tests. Correlation between total scores on
of subjects, and
and difficulty of test material, major
variance confirms that content
the
have significant influences on
level and major of native comparison groups
A discussion of the advantages over
CLOZENTROPY index of English proficiency.
computer
and the major weakness (dependency on
convention al types of tests
the
of the test instrument, sample letters to
assistance in scoring), a sample copy
data are
the scoring program, and other
students. samples of computer output on
included in the report. (AMM)
U.S. DEPARTMENT OF HEALTH, EDUCATION &
WELFARE
OFFICE OF EDUCATION
THIS DOCUMENT HAS BEEN REPRODUCED EXACTLY AS RECEIVED FROM THE
POINTS OF VIEW OR OPINIONS
PERSON OR ORGANIZATION ORIGINATING IT.
STATED DO NOT NECESSARILY REPRESENT OFFICIAL OFFICE OF EDUCATION
POSITION OR POLICY.
FINAL REPORT
Project No. 7-H-010
Grant No. OEG 8-8-070010-2000 (057)
PROFICIENCY TEST
THE DEVELOPMENT OF AN ENGLISH LANGUAGE
PROCEDURE
OF FOREIGN STUDENTS, USING A CLOZENTROPY
October 1968
U.S. DEPARTMENT OF
HEALTH, EDUCATION, AND WELFARE
Office of Education
Bureau of ResearCh
AL 001 598
r.
FINAL REPORT
Project No. 7-H-010
(057)
Grant No. OEG 8-8-070010-2000
PROFICIENCY TEST
THE DEVELOPMENT OF AN ENGLISH LANGUAGE
PROCEDURE
OF FOREIGN STUDENTS, USING A CLOZENTROPY
Donald K. Darnell
University of Colorado
80302
Boulder, Colorado
October 1968
to a grant with the
The research reported herein was performed pursuant
Health, Education, and Welfare.
Office of Education, U.S. Department of
under Government sponsorship are
Contractors undertaking such projects
professional judgment in the conduct
encouraged to express freely their
therefore,
Points of view or opinions stated do not,
of the project.
Education position or policy.
necessarily represent official Office of
U.S. DEPARTMENT OF
HEALTH, EDUCATION, AND WELFARE
Office of Education
Bureau of Researdh
TABLE OF CONTENTS
iv
Preface
1
Summary
2
Background
Chapter I
abnormality-compatibility.
A mathematical definition of
Pilot studies.
Cloze procedure.
12
Sampling and Testing
Chapter II
Foreign
The criterion groups.
The test passages.
Test administration.
student Ss.
17
Results
Chapter III
Effects of six independent
Validity.
Reliability.
ANOVA of native
variables on CLOZENTROPY scores.
Summary.
ANOVA of foreign data.
data.
28
Summary and Conclusions
Chapter IV
34
References
Appendices
36
A sample copy of the test instrument
A.
46
Sample letters to native students
B.
51
Sample letters to foreign students
C.
55
Samples of c,.;mputer output on scoring program
D.
62
Significantly different cell means--native data
E.
64
Significantly different cell means--foreign data
F.
LIST OF TABLES
18
Correlations among CLOZENTROPY, TOEFL, and GPA
Table 1.
20
Analysis of variance of native student data
Table 2.
22
Analysis of variance of foreign student data
Table 3.
ii
LIST OF FIGURES
20
Level by content interaction (native)
Fig. 1
21
Major by content interaction (native)
Fig. 2
21
Major by level by content interaction (native)
Fig. 3
21
Content by difficulty by level interaction (native)
Fig.
.
.
.
23
Content by major interaction (foreign)
Fig. 5
23
Content by difficulty interaction (foreign)
Fig. 6
23
Content by rank of criterion group interaction
Fig. 7
.24
Major by rank of criterion group interaction
Fig. 8
24
Content by major of criterion group interaction
Fig. 9
24
Difficulty by major of criterion group interaction
Fig. 10
24
Content by major by rank of criterion group interaction
Fig. 11
.
Difficulty by content by rank of criterion group
Fig. 12
25
interaction
ty
Difficulty by content by major of criterion group
Fig. 13
25
interaction
Content by rank of criterion group by major of criterion
Fig. 14
25
group interaction
Difficulty by rank of criterion group by major of
Fig. 15
26
criterion group interaction
Major by content by difficulty by rank of criterion
Fig. 16
26
group interaction
Content by difficulty by rank of criterion group by
Fig. 17
26
major of criterion group interaction
PREFACE
This research report may be taken from two different points of view.
Obviously, this is a validation study of a test instrument, but it may
also be viewed as a validation study of a procedure for constructing test
It makes cer-
The procedure is an innovation in testing.
instruments.
tain assumptions about language and its function and establishes an
objective criterion related to those assumptions for the evaluation of
individual linguistic performance.
The specific test created in this study is only an example of the
Although a copy
kind of test that the CLOZENTROPY procedure produces.
of the test instrument appears in Appendix A, a scoring key does not
(1) The simplest
appear in this report for three very good reasons:
hand scoring key imaginable would require the addition of approximately
A complete scoring key would require at
150 pages to this report.
(2) Complete or incomplete, such a scoring
least 300 additional pages.
key would have no practical value, because neither man nor machine could
(3) The use of such a key by anyone not at the
use it effectively.
University of Colorado would be inconsistent with the theory underlying
CLOZENTROPY procedure.
All the essential information which fhe individual user would need
to develop his own CLOZENTROPY test and his awn scoring key is provided
Hopefully, a computer program which would
in detail in this report.
.
facilitate such developments will soon be published.
One could never acknowledge all the people who have contributed to
be singled out
a project of this kind, but there are six people who must
fvr their contributions to this study.
First, my thanks go to Dr. David R. Saunders of the University of
Colorado Department of Psychology for writing a computer program that
His encouragement and concrete assistance
made this project feasible.
made possible the pilot research which led to the proposal for this
His continued assistance in modifying the program for this spe-
study.
cific application went far beyond what one can expect of a colleague.
my appreciation also goes to Mr. Glenn Bracht and Dr. Kenneth Hopkins
Their
of the University of Colorado Laboratory of Educational Research.
advice on research design and assistance with the analysis contributed
immeasurably to the success of this study.
The other three people whom I wish to thank are my Chief assistant
an the project, Stephen Clarke, and two other graduate students, Roger
These three put
Babich and John Boyd, who volunteered their services.
Their work
in many hours of hard work and asked innumerable questions.
made the project possible, and their questions kept it interesting.
Of course, I take full responsibility for any errors that may have
been made in the conduct of this research or in the writing of this
report.
iv
SUMMARY
The Test of English as a Foreign Language (TOEFL) and a CLOZENTROPY
Colorado foreign
test were administered to a sample of 48 University of
The CLOZENTROPY test was also administered to 200 native
students.
speakers of English, students at the University of Colorado.
CLOZENTROPY procedure is a combination of cloze procedure and an
Its product is a
entropy analysis derived from information theory.
student's language patterns
measure of the compatibility of a foreign
with the usage patterns among his native English speaking peers.
Comparable reliability coefficients of approximately .86 were ob-
Subtests in the two
tained for the total scores on the two tests.
batteries were treated as items, and reliability coefficients were not
obtained on the individual subtests.
3 tests indicates that the total scores
Correlation analysis of the 1.
The communality accounts for
on the two batteries are correlated
.833.
This is inter-
almost all of the reliable variance in both of the tests.
preted as positive support for the validity of fhe CLOZENTROPY test.
Neither test was found to have any significant correlation with grade
point average for the 48 foreign students tested.
ANOVA treatment of the data from 200 native students indicated signi-
ficant differences between graduate and undergraduate students and
The level and major variables were
between engineers and non-engineers.
such
found to interact with content and difficulty of the test material,
that the main effects are largely attributable to difficult material and
engineering content.
ANOVA treatment of the data from 40 foreign students indicates no
significant difference between graduate and undergraduate foreign students
and no simple effects of the difference between engineering and non-
There were significant main effects of the message
engineering students.
(level and major
content variable and the two criterion group variables
Interaction effects involve five of the six variables
of native students).
studied.
CLOZENTROPY procedure yields a reliable and valid test of English,
traditional
language proficiency that has a nuMber of advantages over more
Its one severe limitation is the need for computer
test procedures.
assistance in scoring.
CLOZENTROPY procedure seems to be capable of producing highly spe-
cialized language tests for a variety of aChievement levels and fields
Further research along these lines is recommended.
of interest.
-1-
PROFICIENCY TEST
THE DEVELOPMENT OF AN ENGLISH LANGUAGE
CLOZENTROPY PROCEDURE
OF FOREIGN STUDENTS, USING A
Chapter I
Background
the reliability, valid-
This is a report of a study conducted to test
of English language proficiency.
ity, and practicality of a new test
Cloze Procedure (used previously to
The new test employs a variation of
comprehension, and language aptitude) and an entropy
measure readability,
theory which indexes the compatibility
measure derived from information
of a selected criterion group.
of an individual's responses with those
combination of procedures.
The term, CLOZENTROPY, was coined to name this
refer to the
"CLOZENTROPY test" or "CLOZENTROPY battery" will be used to
"CLOZENTROPY procedure" will
particular test developed in this study, and
which might be used to generate
be used to refer to the general procedure
any number of specific tests.
based on the follow-
The rationale or justification for this study is
(1) The primary function of language is communication.
ing assumptions:
by compliance with group
(2) This function is best served within any group
(3) A measure of proficiency in language should
norms of language usage.
of language rather
index the ability to conform to existing group norms
(4) If
language pattern.
than to some prescriptive model or idealized
of proficiency
language norms vary from group to group, the best measure
the group or groups with whom
for an individual would be given in terms of
(5) An ideal measure of language proficiency
he needs to communicate.
ability to exercise freedom of
would take into account an individual's
relatively rigid restrictions.
choice as well as his ability to comply with
allow for new statements within
That is, any natural language, in order to
that language some freedom
the language structure, must permit users of
involves exercising
Proficiency in the use of language, then,
of choice.
boundaries on freedom and is some-
that freedom as well as knowing the
(6) There should
of rules.
thing more than mere imitation or recitation
the security problems of
be alternate forms of a good test to minimize
organizations
suit the specific needs of different
a testing program and to
that might wish to use the test.
of the traditional
Implicit in these assumptions are some criticisms
The better traditional tests
methods of testing language proficiency.
Language Associa-
(including TOEFL, the most recent product of the Modern
multiple choice
tion) are, for the most part, composed of completion or
Although a carefully constructed
items which are scored right or wrong.
reliable and presumably more valid
test of this type is generally more
scoring procedure inevitably
than an essay evaluation, the right-wrong
decision with the case of a slight
equates the grammatically determined
Imposing an ad hoc, subjective, weighting system
stylistic preference.
does not seem to be a satisfactory
to compensate for the initial error
Two sample items from the TOEFL Handbook for
solution for the problem.
Candidates (ETS, 1963) will illustrate the point:
"This ballpoint pen won't write."
1.
it?"
"What's the matter
Answer (b)
(a) for (b) with (c) of (d) by
the subject
"Because he had little education, his knowledge of
2.
was
.
(c) minor
(a) limited (b) small in quantity
Answer (a)
(d) not large at all
important aspect of language,
Although both of these items may index some
in the same way,
the "wrong" answers to item #2 are Obviously not wrong
those in item #1.
nor in the same degree, as
would seem to be
Secondly, the goal of a nationally standardized exam
and specialist group dif-
in conflict with what is known about regional
In order to make national
1959).
ferences in language usage (Malstom,
in the direction
standardization somewhat meaningful, testers are pushed
tests) even thcagh they may be
of formal written English (even in oral
will employ informal oral
well aware that much of the S's communication
English or a dialect.
comparable multiple
Thirdly, the difficulty and cost of constructing
prohibitive for all but large
forms of the traditional type of test is
Consequently, a small local
organizations that specialize in testIng.
subscribe to the
organization which has need of a proficiency test must
their awn test, for which
services of a testing organization or make up
Further,
personnel.
they have neither the time nor the necessary skilled
problems associated with the
the time problems created by the security
That is, a
standardized exam prohibit the use of the best instruments.
involving days or weeks of delay
test which must be sent away for scoring,
vflue to a
between testing and the reporting of scores, is of little
decisions in a matter of hours.
person who must make placement
of language pro-
There can be little doubt about the need for tests
placement, the
The increasing problems of college admission and
ficiency.
languages of all kinds aad,
increasing demand for training in foreign
numbers of foreign students coming to our
most pressing, the increasing
which demand reliable, valid,
colleges and universities are all problems
of their solution.
and practical tests of language proficiency as part
According to an Extract from altifIlliatIrallsILTIsEclency_g_Foreign
1961), a conference
Students (distributed by Educational Testing Service,
This conference
Washington, D.C. in 1961.
on this subject was held in
Language Association of America, the
was jointly sponsored by the Modern
National Association of
Institute of International Education, and the
The following quotation is taken from the pro-
Foreign Student Advisors.
ceedings of that conference:
the desirability
The conference goes on record as recognizing
using care-
of, and urgent need for a comprehensive program
of foreign
fully constructed tests of the English proficiency
insti-
students, suitable and acceptable to all educational
other organizations,
tutions in the United States and to various
chiefly governmental.
proficiency battery to
The conference recommended the development of a
structure, (2) auditory comprehension,
measure (1) control of English
(5) writing ability.
(3) vocabulary. (4) reading comprehension, and
-3-
Such a battery was developed, the TOEFL referred to earlier.
In addi-
tion, the conference recommended further research in language aptitude
testing and measures of oral production skills.
The Test of English as a Foreign Language (TOEFL) has not been
universally accepted.
According to a service memorandum (ETS, 1965),
2,979 foreign students were tested between February of 1964 and May of
A more recent communication from ETS indicates that 34,774
1965.
foreign applicants to U.S. colleges were tested between February of 1964
and April of 1967.
Certainly a number of U.S. colleges that have a
significant foreign student enrollment have not adopted TOEFL as
an
admission requirement.
Perhaps this is due, at least in part, to some
of the criticism outlined above.
Nevertheless, TOEFL is undoubtedly the most authoritative and the
most complete test of English proficiency available today.
For that
reason, it was adopted as the primary criterion measure for assessing
the validity of the CLOZENTROPY test developed in this study.
The TOEFL
and CLOZENTROPY tests were administered to a group of foreign students
at the University of Colorado.
Comparable reliability coefficients and
a matrix of intercorrelations among the components of the two test bat-
teries are given in the results section of this report.
A Mathematical Definition of Abnormalit -Com atibilit
A key element in the deveiopment of a new kind of language proficiency
test was the development of procedures which permit an alternative to the
right-wrong scoring system and to the subjective evaluation of
These
essays.
procedures were derived from information theory.
Shannon and Weaver, in The Mathematical Theory of Communication (1949),
provide us with a numerical expression for the average absolute entropy (H)
of a source or code.
With n independer.- symbols, each with a probability
of occurrence
the average amount of information (entropy) that cah be
transmitted with that symbol set is
-(pi log2 pi + p2 log2 p2 + p3 log2
+ pn log2 pn) or H =
p3 + .
. .
-E pi log2
With two equally probable choices, H has a value of 1.
With four equally
probable choices, this expression has a value of 2.
With four choices
and the probabilities .37,
.25, .21, and .17, the approximate value of
H is 1.9381.
The point, as far as this paper is concerned, is that
one
can descrfbe a set of discrete, independent elements with a figure that
expresses both the number of elements in the set and the probabilities
of all the elements.
The value (H), for any set of equally probable
elements, increases as the number of elements increases and has
a maximum
value for any finite number of elements when they
are equally probable.
A second component can be derived from the preceding discussion, but it
is highlighted by Gleason (1961, p. 377).
"The amount of information in
any signal is the logarithm to the base two of the reciprocal of the prob-
ability of that signal.
I = Log2 1/p."
That is:
That value which was