ERIC ED345781: Implementation and Initial Validation of the APS English Test [and] The APS English-Writing Test at Golden West College: Evidence for Predictive Validity.

39 Pages·1991·1.2 MB·English

by  ERIC

DOCUMENT RESUME
ED 345 781
JC 920 261
AUTHOR
Isonio, Steven
TITLE
Implementation and Initial Validation of the APS
and
English Test
The APS English-Writing Test at
Golden West College: Evidence for Predictive
Validity.
INSTITUTION
Golden West Coll., Huntington Beach, Calif.
PUB DATE
91
NCTE
39p.
PUB TYPE
Reports - Evaluative/Feasibility (142)
-- Statistical
Data (110)
RDRS PRICE
XF01/PCO2 Plus Postage.
DESCRIPTORS
*College English; Community Colleges; Correlation;
Grades (Scholastic); Predictive Neasurement;
Predictive Validity; Predictor Variables; *Reading
Tests; 'Student Evaluation; 'Student Placement;
Tables (Data); Test Results; Test Use; *Test
Validity; Two Year Colleges; Writing (Composition);
'Writing Tests
IDENTIFIERS
Assessment and Placement Services for Comm Coll
ABSTRACT
In May 1991, Golden West College (California)
conducted a validation study of the English
portion of the Assessment
and Placement Services for Community Colleges (APS),
followed by a
predictive validity study in July 1991. The initial
study was
designed to aid in the implementation of the
new test at GWC by
comparing data on APS use at other community colleges
and by testing
placement rules. The follow-up study examined
the relationship
between the APS and eventual course success
as measured by students'
grades. The initial sample for both studies
consisted of 381 students
from 23 classes offered in spring 1991 who had
taken the reading and
writing portions of the APS. Three standard
criterton groups were
identified, consisting of native English
speakers placed in English
009, 010, or 100. Selected findings Included
the iollowing: (1) the
APS appeared able to differentiate
among course levels and was
significantly related to standard
group membership, making a clear
placement recommendation possible for the
majority of students; (2)
ratings of writing samples could aid in
definitive placement of those
students earning unclear APS scores; (3)
the gain in predictability
from using a combination of reading and
writing score information
over using writing alone was insignificaLt; (4) the
analysis of the
relationship of grades to APS sc6res revealed
a low to moderate-sized
predictive validity coefficient, a value
within the expected range;
and (5) a significant difference in APS
scores was found between
successful and unsuccessful students. Data
tables, references and
short appendixes, providing a sample student
survey and a correction
for restricted range of a predictive validity
coefficient, are
included.
(JSP)
*******************0**************************0************************
Reproductions suppliee by EDRS are the Dest
that can be made
from the original document.
***************************************************************ft*******
1
Implementation and Initial Validation of the APS English Test
and
The APS English-Writing Test at Golden West College:
Evidence for Predictive Validity
GOLDEN WEST COLLEGE
1991
S DEPARTMENT CIF EDUCATION
U
ottre of Eaucahonat Research srid omprowemeni
"PERMISSIONTOREPRODUCETHIS
EDUCATIONAL RESOURCES ISWORMATION
MATERIAL HAS BEEN GRANTED BY
CENTER (ERIC{
Min aecumaM has been reproduced an
S. Isonio
the pirson
or organization
received from
hare Ivan rnatie to
improve
Minor changes
optocfacfron QUSIIIy
(loco
Points i.vf view or opimonii %fated in tn
mant do not necessarriv represent ofirrize
TO THE EDUCATIONAL RESOURCES
OERI positron or policy
INFORMATION CENTER (ERIC).-
Steven Isonio, PhD
Advisor, Matriculation Research/Assessment
Cqrv pc!Tn
BTST
L
2
i
I
a
Implementation and Initial Validation of the
APS English Test
GOLDEN WEST COLLEGE
May 1991
Steven bonito, PhD
Advisor, Matriculation Research/Assessment
Validation of the
Implementation and Initial
APS English Test
background,
is usLd to present the
A question and answer format
assessment validation study
procedures and results of an
the study was
The primary intention of
conducted in Spring 2991.
West
for the new test at Golden
to establish placement rules
followup research is
An overview of necessary
College.
presented.
implementation of
iu the adoption and
What steps are involved
I.
into
writing skills for placing students
a new measure of
College?
English courses at Goldet West
recognizing the need for a better
The first steps, namely,
consideration, discussing its
test, identifying a test for
A
adopting it, are complete.
merits, and finally formally
portion of the Assessment and
proposal to adopt the English
Colleges (APS), developed and
Placement Services for Community
Service (ETS) was made in
distributed by the Educational Testing
recommended by a
The test was reviewed and
December 1990.
In February 1991,
department.
subcommittee of the GWC English
faculty voted 14-0 (with one
the English Department full-time
abstention) to adopt the new test.
would
designed to gather information that
A pilot study was
placement rules as well as begin
help with the establishment of
and
the relationship between the APS
the process of documenting
research is
Such assessment validation
relevant criteria.
likely,
Adjustments in placement rules are
necessarily ongoing.
studies are required.
and an asssortment of validation
Stanford TASK for placing
Is the APS a better test than the
2.
students into courses at Golden West?
validity, there is a consensus
While both tests have face
the
of higher level skills than
that the APS taps a broader range
Academic Skills), thereby making it more
TASK (Stanford Test of
Both
into writing courses.
appropriate for placing students
the English department
instruments were thoroughly reviewed by
This review resulted in a strong
assessment subcommittee.
TASK as the objective
recommendation that the APS replace the
The TASK was
English placement test at Golden West.
discrete items"
characterized as "simplistic, concentrating on
Conversely, the APS was
ability".
and evaluating "a low level of
it "features a higher level of
deemed more appropriate in that
global scope."
writing and language skills and has a more
developed specifjcally
Further, unlike the TASK, the APS was
skills'
college students into basic
for placing of community
widely used
Additionally, the APS is now the most
courses.
into English courses in
instrument for placing students
California Community Colleges.
1
What guidelines for use of the APS
3.
are offered by the test
publisher?
ETS provides extensive information about the
construction
and norming of the test in tha technical manual
(College Entrance
Examination Board, 1990).
References to pertinent research are
also provided.
Additionally, the manual contains
a discussion of
selected psychometric properties of
the test.
However, this
instrument, like any placement test,
does not come with
universally applicable placemert
rules.
Local placement rules
must be developed.
What type of placement rules
4.
are used at other schools that
have adopted the APS?
Information about how the instrument
is used at a sample of
other colleges was gathered.
This information suggested models
that would be tested as a part of the pilot
Table 1
study.
presents placement rules implemented
at a sample of four colleges
that use the APS for placing students
into three levels of
English.
To facilitate comparison,
course numbers for the
transferable composition course have
been changed to "100";
similarly, the two
courses below this level are referred to
as
"10" and "9", respectively, although
these may not be the actual
course numbers at the other colleges.
Two schools, American River College
and Rancho Santiago
College use only the objective writing
section of the APS to
place students W.rectly into the
three courses.
Moorpark College
and Santa Barbara City College
incorporate information from
an
essay to clarify placements in certain marginal
ranges.
Specifically, in the
case of Moorpark College, students who
scored in the upper portion of the
range for placement into the
English 10 course can challenge this
placement by having their
essay evaluated.
The student's placement is either
changed to
the English 100 course,
or remains English 10.
Santa Barbara
City College routinely evaluates
essays written by students in
either of three ranges:
those students in a band between English
10 and English 100, those students in
a band between English 9
and English 10, and those
students scoring in the very low
ranges
of the distribution.
There appears to be much consistency
in placement rules used
by these schools.
This information was useful in helping
to
generate reasonable placement rules
for consideration in the
study.
2
1
Table
Rules
Three-tiered Placement
Colleges
at Sample
With APS Writing Test
RANCHO
Score
SANTIAGO
SANTA BARBARA
AM.RIV.
MOORPARX
100
100
100
100
40
100
100
100
100
39
100
100
100
100
38
100
100
100
100
37
100
100
100
100
36
100
100
100
100
36
100
100
100
100
35
100
100
100
100
33
100
100
100
100
32
100
10 OR 100
100
100
31
100
10 OR 100
100
100
30
10
10 OR 100
100
100
29
10
10
100
100
28
10
10
10
10+
27
10
10
10
10+
26
10
10
10
10+
25
10
10
10
10+
24
9
10
10
10
23
10 OR 9
9
10
10
22
9
10 OR 9
10
10
21
9
10 OR 9
9
10
20
10 OR 9
9
9
10
19
9
9
9
10
28
9
9
10
9
17
9
9
9
9
16
9
9
9
9
15
9
9 OR LOWER
9
9
14
9
9 OR LOWER
9
9
13
9
9 OR LOWER
9
9
12
9
9 OR LOWER
9
9
11
9
9 OR LOWER
9
9
10
9
9 OR LOWER
9
9
9
9
9 OR LOWER
9
9
8
9
9 OR LOWER
9
9
7
9
9 OR LOWER
9
9
6
9
9 OR LOWER
9
9
5
9
9 OR LOWER
9
9
4
9
9 OR LOWER
9
9
3
9
9 OR LOWER
9
9
2
9
9 OR LOWER
9
9
3,
indicated, definitive
two placements are
In cases where
rote.
"10+11
of the essay.
an evaluation
placement is made based upon
students
level course, but
the English 10
indicates placement into
placement into
read for possible
that their essay be
may request
the English 100 course.
Describe the design and procedures of the pilot study. what
5.
was expected?
In the Spring 1991 semester, a representative sample of 23
A total of 381
English 9, 10, and 100 classes was identifed.
students from these classes were administered the reading and
Additionally,
writing sections of the APS (see Table 2).
information about the students' placement based upon the TASK
test (previously used objective English placement test), a
wrfting sample produced the first day of instruction, and
instructors' recommendations was gathered.
These
Three "standard" criterion groups were identified.
groups consisted of native English speakers who were placed into
English 9, 10, or 100 and for whom these indicators were in
Certainly this composite criterion is not perfect,
agreement.
but its reliability was enhanced by the convergence of these
Finally, only native speakers of English
various indicators.
were included in the analysis because the APS was designed
Primary language
specifically for placement of such students.
was determined from a supplemental survey administered after the
test (see Appendix A).
Distributions of scores were obtained for the reading and
The relationship
writing sections of APS, and their combination.
between APS scores and course level was determined with the
To the
multiserial correlation coefficient (Jaspen, 1946).
extent that a relationship exists between the APs scores and
standard group membership, the APS can be said to have criterion-
Further, the central points of the three
related validity.
distributions should be at different locations along the
abscissa, ideally with a only a moderate degree of overlap.
Finally, a variety of placement rules were evaluated.
The
ability of these various rules to differentiate among the three
groups (maximizing correct "placements" while minimizing
The score ranges for the most
incorrect ones) was assessed.
effective model would constitute the recommended initial cut
scores.
Is the APS capable of differentiating the three levels of
6.
English courses?
The process of implementing a test and establishing
placement rules entails comparing it to a standard.
The
difficulty arises in that there is no ideal criterion to serve as
Therefore, as noted above, existing information
the standard.
(placement based on the previous English placement test (TASK),
ratings of the first day placement writing samples (PWS), and
students'. self-evaluation implicit in their course selection were
used to de'fine three standard comparison groups.
At a minimum, APS mean scores for the three standard grpups
The standard groups
should be clearly different from each other.
do indeed differ significantly in terms of mean hPS writing score
Figure la depicts these group
[f (2,187) = 79.09, p < .0013.
The reading and writing scores were combined
mean differences.
4
7
Table 2
Course Level
Number of Students Tested by English
Students in
Students
Number of
Course
sis*
a
the
es
Cla
es
ve
44
54
English 9
4
60
156
English 10
8
87
171
English 100
11
191
381
Total
23
placed into their
*Only Native Speakers of English who were
Placement Writing
enrolled class with the TASK-EPT and the
These students
Sample were included in the analysis.
comprise the standard criterion groups.
5
Figure la
Mean APS Writing Score
By Level
30
25
-
English 100
English 010
English 009
Standard Group Level
API Pam, Ipilpg URI
APS Writing Scores
Descriptive Statistics
Standard
Level
- 27
5.13
7
17.82
009
English
- 33
12
4.87
22.92
010
English
- 38
15
4.75
28.87
100
English
(Schmidt, 1971). The
values for each student
by summing the two
the three
significantly across
also differed
mean combined score
These means are
94.50, R < .001).
groups LE (2,185) =
writing
the mean APS
Thus, for both
presented in Figure lb.
reading score, there are
combination with the
score alone and in
differences amonc the three groups.
very striking
such as those just
Differences among the group means
points of the
only that the central
described, however, indicate
to
Therefore, it was necessary
differ.
three distributions
determine the extent
of scores to
examine the full distributions
Figures 2a, 2b, and 2c
distributions overlap.
to which the
010,
distributions for English 009,
depict the APS writing score
to reflect
The ordinate was scaled
and 100, respectively.
across the
facilitate direct comparisons
relative frequency to
among
The striking differences
samples.
three different-sized
reflected by the
writing scores noted above, are
group mean
sample,
For the English 009
distributions.
differencen among the
were 12 to 33
to 27, whereas the ranges
the scores ranged from 7
.21lko 38 for English 100.
for English 10, and
semi-interquartile
also reflected in the
The differences are
These ranges are 15
middle 50% of the distribution.
ranges, the
and 27 to 32 for
to 27 for English 10,
to 21 for English 9, 20
semi-interquartile range is
The advantage of the
English 100.
excluding
"core" of the distribution,
that it reflects the
low motivation
be attributable to very
extreme scores which may
particularly stable.
Therefore, these values are
or luck.
of
depict the distributions
Similarly, Figures 3a, 3b, 3c
standard
for students in the
combined reading and writing scores,
The central points
respectively.
English 9, 10 and 100 groups?
different places on
clearly located at
of the distributions are
to be slightly more
However, there appears
axis.
the APS score
The
levels.
distributions for adjacent course
overlap between
For English 10
English 9 is 25-38.
semi-interguartile range for
48-61.
English 100 the range is
this value is 33-50; for
somewhat
writing section alone appears
In short, the APS
The
English.
the three levels of
able to differentiate among
slightly less
reading scores seems
combination of writing and
able to do so.
in conjunction with
be best utilized
Row can a writing sample
7.
maximizing accuracy of
information (that is,
APS test score
efficiency)?
placement while maintaining
the sample APS
is no consensus among
As notad earlier, there
Some schools do
writing samples.
schools regarding the use of
to supplement
Other schools use them
not use writing samples.
placement
typically to make definitive
the APS scores,
in marginal ranges.
recommendations for students
course
to differentiate among
Since the APS appears able
membership,
related to standard group
levels and is significantly
for the majority of
recommendation is possible
a clear placement
contribute to a
Ratings of writing samples can
students.
that
students who earn APS scores
definitive placement for those
7
AVAILABLE
dEST COPY