ERIC ED462447: Computer-Based Assessment: Can It Deliver on Its Promise? Knowledge Brief.

11 Pages·2001·0.25 MB·English

by  ERIC

DOCUMENT RESUME
TM 033 712
ED 462 447
Rabinowitz, Stanley; Brandt, Tamara
AUTHOR
Computer-Based Assessment: Can It Deliver on Its Promise?
TITLE
Knowledge Brief.
WestEd, San Francisco, CA.
INSTITUTION
Office of Educational Research and Improvement (ED),
SPONS AGENCY
Washington, DC.
2001-00-00
PUB DATE
NOTE
10p.
ED-01-00-0012
CONTRACT
WestEd, 730 Harrison Street, San Francisco, CA 94107-1242.
AVAILABLE FROM
Tel: [phone] (Toll Free); Web site: http://wested.org.
Information Analyses (070)
PUB TYPE
EDRS PRICE MF01/PC01 Plus Postage.
Adaptive Testing; *Computer Assisted Testing; Educational
DESCRIPTORS
Technology; *Educational Testing; Elementary Secondary
Education; *Testing Problems
Access to Computers
IDENTIFIERS
ABSTRACT
Computer-based assessment appears to offer the promise of
radically improving both how assessments are implemented and the quality of
the information they can deliver. However, as many states consider whether to
embrace this new technology, serious concerns remain about the fairness of
the new systems and the readiness of states (and districts and schools) to
support them. This brief describes the potential advantages of a fully
implemented computer-based assessment system and then lays out questions
states must address as they consider the next generation of high-stakes
assessments. Computer-based assessments promise to make obsolete many of the
shortcomings of current high-stakes, statewide assessment systems and to
expand the capacity of such systems to measure rigorous standards in truly
innovative ways. They offer answers to the usual concerns about current
assessments with regard to logistics, content/methodologies, and value.
Standing in the way of computer-based assessments are other logistical
problems, such as the need for back-up machines and materials, delivery of
materials and training of test administrators, and scheduling. Security is a
problem in computer-based testing as it is in conventional testing. Test
equivalence and computer equivalence must be considered, and the largest
hurdle to realizing the promise of computer-based assessment is access.
Moving ahead without addressing the issues raised in this brief will almost
certainly result in flawed assessments.
(SLD)
Reproductions supplied by EDRS are the best that can be made
from the original document.
Computer-Based Assessment: Can It
Deliver on Its Promise?
Knowledge Brief.
Stanley Rabinowitz
Tamara Brandt
West Ed
U.S. DEPARTMENT OF EDUCATION PERMISSION TO REPRODUCE AND
DISSEMINATE THIS MATERIAL HAS BEEN
Office of Educational Research and Improvement
GRANTED BY
EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
This document has been reproduced as
T. Ross
received from the person or organization
originating it.
0
Minor changes have been made to improve
reproduction quality.
TO THE EDUCATIONAL RESOURCES
INFORMATION CENTER (ERIC)
Points of view or opinions stated in this
document do not necessarily represent
official OERI position or policy.
1
BESTCOPYAVALABLE
2
brief
knowledge
4=,
°U7
promise?
E 7
deliver
on its
Can it
It's April, and a class fiill of students is about to tackle the rigorous, high-stakes statewide assessment. Last year they had
all sharpened their number 2 pencils, sat down at their desks, opened thick test booklets, and begun filling in the blanks
lab, taking their exam on computers
on paper answer forms. This year, these students are sitting in the school's computer
connected to the Internet. Two of them, a visually impaired boy and a girl who has severe dyslexia, wear headphones,
working at computers that will read the test aloud. Other students in the school are scheduled to take the test at other
times in the coming weeks. Instead of answering page after page of multiple-choice questions, all these students will write
essays, graph math solutions, manipulate
simulations, and use various computer-based
tools to solve real-world problems.
The computers immediately score students'
responses, including their answers to essay
questions. Based on how a student answers
particular questions, the computer branches to
harder or easier questions, an adaptive process
that yields more detailed information about
what a student does or doesn't know. Because
the tests are adaptive, students can also answer
fewer questions than needed on a traditional
paper assessment, yet the test yields more specific
results. When each student is done, his or her
test scores are automatically emailed to the
teacher and principal. Exam data, sent via the
Internet, are stored in the main computer at the
state department of education, readily available
to schools as needed.
Written by
Stanley Rabinowitz and Tamara Brandt
WestEd
Improving education through
research, development, and service
Illustration by James Yang
Computer-Based Assessment
To some, this vision of a more efficient and
states must address as they consider the next
informative assessment process sounds like a
generation of high-stakes assessment.
fantasy. But many states don't think so, and they are
actively exploring the conversion of their statewide
Technology is no stranger to assessment. In the
assessment systems from pencil-and-paper exams to
middle of the last century, the rise of multiple-
computer-based assessments. In turn, test publishers
choice methodology for large-scale assessment was
are working feverishly to stake their
fueled heavily by the development
claims on that territory. Bold
of high-speed scanners. More
Advances in the speed,
statements abound about the
recently, computer-adaptive
promise of computer-based testing.
models, such as that described in
capacity, and availability
the opening vignette, in which
of computers allow
Can technology-assisted assessment
students are presented with
live up to these promises?
questions tailored to their ability
applications that could
levels, have promised to make
Throughout the relatively brief
assessment more efficient and able
only be imagined less
history of high-stakes assessment,
to target the needs of individual
than a generation ago.
many innovations have come along
students. But past advances pale
that promised to revolutionize the
compared to those of the last
assessment experience. Some
decade, which has seen a rapid
delivered on the promise. For example, innovations
increase in both the use and potential of technology
in the scoring of essays (e.g., scoring rubrics, large
to support assessment. On the hardware side,
scoring centers) have truly changed the face of
advances in the speed, capacity, and availability of
assessment, leading to the large majority of
computers allow applications that could only be
programs (local, state, National Assessment of
imagined less than a generation ago. On the
Educational Progress) incorporating direct writing
software side, developments in database structures,
or constructed-response questions. Other
simulation technologies, and artificial intelligence
innovations, such as performance-based methods
models promise to dramatically improve the
(e.g., portfolios, projects), have proven harder to
efficiency and capabilities of assessment
sustain and have been relegated to low-stakes
administration, scoring, and reporting.
smaller programs or local assessments.
College admissions and certification programs have
Now, with computer-based assessment, comes the
led the way in using the new computer-based
possibility of radically improving both how
technology. The success of these pioneers has caused
assessments are implemented and the quality of the
businesses ranging from the major commercial
information they can deliver. However, as many
testing companies down to one-product start-ups to
states consider whether to embrace the new
spend millions on assessment-related research and
technologies
and as some already have
serious
development. Transferring the emerging
concerns remain about the fairness of the new
technologies fully into the K-12 arena seems an
systems and the readiness of states (and their districts
obvious next step.
and schools) to support them. This Knowledge Brief
first describes the potential advantages of a fully
implemented computer-based assessment system. It
ends more cautiously, laying out a series of questions
4
2
Wes t Ed
information would be needed for schools to develop
Pushing beyond current testing
individual student assistance plans. Equally
limitations: What the future could
desirable would be for schools to obtain results in a
look like
timely manner following administration of the
assessment. Yet states that test in the spring and
whose systems include direct writing or open-
Computer-based assessment promises to:
response items rarely receive the results before the
make obsolete many of the shortcomings of
end of the school year and often don't receive them
current high-stakes, statewide assessment
until the following fall. Equally daunting is the cost
systems; and
and challenge of printing, delivering, and tracking
expand the capacity of such systems to measure
the amount of paper used in high-stakes, statewide
rigorous standards in truly innovative ways.
assessment systems.
State assessments are an essential tool in the student
Computer-based assessment provides answers to all
and school accountability model so prevalent in
of these problems. Using adaptive software, more
detailed content can be tested in a shorter period of
today's reform efforts. In large part because of the
these new
high stakes attached to their results,
time. Results can be instantaneous, even for
programs that require student writing.
assessments have been
New breakthroughs in artificial
subjected to intense scrutiny,
Computer-supported
and a number of problems
intelligence and other models allow
have been identified, most
computer scoring of essays in a
scoring models have
having to do with the limits of
fraction of the time currently required.
Computer-supported scoring models
the methodology. Proponents
met or exceeded
believe that new assessment
have met or exceeded the accuracy of
the accuracy of
technologies can break through
human raters across a range of content
areas. Computer-administered tests
these boundaries, solving many
human raters across a
require no printing and shipping of
of the problems associated
range of content areas.
test booklets, a major expense for
with traditional large-scale
testing programs. When students
examinations.
respond online, no answer forms have
Broadly speaking, criticism of current large-scale
to be shipped either, which cuts costs immediately.
In addition, local staff time is no longer needed to
assessment systems falls into three categories:
logistics, content/methodologies, and value.
process vast amounts of paper.
Content/Methodologies. The rigorous, "world-
Logistics. Because they usually result from a series
of comprämises driven by limited time, resources,
class" standards that have been adopted by many
and methodologies, statewide assessment systems
states stress students' ability to "know and do."
are far from ideal. For example, while schools would
However, with their heavy reliance on multiple-
greatly benefit from detailed diagnostic information
choice items (with some limited constructed-
on each student, the amount of testing time and
response items), many state tests emphasize more of
associated cost required to accomplish this on a
the knowing than the doing. The new technologies
statewide basis would be prohibitive using current
open the door to using a range of methodologies not
methodologies. So while today's assessments provide
currently available, enabling not only the assessment
schools with important programmatic data, more
of higher-order-thinking and more complex skill
3
Computer-Based Assessment
standards, but in formats potentially more engaging
items that fit into that range. From an
than our current test stimuli. Use of CD-ROMS, for
instructional perspective, the more "authentic"
example, will allow students to respond to authentic
computer-based assessment methodologies can be
"real-world" scenarios designed to
readily integrated into classroom
assess problem solving, reasoning,
practice, allowing more of the
New methodologies
and other higher-order skills, in
curriculum-embedded features that
addition to academic content.
were promised by proponents of
hold out the promise
Simulation software already in
performance assessments.
existence for training programs are
of great improvements
equally applicable in an assessment
The new methodologies hold out the
in the quality of data,
context. Graphing software can
promise of great improvements in
completely change the nature of
the quality of data, leading to greater
leading to greater
mathematics tests, allowing
precision and increased validity.
precision and
students to develop and compare
Decreases in measurement error will
alternative solutions to
be beneficial for the complex school
increased validity
.
challenging, real-world problems.
accountability formulas in place in
Procedures such as "knowledge
many states. Automated, adaptive
mapping" allow students to draw
assessments enable real-time
relationships among many concepts and then be
equating, both vertical and horizontal, a feature
scored against several varying "expert" maps.
currently lacking in both state and commercial
Students can create vastly different maps and still
testing programs. This feature will ensure all
earn high scores because it is the reasoning behind
students' results are comparable, whether within a
the relationships that is being tested, not any single
grade or across grades, despite the fact that students
so-called "right" answer.
are tested on different sets of items and prompts.
Value. As noted above, computer-based assessment
results are more immediately available to teachers
What stands in the way:
than the results from current testing systems. This
Questions in need of answers
timeliness allows immediate intervention, whether
it be remediation or enrichment, assignment to
Given the notable advantages described above, the
summer school, or anything else indicated by the
question is not if; but when, all high-stakes state
results. More importantly, the type of information
testing programs will attempt to incorporate the
available from computer-based assessments
emerging technologies. But standing between
promises to be much more valuable. Computer-
technology's promise of significantly more effective
adaptive testing (CAT) techniques, like those
assessment and the fulfillment of that promise are a
described earlier, zoom in on the ability level of
number of equally significant barriers. What follows
each individual, allowing more reliable assessment
is a discussion of the impediments that must be
with fewer items. Current systems require that
eliminated before technology can reinvent high-
students attempt every item even though, for
stakes assessment.
some, the early items are too easy and for others,
the final items are too difficult. Based on how a
More Logistics. Unfortunately, use of the new
student responds to certain early items in an
technologies will not solve all existing logistics
assessment, CAT models quickly identify that
concerns, and it may even create some new ones.
student's ability level and, thereafter, only present
4
WestEd
computer-based assessment. School personnel have
When a pencil point breaks, a student can simply
become reasonably proficient in administering
sharpen it or switch to a new one. The problem of a
pencil-and-paper assessments. But, while
computer crash is much more significant and often
computer-based assessment procedures are
time consuming to resolve. At the very least, the
ultimately less difficult to implement, states must
assessment delivery tool could be decommissioned
not underestimate the training needs in the early
in the middle of the test. There is also the potential
years of any new program. For example, countless
that an entire testing session, along with the
decisions must be made in the assessment
responses of all networked students, could be lost.
administration process: Should files be stored on
Backup procedures are essential, both in terms of
diskettes and sent by mail or should files be sent
safely storing student responses and of having
via the Internet? Will school officials at each site
alternative means to administer the test.
know how to load or access files and ensure
uniform assessment conditions? Might certain
Because schools differ tremendously in the
software features need to be disabled (e.g., a
hardware and software they have, system
spelling or grammar checker for a writing test),
incompatibility is quite common whenever a new
and does anyone in the school know how to do so?
product line is introduced. In developing
assessment systems that rely on technology, states
Scheduling is the final hurdle to overcome. Few
need either to make sure the test software works on
schools have sufficient computer resources to test
several different platforms or to require and support
all students at once. How long it
a single platform for all schools.
took before all students were
Both options can be quite costly;
tested would depend entirely
the latter could exacerbate
on a school's resources, but
pockets of computer shortages
it's conceivable that it could
and inequities.
take from several days to
several weeks. In such cases,
The test administration process
security, equivalence, and access
has two basic components:
issues would arise. Each is
format of test stimulus (e.g., test
discussed below.
booklets) and format of student
response (e.g., student answer
Security. All current testing
sheets). Currently, most
programs fear the copy machine.
programs use paper for both.
Complex security procedures are
Some programs have begun
now standard practice in most
delivering tests on computer but
states, driven in part by the high
continue to have students
now associated with statewide assessment.
respond on paper. This limited approach
stakes
is safer
That said, security breaches in paper-based
than relying entirely on computers because it
assessment systems, while not uncommon, tend at
reduces the likelihood of a computer crash, but it
least to be localized and confinable. In contrast,
does not take full advantage of the technology.
with new digital technologies, a simple push of a
Paper responses must then be shipped and scanned,
button could send "secure" test forms literally
causing the delays and costs described earlier.
around the world. While secure information is
claimed to be "read only," stories of identity theft
Delivery of materials and training of
and hackers into the Pentagon missile systems
administrators remain necessary steps with
r7
5
Computer-Based Assessment
should cause state test directors to wonder how safe
administered in one day or spread across more days?
any so-called secure files could ever be.
Should items be presented one at a time on the
computer screen or should students be allowed to
Another security concern would
scroll across all items? Finally, for
exist at each administration site.
any of these questions, how might
At the most basic level,
With the necessary extension of
the answers differ according to
the testing window, students could
students' age, sex, race, ethnicity,
we need to know that
more easily obtain advanced
language, or disability?
information on actual test items
a test item presented
from those who took the test
Content equivalence would also
"on screen" measures
earlier. Computer labs would, of
need to be addressed. At the most
course, need to be supervised. But
basic level, we need to know that a
the same knowledge
equally important, states might
test item presented "on screen"
and skills as its
need to develop several forms of
measures the same knowledge and
tests, drawing from much larger
skills as its paper counterpart. And
paper counterpart.
item banks than currently exist.
what should be done about such
However, few states have the
features as a spell check, which is
resources to develop sufficient
routinely available in word-
item banks for security purposes
or to take
processing packages? (Even making dictionaries
advantage of the type of CAT models described
available would require much more action and time
earlier. Unfortunately, those states that do develop
from students than is required for using a
large item banks would, in doing so, exacerbate the
computer's spelling feature.) In the end, we must be
equivalence problem discussed next.
able to say that content is identical across all
delivery avenues.
Equivalence. Several significant concerns about test
equivalence
in administration setting, content,
Additionally, while computer-based assessment is
and results
need to be allayed before states
expected to expand the types of standards that can
should jump fully into computer-based assessment.
be efficiently measured in large-scale assessment,
Equivalence is important for a number of reasons,
states need to be sure they don't lose the ability to
not least of which is the probability that, due to
measure basic skills. The issues here closely parallel
resource issues, many states might need to roll out a
concerns raised for many years about the use of
new technology-based assessment system in phases.
calculators on mathematics tests. The increased use
In this scenario, some students in the state would be
of calculators led to math items typically being
assessed with the new technology while others were
classified as calculator-neutral or calculator-
assessed using traditional paper-based methodology.
enhanced, depending on the degree and type of
calculation they require. Should future items be
Ensuring administration equivalence between a
similarly classified according to how they would be
computer-based assessment system and the
affected by mode of delivery (e.g., technology-
traditional pencil-and-paper system will require
neutral)? This question would be especially
more research. Among the questions still needing to
important for the many states likely to begin their
be answered: Are the number and timing of breaks
move toward computer-based assessment with
needed during the assessment comparable from one
mixed-delivery models, in which some students
medium to the other? How many test items should
would receive paper-and-pencil tests while others
be included in any given session? Should tests be
received computer-based tests.
0
6
WestEd
urban and rural students, among others, must all
Finally, there is the issue of results equivalency. In
the current high-stakes environment, any
share the same comfort level if the emerging
perception (or reality) that one mode of assessment
methods are to be judged fair and defensible.
delivery or response results in higher scores than
another will undermine the fairness of the system.
Yet not all students have equal access to technology,
In CAT designs, for example, all students can
either for instruction or assessment purposes. Nor
actually be taking different tests; in this scenario,
are all teachers equally comfortable incorporating
technology into their practice. Exacerbating the
aggregating assessment results for high-stakes
accountability purposes could be difficult both to
problem are the vast differences in students' access
do and to explain. Even if we could develop
to computers outside of school. Many homes do
not have a computer, let alone Internet connection.
formulas to statistically adjust various formats to
make them equivalent, might the public not view
To the extent these disparities exist across
this as a manipulation of scores to reach
socioeconomic, ethnic, and racial lines,
predetermined achievement goals or hide
performance differences on computer-based
performance disparities? If all students took the test
assessments would likely reflect or even increase the
online, it would still be important to equate results
current achievement gap. States must determine
across different years and formats lest we lose the
how to provide equal (or at least sufficient) access to
ability to track trends, reward gains, and address the
technology for both instruction and assessment
needs of those who continue to lag behind.
before computer-based assessments can be
considered a viable alternative to the status quo.
Access. The largest hurdle to realizing the promise
Given the inability to do so over the past 20 years
with calculators, a much less costly item, serious
of computer-based assessment is access, an issue
that bumps right up against the inequities of
questions remain as to when equal access to
funding and resource allocation evident throughout technology might be achieved, permitting
the public education system.
widespread implementation of innovative, emerging
technologies. Also, with computer technology, as
with calculators, attention must be paid to yet
The first access factor is the need to assess using a
mode of instruction the student commonly
another type of equivalency: the power of the
experiences. Take the content of the assessment. If particular technology available. Currently, testing
the first time students encountered that content
officials worry about students who work with four-
online was during the actual test administration, the
function calculators having to compete with those
results would not be valid. Just as curriculum and whose calculators include the latest graphing
instruction must incorporate the content standards
capabilities. The potential for disparity is that much
to properly prepare students for tests, so, too, must
greater when dealing with varying hardware
students be familiar with the actual technology used
capacity and software packages.
for assessment if such assessments are to be fair
measures of student achievement.
Conclusion
Interestingly, a major impetus for many states to
explore the new technologies has been the desire to
Some revolutions are inevitable. Computer use is so
accommodate special populations, as required by the
central to today's society and its potential for
federal Individual with Disabilities Education Act
improving assessment so great that it seems only a
and other state and local statutes and regulations.
matter of time before computer-based assessment
English language learners, Title I students, and
defines the test-administration process. States must
9
7
a nonprofit research, development, and service
WestEd,
plan now for that eventuality because, while the end of the road
agency, works with education and other communities to
may already be determined, our ability to transport all segments
promote excellence, achieve equity, and improve
of society safely and fairly to that point is by no means
learning for children, youth, and adults. While WestEd
guaranteed. Moving ahead without addressing the questions
serves the states of Arizona, California, Nevada, and Utah
as one of the nation's Regional Educational Laboratories,
raised in this brief will almost certainly result in flawed, unfair,
our agency's work extends throughout the United States
invalid (and most likely, illegal) assessment systems. Failure to
and abroad. It has 16 offices nationwide, from
move at all, however, threatens to deprive education of the many
Washington and Boston to Arizona, Southern California,
promises of computer-based assessment, most particularly,
and its headquarters in San Francisco.
increased content coverage, decreased timelines, more defensible
For more information about WestEd, visit our Web site:
technical characteristics, and more authentic approaches to
WestEd.org, call 415.565.3000 or, toll-free,
instruction and assessment, as well as the assurance that all
(1.877) 4-WestEd, or write:
students, including those with special needs, will benefit.
WestEd
730 Harrison Street
1
San Francisco, CA 941 07-1 242
\
/ /
This report was produced in whole or in part with funds from
the Office of Educational Research and Improvement,
\ Visit 616 r
U.S. Department of Education, under contract
#ED-01-00-0012. Its contents do not necessarily reflect the
est
views or policies of the Department of Education.
2001 WestEd. All rights reserved.
Non-Profit
Wes
d®
U.S. Postage
PAID
Los Alamitos, CA
730 Harrison Street
90720
San Francisco
Permit No. 87
California 94107-1242
Address service requested