in the process of arrival

119 Pages·2005·1.82 MB·English

A FUTURE
Alan R. Taylor

IN THE PROCESS
OF ARRIVAL
Using Computer Technologies for
the Assessment of Student Learning
Excellence in Education
SOCIETY FOR THE ADVANCEMENT OF
A FUTURE IN THE
PROCESS OF ARRIVAL
Using Computer Technologies for
the Assessment of Student Learning
Alan R. Taylor
A FUTURE
IN THE
PROCESS
OF ARRIVAL
Using Computer Technologies for
the Assessment of Student Learning
Society for the Advancement of
Excellence in Education
4
Copyright © 2005 Society for the Advancement of Excellence in Education
____________________________________
(SAEE)
A FUTURE IN THE
PROCESS OF ARRIVAL The views expressed in this report are those of the author and not necessarily
those of SAEE. Permission is hereby granted by SAEE to reproduce this docu-
ment for non-profit and educational purposes, provided that the author and
the publisher are acknowledged.
Library and Archives Canada Cataloguing in Publication
Taylor, Alan Richard, 1940-
A future in the process of arrival : using computer technologies for the assess-
ment of student learning / by Alan R. Taylor.
(SAEE research series ; 22)
Includes bibliographical references.
ISBN 0-9737755-1-3
1.Educational tests and measurements—Computer programs. 2.Grading and
marking (Students)—Computer programs. 3.Achievement tests—Computer
programs.
I.Society for the Advancement of Excellence in Education. II.Title. III.Series.
LB3060.5.T39 2005 371.26’0285 C2005-902621-9
SOCIETY FOR THE ADVANCEMENT OF
EXCELLENCE IN EDUCATION
The Society is an independent non-profit Canadian education research agency
founded in 1996. The mission of the Society is to encourage excellence in
public education through the provision of rigorous, non-partisan research.
The Society is particularly interested in assisting research that may shed in-
sight on innovative practices leading to successful learning outcomes. With
generous assistance from eight Canadian foundations and other funding part-
ners, the SAEE has commissioned 26 studies on schooling practices in Canada.
The Society is a registered Canadian charity and may provide official tax re-
ceipts for donations to its research work.
For additional copies of this report, please contact:
SOCIETY FOR THE ADVANCEMENT OF
EXCELLENCE IN EDUCATION
225 - 1889 Springfield Road, Kelowna, B.C. V1Y 5V5
Telephone [phone]
Fax [phone]
E-mail: [email protected]
Website http://www.saee.ca
Acknowledgements 5
____________________________________
I would like to express appreciation to the Max Bell Foundation and the
Society for the Advancement of Excellence in Education (SAEE) for spon-
soring the Technology Assisted Student Assessment Research Institute and
providing the research grant in support of this publication. Special thanks
goes to Helen Raham, Executive Director of SAEE, who initiated estab-
lishment of TASA and provided editorial guidance for this publication.
The following members of the TASA Advisory Committee are also acknowl-
edged for their constructive advice on the survey of Provinces and Territo-
ries and an earlier draft of this document: Bill Allen, Nelson Publishing;
Barry Anderson, BC Ministry of Education; Kathryn Barker, FutureEd;
Robert MacDonald, Media Futures Institute.
The following contacts at provincial, territorial and district levels also made
important contributions to this report.
British Columbia Quebec
• Britta Gundersen-Bryden, Director, Examina- • Laila Touma-Valin, Director, Evaluation Branch,
tions and Assessment Branch, Ministry of Educa- Ministere de l’Education
tion • Anne Doucet, Evaluation Branch, Ministere de
• Jim Gaskill, Manager, Examinations and Assess- l’Education
ment Branch, Ministry of Education • Thomas Stenzel, Ped Consultant & Local RECIT
• Stewart Hercus, Superintendent, Sunshine Coast Animator, Riverside School Board
School District New Brunswick
Alberta • Cary Grobe, Director of Evaluation, Department
• Donna Smith, Executive Director, Learner Assess- of Education
ment, Alberta Education Nova Scotia
Saskatchewan • Vince Warner, Director Evaluation Services Divi-
• Rick Johnson, Director, Assessment for Learning sion, Department of Education
Unit, Saskatchewan Learning Prince Edward Island
• Sue Amundred, A/Executive Director, E-Learn- • Cindy Wood, Consultant Program Evaluation &
ing Branch, Saskatchewan Learning Student Assessment, Department of Education
Manitoba • Guy Albert, Program Evaluation and Student As-
• Ken Clark, Coordinator Assessment and Evalua- sessment, Department of Education
tion Branch, Manitoba Education Newfoundland
• John Finch, Technology Coordinator Mountain • Marian Fushell, Director of Evaluation, Depart-
View School Division ment of Eduction
Ontario Northwest Territories
• Rick Jones, Coordinator International, National, • Ronda Nicklen, Coordinator of Assessment and
and Special Projects, Educational Quality and Evaluation, Department of Education
Accountability Office (EQAO) Nunavut
• Michael O’Gorman, Manager, Assessment Unit • Brian Yamamura, Coordinator, Mathematics,
Ministry of Education Science &Technology, Dept. of Education
6 About the Author
____________________________________
A FUTURE IN THE Dr. Taylor is internationally recognized as a testing and measurement ex-
PROCESS OF ARRIVAL pert. He designed and implemented numerous program assessments in
BC. From 1994-2002 he was the National Research Coordinator for Canada
in the Third International Mathematics and Science Study. His adminis-
trative experience includes positions as Director of Student Assessment
and Examinations for British Columbia and Director of Curriculum, As-
sessment and Research for a large urban school district. A former UBC
adjunct professor, he is currently vice president of Raven Research Associ-
ates and working on a variety of international research contracts in assess-
ment design. Co-author of Student Assessment in Canada (2001, SAEE), Dr.
Taylor is Director of TASA Institute.
About the TASA Institute
The TASA Institute is a research initiative of the Society for the Advance-
ment of Excellence in Education (SAEE). Established in 2004 with a re-
search grant from Max Bell Foundation, the mission of TASA Institute is
to study and advance knowledge in the development and application of
assessment technology in the Canadian public education system.
The purposes of the Institute are:
1. To document trends, leading-edge prototypes, evidence regarding their
effectiveness, best practice, and implications for policy in the field of tech-
nology-delivered student assessment.
2. To develop a next-generation assessment toolset and process, leveraging
the considerable strengths of computer and online technologies.
3. To collaborate with Ministries of Education, school districts, testing agen-
cies and international researchers in the piloting and evaluation of com-
puter assisted assessment models.
4. To serve as a clearinghouse for research and provide a source of exper-
tise to schools, districts, and ministries/departments of education on the
design, implementation, and use of computer based assessment.
For more information, see: http://www.tasainstitute.com
Table of Contents 7
____________________________________
1. Introduction / 9
2. Design of the Study / 13
3. Designs and Applications of Computers in
Assessment / 17
4. An Overview of Initial Findings / 25
5. The Canadian Context for Computer-Based
Assessment / 39
6. Key CBA Initiatives in the United States / 55
7. Innovations in Computer-Based Testing / 67
8. Converting to a Computer-Based Format / 93
9. Key Findings, Implications, and
Recommendations / 101
References / 111
Appendices / 114
8 Tables and Figures
____________________________________
A FUTURE IN THE List of Tables
PROCESS OF ARRIVAL Table 2.1 Survey on the Use of Computer Technologies to
Assess Student Learning in Canada 15
Table 3.1 Advantages and Disadvantages of Computer
Test Designs 24
Table 5.1 Electronic Examinations in British Columbia 42
Table 5.2 Ratios of Students per Internet Connected
Computer 52
Table 5.3 Location of Internet Connected Computers 53
Table 6.1 Summary of State Activities in Technology-Based
Assessment 60
List of Figures
Figure 4.1 Summary of Administrative Advantages of CAT 29
Figure 4.2 Item Banking Software Usability Evaluation Sheet 30
Figure 7.1 Summary of Features of Major Essay Scoring Programs 71
Figure 7.2 Scannable Answer Form 74
Figure 7.3 Scanning Handwritten Numerical Responses 76
Figure 7.4 Scanning Locations in the Coordinate Plane 77
Figure 7.5 Steps in Design of a WLAN 87
Figure 7.6 Start-up List for Portfolios 91
Figure 8.1 Pilot Stages 97
Figure 8.2 Software Modules for the Pilot 98
Figure 8.3 Design of a Multi-dimensional Data Base 99
1. 9
____________________________________
Introduction
Welcome to the future of student assessment! It is a world where the link
between instruction and assessment is seamless, a world where student
achievement is measured in a virtual milieu, replicating the learning envi-
ronment of the student. It is a world utilizing achievement measures that
introduce “hands on” or experiential scenarios through simulations and
modeling, a world in which organization and process skills are measured The link between
through enhanced item formats and the use of multi-media, a world in assessment and in-
which both student and teacher receive instantaneous feedback in time to struction should be
make a difference. Indeed, it is a world where assessment informs instruc- seamless.
tion in a timely fashion, not only in terms of effectiveness through com-
parative analysis and links to standards, but also through guidance for
student remediation, for course planning, and for identifying and apply-
ing the best and the most effective instructional techniques. How can all
of this be attained? It can be realized through the potential of computer
technologies in the assessment of student learning.
The Impact of Technology
Technology touches us all in a profound way. In our daily lives, in business
transactions, and in our means of communication. Airlines, for example,
prefer to issue electronic tickets; banks encourage online transactions;
companies realize savings in purchasing and managing inventory levels
through computerized systems; and many educational institutions and
companies offer online, instructor-led courses through e-learning. Job
applicants may spend their initial interviews relaxing at home with a morn-
ing coffee while computers determine whether or not they are fit for a
position (Shropshire, 2005). Iris scans are now being used to establish the
As technology becomes
identity of travellers flying between Canada and the United States in the
an integral component
FastTrack Border Initiative.
of what and how
students learn, its use
It is predictable that in a world where technology has had such far-reach-
as an essential tool for
ing effects on consumers, the workplace, the business world, and in the
student assessment is
delivery of educational programs, that significant efficiency can also be
inevitable.
gained in the assessment of student learning. The advance of technology
is inevitable due to its exponential growth in capability. As technology
becomes an integral component of what and how students learn, its use as
an essential tool for student assessment is inevitable.
10 Computer Technologies in
____________________________________
Assessment
A FUTURE IN THE
PROCESS OF ARRIVAL The assessment of student learning through applications of computer tech-
nology has become a hot topic among educators in recent years. It has
gained momentum in North America and elsewhere in the world as ad-
vances in technology and related software products make it possible to
more closely match instruction and assessment, to realize major gains in
relevance and efficiency, and to provide students with greater access and
better mediums through which to demonstrate their learning.
Inherent in the use of computer assessment technologies is the capability
for students to gain greater ownership of their learning. For example,
formative measures are available for self assessment; tests can be taken on
demand for qualification or placement in a stream; simulations and audio
visual features add motivating realism to the process; e-portfolios allow
students to reflect on personal growth in competencies demonstrated.
Such attributes are powerful metacognative tools, encouraging and sup-
porting students to take greater ownership of and responsibility for their
own learning.
Advances in technology and new developments in the field of educational
measurement have generated significant advantages for the computer-
based testing model. Only recently, for example, have effective programs
been developed that provide for secure interactive access and online scor-
ing of both multiple-choice and written response questions. As a result,
Among the benefits of
computer-based strategies are currently being utilized by a growing number
CBA are:
of testing agencies to achieve improvement in test quality, in examination
• closer match be-
delivery, and in reporting.
tween curriculum
and instruction
The potential for accrued benefits has resulted in an explosion of interest
• more extensive use of
in technology as a tool for better assessment. What advantages does this
existing banks of
approach bring to the testing arena? The following are among the many
items
types of gains that can be achieved:
• greater precision
• measurement of
• a closer match between curriculum and instruction through item en-
other types of learn-
hancement – simulations, models, sound, etc.
ing outcomes
• more extensive use of existing banks of items
• cost savings
• greater precision of measures through capacity to adapt to individual
• greater student
student competency levels
access
• ability to measure learning outcomes not possible through paper and
• improved turnaround
pencil tests
time
• cost savings and increased reliability in marking
• greater access for students through examination on demand