Computerized Adaptive Testing: From Inquiry to Operation

324 Pages·1998·28.805 MB·English

by  William A. Sands, Brian K. Waters, James R. McBride

Edi
ted br y
Sands
(
A
Brian K. Waters
M id id
Digitized by the Internet Archive
In 2022 with funding from
Kahle/Austin Foundation
https://archive.org/details/computerizedadapO0000unse
Computerized
Adaptive Testing
From Inquiry to Operation
Edited by
William A. Sands
Brian K. Waters
James f. McBride
~~PISCARDED
WIDENER UNIVERSITY
AMERICAN PSYCHOLOGICAL ASSOCIATION
Washington, DC
Copyright © 1997 by the American Psychological Association. All rights reserved.
Except as permitted under the United States Copyright Act of 1976, no part of
this publication may be reproduced or distributed in any form or by any means,
or stored in a database or retrieval system, without the prior written permission
of the publishers. Exceptions to this copyright are chapters 8, 9, 10, 11, 12 13,
14, 15, 17, 18, 19, 20, 21, 22, 24, and 26. These chapters were prepared by
government employees and are, therefore, in the public demain.
Published by
American Psychological Association
750 First Street, NE
Washington, DC 20002
Copies may be ordered from
APA Order Department
P.O. Box 92984
Washington, DC 20090-2984
In the United Kingdom and Europe, copies may be ordered from
American Psychological Association
3 Henrietta Street
Covent Gardens, London
WC2E 8LU England
Typeset in Century Schoolbook by G&S Typesetters, Inc., Austin, TX
Printer: Braceland, Philadelphia, PA
Cover design: Minker Design, Bethesda, MD
Technical/production editor: Edward B. Meidenbauer
Library of Congress Cataloging-in-Publication Data
Computerized adaptive testing : frominquirytooperation / edited by
William A. Sands, Brian K. Waters, and James R. McBride.
p. cm.
Includes bibliographical references and index.
ISBN 1-55798-442-5 (acid-free paper)
1. Psychological tests—Data processing. 2. Computer adaptive
testing. I. Sands, William A. II. Waters, Brian K. III. McBride,
éS —S
»
BF176.2.C65 1997
‘0285 —de21 97-25695
CIP
British Cataloguing-in-Publication Data
A CIP record is available from the British Library
pone’ URRY ¥E aS
Printed in the United States of America & oS
First edition pi eer ier
\ LIBRARY }
he
SM esta 4 tale
Contents
About the Editors and Contributors vii
Preface xill
Foreword XV
W. S. Sellman and Jane M. Arabian
Part I Background
Introduction to ASVAB and CAT
W. A. Sands and Brian K. Waters
Policy and Program Management Perspective 11
Clessen J. Martin and C. Richard Hoshaw
R&D Laboratory Management Perspective 21
Martin F. Wiskoff
Technical Perspective 29
James R. McBride
Part II Evaluating the Concept of CAT 45
Research Antecedents of Applied Adaptive Testing 47
James R. McBride
The Marine Corps Exploratory Development Project: 1977-1982 59
James R. McBride
The Computerized Adaptive Screening Test 69
W. A. Sands, Paul A. Gade, and Deirdre J. Knapp
CONTENTS rae
Part III First Generation—The Experimental CAT-ASVAB System 81
Preliminary Psychometric Research for CAT-ASVAB: Selecting an
Adaptive Testing Strategy 83
James R. McBride, C. Douglas Wetzel, and Rebecca D. Hetter
Development of the Experimental CAT-ASVAB System Dik
John H. Wolfe, James R. McBride, and J. Bradford Sympson
Validation of the Experimental CAT-ASVAB System 103
Daniel O. Segall, Kathleen E. Moreno, William F. Kieckhaefer,
Frank L. Vicino, and James R. McBride
Part EV. Second Generation—Full-Scale CAT-ASVAB Systems 115
ihe Item Pool Development and Evaluation Ly
Daniel O. Segall, Kathleen E. Moreno, and Rebecca D. Hetter
1 Psychometric Procedures for Administering CAT-ASVAB 131
Daniel O. Segall, Kathleen E. Moreno, Bruce M. Bloxom,
and Rebecca D. Hetter
13. Item Exposure Control in CAT-ASVAB 141
Rebecca D. Hetter and J. Bradford Sympson
14. AcaAP Hardware Selection, Software Development, and
Acceptance Testing 145
Bernard Rafacz and Rebecca D. Hetter
15. Human Factors in the CAT System: A Pilot Study 157
Frank L. Vicino and Kathleen E. Moreno
16. Evaluating Item Calibration Medium in Computerized
Adaptive Testing 161
Rebecca D. Hetter, Daniel O. Segall, and Bruce M. Bloxom
Lh Reliability and Construct Validity of CAT-ASVAB 169
Kathleen E. Moreno and Daniel O. Segall
18; Evaluating the Predictive Validity of CAT-ASVAB 175
John H. Wolfe, Kathleen E. Moreno, and Daniel O. Segall
19: Equating the CAT-ASVAB 181
Daniel O. Segall
20. CAT-ASVAB Operational Test and Evaluation ASf y
Kathleen E. Moreno
1v CONTENTS
Part V Third Generation—Building a System for Nationwide
Implementation 207
21. Development of a System for Nationwide Implementation 209
Vincent Unpingco, Irwin Hom, and Bernard Rafacz
22. The Psychometric Comparability of Computer Hardware 219
Daniel O. Segall
23. CAT-ASVAB Cost and Benefit Analyses 227
Lauress L. Wise, Linda T. Curran, and James R. McBride
Part VI Current Status and Beyond 237
24. Expanding the Content of CAT-ASVAB: New Tests and Their Validity 239
John H. Wolfe, David L. Alderton, Gerald E. Larson, Bruce M.
Bloxom, and Lauress L. Wise
25. Dissemination of CAT-ASVAB Technology 251
James R. McBride
26. Current and Future Challenges 257
Daniel O. Segall and Kathleen E. Moreno
References 271
List of Acronyms 279
Index 283
CONTENTS Vv
nbc sonnel Sanite pemticae a Sh
on = shane o
ee
C3e ts
ileal.
Acai Taphuee Pte © 6062 @ es Ign teas
iooqirar (hens | . <aryorrad Emil 16
Darvethd fem 0OOGES: A
| evtvall
* Hite Oratp®GieuD / opr
fonab}. wv o@* — 4, _
Cass ne | _— ra _ 2.2.2 & an
Miisg@la Toe = : =
irieiedf h YE. hao =Gye ots be wa”
fb Saag Rb ineD 2) ee fire
ApAle® © oie =o=-= 7 s. De.
Asher ey uct ~~ a= cael
al OD ae -—F gee 7 Seaspo Com
bw ag Ses as @ es 4
Pimngled# > Sql ;
2p|j aa] Oe a
Paphos @ Jams
About the Editors
and Contributors
The Editors fication; survey design and analyses; computer-
based vocational guidance; artificial neural net-
works; and, expert and decision support systems.
W. A. (Drew) Sands has spent most of his career
in military personnel research. He earned a B.S. Brian K. Waters is Program Manager of the
in Social Sciences and an M.A. in Counseling and Manpower Analysis Program of the Human Re-
Testing Psychology from The American Univer- sources Research Organization (HumRRO). He
sity in Washington, DC. In 1967, he joined the joined HumRRO in 1980, after retiring from the
Naval Personnel Research and Development Labo- Air Force, where he taught and was Director: of
ratory in Washington as a personnel research psy- Evaluation at the Air War College, was an R&D
chologist. In 1973, Mr. Sands transferred to the manager and researcher with the Air Force Hu-
Navy Personnel Research and Development Cen- man Resources Laboratory and a navigator. He
ter (NPRDC) in San Diego, CA. In 1980, he became holds a Ph.D. and M.S. in Educational Measure-
the Head of NPRDC’s Computerized Personnel Ac- ment and Testing from Florida State University,
cessioning Systems Branch of the Personnel Sys- and an M.B.A. from Southern Illinois University.
tems Department. He managed the R&D team His doctoral dissertation in 1974 was one of the
that developed the Navy Personnel Accessioning earliest empirical studies of CAT, and he has over
System (NPAS) and the Computerized Adaptive 20 years’ experience with CAT R&D. He is a fellow
Screening Test (CAST). In 1983, he became Head of of the American Psychological Association (APA)
the Computerized Testing and Accessioning Divi- and a former President of the APA Division of Mili-
sion in the Personnel Systems Department, which tary Psychology. He has authored over 100 journal
was focused on the CAT version of the Armed Ser- articles, books, and professional papers, primarily
vices Vocational Aptitude Battery (CAT-ASVAB). In dealing with the selection, classification, and test-
March 1986, he became Director of the Personnel ing of military and civilian personnel.
Systems Department at NPRDC, where he planned,
directed, and evaluated the overall scientific re- James R. McBride is a Principal Scientist with
search program in personnel screening, selection, HumRRO. A research psychologist, he has been
classification, and performance assesSment. As involved in research and development related to
the Officer-in-Charge, he had the lead laboratory CAT since 1972. During his doctoral studies in
(NPRDC) responsibility for the Joint-Service CAT- psychometric methods at the University of Min-
ASVAB Program. Mr. Sands retired from civil ser- nesota, he was a research assistant to David J.
vice in March 1994, returned to the Washington, Weiss, and participated in Weiss’ pioneering CAT
DC area, and is conducting research for various research for the Office of Naval Research. Since
organizations as a consultant with Chesapeake Re- completing doctoral training in 1976, he has done
search Applications. He has authored over 110 book test development and personnel research for the
chapters, journal articles, technical reports, and Army Research Institute for the Behavioral and
professional presentations in psychological test- Social Sciences (ARI), NPRDC, The Psychological
ing (paper-and-penci! and computerized adaptive Corporation, and HumRRO. At NprDc, he was
tests); personnel screening, selection, and classi- Principal Investigator on a variety of CAT-related
ABOUT THE EDITORS AND CONTRIBUTORS - vil
projects ranging from the exploratory development Jane M. Arabian is Assistant Director for Enlist-
work that provided the first empirical demonstra- ment Standards in the Directorate for Accession
tion of CAT’s efficiency for military personnel test- Policy, Office of the Secretary of Defense. She plans
ing, to the design and development of prototype and formulates policy on military enlistment stan-
systems intended for nationwide administration dards (i.e., aptitude, education, moral character,
of computerized adaptive versions of the ASVAB. age, citizenship) as well as the DoD Student Test-
At NPRDC, he designed and directed the develop- ing Program. Dr. Arabian also serves as the Ex-
ment of the first complete computerized systems ecutive Secretary for the Defense Advisory Com-
for adaptive ASVAB administration. At the time of mittee on Military Personnel Testing. She is an
his departure from NPRDC, he was Director of the author of over 30 journal articles, professional pa-
Personnel Systems Department, with responsibil- pers, and technical reports in the areas of experi-
ity for the entire spectrum of scientific research re- mental psychology, health psychology, and mili-
lated to Navy personnel selection, classification, tary selection and classification testing and has
and testing. He joined The Psychological Corpora- presented formal research papers and invited ad-
tion in 1984, as Director of its Computer-Based dresses at over 50 professional conferences. Dr.
Testing Group; later, his responsibilities there ex- Arabian earned her Ph.D. in Experimental Psy-
tended to all development and research related to chology at the University of Toronto.
tests designed for personnel assessment in busi-
ness, government, and career development. Be- Bruce M. Bloxom obtained a Ph.D. in Psychomet-
tween 1984 and 1990, he designed and directed de- rics from the University of Washington in 1966.
velopment of a number of computer-based testing He is presently a research psychologist at the De-
systems, including the first commercial applica- fense Manpower Data Center (DMDC), Monterey,
tion of CAT: The Computerized Adaptive Edition California, where he has served as Chief of the
of the Differential Aptitude Tests. Since joining Quality Control and Analysis Branch as well as
HumRRO in 1990, he has continued his involve- Acting Chief of the Personnel Testing Division.
ment in R&D on computer-based testing in gen- Prior to his time at DMDC, he was Professor of Psy-
eral, and CAT in particular. He directed the devel- chology at Vanderbilt University.
opment of one of the first CAT systems used for
personnel selection in industry for a Fortune 100 Linda T. Curran was a psychologist with the
HumRRO client. He has provided consulting ser- Personnel Testing Division of the DoD Manpower
vices in computer-based testing to several other Data Center in Arlington, Virginia from 1991 un-
private sector firms, and has been a member of til she died in June 1997. She was responsible for
an expert panel advising the U.S. Department of coordination between DoD staff in the Washing-
Labor on the development and evaluation of a ton area and the Testing Division staff in Monte-
CAT version of the General Aptitude Test Battery rey, California. She received her doctorate from
(GATB). He is currently directing the HumRRO the University of Texas in Educational Psychology
project team responsible for modifying the Army’s (specializing in psychometrics) and a Bachelor’s
Computerized Adaptive Screening Test for use by degree from Baylor University. She chaired two
all of the Armed Services. groups: One reviews technical and policy matters
related to the research and development of the
ASVAB and the other is involved in the nationwide
The Contributors implementation of CAT-ASVAB. She also was the
manager of the program to norm the CAT-ASVAB
and an interest measure called the Interest-
David L. Alderton is the author or co-author Finder. Her experience included conducting test-
of numerous journal articles on individual differ- ing and training research for five years at the Air
ences in spatial ability, problem solving, and intel- Force Human Resources Laboratory, consulting
ligence. After receiving his Ph.D. in Educational with Jeanneret and Associates to develop selection
Psychology from the University of Santa Barbara tests for various oil companies for three years, and
in 1986, Dr. Alderton joined the NPRDC staff, developing and validating tests for the University
where he played a major role in several projects to of Texas for seven years.
develop and validate new tests. He now works at
the Centers for Disease Control and Prevention, Paul A. Gade is Chief of the Organization and
where he is conducting behavioral surveillance re- Personnel Resources Research Unit at the U.S.
search for the prevention of HIV infection. Army Research Institute for the Behavioral and
villi ABOUT THE EDITORS AND CONTRIBUTORS