The Psychometrics of Standard Setting: Connecting Policy and Test Scores

275 Pages·2022·17.926 MB·English

by  Mark Reckase

The Psychometrics of
Standard Setting
This book provides a unifying structure for the activities that fall under the
process typically called “standard setting” on tests of proficiency. Standard
setting refers to the methodology used to identify performance standards on
tests of proficiency. The results from standard setting studies are critical for
supporting the use of many types of tests. The process is frequently applied
to educational, psychological, licensure/certification, and other types of tests
and examination systems. The literature on procedures for standard setting
is extensive, but the methodology for standard setting has evolved in a hap-
hazard way over many decades without a unifying theory to support the
evaluation of the methods and the validation of inferences made from the
standards. This book provides a framework for going beyond specific stan-
dard setting methods to gain an understanding of the goals for the methods
and how to evaluate whether the goals have been achieved.
The unifying structure provided in this book considers policy that calls
for the existence of performance standards, the relationship of proficiency
test design to the policy, and tasks assigned to subject matter experts to help
them convert the policy to estimates of locations on the reporting score scale
for the test. The book provides guidance on how to connect the psychometric
aspects of the standard setting process to the intentions of policy makers as
expressed in policy statements. Furthermore, the structure is used to support
validity arguments for inferences made when using standards. Examples are
provided to show how the unifying structure can be used to evaluate and
improve standard setting methodology.
Statistics in the Social and Behavioral Sciences Series
Series Editors
Jeff Gill, Steven Heeringa, Wim J. van der Linden, Tom Snijders
Recently Published Titles
Big Data and Social Science: Data Science Methods and Tools for Research and
Practice, Second Edition
Ian Foster, Rayid Ghani, Ron S. Jarmin, Frauke Kreuter and Julia Lane
Understanding Elections through Statistics: Polling, Prediction, and Testing
Ole J. Forsberg
Analyzing Spatial Models of Choice and Judgment, Second Edition
David A. Armstrong II, Ryan Bakker, Royce Carroll, Christopher Hare, Keith T. Poole and
Howard Rosenthal
Introduction to R for Social Scientists: A Tidy Programming Approach
Ryan Kennedy and Philip Waggoner
Linear Regression Models: Applications in R
John P. Hoffman
Mixed-Mode Surveys: Design and Analysis
Jan van den Brakel, Bart Buelens, Madelon Cremers, Annemieke Luiten, Vivian Meertens,
Barry Schouten and Rachel Vis-Visschers
Applied Regularization Methods for the Social Sciences
Holmes Finch
An Introduction to the Rasch Model with Examples in R
Rudolf Debelak, Carolin Stobl and Matthew D. Zeigenfuse
Regression Analysis in R: A Comprehensive View for the Social Sciences
Jocelyn H. Bolin
Intensive Longitudinal Analysis of Human Processes
Kathleen M. Gates, Sy-Min Chow, and Peter C. M. Molenaar
Applied Regression Modeling: Bayesian and Frequentist Analysis of Categorical
and Limited Response Variables with R and Stan
Jun Xu
The Psychometrics of Standard Setting: Connecting Policy and Test Scores
Mark D. Reckase
For more information about this series, please visit: https://www .routledge .com /
Chapman- -HallCRC -Statistics -in -the -Social -and -Behavioral -Sciences /book -series /
CHSTSOBESCI
The Psychometrics of
Standard Setting
Connecting Policy and Test Scores
Mark D. Reckase
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks
does not warrant the accuracy of the text or exercises in this book. This book’s use or discussion of
MATLAB® software or related products does not constitute endorsement or sponsorship by The
MathWorks of a particular pedagogical approach or particular use of the MATLAB® software.
First edition published 2023
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
CRC Press is an imprint of Taylor & Francis Group, LLC
© 2023 Taylor & Francis Group, LLC
Reasonable efforts have been made to publish reliable data and information, but the author and pub-
lisher cannot assume responsibility for the validity of all materials or the consequences of their use.
The authors and publishers have attempted to trace the copyright holders of all material reproduced
in this publication and apologize to copyright holders if permission to publish in this form has not
been obtained. If any copyright material has not been acknowledged please write and let us know so
we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information
storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, access www .copyright
.com or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. For works that are not available on CCC please contact mpkbookspermissions @tandf
.co. uk
Trademark notice: Product or corporate names may be trademarks or registered trademarks and are
used only for identification and explanation without intent to infringe.
ISBN: 9781498722117 (hbk)
ISBN: 9781032422619 (pbk)
ISBN: 9780429156410 (ebk)
DOI: 10.1201/9780429156410
Typeset in Palatino
by Deanta Global Publishing Services, Chennai, India
This book is dedicated to Charlene Repetny Reckase who set the standard
for me for what it meant to be a good and generous human being.

Contents
Preface ......................................................................................................................xi
Author biography ................................................................................................xiii
1 Conceptual Overview for a Theory of Standard Setting .......................1
1.1 The Prevalence of Standards and Standard Setting ........................1
1.2 Definitions of a Standard .....................................................................4
1.3 The Differing Contexts for Setting a Standard .................................7
1.4 Who Defines the Standard? .................................................................8
1.5 Qualifications of the Persons Who Participate in the
Standard Setting Process .....................................................................9
1.6 The Standard Setting Process as a Data Collection Design ..........11
1.7 Calculating the Value of the Standard .............................................12
1.8 Evaluation of the Results of a Standard Setting Process ...............14
1.9 Is a Performance Standard an Instructional Goal? ........................15
1.10 The Goals for This Book .....................................................................16
2 A Theory of Standard Setting ....................................................................17
2.1 Introduction to a Theory of Standard Setting ................................17
2.2 Basic Concepts .....................................................................................18
2.2.1 Agency .....................................................................................18
2.2.2 Policy Definition.....................................................................19
2.2.3 Content-Specific Definition and Test Specifications .........20
2.2.4 Kernel of the Standard Setting Process ..............................22
2.2.5 Estimate of the Intended Standard......................................24
2.2.6 Feedback to Panelists ............................................................25
2.2.7 Approval of the Estimated Standard ..................................26
2.3 Documentation of the Standard Setting Process ...........................27
2.4 The Validity Argument ......................................................................27
2.5 Some Fanciful Examples of Standard Setting .................................28
2.5.1 How Much Oxygen Is Needed for Healthy Living? .........28
2.5.2 Understanding of the Functioning of Passenger
Automobiles ............................................................................30
2.6 Summary and Organization of Subsequent Chapters ..................32
3 Agency, Policy Definitions, Content-Specific Definitions and
Minimum Qualifications ............................................................................35
3.1 The Agency and Its Role in Setting Performance Standards .......35
3.1.1 Involvement ............................................................................35
3.1.2 Expertise ..................................................................................36
3.1.3 Heterogeneity .........................................................................36
vii
viii Contents
3.1.4 Categorization of Agencies ...................................................37
3.1.4.1 State of Michigan Legislature ...............................37
3.1.4.2 National Assessment Governing Board .............38
3.1.4.3 National Council of State Boards of Nursing .....39
3.1.5 Influence of Agency Characteristics on Standard
Setting ......................................................................................40
3.2 Policy Definition (D ) ..........................................................................41
P
3.3 Content-Specific Definition (D ) .......................................................43
C
3.4 Minimum Qualifications Definition (minD ) ................................44
C
3.5 Summary ..............................................................................................45
4 The Construct and the Test Designed to Measure the Construct .......47
4.1 The Target for Test Design and Development ................................47
4.2 The Unidimensional Model for Test Development ........................49
4.3 Domain Sampling Model for Test Development ............................52
4.4 Implications of Test Design for Standard Setting ..........................54
4.5 Summary ..............................................................................................55
5 The Kernel of the Standard Setting Process ...........................................57
5.1 Introduction .........................................................................................57
5.2 Examples of Standard Setting Kernels ............................................59
5.2.1 The Bookmark Kernel ...........................................................60
5.2.2 The Angoff Kernel .................................................................73
5.2.3 The Body of Work Kernel .....................................................79
5.2.4 The Contrasting Groups Kernel...........................................86
5.3 Criteria for Selecting a Standard Setting Kernel ............................89
5.3.1 Criterion Number 1 ...............................................................89
5.3.2 Criterion Number 2 ...............................................................90
6 Panelist Selection and Training .................................................................93
6.1 Introduction .........................................................................................93
6.2 The Influence of Panelists’ Characteristics on the Estimated
Standard ...............................................................................................94
6.3 Selecting and Recruiting Panelists ...................................................97
6.4 Statistical Considerations for Selecting Panelists ...........................99
6.5 Training of Panelists .........................................................................103
6.5.1 Presenting the “Big Picture” ..............................................106
6.5.2 Training on Policy and Content Definitions
(D and D ) ...........................................................................107
P C
6.5.3 Training on the Language of the Reporting Score Scale .....107
6.5.4 Training on the Tasks Required of the Standard
Setting Kernel .......................................................................108
6.5.5 Planning and Timing ..........................................................109
6.5.6 Evaluating Training .............................................................112
6.6 Conclusions ........................................................................................113
Contents ix
7 Feedback and Auxiliary Information .....................................................115
7.1 Introduction .......................................................................................115
7.1.1 Categorizations of Feedback and Auxiliary
Information ...........................................................................116
7.1.2 Purposes for Providing Feedback .....................................118
7.2 Types of Feedback .............................................................................121
7.2.1 Feedback about Understanding of the Standard
Setting Kernel .......................................................................122
7.2.2 Feedback about the Consequences of the Use of the
Estimated Cut Score.............................................................132
7.2.3 Other Types of Feedback ....................................................135
7.3 Auxiliary Information ......................................................................138
7.4 Timing of Feedback ..........................................................................141
7.5 Impact of Feedback on the Standard Setting Process ..................141
7.6 Conclusions and Recommendations ..............................................142
8 Estimating the Standard ...........................................................................145
8.1 Introduction .......................................................................................145
8.2 Assumptions Underlying the Estimation Processes ....................148
8.2.1 Reporting Score Scale ..........................................................148
8.2.2 Independence of Information from Panelists ..................149
8.2.3 Other Assumptions .............................................................151
8.3 The Value that Is Estimated .............................................................151
8.3.1 Estimation Based on Contrasting Groups Data
Collection ..............................................................................154
8.3.2 Estimation Based on Test Item Judgment Data ...............174
8.3.2.1 Effect of Item Parameter Estimation Error
on Estimates of Standards ...................................183
8.3.2.2 Effect of Panelists’ Errors in Estimation of
Conditional Probabilities ....................................185
8.3.2.3 Including Panelists’ Interpretation of
D and minD in the Estimation of the
c C
Intended Standard ...............................................187
8.3.3 Estimation Using Compromise Methods .........................189
8.4 Recent Developments .......................................................................196
8.5 Cautions about Generalizability .....................................................197
8.6 Summary and Conclusion ...............................................................198
9 Evaluating the Standard Setting Process ..............................................201
9.1 Introduction .......................................................................................201
9.2 Types of Evidence ..............................................................................203
9.2.1 Conceptual Coherence Evidence .......................................203
9.2.1.1 The Policy Definition (D ) ...................................205
P
9.2.1.2 The Content-Specific Definition (D ).................207
C