Statistical Test Theory for the Behavioral Sciences

282 Pages·2007·4.367 MB·English

by  Dato N. M. de Gruijter, Leo J. Th. van der Kamp

C9586_C000.fm Page i Thursday, July 12, 2007 10:43 AM
C9586_C000.fm Page ii Thursday, July 12, 2007 10:43 AM
C9586_C000.fm Page iii Thursday, July 12, 2007 10:43 AM
C9586_C000.fm Page iv Thursday, July 12, 2007 10:43 AM
Chapman & Hall/CRC
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2008 by Taylor & Francis Group, LLC
Chapman & Hall/CRC is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed in the United States of America on acid-free paper
10 9 8 7 6 5 4 3 2 1
International Standard Book Number-13: 978-1-58488-958-8 (Hardcover)
This book contains information obtained from authentic and highly regarded sources. Reprinted
material is quoted with permission, and sources are indicated. A wide variety of references are
listed. Reasonable efforts have been made to publish reliable data and information, but the author
and the publisher cannot assume responsibility for the validity of all materials or for the conse-
quences of their use.
No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any
electronic, mechanical, or other means, now known or hereafter invented, including photocopying,
microfilming, and recording, or in any information storage or retrieval system, without written
permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.
copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC)
222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that
provides licenses and registration for a variety of users. For organizations that have been granted a
photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Gruijter, Dato N. de.
Statistical test theory for the behavioral sciences / Dato N.M. de Gruijter and
Leo J. Th. van der Kamp.
p. cm. -- (Statistics in the social and behavioral sciences series ; 2)
Includes bibliographical references and index.
ISBN-13: 978-1-58488-958-8 (alk. paper)
1. Social sciences--Mathematical models. 2. Social sciences--Statistical
methods. 3. Psychometrics. 4. Psychological tests. 5. Educational tests and
measurements. I. Kamp, Leo J. Th. van der. II. Title.
H61.25.G78 2008
519.5--dc22 2007017631
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
C9586_C000.fm Page v Thursday, July 12, 2007 10:43 AM
Table of Contents
Chapter 1 Measurement and Scaling...............................................1
1.1 Introduction..................................................................................1
1.2 Definition of a test.......................................................................1
1.3 Measurement and scaling...........................................................2
Exercises..................................................................................................7
Chapter 2 Classical Test Theory.......................................................9
2.1 Introduction..................................................................................9
2.2 True score and measurement error............................................9
2.3 The population of persons.........................................................12
Exercises................................................................................................14
Chapter 3 Classical Test Theory and Reliability...........................15
3.1 Introduction................................................................................15
3.2 The definition of reliability and the standard
error of measurement................................................................15
3.3 The definition of parallel tests..................................................17
3.4 Reliability and test length........................................................19
3.5 Reliability and group homogeneity...........................................20
3.6 Estimating the true score.........................................................21
3.7 Correction for attenuation.........................................................23
Exercises................................................................................................23
Chapter 4 Estimating Reliability....................................................25
4.1 Introduction................................................................................25
4.2 Reliability estimation from a single administration
of a test.......................................................................................26
4.3 Reliability estimation with parallel tests................................36
4.4 Reliability estimation with the test–retest method................36
4.5 Reliability and factor analysis..................................................37
C9586_C000.fm Page vi Thursday, July 12, 2007 10:43 AM
4.6 Score profiles and estimation of true scores............................37
4.7 Reliability and conditional errors of measurement................42
Exercises................................................................................................44
Chapter 5 Generalizability Theory.................................................47
5.1 Introduction................................................................................47
5.2 Basic concepts of G theory........................................................48
5.3 One-facet designs, the p × i design
and the i : p design....................................................................50
5.3.1 The crossed design........................................................50
5.3.2 The nested i : p design.................................................54
5.4 The two-facet crossed p × i × j design......................................55
5.5 An example of a two-facet crossed p × i × j design:
The generalizability of job performance measurements.........59
5.6 The two-facet nested p × (i : j) design......................................60
5.7 Other two-facet designs.............................................................62
5.8 Fixed facets................................................................................64
5.9 Kinds of measurement errors...................................................67
5.10 Conditional error variance........................................................73
5.11 Concluding remarks...................................................................74
Exercises................................................................................................75
Chapter 6 Models for Dichotomous Items......................................79
6.1 Introduction................................................................................79
6.2 The binomial model...................................................................80
6.2.1 The binomial model in a homogeneous
item domain..................................................................82
6.2.2 The binomial model in a heterogeneous
item domain..................................................................87
6.3 The generalized binomial model...............................................88
6.4 The generalized binomial model and item
response models.........................................................................91
6.5 Item analysis and item selection..............................................92
Exercises................................................................................................98
Chapter 7 Validity and Validation of Tests..................................101
7.1 Introduction..............................................................................101
7.2 Validity and its sources of evidence.......................................103
7.3 Selection effects in validation studies....................................106
C9586_C000.fm Page vii Thursday, July 12, 2007 10:43 AM
7.4 Validity and classification.......................................................108
7.5 Selection and classification with more than
one predictor.............................................................................115
7.6 Convergent and discriminant validation: A strategy
for evidence-based validity......................................................118
7.6.1 The multitrait–multimethod approach.....................119
7.7 Validation and IRT..................................................................121
7.8 Research validity: Validity in empirical behavioral
research....................................................................................122
Exercises..............................................................................................123
Chapter 8 Principal Component Analysis, Factor
Analysis, and Structural Equation
Modeling: A Very Brief Introduction..........................125
8.1 Introduction..............................................................................125
8.2 Principal component analysis (PCA)......................................125
8.3 Exploratory factor analysis.....................................................127
8.4 Confirmatory factor analysis and structural
equation modeling....................................................................130
Exercises..............................................................................................132
Chapter 9 Item Response Models.................................................133
9.1 Introduction..............................................................................133
9.2 Basic concepts..........................................................................134
9.2.1 The Rasch model........................................................135
9.2.2 Two- and three-parameter logistic models...............136
9.2.3 Other IRT models.......................................................139
9.3 The multivariate normal distribution
and polytomous items..............................................................143
9.4 Item-test regression and item response models....................146
9.5 Estimation of item parameters...............................................148
9.6 Joint maximum likelihood estimation for item
and person parameters............................................................150
9.7 Joint maximum likelihood estimation
and the Rasch model...............................................................151
9.8 Marginal maximum likelihood estimation.............................153
9.9 Markov chain Monte Carlo.....................................................154
9.10 Conditional maximum likelihood estimation
in the Rasch model..................................................................156
C9586_C000.fm Page viii Thursday, July 12, 2007 10:43 AM
9.11 More on the estimation of item parameters..........................157
9.12 Maximum likelihood estimation of person
parameters................................................................................160
9.13 Bayesian estimation of person parameters...........................162
9.14 Test and item information.......................................................162
9.15 Model-data fit...........................................................................167
9.16 Appendix: Maximum likelihood estimation
of θ in the Rasch model...........................................................170
Exercises..............................................................................................174
Chapter 10 Applications of Item Response Theory.....................177
10.1 Introduction..............................................................................177
10.2 Item analysis and test construction.......................................179
10.3 Test construction and test development................................180
10.4 Item bias or DIF......................................................................182
10.5 Deviant answer patterns.........................................................189
10.6 Computerized adaptive testing (CAT)....................................191
10.7 IRT and the measurement of change.....................................194
10.8 Concluding remarks.................................................................195
Exercises..............................................................................................197
Chapter 11 Test Equating..............................................................199
11.1 Introduction..............................................................................199
11.2 Some basic data collection designs
for equating studies.................................................................202
11.2.1 Design 1: Single-group design...................................202
11.2.2 Design 2: Random-groups design..............................203
11.2.3 Design 3: Anchor-test design.....................................203
11.3 The equipercentile method......................................................204
11.4 Linear equating........................................................................207
11.5 Linear equating with an anchor test.....................................208
11.6 A synthesis of observed score equating
approaches: The kernel method..............................................212
11.7 IRT models for equating..........................................................212
11.7.1 The Rasch model........................................................213
11.7.2 The 2PL model............................................................214
11.7.3 The 3PL model............................................................215
11.7.4 Other models...............................................................216
C9586_C000.fm Page ix Thursday, July 12, 2007 10:43 AM
11.8 Concluding remarks.................................................................216
Exercises..............................................................................................219
Answers..............................................................................................221
References.........................................................................................235
Author Index....................................................................................255
Subject Index...................................................................................261