Diagnostic Measurement

370 Pages·05.047 MB·English

by  Andre A. Rupp

ebook
THE GUILFORD PRESS
Diagnostic Measurement
Methodology in the Social Sciences
David A. Kenny, Founding Editor
Todd D. Little, Series Editor
This series provides applied researchers and students with analysis and research design books
that emphasize the use of methods to answer research questions. Rather than emphasizing
statistical theory, each volume in the series illustrates when a technique should (and should not)
be used and how the output from available software programs should (and should not) be
interpreted. Common pitfalls as well as areas of further development are clearly articulated.
SPECTRAL ANALYSIS OF TIME-SERIES DATA
Rebecca M. Warner
A PRIMER ON REGRESSION ARTIFACTS
Donald T. Campbell and David A. Kenny
REGRESSION ANALYSIS FOR CATEGORICAL MODERATORS
Herman Aguinis
HOW TO CONDUCT BEHAVIORAL RESEARCH OVER THE INTERNET:
A BEGINNER’S GUIDE TO HTML AND CGI/PERL
R. Chris Fraley
PRINCIPLES AND PRACTICE OF STRUCTURAL EQUATION MODELING, SECOND EDITION
Rex B. Kline
CONFIRMATORY FACTOR ANALYSIS FOR APPLIED RESEARCH
Timothy A. Brown
DYADIC DATA ANALYSIS
David A. Kenny, Deborah A. Kashy, and William L. Cook
MISSING DATA: A GENTLE INTRODUCTION
Patrick E. McKnight, Katherine M. McKnight, Souraya Sidani, and Aurelio José Figueredo
MULTILEVEL ANALYSIS FOR APPLIED RESEARCH: IT’S JUST REGRESSION!
Robert Bickel
THE THEORY AND PRACTICE OF ITEM RESPONSE THEORY
R. J. de Ayala
THEORY CONSTRUCTION AND MODEL-BUILDING SKILLS:
A PRACTICAL GUIDE FOR SOCIAL SCIENTISTS
James Jaccard and Jacob Jacoby
DIAGNOSTIC MEASUREMENT: THEORY, METHODS, AND APPLICATIONS
André A. Rupp, Jonathan Templin, and Robert A. Henson
DIAGNOSTIC
MEASUREMENT
Theory, Methods, and Applications
André A. Rupp
Jonathan Templin
Robert A. Henson
Series Editor’s Note by Todd D. Little
THE GUILFORD PRESS
New York London
©2010 The Guilford Press
A Division of Guilford Publications, Inc.
72 Spring Street, New York, NY 10012
www.guilford.com
All rights reserved
No part of this book may be reproduced, translated, stored in a retrieval system, or
transmitted, in any form or by any means, electronic, mechanical, photocopying,
microfi lming, recording, or otherwise, without written permission from the Publisher.
Printed in the United States of America
This book is printed on acid-free paper.
Last digit is print number: 9 8 7 6 5 4 3 2 1
Library of Congress Cataloging-in-Publication Data
Rupp, André A.
Diagnostic measurement : theory, methods, and applications / André A. Rupp,
Jonathan Templin, Robert A. Henson.
p. cm. — (Methodology in the social sciences)
Includes bibliographical references and index.
ISBN 978-1-60623-528-7 (hardcover : alk. paper) — ISBN 978-1-60623-527-0
(pbk. : alk. paper)
1. Psychometrics. 2. Cognition—Testing. I. Templin, Jonathan. II. Henson,
Robert A. III. Title.
BF39.R785 2010
150.28′7—dc22 2009045966
Series Editor’s Note
Diagnostic classifi cation models (DCMs) have evolved into powerful tools over the last 20
years, particularly in the areas of educational and psychological measurement. These models
are ideally suited to disentangle the multidimensional content of many assessment tools to
provide unambiguous information about respondents that can guide decisions and help
specialists make accurate diagnoses about them. From complex cognitive stimuli in educa-
tional measurement to response-contingent clinical assessments, an appropriate DCM is an
essential tool to accurately classify and ultimately diagnose precisely where and in what way
a respondent is defi cient. The precision and accuracy of empirically grounded decisions with
modern DCMs will become the delight of intervention-minded researchers spanning educa-
tional, clinical, and personnel assessment.
For the uninitiated, the topic of designing diagnostic assessments and modeling and
resulting response data with DCMs may sound daunting. Fortunately, Rupp, Templin, and
Henson have provided an accessible, clear, and thorough road map for all of us to follow,
regardless of expertise. If you are new to the idea of DCMs, these accomplished scholars
give us a fresh and clear introduction to the different kinds of DCMs that are currently avail-
able, how they can be used, and how the resulting parameters can be used to guide decision
making. If you are experienced in the use of these models, you’ll fi nd the depth of scholar-
ship rewarding; these authors may even spark some new insights for you—they certainly did
for me!
This authorial team of rising stars in the fi eld of quantitative methods is also careful to
point out current limitations and potential pitfalls of diagnostic measurement techniques.
They highlight the critical importance of linking together measurement tools, statistical mod-
els, and theoretical models. Such linkages among diagnostic assessment design and DCM
analysis (and the resulting interpretations) can create compelling evidentiary narratives about
the diagnostic states of respondents. Clearly, should any one of these components be defi -
cient or should their alignment be weak, the empirical results and resulting decisions will be
v
vi Series Editor’s Note
less than satisfactory and only be weakly defensible. When properly designed and executed,
on the other hand, the results of diagnostic assessments analyzed with DCMs can unveil a
wealth of information that will allow interventionists to create a tailored protocol to amelio-
rate identifi ed defi ciencies.
As the praise from reviewers of this book attest, this will be one of those books that you
will keep at the ready, as it develops the patina of a well-utilized resource. Read it and use it
to help make a difference in your fi eld.
TODD D. LITTLE
Wit’s End
Lake Side, Montana
Preface
The book that you are holding in your hands is dedicated to the theory, methods, and ap-
plications of an important class of statistical models called diagnostic classifi cation models
(DCMs). The book is being published at an important time for the fi eld of diagnostic assess-
ment more generally and data analysis with DCMs more specifi cally. Specialists are increas-
ingly realizing the relative strengths of DCMs as well as their weaknesses and limitations, and
are rediscovering the importance of thoughtful and principled assessment design. This new
awareness becomes even more apparent when complex multidimensional measurement
models like DCMs are applied to diagnostic assessment data. Hence, engaging in critical dia-
logue with a common basis of knowledge and principles for reasoning is clearly needed. We
sincerely hope that this book can facilitate the creation of such a basis and that those who
read this book will fi nd it a useful guide for thinking, communicating, and acting in the fi eld
of diagnostic assessment.
THE ORGANIZATION OF THIS BOOK
We have divided this book into three main sections labeled “Theory,” “Methods,” and
“Applications.”
The “Theory” Section
In Part I, “Theory,” we focus in detail on basic considerations for developing, implementing,
and reporting results from diagnostic assessments in practice in contexts when data should
be analyzed with DCMs. Specifi cally, in Chapter 2 we discuss foundational concepts for
applying DCMs to diagnostic assessment data, which includes a description of an imple-
mentation process for diagnostic assessments, a review of two principled assessment design
frameworks, and a discussion of key validity frameworks. In Chapter 3 we discuss theoretical
underpinnings and motivations for using DCMs to analyze diagnostic assessment data, which
vii
viii Preface
are framed by key characteristics of DCMs, extended in discussions of diagnostic rules based
on empirical evidence and standard-setting, and rounded off by contexts for which DCM-
based analyses are appropriate. In Chapter 4, we discuss one key aspect of DCMs, namely,
how they measure the attributes of interests. We discuss the nature of the attributes that they
are measuring, how attribute relationships are represented in attribute hierarchies, how at-
tribute profi les arising from DCMs should be reported, and how cognitive processing models
for identifying attributes can be developed in practice.
The “Methods” Section
In Part II, “Methods,” we focus in detail on the statistical characteristics of a wide variety of
DCMs. Specifi cally, in Chapter 5 we discuss the relationship of DCMs with other latent-
variable models by juxtaposing key characteristics of these models, presenting a detailed
taxonomy of DCMs based on these key characteristics, and discussing related classifi cation
approaches. In Chapter 6 we present the model structure of six core DCMs organized accord-
ing to compensatory and noncompensatory models, where compensation type is one rele-
vant characteristic of DCMs. For each model we present the model equation, explain all
components in the equation in detail, and show, via real-life applications, what parameter
estimates for these models look like. In Chapter 7 we present a more general modeling frame-
work for DCMs based on log-linear models. The framework allows us to specify not only the
DCMs that we presented in Chapter 6 but also a wide variety of additional DCMs. In Chapter
8 we discuss different ways of simplifying the estimation of the attribute parameters in DCMs
that represent the base rates of attribute mastery and correlations between attributes in a
population of respondents, which is referred to as the structural parameterization of DCMs.
This chapter concludes the middle section of the book, which focuses on the mathematical
structure of DCMs and their relationship to other measurement models with latent
variables.
The “Applications” Section
In Part III, “Applications,” we explore different components that are essential for using DCMs
in practice. Specifi cally, in Chapter 9 we illustrate how DCMs can be estimated with the
software program Mplus (Muthén & Muthén, 1998–2010) within the common modeling
framework that is described in Chapter 7. We have selected this program because it is a very
fl exible, commercial general-purpose estimation program for latent-variable models, which
already contains a wide variety of advanced routines for estimating DCMs. To illustrate the
concepts presented in the following chapters we frequently refer back to examples calibrated
with Mplus to make the links between Chapter 9 and the other chapters explicit.
In Chapter 10 we discuss how to estimate the attribute profi les of the respondent based
on the item and structural parameters in DCMs. We present two key methods for classifying
the respondents within a Bayesian estimation framework, and we discuss the advantages and
disadvantages of these methods along with their sensitivity to different estimation settings.
Moreover, we illustrate how the estimation of attribution profi les is practically done in Mplus