Generalizability Theory

543 Pages·2001·11.573 MB·English

by  Robert L. Brennan (auth.)

Statistics for Social Science
and Public Policy
Advisors:
S.E.Fienberg D.Lievesley 1.Rolph
Springer-Verlag Berlin Heidelberg GmbH
Statistics forSocial Science andPublic Policy
Brennan:GeneralizabilityTheory.
Devlin/Fienberg/Resnick/Raeder(Eds.):Intelligence, Genes,andSuccess: Scientists
RespondtoTheBellCurve.
Finkelstein/Ievin:StatisticsforLawyers,SecondEdition.
Gastwirth(Ed.):StatisticalScienceintheCourtroom.
HandcockiMorris:RelativeDistributionMethodsintheSocialSciences.
JohnsoniAlbert: OrdinalDataModeling.
Morton/Rolph: PublicPolicyandStatistics:CaseStudiesfromRAND.
Zeisel/Kaye:ProveItwithFigures:EmpiricalMethodsinLawandLitigation.
Robert L. Brennan
Generalizability Theory
Springer
RobertL. Brennan
IowaTestingPrograms
UniversityofIowa
IowaCity, IA52242-1529
USA
Advisors:
StephenE.Fienberg DeniseLievesley
DepartmentofStatistics Institutefor Statistics
CarnegieMellon University Room H.113
Pittsburgh, PA 15213 UNESCO
USA 7 Placede Fontenoy
75352 Paris 07 SP
France
lohn Rolph
DepartmentofInformation and
Operations Management
Graduate SchoolofBusiness
UniversityofSouthern California
LosAngeles, CA90089
USA
LibraryofCongressCataloging-in-PublicationData
Brennan,Robert L.
Generalizabilitytheory/Robert L. Brennan
p. cm.- (Statisticsforsocial science andpublic policy)
Includes bibliographiealreferences(p. )and indexes.
I. Psychometries. 2. Psychology-Statisticalmethods. 3. Analysisof
variance. 1. Title. H. Series.
BF39.B755 2001
150'.1'5I95-dc2I 2001032009
Printedonacid-free paper.
©2001Springer-VerlagBerlinHeidelberg
OriginallypublishedbySpringer-VerlagBerlinHeidelbergNewYorkin200I.
SoftcoverreprintofthehardcoverIstedition200I
All rights reserved.This work may not be translated or copied in whole or in part without the
written permission of the publisher (Springer-Verlag Berlin Heidelberg GmbH),
except for brief excerpts in connection with reviews or scholarly
analysis. Use in connection with any form of information storage and retrieval, electronic
adaptation,computersoftware,or bysimilarordissimilarmethodology now known or hereafter
developedisforbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even
if the former are not especially identified, is not to be taken as a sign that such names, as
understoodbytheTrade Marks and Merchandise MarksAct, may accordinglybeused freely by
anyone.
ProductionmanagedbyAllanAbrams; manufacturingsupervisedbyJeffrey Taub.
Photocomposedcopy preparedbytheauthor using It<TEX.
987 654 3 2 I
ISBN978-1-4419-2938-9 ISBN978-1-4757-3456-0(eBook)
DOI10.1007/978-1-4757-3456-0
Springer-Verlag NewYork Berlin Heidelberg
Amember0/BertelsmannSpringerScience+BusinessMedia GmbH
To Cicely
Preface
In 1972 a monograph by Cronbach, Gleser, Nanda, and Rajaratnam was
published entitled The Dependability of Behavioral Measurements. That
bookincorporated,systematized,and extended their previousresearchinto
what came to be called generalizability theory, which liberalizes classical
test theory, in part through the application of analysis of variance proce
dures that focusonvariancecomponents.Generalizabilitytheoryisperhaps
the most broadly defined measurement model currently in existence, and
the Cronbach et al. (1972) treatment ofthe theory represents a major con
tribution to psychometrics. However,as Cronbach et al. (1972, p. 3) state,
their book is "complexlyorganized and by no meanssimpleto follow" and,
ofcourse, it is nearly 30years old.
In 1983, ACT, Inc. published my monograph entitled Elements of Gen
eralizability Theory,withaslightlyrevised version appearingin 1992.That
treatment is considerably less comprehensive than Cronbach et al. (1972)
but still detailed enough to convey much ofthe richness ofthe theory and
to facilitate its application. However, the 1983/1992 monograph is essen
tiallytwodecadesold,it does not cover multivariategeneralizability theory
in depth,and it does not incorporate recent developments instatisticsthat
bear upon the estimation of variance components. Also, of course, there
have been numerous developments in generalizability theory in the last 20
years.
This book provides a much more comprehensive and up-to-date treat
ment of generalizability theory.It covers allofthe major topics that have
been discusscd in generalizability theory, as weIlas some new ones. In ad-
viii Preface
dition, it provides asynthesisofthose partsofthe statistiealliteraturethat
are directly applicable to generalizability theory.
The principalintendedaudience ismeasurement practitionersand upper
level graduate students in the behavioral and social sciences, particularly
education and psychology.Generalizability theory has broader applicabil
ity, however. Indeed, it might be used in virtually any field that attends
to measurements and their errors. Readers willbenefit from some familiar
ity with classieal test theory and analysis of variance, but the treatment
of most topics does not presume specific background. In particular, vari
ance components are a central focus of generalizability theory, but it is
not assumed that readers are familiar with them or with procedures for
estimating them.
Although the statistieal aspects of generalizability theory are undeni
ably important,perhaps the most distinguishingfeature ofthe theory isits
conceptual framework, whieh permits a multifaceted perspective on mea
surement error and its components. What makes generalizability theory
both challenging and useful is that it marries this rieh conceptual frame
work with powerful, but sometimes complicated, statistieal procedures.
This book gives substantial attention to both aspects of generalizability
theory-the conceptual framework and the statistieal machinery. However,
the book per seisneither a treatise on the philosophy ofmeasurement, nor
a textbook on statistieal procedures. Rather, it integrates those parts of
both topics that bear upon generalizability theory.
Precursors to generalizability theory are evident in papers written as
long aga as the 1930s.However,generalizability theory per se is relatively
new, it is evolving, and there are a few somewhat different perspectives
on the theory. Most ofthese perspectives are complementary, or might be
viewedasspecial casesorextensions.Evenso,Ijudgedit necessary to adopt
one principal perspective and maintain it throughout this book. That per
spective is closelyaligned with Cronbach et al. (1972), but there are some
occasional differences. For example, except for the last chapter, this book
does not emphasize regressed score estimates of universe scores nearly as
muchas Cronbach et al. (1972).Also,thereare somenotationaldifferences,
especially in those chapters that treat multivariate generalizability theory.
There are three sets ofchapters in this book. They are ordered in terms
of increasing complexity. The fundamentals of univariate generalizability
theory are contained in Chapters 1to 4. They might be used as part of a
graduate--levelcourseinadvanced measurement.Additional, morechalleng
ingtopies in univariatetheoryare coveredin Chapters 5to 8,and Chapters
9to 12provide myownperspectiveon multivariate generalizabilitytheory.
The treatment of multivariate generalizability theory is inspired by the
workof Cronbach et al. (1972), but there are notieeable differencesin em
phasis, coverage, and notational conventions. I have tried to provide the
reader with different ways of thinking about multivariate generalizability
theory,and I have tried to illustrate its similarities to and differencesfrom
Preface
IX
univariate theory. An important goal of this book is to make multivariate
generalizability theory more accessible to practitioners.
More consideration is given to reliability-likecoefficients than is necessi
tated by the theory. However, in my experience, many students and mea
surement practitioners have great difficulty, at least initially, in appreciat
ing the applicability and usefulness of generalizability theory unless they
can relate some of its results to classical reliability coefficients. For this
reason, such coefficients are actively considered, although the magnitudes
of variance components, and particularly error variances,are clearly more
important.
Manyofthetopicscoveredherecould betreated usingmatrixoperators.
With the exception of one appendix, however, matrix operators are not
employed, because doingso would render the content inaccessible to many
students and practitioners who might benefit from the theory.
I am grateful to ACT, Inc., for permitting me to use parts of Brennan
(1992a). That monograph clearly infiuenced my treatment of Chapters 2
to 5 and several appendices. Also, Chapter 1is largely a revised version of
Brennan (1992b) used with the permission of the publisher, the National
Council on Measurement in Education, and parts of Section 5.4 are from
Brennan (1998) used with permission of the publisher, Sage. I am also
grateful to ACT, Inc. for permitting me access to ACT Assessment data
used for various multivariate examples in the later chapters of this book,
toSuzanneLanefor permitting meto usethe QUASAR data referenced in
Section 5.4, to Clare Kreiter for the opportunity to analyze data discussed
in Section 8.3, and to Judy Ru at Iowa Testing Programs (ITP) for her
assistance with ITP data.
I especiallywant to acknowledgethe considerablebenefit I havereceived
over the last 30 years from numerous communications with Lee Cronbach.
Also, I am particularly gratefulto Michael Kane, whose research,insights,
criticisms, and support have contributed greatly to my own thinking, re
search, and writings about generalizability theory. I have benefited as weil
from joint research with Xiaohong Gao, especially in the area of perfor
mance assessments.
Others who have infiuenced my workinclude David Jarjoura, Joe Crick,
Richard Shavelson, Noreen Webb, Gerald Gillmore, and Dean Colton. Fi
nally,Iwant to thank mystudents, especiallyWon-Chan Lee,Scott Bishop,
Guemin Lee, Dong-In Kim, Janet Mee, Ping Yin, and Steven Rattenborg.
They have assisted me in numerous ways. In particular, their questions
and comments have often infiuenced how I think about and present the
theory. My thanks to all of them. Finally, I am grateful to my secretary,
Sue Wollrab,for her help in preparing the manuscript.
Iowa City, IA Robert L. Brennan
January, 2001
Contents
Preface vii
Principal Notational Conventions xix
1 Introduction 1
1.1 Framework of Generalizability Theory . . . . . . . . . . .. 4
1.1.1 Universe of Admissible Observations and G Studies. 5
1.1.2 Infinite Universe of Generalization and D Studies .. 8
1.1.3 Different Designs and/or Universesof Generalization 13
1.1.4 Other Issues and Applications . 17
1.2 Overview of Book . 18
1.3 Exercises .... . 19
2 Single-Facet Designs 21
2.1 G Study p x i Design . . . . . . . . . . . . . . . 22
2.2 G Study Variance Components for p x i Design 24
2.2.1 Estimating Variance Components . 25
2.2.2 Synthetic Data Example . 28
2.3 D Studies for the p x I Design 29
2.3.1 Error Variances . . . . . . 31
2.3.2 Coefficients.... .... 34
2.3.3 Synthetic Data Example . 35