Handbook of Multimethod Measurement in Psychology

446 Pages·2006·51.075 MB·English

by  PsycBooks (Ovid)

Handbook of Multimethod Measurement in Psychology

Editors: Eid, Michael & Diener, Ed
Hardcover: 553 pages
Publisher: American Psychological Association (APA)
Publication Date: January, 2006
Language: English
ISBN-10: 1591473187
ISBN-13: 978-1591473183

Additional Information
Categories: Psychology: Assessment, Testing & Measurement
Subjects: Research Psychology

Summary:

Handbook of Multimethod Measurement in Psychology is the first integrative
guide to theoretical, methodological, and applied aspects of multimethod
measurement in psychological research. Although the multitrait-multimethod
analysis and multimethod research strategies have become increasingly important in
all areas of psychological research, comprehensible handbooks of multimethod
measurement that integrate all phases of the data-gathering and assessment process
have been missing. Handbook of Multimethod Measurement in Psychology fills this
long-standing gap. Researchers and students of research who want to run a
multimethod study or who want to understand a published study no longer must
page through methodological journals trying to piece together an understanding of
multimethod approaches. This volume provides a single, understandable resource
for learning the meaning, advantages, and limitations of different methods.

The volume brings together outstanding researchers from all areas of psychology.
Contributors give detailed explanations about exactly what has to be considered
when conducting a multimethod study, what the strength of the convergent validity
and method-specificity in different areas of research is, and what methods should be
selected for proving validity in specific research contexts. The volume discusses
theoretical concepts, then lays out the most important measurement tools—
including, among many others, global self-assessment, informant assessment,
observational methods, and nonreactive methods—followed by statistical models
for analyzing multitrait-multimethod data. Contributors also explore applications of
multimethod research in different areas of psychology as well as broader
perspectives on the meaning and purpose of multimethod approaches.

Destined to be a classic, Handbook of Multimethod Measurement in Psychology
provides an indispensable resource for researchers and students in psychology
seeking to enrich their work by using multimethod approaches.
CONTENTS

Preface

I. Theoretical Concepts

1. Introduction: The need for multimethod measurement in psychology
2. Conceptual, theoretical, and historical foundations of multimethod assessment

II. Assessment Methods

3. Global self-assessment
4. Informant assessment
5. Momentary capture of real-world data
6. Web based methods
7. Computer-based testing
8. Ability tests
9. Catching the mind in action: Implicit methods in personality research and assessment
10. Sequential observational methods
11. Quantitative text analysis
12. Multilevel Analysis: Physiological and biochemical methods
13. Brain imaging and related methods
14. Nonreactive methods in psychological research
15. Experimental methods of psychological assessment

III. Methods of Data Analysis

16. Methodological approaches for analyzing multimethod data
17. Assessing multimethod association with categorical variables
18. Multimethod item response theory
19. Multilevel models for multimethod measurements
20. Structural equation models for multitrait-multimethod data
21. Longitudinal methods

IV. Applications

22. Using multiple methods in personality psychology
23. Measuring emotions: Implications of a multi-method perspective
24. The evolution of concepts in research on human memory
25. Applying a multi-method perspective to the study of developmental psychology
26. Multimethod approaches in social psychology: Between and within method replication and
multimethod assessment
27. Clinical psychology: Construct validation with multiple sources of information and multiple
settings
28. Multimethod Approaches in health psychology
29. Multiple methods in industrial and organizational psychology: Expanding “methods” to include
longitudinal designs
30. A Multiple Method Perspective on Self-concept Research in Educational Psychology: A
Construct Validity Approach
31. The Finale: Take-home messages from the editors
TABLE OF CONTENTS

SECTION I:
Theoretical Concepts

CHAPTER 1 CHAPTER 2
Eid, Michael; Diener, Ed Schmitt, Manfred
Introduction: The Need for Conceptual, Theoretical, and Historical
Multimethod Measurement in Foundations of Multimethod
Psychology Assessment
pp 3-8 pp 9-25

SECTION II:
Assessment Methods

CHAPTER 3 CHAPTER 8
Lucas, Richard E.; Baird, Brendan M. Lubinski, David
Global Self-Assessment Ability Tests
pp 29-42 pp 101-114

CHAPTER 4 CHAPTER 9
Neyer, Franz J. Robinson, Michael D.; Neighbors,
Informant Assessment Clayton
pp 43-59 Catching the Mind in Action: Implicit
Methods in Personality Research and
CHAPTER 5 Assessment
Stone, Arthur A.; Litcher-Kelly, pp 115-125
Leighann
Momentary Capture of Real-World CHAPTER 10
Data Bakeman, Roger; Gnisci, Augusto
pp 61-72 Sequential Observational Methods
pp 127-140
CHAPTER 6
Reips, Ulf-Dietrich CHAPTER 11
Web-Based Methods Mehl, Matthias R.
pp 73-85 Quantitative Text Analysis
pp 141-156
CHAPTER 7
Drasgow, Fritz; Chuah, Siang Chee
Computer-Based Testing
pp 87-100
CHAPTER 12 CHAPTER 14
Berntson, Gary G.; Cacioppo, John T. Fritsche, Immo; Linneweber, Volker
Multilevel Analysis: Physiological and Nonreactive Methods in Psychological
Biochemical Measures Research
pp 157-172 pp 189-203

CHAPTER 15
CHAPTER 13 Erdfelder, Edgar; Musch, Jochen
Zald, David H; Curtis, Clayton Experimental Methods of Psychological
Brain Imaging and Related Methods Assessment
pp 173-187 pp 205-220

SECTION III:
Methods of Data Analysis
CHAPTER 16 CHAPTER 19
Eid, Michael Hox, Joop; Maas, Cora Multilevel
Methodological Approaches for Models for Multimethod
Analyzing Multimethod Data Measurements
pp 223-230 pp 269-281

CHAPTER 17 CHAPTER 20
Nussbeck, Fridtjof W. Assessing Eid, Michael; Lischetzke, Tanja;
Multimethod Association With Nussbeck, Fridtjof W.
Categorical Variables Structural Equation Models for
pp 231-247 Multitrait-Multimethod Data
pp 283-299
CHAPTER 18
Rost, Jurgen; Walter, Oliver CHAPTER 21
Multimethod Item Response Theory Khoo, Siek-Toon; West, Stephen G;
pp 249-268 Wu, Wei; Kwok, Oi-Man
Longitudinal Methods
pp 301-317
SECTION IV:
Applications

CHAPTER 22 CHAPTER 27
Roberts, Brent W; Harms, Peter; Burns, G. Leonard; Haynes, Stephen N.
Smith, Jennifer L; Wood, Dustin; Clinical Psychology: Construct
Webb, Michelle Validation With Multiple Sources of
Using Multiple Methods in Personality Information and Multiple Settings
Psychology pp 401-418
pp 321-335
CHAPTER 28
CHAPTER 23 Knauper, Barbel; Klein, Rupert
Larsen, Randy J; Prizmic-Larsen, Multimethod Approaches in Health
Zvjezdana Psychology
Measuring Emotions: Implications of a pp 419-427
Multimethod Perspective
pp 337-351 CHAPTER 29
Miner, Andrew G.; Hulin, Charles L.
CHAPTER 24 Multimethods in Industrial and
Benjamin, Aaron S Multimethod Organizational Psychology: Expanding
Approaches to the Study of Cognition: "Methods" to Include Longitudinal
The Evolution of Concepts in Research Designs
on Human Memory pp 429-439
pp 353-369
CHAPTER 30
CHAPTER 25 Marsh, Herbert W.; Martin, Andrew J.;
Morris, Amanda Sheffield; Robinson, Hau, Kit-Tai
Lara R; Eisenberg, Nancy Applying a A Multimethod Perspective on Self-
Multimethod Perspective to the Study Concept Research in Educational
of Developmental Psychology Psychology: A Construct Validity
pp 371-384 Approach
pp 441-456
CHAPTER 26
Smith, Richard H.; Harris, Monica J. CHAPTER 31
Multimethod Approaches in Social Diener, Ed; Eid, Michael
Psychology: Between- and Within- The Finale: Take-Home Messages From
Method Replication and Multimethod the Editors
Assessment pp 457-463
pp 385-400
CHAPTER 1
INTRODUCTION: THE NEED
FOR MULTIMETHOD MEASUREMENT
IN PSYCHOLOGY
Michael Bid and Ed Diener
We must measure what can be measured, and make measurable what cannot be measured.
—Galileo Galilei (1610)
The history of empirical psychology is one of mak- And there are previously popular methods (e.g.,
ing measurable what most laypeople think cannot unobtrusive and nonreactive measures), which must
be measured—emotions, cognitions, motivations, be preserved and revived as valuable measurement
personality traits, and so forth. As in all other tools that offer insights not otherwise obtained.
empirical and natural sciences, the progress of psy- Although each epoch has its own scientific para-
chology is closely and inextricably linked to the digms and methods that fit better than other meth-
development of new and more refined methods for ods, it would be unwise to stake scientific insight
measuring psychological concepts. New technical on just one. A multimethod approach offers
developments (e.g., modern methods of brain imag- insights into scientific phenomena and can con-
ing or biochemical analyses) allow deeper insights tribute to confirming psychological theories in a
into psychological processes than ever expected at way a single-method approach cannot (Schmitt, this
the end of the 19th century when psychology began volume, chap. 2). There are at least two reasons
establishing itself as an empirical science. Modern why psychology research and applied work make a
computer technology enables traditional psycholog- multimethod research necessary: the multicompo-
ical methods (e.g., ability testing, behavior observa- nential structure of psychological phenomena and
tion, text analysis, and reaction time measures) to the validity of a research program.
come into full flower and makes it possible to real-
ize old dreams like measuring individuals in their
MULTICOMPONENTIAL AND MULTILEVEL
everyday lives far away from the anonymity and
STRUCTURE OF PSYCHOLOGICAL
artificiality of the psychological laboratory. Modern
PHENOMENA
communication tools like the Internet make it pos-
sible to conduct experiments around the world Psychological phenomena usually consist of many
across borders closed to researchers. facets. Emotions, for example, refer not only to the
Besides the rapid progress that technological conscious representation of the feeling itself ("I am
revolutions cause throughout the sciences, each happy") but also to many other changes in the indi-
science has its classical standard measures that with- vidual (Davidson, Goldsmith, & Scherer, 2003;
stand the tide of technological progress, almost Kahneman, Diener, & Schwarz, 1999; Larsen &
unchanged. Just as a medical doctor does not aban- Prizmic-Larsen, this volume, chap. 23). An individ-
don listening to lung sounds with instruments that ual feeling happiness might jump up, feel an inner
have not fundamentally changed over the years, vir- ease, and even embrace and kiss passersby. More-
tually no psychologist discards the treasure chests of over, muscle changes in the face might accompany
self- and informant-reports, even though the way he this feeling, the brain might produce endorphins,
or she uses them also remains virtually unchanged. and the individual may likely entertain positive
Eid and Diener
thoughts. This simple example shows that a psy- For example, while two individuals may feel pride
chological phenomenon has many facets. To under- after receiving a compliment, one might jump for
stand these emotional reactions it is necessary to joy, while the other quietly sits down. Analyzing
have various, appropriate research methods to ana- individual differences in the associations between
lyze the diverse facets. A method that measures the components might reveal that the first person
muscle movements in the face is not appropriate to grew up in a culture in which pride is a highly
assess endorphin levels in the brain. Moreover, a appreciated emotion (e.g., the United States),
method for determining the endorphin level is whereas the other was raised in a culture in which
probably not useful for assessing subjective feelings. pride is undesirable and should not be expressed,
A thorough understanding of an emotional reaction for example in East Asian cultures (e.g., Eid &
requires a set of appropriate multimethod tools to Diener, 2001). Hence, combining multimethod
measure the different facets of the phenomena. approaches for analyzing individual differences in
Multilevel analyses are a related example of the covariation of different components of a multi-
multimethod research programs. Berntson and component phenomenon may help us understand
Cacioppo (this volume, chap. 12) define multilevel individual and social regulation processes.
analyses as a subset of multimethod approaches These simple examples show that a multimethod
where the measures, constructs, and theories research program is necessary for a thorough
extend across levels of organization—from the psy- description of multicomponent phenomena, as well
chological to the physiological to the cellular and as for analyzing the different components of phe-
ultimately to the gene and beyond.1 To assess the nomena to detect general and individual rules of
different levels, different methods are needed. behavior. A classic example of multimethod
Hence, the first aim of using multimethod research strategies is Murray's (1938) famous
approaches is the precise description of the multi- Explorations in Personality, where he used such
component and multilevel phenomena that are the diverse methods as aptitude tests, projective tests,
focus of the behavioral sciences. questionnaires, interviews, and so forth to learn
A second aim of multimethod research is provid- more about the different components of the person-
ing information for detecting general associations ality of the participants of his study.
between different components and levels of a phe-
nomenon. For example, to analyze the reasons why
VALIDITY
happy individuals might be healthier, research must
show a link between the feeling component and rel- Validity, one of the key issues of research, concerns
evant physiological measures that explain individ- the question whether the inferences drawn from the
ual differences in health. Insight into these results of a study are true or not (Shadish, Cook, &
processes can be obtained by multimethod research Campbell, 2002). In particular, with respect to
programs. However, general relations between the measurement methods, validity represents the
different components form only one side of the degree to which the adequacy and appropriateness
coin. Beyond general associations individual differ- of inferences and actions based on the results of a
ences must be considered because not all individu- measurement device are supported by empirical evi-
als behave in the same way. If an emotional reaction dence and theoretical rationales (Messick, 1989).
were patterned in a uniform way, measuring one Multimethod research plays a key role in the valida-
component would suffice when predicting other tion process. In their groundbreaking article, "Con-
components. However, strong individual differences vergent and discriminant validation by the
often exist when exploring different components. multitrait-multimethod matrix," Campbell and
'In this handbook the term multilevel analyses will also be used for statistical methods for analyzing nested data (e.g., students nested within
classes, measurement occasions nested within individuals, etc). There are strong differences between multilevel analyses as a research program for
measuring different determinants of behavior and multilevel analysis as statistical method. However, the appropriate meaning will be clearly deter-
mined by the context.
Introduction
Fiske (1959) described the cornerstones of a multi- or severity errors and halo effects can affect peer rat-
trait-multimethod research program regarding the ings, and peer- as well as self-rating might also be
validation process. The basic promises of the multi- distorted by social desirability effects (Neyer, this
trait-multimethod approach have strongly influ- volume, chap. 4). All these forms of bias and distor-
enced the process of exploring validity. First, tion can cause small convergent validity coefficients.
Campbell and Fiske pointed out that several meth- Therefore, Westen and Rosenthal (2003) recom-
ods are needed to appropriately analyze validity, mend quantifying construct validity by comparing
and these different methods should converge in the the observed patterns of correlations with the theo-
measurement of the same trait. The convergence of retically expected patterns of correlations. They con-
different independent methods indicates convergent tend that if a good theoretical reason for expecting
validity. Second, they convincingly argued that dis- lower correlations between multiple measures exists,
criminant validity must be shown before introducing and this pattern of correlations can be empirically
a new construct into science. Third, Campbell and confirmed, modest degrees of convergence can con-
Fiske clarified that a score on a psychological vari- firm construct validity.
able not only reflects the psychological construct High convergent validity is not always the goal
under consideration, but also reflects systematic of research. Take, for example, a questionnaire
method-specific influences. Fourth, they demon- measuring different facets of marital satisfaction.
strated the necessity of including at least two differ- Spouses rate their own satisfaction and also their
ent methods in psychological studies to separate perception of the satisfaction of their spouse. If the
trait from method influences. Hence, for a complete aim of the test construction process was to develop
understanding of psychological processes it is nec- a questionnaire that detects deficiencies in
essary to apply a multimethod research strategy. intraspouse perception and communication
Therefore, the multitrait-multimethod analysis has processes, the items with the lowest convergences
become an essential strategy for proving the con- might be the most interesting. In other words,
struct validity of psychological measures. method influences are not inevitably unwanted ran-
Convergent validity is a core aspect of validity, dom disturbances (e.g., measurement error) but
and validation research programs have been focused they can indicate valid and valuable information. A
for a long time on seeking high convergent validity deeper understanding of method influences can
coefficients. Although high validity coefficients are enlarge our knowledge of the construct under con-
desirable many reasons explain why convergent sideration, and this knowledge, in turn, can help
validity coefficients are often lower than hoped. For explain method effects, correct for method effects,
example, if one compares physiological measures and plan and conduct studies in which method
with other measures one must contend with individ- effects are minimized or—depending on the aim of
ual response-uniqueness (e.g., Berntson & the study considered—maximized. Beyond the tra-
Cacioppo, 2004). Not all individuals react to stimu- ditional search for maximum convergent validity, a
lus in the same way, and this response specificity thorough analysis of method influences might tell a
can lower convergence when measured with a corre- more interesting story of the construct under con-
lation coefficient. Moreover, if one wants to compare sideration. Hence, a multimethod study should
a self-rating with a peer-rating, one often uncovers always have two facets: first, the proof of conver-
medium-sized correlation coefficients. In comparing gent validity on the basis of theoretical expecta-
self- and other-ratings one must recognize rater tions, and second, the analysis of the nature of
biases (Hoyt, 2000). Raters may not only interpret method-specific influences. Whereas multimethod
scale items differently but might also have opportu- studies intend to meet the first goal, the second
nities to observe different behavior, they might use goal is often not considered when planning the
different indicators of behavior, and they might link study's design. A careful analysis of method effects
the indicators to the response scale in a different requires the inclusion of variables that may explain
way (Hoyt, 2000; Kenny, 1991). Moreover, leniency method influences, and that might suppress
Eid and Diener
method-specific effects to enhance convergent This selection was guided by the goal of presenting
validity. This makes a thorough knowledge of meas- methods at the basic level, which can be combined
urement methods necessary for all researchers. to understand more complex assessment strategies.
For example, you will not find a chapter on inter-
AIMS OF THIS HANDBOOK AND OVERVIEW viewing, as an interview situation combines many
more basic methods (e.g., self-report, observational
Because multimethod research and assessment
methods, informant assessment, and text-analysis of
strategies are superior to monomethod approaches,
the transcript of the interview). The handbook
this handbook aims to provide the reader with the
focuses more on assessment methods than on
necessary knowledge to plan and conduct multi-
research methods in general. It follows the tradition
method studies and to analyze multimethod data.
of Campbell and Fiske's (1959) multimethod
We present the most important measurement meth-
approach; therefore, we present methods that can
ods and show their applications (Part II). Further-
be used to measure human behavior, attitudes, and
more, we present the most up-to-date methods for
feeling. We do not present research methods to test
analyzing multimethod data (Part III). Finally, sev-
theories without assessing humans (e.g., animal
eral chapters discuss applications of multimethod
studies and computer simulation techniques).
research programs in different areas of psychology
These methods may hold importance for multi-
(Part IV). These chapters show how multimethod
method research in general, but they are less signifi-
research programs can be successfully applied and
cant for assessment purposes. We use the term
discuss problems related to the implementation of
multimethod, in most cases, in the sense of applying
multimethod strategies. Although these chapters
different methods for measuring human beings
focus on single subdisciplines of psychology, they
although some chapters also refer to multimethod
also discuss issues relevant to other fields. Because
research programs in the sense of applying different
the ways multimethod strategies are implemented
research strategies (e.g., experimental vs. nonexper-
differ between subdisciplines, we strongly recom-
imental research).
mend consulting applications chapters in domains
Moreover, we focus on widely applied and estab-
different from those in which one is primarily inter-
lished methods, including more recent develop-
ested. Examining methods in other subdisciplines
ments (like brain imaging). Some new methods
gives readers new ideas about how to improve their
may have a high potential for psychological assess-
own research and how to develop new and innova-
ment and measurement but are less established,
tive research programs.
with a status more comparable to research methods.
Some of these methods include virtual environment
Part I: Theoretical Concepts
technology (Blascovich et al., 2002) and molecular
The first part of the handbook introduces the genetic analysis (e.g., Caspi et al., 2003) and are
importance of multimethod assessment in psychol- not considered in this volume.
ogy, the philosophical and historical aspects of mul- The handbook covers the most basic assessment
timethod research strategies, and the different methods that are relevant for a thorough under-
concepts of consistency and method specificity. standing of human behavior, attitudes, and feelings.
These include self-report (Lucas & Baird, this vol-
Part II: Assessment Methods ume, chap. 3), informant assessment (Neyer, this
The second part presents the current state of the volume, chap. 4), ability tests (Lubinski, this vol-
art—the most important measurement methods in ume, chap. 8), implicit methods (Robinson &
modern psychology, which comprise traditional Neighbors, this volume, chap. 9), observational
methods like self-report as well as the most recent methods (Bakeman & Gnisci, this volume, chap.
developments in brain imaging and Web-based 10), physiological and biochemical methods
methods. Not all methods applied in psychology (Berntson & Cacioppo, this volume, chap. 12),
can be presented, and selections had to be made. functional neuroimaging (Zald & Curtis, this vol-