An Introduction to Applied Multivariate Analysis

498 Pages·2008·2.08 MB·English

by  Tenko Raykov

Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page i 2.2.2008 2:54pm CompositorName:BMani
An Introduction
to Applied
Multivariate
Analysis
Tenko Raykov George A. Marcoulides
New York London
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page ii 2.2.2008 2:54pm CompositorName:BMani
Routledge Routledge
Taylor & Francis Group Taylor & Francis Group
270 Madison Avenue 2 Park Square
New York, NY 10016 Milton Park, Abingdon
Oxon OX14 4RN
© 2008 by Taylor & Francis Group, LLC
Routledge is an imprint of Taylor & Francis Group, an Informa business
Printed in the United States of America on acid-free paper
10 9 8 7 6 5 4 3 2 1
International Standard Book Number-13: 978-0-8058-6375-8 (Hardcover)
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, trans-
mitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter
invented, including photocopying, microfilming, and recording, or in any information storage or retrieval
system, without written permission from the publishers.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Introduction to applied multivariate analysis / by Tenko Raykov & George A.
Marcoulides.
p. cm.
Includes bibliographical references and index.
ISBN-13: 978-0-8058-6375-8 (hardcover)
ISBN-10: 0-8058-6375-3 (hardcover)
1. Multivariate analysis. I. Raykov, Tenko. II. Marcoulides, George A.
QA278.I597 2008
519.5’35--dc22 2007039834
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the Psychology Press Web site at
http://www.psypress.com
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page iii 2.2.2008 2:54pm CompositorName:BMani
Contents
Preface...............................................................................................................ix
Chapter 1 Introduction to Multivariate Statistics
1.1 Definition of Multivariate Statistics .............................................1
1.2 Relationship of Multivariate Statistics
to Univariate Statistics..................................................................5
1.3 Choice of Variables and Multivariate Method,
and the Concept of Optimal Linear Combination.......................7
1.4 Data for Multivariate Analyses ....................................................8
1.5 Three Fundamental Matrices in Multivariate Statistics............11
1.5.1 Covariance Matrix......................................................................12
1.5.2 Correlation Matrix......................................................................13
1.5.3 Sums-of-Squares and Cross-Products Matrix........................15
1.6 Illustration Using Statistical Software........................................17
Chapter 2 Elements of Matrix Theory
2.1 Matrix Definition.........................................................................31
2.2 Matrix Operations, Determinant, and Trace..............................33
2.3 Using SPSS and SAS for Matrix Operations..............................46
2.4 General Form of Matrix Multiplications With Vector,
and Representation of the Covariance, Correlation,
and Sum-of-Squares and Cross-Product Matrices.....................50
2.4.1 Linear Modeling and MatrixMultiplication..........................50
2.4.2 ThreeFundamental Matricesof Multivariate Statistics
in Compact Form.......................................................................51
2.5 Raw Data Points in Higher Dimensions, and Distance
Between Them..............................................................................54
Chapter 3 Data Screening and Preliminary Analyses
3.1 Initial Data Exploration...............................................................61
3.2 Outliers and the Search for Them...............................................69
3.2.1 Univariate Outliers.....................................................................69
3.2.2 Multivariate Outliers.................................................................71
3.2.3 Handling Outliers: ARevisit....................................................78
3.3 Checking of Variable Distribution Assumptions.......................80
3.4 Variable Transformations............................................................83
iii
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page iv 2.2.2008 2:54pm CompositorName:BMani
iv
Chapter 4 Multivariate Analysis of Group Differences
4.1 A Start-Up Example ....................................................................99
4.2 A Definition of the Multivariate Normal Distribution............101
4.3 Testing Hypotheses About a Multivariate Mean.....................102
4.3.1 The Case of Known CovarianceMatrix................................103
4.3.2 The Case of Unknown CovarianceMatrix...........................107
4.4 Testing Hypotheses About Multivariate Means of
Two Groups...............................................................................110
4.4.1 Two Relatedor MatchedSamples
(Change OverTime)................................................................110
4.4.2 Two Unrelated(Independent) Samples................................113
4.5 Testing Hypotheses About Multivariate Means
in One-Way and Higher Order Designs (Multivariate
Analysis of Variance, MANOVA).............................................116
4.5.1 Statistical Significance VersusPracticalImportance...........129
4.5.2 Higher OrderMANOVA Designs.........................................130
4.5.3 Other Test Criteria...................................................................132
4.6 MANOVA Follow-Up Analyses...............................................143
4.7 Limitations and Assumptions of MANOVA............................145
Chapter 5 Repeated Measure Analysis of Variance
5.1 Between-Subject and Within-Subject Factors
and Designs................................................................................148
5.2 Univariate Approach to Repeated Measure Analysis.............150
5.3 Multivariate Approach to Repeated Measure Analysis..........168
5.4 Comparison of Univariate and Multivariate Approaches
to Repeated Measure Analysis..................................................179
Chapter 6 Analysis of Covariance
6.1 Logic of Analysis of Covariance...............................................182
6.2 Multivariate Analysis of Covariance........................................192
6.3 Step-Down Analysis (Roy–Bargmann Analysis).....................198
6.4 Assumptions of Analysis of Covariance..................................203
Chapter 7 Principal Component Analysis
7.1 Introduction...............................................................................211
7.2 Beginnings of Principal Component Analysis.........................213
7.3 How Does Principal Component Analysis Proceed?...............220
7.4 Illustrations of Principal Component Analysis .......................224
7.4.1 Analysis of the Covariance Matrix S (S)of the
Original Variables....................................................................224
7.4.2 Analysis of the Correlation Matrix P(R) of the
Original Variables....................................................................224
7.5 UsingPrincipalComponentAnalysisinEmpiricalResearch.........234
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page v 2.2.2008 2:54pm CompositorName:BMani
v
7.5.1 Multicollinearity Detection.....................................................234
7.5.2 PCA With Nearly UncorrelatedVariablesIs
Meaningless...............................................................................235
7.5.3 Can PCABe Used asaMethod for Observed Variable
Elimination?..............................................................................236
7.5.4 WhichMatrixShouldBe Analyzed?.....................................236
7.5.5 PCA as a HelpfulAid in Assessing Multinormality..........237
7.5.6 PCA as‘‘Orthogonal’’ Regression.........................................237
7.5.7 PCA Is Conductedvia FactorAnalysis Routinesin
Some Software..........................................................................237
7.5.8 PCA as a Rotation of Original Coordinate Axes.................238
7.5.9 PCA as a Data Exploratory Technique.................................238
Chapter 8 Exploratory Factor Analysis
8.1 Introduction...............................................................................241
8.2 Model of Factor Analysis..........................................................242
8.3 How Does Factor Analysis Proceed?........................................248
8.3.1 FactorExtraction......................................................................248
8.3.1.1 Principal Component Method.................................248
8.3.1.2 MaximumLikelihood Factor Analysis...................256
8.3.2 FactorRotation.........................................................................262
8.3.2.1 Orthogonal Rotation.................................................266
8.3.2.2 Oblique Rotation.......................................................267
8.4 Heywood Cases.........................................................................273
8.5 Factor Score Estimation.............................................................273
8.5.1 Weighted Least SquaresMethod
(Generalized LeastSquares Method)....................................274
8.5.2 Regression Method..................................................................274
8.6 Comparison of Factor Analysis and Principal
Component Analysis.................................................................276
Chapter 9 Confirmatory Factor Analysis
9.1 Introduction...............................................................................279
9.2 A Start-Up Example ..................................................................279
9.3 Confirmatory Factor Analysis Model.......................................281
9.4 Fitting Confirmatory Factor Analysis Models.........................284
9.5 ABriefIntroductiontoMplus,andFittingthe
ExampleModel.........................................................................................287
9.6 Testing Parameter Restrictions in Confirmatory Factor
Analysis Models........................................................................298
9.7 Specification Search and Model Fit Improvement...................300
9.8 Fitting Confirmatory Factor Analysis Models to the
Mean and Covariance Structure...............................................307
9.9 Examining Group Differences on Latent Variables.................314
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page vi 2.2.2008 2:54pm CompositorName:BMani
vi
Chapter 10 Discriminant Function Analysis
10.1 Introduction.............................................................................331
10.2 What Is Discriminant Function Analysis?..............................332
10.3 Relationship of Discriminant Function Analysis to Other
Multivariate Statistical Methods.............................................334
10.4 Discriminant Function Analysis With Two Groups..............336
10.5 Relationship Between Discriminant Function and
Regression Analysis With Two Groups.................................351
10.6 Discriminant Function Analysis With More Than
Two Groups.............................................................................353
10.7 Tests in Discriminant Function Analysis ...............................355
10.8 Limitations of Discriminant Function Analysis.....................364
Chapter 11 Canonical Correlation Analysis
11.1 Introduction.............................................................................367
11.2 How Does Canonical Correlation Analysis Proceed? ...........370
11.3 Tests and Interpretation of Canonical Variates.....................372
11.4 Canonical Correlation Approach to Discriminant
Analysis....................................................................................384
11.5 Generality of Canonical Correlation Analysis.......................389
Chapter 12 An Introduction to the Analysis
of Missing Data
12.1 Goals of Missing Data Analysis..............................................391
12.2 Patterns of Missing Data.........................................................392
12.3 Mechanisms of Missing Data..................................................394
12.3.1 Missing Completely atRandom........................................396
12.3.2 Missing at Random..............................................................398
12.3.3 Ignorable Missingness and Nonignorable
Missingness Mechanisms....................................................400
12.4 Traditional Ways of Dealing With Missing Data ...................401
12.4.1 Listwise Deletion..................................................................402
12.4.2 Pairwise Deletion.................................................................402
12.4.3 Dummy VariableAdjustment............................................403
12.4.4 Simple Imputation Methods...............................................403
12.4.5 Weighting Methods.............................................................405
12.5 Full Information Maximum Likelihood
and Multiple Imputation.........................................................406
12.6 Examining Group Differences and Similarities
in the Presence of Missing Data...............................................407
12.6.1 Examining Group Mean Differences With
IncompleteData...................................................................410
12.6.2 Testing for GroupDifferences in the Covariance
and Correlation MatricesWith Missing Data..................427
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page vii 2.2.2008 2:54pm CompositorName:BMani
vii
Chapter 13 Multivariate Analysis of Change Processes
13.1 Introduction.............................................................................433
13.2 Modeling Change Over Time With Time-Invariant
and Time-Varying Covariates.................................................434
13.2.1 Intercept-and-Slope Model.................................................435
13.2.2 Inclusion of Time-Varying and Time-Invariant
Covariates..............................................................................436
13.2.3 An Example Application.....................................................437
13.2.4 Testing Parameter Restrictions...........................................442
13.3 Modeling General Forms of Change Over Time.....................448
13.3.1 Level-and-Shape Model.......................................................448
13.3.2 Empirical Illustration...........................................................450
13.3.3 Testing Special Patterns of Growthor Decline................455
13.3.4 Possible Causes of Inadmissible Solutions.......................459
13.4 Modeling Change Over Time With Incomplete Data............461
Appendix: Variable Naming and Order for Data Files............467
References..........................................................................................469
Author Index.....................................................................................473
Subject Index.....................................................................................477
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page viii 2.2.2008 2:54pm CompositorName:BMani
Raykov/IntroductiontoAppliedMultivariateAnalysis RT20712_C000 FinalProof page ix 2.2.2008 2:54pm CompositorName:BMani
Preface
Having taught applied multivariate statistics for a number of years, we
have been impressed by the broad spectrum of topics that one may be
expectedtotypicallycoverinagraduatecourseforstudentsfromdepart-
ments outside of mathematics and statistics. Multivariate statistics has
developed over the past few decades into a very extensive field that is
hard to master in a single course, even for students aiming at methodo-
logical specialization in commonly considered applied fields, such as
those within the behavioral, social, and educational disciplines. To meet
this challenge, we tried to identify a core set of topics in multivariate
statistics, which would be both of fundamental relevance for its under-
standing and at the same time would allow the student to move on to
more advanced pursuits.
This book is a result of this effort. Our goal is to provide a coherent
introduction to applied multivariate analysis, which would lay down the
basics of the subject that we consider of particular importance in many
empirical settings in the social and behavioral sciences. Our approach is
based in part on emphasizing, where appropriate, analogies between
univariate statistics and multivariate statistics. Although aiming, in prin-
ciple,atarelativelynontechnical introductiontothesubject,wewerenot
able to avoid the use of mathematical formulas, but we employ these
primarily in their definitional meaning rather than as elements of proofs
orrelatedderivations.Thetargetedaudiencewhowillfindthisbookmost
beneficialconsists primarilyofgraduatestudents,advancedundergradu-
ate students, and researchers in the behavioral, social, as well as educa-
tional disciplines, who have limited or no familiarity with multivariate
statistics. As prerequisites for this book, an introductory statistics course
with exposure to regression analysis is recommended, as is some famil-
iarity with twoofthemostwidelycirculated statistical analysissoftware:
SPSS and SAS.
Without the use of computers, we find that an introduction to applied
multivariate statistics is not possible in our technological era, and so we
employ extensively these popular packages, SPSS and SAS. In addition,
forthepurposesofsomechapters,weutilizethelatentvariablemodeling
program Mplus, which is increasingly used across the social and behav-
ioralsciences.Onthebookspecificwebsite,www.psypress.com=applied-
multivariate-analysis, we supply essentially all data usedin the text. (See
Appendixfornameofdatafileandofitsvariables,aswellastheirorderas
ix