Table Of Content

'I AppMluiletiMdve athrioadtes fDoarAta n alysts DalEl.aJ so hnson KansSatsaU tnei versity DuxburyP ress AnI mproifBn rto oksP/uCbolliseCh oimnpga ny ® I(I)P AnI nternaTthioomnsaPolun b lishing Company PaciGfirco v•eA lba•nB ye lmo•n Bto nn•B ost•oC ni ncin•n Daettir• oJ ioth anne•sL bounrdgo n Madr•i Mde lbou•rn Mee xiCciot •y NeYwo r•k P ar•iS si ngap•To orkey •oT orontWoa sh•i ngton SponsorinCga rEoCdlriyotnco kre:t t PermiEsdsiitoMonarsyC: l ark MarkeTteianMmga: r Pceyr mMaanr,g /a'raertk s CovDeers iRgonRy:. N euhaus EditAosrsiiaslKt iamnbteR:ra bl1y1 m TypeseStpteicntPgru:ub mlS ieshrevri ces ProduEcdtiitoNonar n:Vc eyl tlraus Cover PCroiunrtiienrg : ManusEcdriitpLotor r:rP eatltaag i PrinatniBdni gn dCionugr:i er COPYRIGH1T9©9b 8yB rooks/ColCeo mPpuabnlyi shing A DiviosfIi notne rnaTthioomnsPaoulnb liIsnhci.n g Id)P ThIeT lPo giaosr egisttreardeeudms aehrdek r uenidnle irc ense. DuxbPurreyas nstd h lee laofga ort er adeumsaehrdek rsue nidnle irc ense. Fomro rien formcaotnitoanc,t : BROOKS/CPOULBEL ISHING InternTahtoimosEnodanil t ores COMPANY Sene5c3a 51F1o rLeosdtRg oea d CoPlo.l anco PacGirfiocv CeA9, 3 950 115M6e0x iDc.Fo .,M, e xico USA InternTahtoimosPnouanbl l iGsmhbiHn g InternaTthioomnsPaoulnb liEsuhrionpge KonigswSitnrta4es1rs8ee r BerksHhoiur1se6e 8 -173 532B2o7n n HigHho lborn Germany LondWoCnl AVA 7 England International TAhsoimas on Publishing 22H1e ndeRrosaodn ThomNaesl sAouns tralia #05-H1e0n deBrusiolnd ing 10D2o dSdts reet Singa0p3o1r5e SouMtehl bou3rn2e0,5 VictAoursitar,a lia InternTahtoimosPnouanbl l iJsahpianng HirakaKwyaocwhao B3uFi lding, NelsCoann ada 2-2H-i1r akawacho 112B0i rchmount Road ChiyodTao-kky1uo0, 2 ScarboOrnotuagrhi,o Japan CanaMdlaK5 G4 Alrli grhetsse Nrovp eador.ftt hwiosr mka by er eprodsutcoierndaer , de trsiyesvtoaerlm , transcirnai nfbyoe rdom,rb ya nmye ans-elemcetcrhoapnnhiioccta,lo ,c orepcyoirndgi,n g, oro therwiset-hpweri itwohrroi utptte ernm iosfts hipeou nb liBsrhoeork,sP /uCbollies hing CompaPnayc,Gi rfiocv Cea,l ifornia 93950. Prinitnte hUden itSetdao tfAe mse rica. 10 9 8 7 6 5 4 3 2 1 LibroafCr oyn grCeastsa Joging-Diant-aP ublication JohnDsaolnEl,.a (,sd ate] Applmiueldt ivarifaotdrea tamane atlIhyD osadtlsEsl . a s Johnson. p.cm. Inclbuidbelsi ogrreafpehriecniacnled se xa.n d ISB0 -534-23796-7 1.MultivaanrailayItsT.eii st.l e. QA278.1J969185 5!9.'553-dc21 974-7133 CIP Contents 1. APPLIED MULTIVARIATE METHODS 1 1.1 An Overview of Multivariate Methods 1 2 Variable- and Individual-Directed Techniques Creating New Variables 2 Principal Components Analysis 3 Factor Analysis 3 Discriminant Analysis 4 Canonical Discriminant Analysis 5 Logistic Regression 5 Cluster Analysis 5 Multivariate Analysis of Variance 6 Canonical Variates Analysis 7 Canonical Correlation Analysis 7 Where to Find the Preceding Topics 7 1.2 Two Examples 8 Independence of Experimental Units 11 1.3 Types of Variables 11 u 1.4 Data Matrices and Vectors Variable Notation 13 Data Matrix 13 Data Vectors 13 Data Subscripts 14 1.5 The Multivariate Normal Distribution 15 Some Definitions 15 Summarizing Multivariate Distributions 16 Mean Vectors and Variance-Covariance Matrices 16 Correlations and Correlation Matrices 17 The Multivariate Normal Probability Density Function 19 Bivariate Normal Distributions 19 iii iv Contents 1.6S tatiCsotmipcuatli2 n2g Cautions About Computer Usage 22 Missing Values 22 Replacing Missing Values by Zeros 23 Replacing Missing Values by Averages 23 Removing Rows of the Data Matrix 23 Sampling Strategies 24 Data Entry Errors and Data Verification 24 1.7M ultivOaurtilait2ee5r s Locating Outliers 25 Dealing with Outliers 25 Outliers May Be Influential 26 1.8M ultivSaurmimaSattreay t is2t6i cs 1.9S tandaDradtaianz deZ/dS o cro re2s7 Exercises 28 2. SAMPLE CORRELATIONS 35 2.1S tatiTsetsaitncsCd ao ln fidIenntceer v3a5l s Are the Correlations Large Enough to Be Useful? 36 Confidence Intervals by the Chart Method 36 Confidence Intervals by Fisher's Approximation 38 Confidence Intervals by Ruben's Approximation 39 Variable Groupings Based on Correlations 40 Relationship to Factor Analysis 46 2.2S ummar4y6 Exercises 47 3. MULTIVARIATE DATA PLOTS 55 3.1T hree-DiDmaetnPasl ioot5ns5a l 3.2P lootfHs i ghDeirm ensDiaotnaa5 l9 Chernoff Faces 61 Star Plots and Sun-Ray Plots 63 Contents V AndrePwlso't 6s5 Side-bSyc-aStPitldeoert 6 s6 3.3 Plotting to Check for Multivariate Normality 67 Summar7y3 Exerci7s3e s 4. EIGENVALUES AND EIGENVECTORS 77 4.1 Trace and Determinant 77 Exampl7e8s 4.2 Eigenvalues 78 4.3 Eigenvectors 79 PosiDteifivneai ntPdeo siSteimvied eMfiantirtiec8 e0s 4.4 Geometric Descriptions (p = 2) 82 Vecto8r2s BivarNioartmDeai ls tribu8t3i ons 4.5 Geometric Descriptions (p = 3) 87 Vector8s7 TrivaNroiramDtaiels tribu8t7i ons 4.6 Geometric Descriptions (p > 3) 90 Summar9y1 Exerci9s1e s 5. PRINCIPAL COMPONENTS ANALYSIS 93 5.1 Reasons for Using Principal Components Analysis 93 DatSac reen9i3n g Cluster9i5n g DiscriAmnianlayns9ti5 s Regress9i5o n 5.2 Objectives of Principal Components Analysis 96 5.3 Principal Components Analysis on the Variance-Covariance Matrix I 96 PrinCcoimppaoln Secnotr e9s8 ComponLeonatdV iencgt or9s8 Vi Contents 5.4 Estimation of Principal Components 99 EstimoafPt riionnCc oimppaolSn ceonrte 9s9 5.5 Determining the Number of Principal Components 99 Meth1o d1 00 Meth2o d1 00 5.6 Caveats 107 5.7 PCA on the Correlation Matrix P 109 PrinCcoimppaolSn ceonrte 1s1 0 CompoCnoernrte Vleacttioo1rn1s 0 SampCloer reMlaattriio1xn1 0 DetermtihNneui mnbgoe fPr r inCcoimppaoln e1n1t0s 5.8 Testing for Independence of the Original Variables 111 5.9 Structural Relationships 111 5.10 Statistical Computing Packages 112 SARS Proced1u1r2e PRINCOMP PrinCcoimppaolnA ennatlUsys siiFnsag c Atnoarl ysis Progra1m1s8 PCAw itShP SS's Proced1u2r4e FACTOR Summar1y4 2 Exerci1s4e2s 6. FACTOR ANALYSIS 147 6.1 Objectives of Factor Analysis 147 6.2 Caveats 148 6.3 Some History of Factor Analysis 148 6.4 The Factor Analysis Model 150 Assumpt1i5o0n s MatFroirxom ft hFea cAtnoarl Myosdiesl1 51 DefiniotfFi aocnAtsno arl Tyesrimsi no1l5o1g y 6.5 Factor Analysis Equations 151 Nonuniqoufte hnFeea scst o1r5s2 6.6 Solving the Factor Analysis Equations 153 Contents VII 6.7 ChoostihnAegp propNruimabtoeefrF actor1s5 5 Subjective Criteria l56 Objective Criteria 156 6.8 CompuStoelru toifto hnFesa ctAonra lyEsqiusa tio1n5s7 Principal Factor Method on R 158 Principal Factor Method with Iteration 159 6.9 RotatFiancgt or1s7 0 Examples (m= 2) 171 Rotation Methods 172 The Varimax Rotation Method 173 6.10O bliRqoutea tMieotnh od1s7 4 6.11F actSocro re1s8 0 Bartlett's Method or the Weighted Least-Squares Method 181 Thompson's Method or the Regression Method 181 Ad Hoc Methods 181 Summary 212 Exercises 213 7.D ISCRIMAINNAALNYTS IS 217 7.1 DiscrimfionTraw toMi uolnt ivNaorrimaPatolep ulati2o1n7s A Likelihood Rule 218 The Linear Discriminant Function Rule 218 A Mahalanobis Distance Rule 218 A Posterior Probability Rule 218 Sample Discriminant Rules 219 Estimating Probabilities of Misclassification 220 Resubstitution Estimates 220 Estimates from Holdout Data 220 Cross-Validation Estimates 221 7.2 CosFtu nctainoPdnr si Porro bab(iTlwiPoto ipeusl ati2o2n9s ) 7.3 A GeneDriaslc rimRiunl(aeTn wtPo o pulati2o3n1s ) A Cost Function 232 Prior Probabilities 232 Viii Contents Average Cost of Misclassification 232 A Bayes Rule 233 Classification Functions 233 Unequal Covariance Matrices 233 Tricking Computing Packages 234 7.4 Discriminant Rules (More than Two Populations) 235 Basic Discrimination 238 7.5 Variable Selection Procedures 245 Forward Selection Procedure 245 Backward Elimination Procedure 246 Stepwise Selection Procedure 246 Recommendations 247 Caveats 247 7.6 Canonical Discriminant Functions 255 The First Canonical Function 256 A Second Canonical Function 257 Determining the Dimensionality of the Canonical Space 260 Discriminant Analysis with Categorical Predictor Variables 273 7.7 Nearest Neighbor Discriminant Analysis 275 7.8 Classification Trees 283 Summary 283 Exercises 283 8. LOGISTIC REGRESSION METHODS 287 8.1 Logistic Regression Model 287 8.2 The Logit Transformation 287 Model Fitting 288 8.3 Variable Selection Methods 296 8.4 Logistic Discriminant Analysis (More Than Two Populations) 301 Logistic Regression Models 301 Model Fitting 302 I Another SAS LOGISTIC Analysis 314 Exercises 316 ConteinXt s CLUSTER ANALYSIS 319 9. 9.1 Measures of Similarity and Dissimilarity 319 Ruler Distance 319 Standardized Ruler Distance 320 A Mahalanobis Distance 320 Dissimilarity Measures 320 9.2 Graphical Aids in Clustering 321 Scatter Plots 321 Using Principal Components 322 Andrews' Plots 322 Other Methods 322 9.3 Clustering Methods 322 Nonhierarchical Clustering Methods 323 Hierarchical Clustering 323 Nearest Neighbor Method 323 A Hierarchical Tree Diagram 325 Other Hierarchical Clustering Methods 326 Comparisons of Clustering Methods 327 Verification of Clustering Methods 327 How Many Clusters? 327 Beale's F-TyStpatiestic 328 A Pseudo Hotelling's T2T est 329 The Cubic Clustering Criterion 329 Clustering Order 334 Estimating the Number of Clusters 339 Principal Components Plots 348 Clustering with SPSS 355 SAS's Procedure 369 FASTCLUS 9.4 Multidimensional Scaling 385 Exercises 395 10. MEAN VECTORS AND VARIANCE-COVARIANCE MATRICES 397 10.1 Inference Procedures for Variance-Covariance Matrices 397 A Test for a Specific Variance-Covariance Matrix 398 A Test for Sphericity 400