Linear Models and the Relevant Distributions and Matrix Algebra

539 Pages·2018·5.04 MB·English

by  David A. Harville

Linear Models and the
Relevant Distributions
and Matrix Algebra
CHAPMAN & HALL/CRC
Texts in Statistical Science Series
Series Editors
Joseph K. Blitzstein, Harvard University, USA
Julian J. Faraway, University of Bath, UK
Martin Tanner, Northwestern University, USA
Jim Zidek, University of British Columbia, Canada
Statistical Theory: A Concise Introduction Problem Solving: A Statistician’s Guide,
F. Abramovich and Y. Ritov Second Edition
C. Chatfield
Practical Multivariate Analysis, Fifth Edition
A. Afifi, S. May, and V.A. Clark Statistics for Technology: A Course in Applied
Statistics, Third Edition
Practical Statistics for Medical Research
C. Chatfield
D.G. Altman
Analysis of Variance, Design, and Regression :
Interpreting Data: A First Course
Linear Modeling for Unbalanced Data,
in Statistics
Second Edition
A.J.B. Anderson
R. Christensen
Introduction to Probability with R
K. Baclawski Bayesian Ideas and Data Analysis: An
Introduction for Scientists and Statisticians
Linear Algebra and Matrix Analysis for
R. Christensen, W. Johnson, A. Branscum,
Statistics
and T.E. Hanson
S. Banerjee and A. Roy
Modelling Binary Data, Second Edition
Modern Data Science with R
D. Collett
B. S. Baumer, D. T. Kaplan, and N. J. Horton
Modelling Survival Data in Medical Research,
Mathematical Statistics: Basic Ideas and
Third Edition
Selected Topics, Volume I,
D. Collett
Second Edition
P. J. Bickel and K. A. Doksum Introduction to Statistical Methods for
Clinical Trials
Mathematical Statistics: Basic Ideas and
T.D. Cook and D.L. DeMets
Selected Topics, Volume II
P. J. Bickel and K. A. Doksum Applied Statistics: Principles and Examples
Analysis of Categorical Data with R D.R. Cox and E.J. Snell
C. R. Bilder and T. M. Loughin Multivariate Survival Analysis and Competing
Statistical Methods for SPC and TQM Risks
D. Bissell M. Crowder
Introduction to Probability Statistical Analysis of Reliability Data
J. K. Blitzstein and J. Hwang M.J. Crowder, A.C. Kimber,
T.J. Sweeting, and R.L. Smith
Bayesian Methods for Data Analysis,
Third Edition An Introduction to Generalized
B.P. Carlin and T.A. Louis Linear Models, Third Edition
A.J. Dobson and A.G. Barnett
Second Edition
R. Caulcutt Nonlinear Time Series: Theory, Methods, and
Applications with R Examples
The Analysis of Time Series: An Introduction,
R. Douc, E. Moulines, and D.S. Stoffer
Sixth Edition
C. Chatfield Introduction to Optimization Methods and
Their Applications in Statistics
Introduction to Multivariate Analysis
B.S. Everitt
C. Chatfield and A.J. Collins
Extending the Linear Model with R: Mathematical Statistics
Generalized Linear, Mixed Effects and K. Knight
Nonparametric Regression Models,
Introduction to Functional Data Analysis
Second Edition
P. Kokoszka and M. Reimherr
J.J. Faraway
Introduction to Multivariate Analysis:
Linear Models with R, Second Edition
Linear and Nonlinear Modeling
J.J. Faraway
S. Konishi
A Course in Large Sample Theory
Nonparametric Methods in Statistics with SAS
T.S. Ferguson
Applications
Multivariate Statistics: A Practical O. Korosteleva
Approach
Modeling and Analysis of Stochastic Systems,
B. Flury and H. Riedwyl
Third Edition
Readings in Decision Analysis V.G. Kulkarni
S. French
Exercises and Solutions in Biostatistical Theory
Discrete Data Analysis with R: Visualization L.L. Kupper, B.H. Neelon, and S.M. O’Brien
and Modeling Techniques for Categorical and
Exercises and Solutions in Statistical Theory
Count Data
L.L. Kupper, B.H. Neelon, and S.M. O’Brien
M. Friendly and D. Meyer
Design and Analysis of Experiments with R
Markov Chain Monte Carlo:
J. Lawson
Stochastic Simulation for Bayesian Inference,
Design and Analysis of Experiments with SAS
Second Edition
J. Lawson
D. Gamerman and H.F. Lopes
A Course in Categorical Data Analysis
Bayesian Data Analysis, Third Edition
T. Leonard
A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson,
A. Vehtari, and D.B. Rubin Statistics for Accountants
S. Letchford
Multivariate Analysis of Variance and
Repeated Measures: A Practical Approach for Introduction to the Theory of Statistical
Behavioural Scientists Inference
D.J. Hand and C.C. Taylor H. Liero and S. Zwanzig
Practical Longitudinal Data Analysis Statistical Theory, Fourth Edition
D.J. Hand and M. Crowder B.W. Lindgren
Linear Models and the Relevant Distributions Stationary Stochastic Processes: Theory and
and Matrix Algebra Applications
D.A. Harville G. Lindgren
Logistic Regression Models Statistics for Finance
J.M. Hilbe E. Lindström, H. Madsen, and J. N. Nielsen
Richly Parameterized Linear Models: The BUGS Book: A Practical Introduction to
Additive, Time Series, and Spatial Models Bayesian Analysis
Using Random Effects D. Lunn, C. Jackson, N. Best, A. Thomas, and
J.S. Hodges D. Spiegelhalter
Statistics for Epidemiology Introduction to General and Generalized
N.P. Jewell Linear Models
H. Madsen and P. Thyregod
Stochastic Processes: An Introduction,
Third Edition Time Series Analysis
P.W. Jones and P. Smith H. Madsen
The Theory of Linear Models Pólya Urn Models
B. Jørgensen H. Mahmoud
Pragmatics of Uncertainty Randomization, Bootstrap and Monte Carlo
J.B. Kadane Methods in Biology, Third Edition
B.F.J. Manly
Principles of Uncertainty
J.B. Kadane Statistical Regression and Classification: From
Linear Models to Machine Learning
Graphics for Statistics and Data Analysis with R
N. Matloff
K.J. Keen
Introduction to Randomized Controlled Statistical Methods for Spatial Data Analysis
Clinical Trials, Second Edition O. Schabenberger and C.A. Gotway
J.N.S. Matthews
Bayesian Networks: With Examples in R
Statistical Rethinking: A Bayesian Course with M. Scutari and J.-B. Denis
Examples in R and Stan
Large Sample Methods in Statistics
R. McElreath
P.K. Sen and J. da Motta Singer
Statistical Methods in Agriculture and
Introduction to Statistical Methods for
Experimental Biology, Second Edition
Financial Models
R. Mead, R.N. Curnow, and A.M. Hasted
T. A. Severini
Statistics in Engineering: A Practical Approach
Spatio-Temporal Methods in Environmental
A.V. Metcalfe
Epidemiology
Statistical Inference: An Integrated Approach, G. Shaddick and J.V. Zidek
Second Edition
Decision Analysis: A Bayesian Approach
H. S. Migon, D. Gamerman, and
J.Q. Smith
F. Louzada
Analysis of Failure and Survival Data
Beyond ANOVA: Basics of Applied Statistics
P. J. Smith
R.G. Miller, Jr.
Applied Statistics: Handbook of GENSTAT
A Primer on Linear Models Analyses
J.F. Monahan
E.J. Snell and H. Simpson
Stochastic Processes: From Applications to
Applied Nonparametric Statistical Methods,
Theory
Fourth Edition
P.D Moral and S. Penev
P. Sprent and N.C. Smeeton
Applied Stochastic Modelling, Second Edition
Data Driven Statistical Methods
B.J.T. Morgan
P. Sprent
Elements of Simulation
Generalized Linear Mixed Models:
B.J.T. Morgan
Modern Concepts, Methods and Applications
Probability: Methods and Measurement W. W. Stroup
A. O’Hagan
Survival Analysis Using S: Analysis of
Introduction to Statistical Limit Theory Time-to-Event Data
A.M. Polansky M. Tableman and J.S. Kim
Applied Bayesian Forecasting and Time Series Applied Categorical and Count Data Analysis
Analysis W. Tang, H. He, and X.M. Tu
A. Pole, M. West, and J. Harrison
Elementary Applications of Probability Theory,
Statistics in Research and Development, Second Edition
Time Series: Modeling, Computation, and H.C. Tuckwell
Inference
Introduction to Statistical Inference and Its
R. Prado and M. West
Applications with R
Essentials of Probability Theory for M.W. Trosset
Statisticians
Understanding Advanced Statistical Methods
M.A. Proschan and P.A. Shaw
P.H. Westfall and K.S.S. Henning
Introduction to Statistical Process Control
Statistical Process Control: Theory and
P. Qiu
Practice, Third Edition
Sampling Methodologies with Applications G.B. Wetherill and D.W. Brown
P.S.R.S. Rao
Generalized Additive Models:
A First Course in Linear Model Theory An Introduction with R, Second Edition
N. Ravishanker and D.K. Dey S. Wood
Essential Statistics, Fourth Edition Epidemiology: Study Design and
D.A.G. Rees Data Analysis, Third Edition
M. Woodward
Stochastic Modeling and Mathematical
Statistics: A Text for Statisticians and Practical Data Analysis for Designed
Quantitative Scientists Experiments
F.J. Samaniego B.S. Yandell
Texts in Statistical Science
Linear Models and the
Relevant Distributions
and Matrix Algebra
David A. Harville
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2018 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
Version Date: 20180131
International Standard Book Number-13: 978-1-138-57833-3 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity
of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized
in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying,
microfilming, and recording, or in any information storage or retrieval system, without written permission from the
publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Names: Harville, David A., author.
Title: Linear models and the relevant distributions and matrix algebra /
David A. Harville.
Description: Boca Raton : CRC Press, 2018. | Includes bibliographical
references and index.
Identifiers: LCCN 2017046289 | ISBN 9781138578333 (hardback : alk. paper)
Subjects: LCSH: Matrices--Problems, exercises, etc. | Mathematical
statistics--Problems, exercises, etc.
Classification: LCC QA188 .H3798 2018 | DDC 512.9/434--dc23
LC record available at https://lccn.loc.gov/2017046289
Visit the e-resources at: https://www.crcpress.com/9781138578333
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Preface ix
1 Introduction 1
1.1 LinearStatisticalModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 RegressionModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 ClassificatoryModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 HierarchicalModelsandRandom-EffectsModels . . . . . . . . . . . . . . . . . 7
1.5 StatisticalInference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.6 AnOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2 MatrixAlgebra:APrimer 23
2.1 TheBasics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2 PartitionedMatricesandVectors . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3 Traceofa(Square)Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4 LinearSpaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.5 InverseMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.6 RanksandInversesofPartitionedMatrices . . . . . . . . . . . . . . . . . . . . . 44
2.7 OrthogonalMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.8 IdempotentMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.9 LinearSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.10 GeneralizedInverses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.11 LinearSystemsRevisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.12 ProjectionMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.13 QuadraticForms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.14 Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
BibliographicandSupplementaryNotes . . . . . . . . . . . . . . . . . . . . . . 85
3 RandomVectorsandMatrices 87
3.1 ExpectedValues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.2 Variances,Covariances,andCorrelations . . . . . . . . . . . . . . . . . . . . . . 89
3.3 StandardizedVersionofaRandomVariable . . . . . . . . . . . . . . . . . . . . 97
3.4 ConditionalExpectedValuesandConditionalVariancesandCovariances . . . . 100
3.5 MultivariateNormalDistribution . . . . . . . . . . . . . . . . . . . . . . . . . . 101
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
BibliographicandSupplementaryNotes . . . . . . . . . . . . . . . . . . . . . . 122
4 TheGeneralLinearModel 123
4.1 SomeBasicTypesofLinearModels . . . . . . . . . . . . . . . . . . . . . . . . 124
4.2 SomeSpecificTypesofGauss–MarkovModels(withExamples) . . . . . . . . . 129
4.3 Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
4.4 HeteroscedasticandCorrelatedResidualEffects . . . . . . . . . . . . . . . . . . 136
4.5 MultivariateData . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
viii Contents
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
BibliographicandSupplementaryNotes . . . . . . . . . . . . . . . . . . . . . . 162
5 EstimationandPrediction:ClassicalApproach 165
5.1 LinearityandUnbiasedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5.2 TranslationEquivariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
5.3 Estimability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.4 TheMethodofLeastSquares . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
5.5 BestLinearUnbiasedorTranslation-EquivariantEstimationofEstimableFunctions
(undertheG–MModel) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
5.6 SimultaneousEstimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
5.7 EstimationofVariabilityandCovariability . . . . . . . . . . . . . . . . . . . . . 198
5.8 Best(Minimum-Variance)UnbiasedEstimation . . . . . . . . . . . . . . . . . . 211
5.9 Likelihood-BasedMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
5.10 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
BibliographicandSupplementaryNotes . . . . . . . . . . . . . . . . . . . . . . 252
6 SomeRelevantDistributionsandTheirProperties 253
6.1 Chi-Square,Gamma,Beta,andDirichletDistributions . . . . . . . . . . . . . . 253
6.2 NoncentralChi-SquareDistribution . . . . . . . . . . . . . . . . . . . . . . . . 267
6.3 CentralandNoncentralF Distributions . . . . . . . . . . . . . . . . . . . . . . 281
6.4 Central,Noncentral,andMultivariatet Distributions . . . . . . . . . . . . . . . 290
6.5 MomentGeneratingFunctionoftheDistributionofOneorMoreQuadraticForms
orSecond-DegreePolynomials(inaNormallyDistributedRandomVector) . . . 303
6.6 Distribution of Quadratic Forms or Second-Degree Polynomials (in a Normally
DistributedRandomVector):Chi-Squareness . . . . . . . . . . . . . . . . . . . 308
6.7 The Spectral Decomposition, with Application to the Distribution of Quadratic
Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
6.8 MoreontheDistributionofQuadraticFormsorSecond-DegreePolynomials(ina
NormallyDistributedRandomVector) . . . . . . . . . . . . . . . . . . . . . . . 326
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
BibliographicandSupplementaryNotes . . . . . . . . . . . . . . . . . . . . . . 349
7 ConfidenceIntervals(orSets)andTestsofHypotheses 351
7.1 “SettingtheStage”: ResponseSurfacesintheContextofaSpecificApplicationand
inGeneral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
7.2 AugmentedG–MModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
7.3 TheF Test(andCorrespondingConfidenceSet)andaGeneralizedS Method . . 364
7.4 SomeOptimalityProperties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
7.5 One-Sidedt TestsandtheCorrespondingConfidenceBounds . . . . . . . . . . . 421
7.6 TheResidualVariance2: ConfidenceIntervalsandTestsofHypotheses . . . . 430
7.7 Multiple Comparisons and Simultaneous Confidence Intervals: Some Enhance-
ments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
7.8 Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
BibliographicandSupplementaryNotes . . . . . . . . . . . . . . . . . . . . . . 502
References 505
Index 513
Preface
Linearstatisticalmodelsprovidethetheoreticalunderpinningsformanyofthestatisticalprocedures
in common use. In deciding on the suitability of one of those procedures for use in a potential
application, it would seem to be important to know the assumptions embodied in the underlying
modelandthetheoreticalpropertiesoftheprocedureasdeterminedonthebasisofthatmodel.In
fact, the valueof such knowledgeis notlimited to its value in decidingwhetheror notto use the
procedure.When(asisfrequentlythecase)oneormoreoftheassumptionsappeartobeunrealistic,
suchknowledgecanbeveryhelpfulindevisingasuitablymodifiedprocedure—asituationofthis
kindisillustratedinSection7.7f.
Knowledgeofmatrixalgebrahasineffectbecomeaprerequisiteforreadingmuchoftheliterature
pertainingtolinearstatisticalmodels.Theuseofmatrixalgebrainthisliteraturestartedtobecome
commonplaceinthemid1900s.AmongtheearlyadopterswereScheffé(1959),Graybill(1961),Rao
(1965),andSearle(1971).Whenitcomestoclarityandsuccinctnessofexposition,theintroduction
of matrixalgebrarepresenteda greatadvance.However,thosewithoutan adequateknowledgeof
matrixalgebrawereleftataconsiderabledisadvantage.
Amongtheproceduresformakingstatisticalinferencesareonesthatarebasedonanassumption
thatthedatavectoristherealizationofarandomvector,sayy,thatfollowsalinearstatisticalmodel.
Thepresentvolumediscussesproceduresofthatkindandthepropertiesofthoseprocedures.Included
inthecoveragearevariousresultsfrommatrixalgebraneededtoeffectanefficientpresentationofthe
proceduresandtheirproperties.Alsoincludedinthecoveragearetherelevantstatisticaldistributions.
Someofthesupportingmaterialonmatrixalgebraandstatisticaldistributionsisinterspersedwith
thediscussionoftheinferentialproceduresandtheirproperties.
Twoclassicalproceduresaretheleastsquaresestimator(ofanestimablefunction)andtheF test.
TheleastsquaresestimatorisoptimalinthesensedescribedinaresultknownastheGauss–Markov
theorem.TheGauss–Markovtheoremhasarelativelysimpleproof.Resultsontheoptimalityofthe
F testarestatedandprovedherein(inChapter7); theproofsoftheseresultsarerelativelydifficult
andless“accessible”—referenceissometimesmadetoWolfowitz’s(1949)proofsofresultsonthe
optimalityoftheF test,whichare(atbest)extremelyterse.
TheF testisvalidunderanassumptionthatthedistributionoftheobservablerandomvectoryis
multivariatenormal.However,thatassumptionisstrongerthannecessary.Ascanbediscernedfrom
resultslikethosediscussedbyFang,Kotz,andNg(1990),ashasbeenpointedoutbyRavishanker
and Dey (2002, sec. 5.5), and is shown herein, the F test and various related proceduresdepend
ony onlythrougha(possiblyvector-valued)functionofy whosedistributionisthesameforevery
distributionofythatis“ellipticallysymmetric,”sothatthoseproceduresarevalidnotonlywhenthe
distributionofy ismultivariatenormalbutmoregenerallywhenthedistributionofy iselliptically
symmetric.
Thepresentvolumeincludesconsiderablediscussionofmultiplecomparisonsandsimultaneous
confidenceintervals.Atonetime,theuseofthesekindsofprocedureswasconfinedtosituationswhere
therequisitepercentagepointswerethoseofadistribution(likethedistributionoftheStudentized
range) that was sufficiently tractable that the percentage points could be computed by numerical
means. The percentagepointscouldthen be tabulated or could be recomputedon an “as needed”
basis. An alternative whose use is not limited by considerations of “numerical tractability” is to