Linear Algebra and Linear Models

176 Pages·2012·1.32 MB·English

by  R.B. Bapat (auth.)

Universitext
Universitext
SeriesEditors:
SheldonAxler
SanFranciscoStateUniversity,SanFrancisco,CA,USA
VincenzoCapasso
UniversitàdegliStudidiMilano,Milan,Italy
CarlesCasacuberta
UniversitatdeBarcelona,Barcelona,Spain
AngusMacIntyre
QueenMary,UniversityofLondon,London,UK
KennethRibet
UniversityofCalifornia,Berkeley,Berkeley,CA,USA
ClaudeSabbah
CNRS,ÉcolePolytechnique,Palaiseau,France
EndreSüli
UniversityofOxford,Oxford,UK
WojborA.Woyczynski
CaseWesternReserveUniversity,Cleveland,OH,USA
Universitext is a series of textbooks that presents material from a wide variety
of mathematical disciplines at master’s level and beyond. The books, often well
class-tested by their author, may have an informal, personal, even experimental
approach to their subject matter. Some of the most successful and established
books in the series have evolved through several editions, always following the
evolutionofteachingcurricula,intoverypolishedtexts.
Thus as research topics trickle down into graduate-level teaching, first textbooks
writtenfornew,cutting-edgecoursesmaymaketheirwayintoUniversitext.
Forfurthervolumes:
www.springer.com/series/223
R.B. Bapat
Linear Algebra
and Linear Models
Third Edition
Prof.R.B.Bapat
IndianStatisticalInstitute
NewDelhi
India
Aco-publicationwiththeHindustanBookAgency,NewDelhi,licensedforsaleinallcountriesoutside
ofIndia.SoldanddistributedwithinIndiabytheHindustanBookAgency,P19GreenParkExtn.,New
Delhi110016,India
©HindustanBookAgency2011
HBAISBN978-93-80250-28-1
ISSN0172-5939 e-ISSN2191-6675
Universitext
ISBN978-1-4471-2738-3 e-ISBN978-1-4471-2739-0
DOI10.1007/978-1-4471-2739-0
SpringerLondonDordrechtHeidelbergNewYork
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressControlNumber:2012931413
MathematicsSubjectClassification: 15A03,15A09,15A18,62J05,62J10,62K10
Firstedition:1993byHindustanBookAgency,Delhi,India
Secondedition:2000bySpringer-VerlagNewYork,Inc.,andHindustanBookAgency
©Springer-VerlagLondonLimited2012
Apartfromanyfairdealingforthepurposesofresearchorprivatestudy,orcriticismorreview,asper-
mittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,
storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublish-
ers,orinthecaseofreprographicreproductioninaccordancewiththetermsoflicensesissuedbythe
CopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethosetermsshouldbesentto
thepublishers.
Theuseofregisterednames,trademarks,etc.,inthispublicationdoesnotimply,evenintheabsenceofa
specificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandthereforefree
forgeneraluse.
Thepublishermakesnorepresentation,expressorimplied,withregardtotheaccuracyoftheinformation
containedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsoromissions
thatmaybemade.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Themainpurposeofthepresentmonographistoprovidearigorousintroductionto
thebasicaspectsofthetheoryoflinearestimationandhypothesistesting.Thenec-
essary prerequisites in matrices, multivariate normal distribution, and distribution
ofquadraticformsaredevelopedalongtheway.Themonographisprimarilyaimed
atadvancedundergraduateandfirst-yearmaster’sstudentstakingcoursesinlinear
algebra, linear models, multivariate analysis, and design of experiments. It should
alsobeofusetoresearchersasasourceofseveralstandardresultsandproblems.
Some features in which we deviate from the standard textbooks on the subject
areasfollows.
Wedealexclusivelywithrealmatrices,andthisleadstosomenonconventional
proofs.Oneexampleistheproofofthefactthatasymmetricmatrixhasrealeigen-
values. We rely on ranks and determinants a bit more than is done usually. The
developmentinthefirsttwochaptersissomewhatdifferentfromthatinmosttexts.
It is not the intention to give an extensive introduction to matrix theory. Thus,
severalstandardtopicssuchasvariouscanonicalformsandsimilarityarenotfound
here.Weoftenderiveonlythoseresultsthatareexplicitlyusedlater.Thelistoffacts
inmatrixtheorythatareelementary,elegant,butnotcoveredhereisalmostendless.
Weputagreatdealofemphasisonthegeneralizedinverseanditsapplications.
Thisamountstoavoidingthe“geometric”orthe“projections”approachthatisfa-
voredbysomeauthorsandtakingrecoursetoamorealgebraicapproach.Partlyasa
personalbias,Ifeelthatthegeometricapproachworkswellinprovidinganunder-
standingofwhyaresultshouldbetruebuthaslimitationswhenitcomestoproving
theresultrigorously.
Thefirstthreechaptersaredevotedtomatrixtheory,linearestimation,andtests
of linear hypotheses, respectively. Chapter 4 collects several results on eigenval-
uesandsingularvaluesthatarefrequentlyrequiredinstatisticsbutusuallyarenot
proved in statistics texts. This chapter also includes sections on principal compo-
nentsandcanonicalcorrelations.Chapter5preparesthebackgroundforacoursein
designs, establishing the linear model as the underlying mathematical framework.
The sections on optimality may be useful as motivation for further reading in this
research area in which there is considerable activity at present. Similarly, the last
v
vi Preface
chaptertriestoprovideaglimpseintotherichnessofatopicingeneralizedinverses
(rankadditivity)thathasmanyinterestingapplicationsaswell.
Several exercises are included, some of which are used in subsequent develop-
ments. Hints are provided for a few exercises, whereas reference to the original
sourceisgiveninsomeothercases.
IamgratefultoProfessor AlokeDey,H.Neudecker,K.P.S.BhaskaraRao,and
Dr. N. Eagambaram for their comments on various portions of the manuscript.
ThanksarealsoduetoB.Ganeshanforhishelpingettingthecomputerprintoutsat
variousstages.
Aboutthe SecondEdition
This is a thoroughly revised and enlarged version of the first edition. Besides cor-
recting the minor mathematical and typographical errors, the following additions
havebeenmade:
1. A few problems have been added at the end of each section in the first four
chapters.Allthechaptersnowcontainsomenewexercises.
2. Completesolutionsorhintsareprovidedtoseveralproblemsandexercises.
3. Two new sections, one on the “volume of a matrix” and the other on the “star
order,”havebeenadded.
Aboutthe ThirdEdition
Inthiseditionthematerialhasbeencompletelyreorganized.Thelinearalgebrapart
isdealtwithinthefirstsixchapters.Thesechaptersconstituteafirstcourseinlinear
algebra,suitableforstatisticsstudents,orforthoselookingforamatrixapproachto
linearalgebra.
We have added a chapter on linear mixed models. There is also a new chapter
containing additional problems on rank. These problems are not covered in a tra-
ditional linear algebra course. However we believe that the elegance of the matrix
theoreticapproachtolinearalgebraisclearlybroughtoutbyproblemsonrankand
generalizedinverseliketheonescoveredinthischapter.
I thank the numerous individuals who made suggestions for improvement and
pointed out corrections in the first two editions. I wish to particularly mention
N.EagambaramandJeffStuartfortheirmeticulouscomments.IalsothankAloke
DeyforhiscommentsonapreliminaryversionofChap.9.
NewDelhi,India RavindraBapat
Contents
1 VectorSpacesandSubspaces . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 VectorSpaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 BasisandDimension . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Rank,InnerProductandNonsingularity . . . . . . . . . . . . . . . . 9
2.1 Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 InnerProduct. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 Nonsingularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 FrobeniusInequality . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 EigenvaluesandPositiveDefiniteMatrices . . . . . . . . . . . . . . . 21
3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2 TheSpectralTheorem . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3 SchurComplement . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4 GeneralizedInverses . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2 MinimumNormandLeastSquaresg-Inverse . . . . . . . . . . . . 33
4.3 Moore–PenroseInverse . . . . . . . . . . . . . . . . . . . . . . . 34
4.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
5 InequalitiesforEigenvaluesandSingularValues . . . . . . . . . . . 37
5.1 EigenvaluesofaSymmetricMatrix . . . . . . . . . . . . . . . . . 37
5.2 SingularValues . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.3 MinimaxPrincipleandInterlacing . . . . . . . . . . . . . . . . . 41
5.4 Majorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.5 VolumeofaMatrix . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
vii
viii Contents
6 RankAdditivityandMatrixPartialOrders . . . . . . . . . . . . . . 51
6.1 CharacterizationsofRankAdditivity . . . . . . . . . . . . . . . . 51
6.2 TheStarOrder . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7 LinearEstimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
7.1 LinearModel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
7.2 Estimability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.3 ResidualSumofSquares . . . . . . . . . . . . . . . . . . . . . . 66
7.4 GeneralLinearModel . . . . . . . . . . . . . . . . . . . . . . . . 72
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
8 TestsofLinearHypotheses. . . . . . . . . . . . . . . . . . . . . . . . 79
8.1 MultivariateNormalDistribution . . . . . . . . . . . . . . . . . . 79
8.2 QuadraticFormsandCochran’sTheorem . . . . . . . . . . . . . . 83
8.3 One-WayandTwo-WayClassifications . . . . . . . . . . . . . . . 86
8.4 LinearHypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . 90
8.5 MultipleCorrelation . . . . . . . . . . . . . . . . . . . . . . . . . 92
8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
9 LinearMixedModels. . . . . . . . . . . . . . . . . . . . . . . . . . . 99
9.1 FixedEffectsandRandomEffects. . . . . . . . . . . . . . . . . . 99
9.2 MLandREMLEstimators . . . . . . . . . . . . . . . . . . . . . 102
9.3 ANOVAEstimators . . . . . . . . . . . . . . . . . . . . . . . . . 107
9.4 PredictionofRandomEffects . . . . . . . . . . . . . . . . . . . . 112
9.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
10 MiscellaneousTopics . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
10.1 PrincipalComponents . . . . . . . . . . . . . . . . . . . . . . . . 115
10.2 CanonicalCorrelations. . . . . . . . . . . . . . . . . . . . . . . . 116
10.3 ReducedNormalEquations . . . . . . . . . . . . . . . . . . . . . 117
10.4 TheC-Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
10.5 E-,A-andD-Optimality . . . . . . . . . . . . . . . . . . . . . . . 120
10.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
11 AdditionalExercisesonRank . . . . . . . . . . . . . . . . . . . . . . 129
12 HintsandSolutionstoSelectedExercises. . . . . . . . . . . . . . . . 135
13 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
Chapter 1
Vector Spaces and Subspaces
1.1 Preliminaries
Inthischapterwefirstreviewcertainbasicconcepts.Weconsideronlyrealmatri-
ces.Althoughourtreatmentisself-contained,thereaderisassumedtobefamiliar
with the basic operations on matrices. We also assume knowledge of elementary
propertiesofthedeterminant.
Anm×nmatrixconsistsofmnrealnumbersarrangedinmrowsandncolumns.
Theentryinrowiandcolumnj ofthematrixAisdenotedbya .Anm×1matrix
ij
is called a column vector of order m; similarly, a 1×n matrix is a row vector of
ordern.Anm×nmatrixiscalledasquarematrixifm=n.
IfAandB arem×nmatrices,thenA+B isdefinedasthem×nmatrixwith
(i,j)-entrya +b .IfAisamatrixandcisarealnumberthencAisobtainedby
ij ij
multiplyingeachelementofAbyc.
IfAism×pandB isp×n,thentheirproductC=AB isanm×nmatrixwith
(i,j)-entrygivenby
(cid:2)p
c = a b .
ij ik kj
k=1
Thefollowingpropertieshold:
(AB)C=A(BC),
A(B+C)=AB+AC,
(A+B)C=AC+BC.
Thetransposeofthem×nmatrixA,denotedbyA(cid:2),isthen×mmatrixwhose
(i,j)-entryisa .Itcanbeverifiedthat(A(cid:2))(cid:2)=A,(A+B)(cid:2)=A(cid:2)+B(cid:2)and(AB)(cid:2)=
ji
(cid:2) (cid:2)
B A.
A good understanding of the definition of matrix multiplication is quite useful.
We note some simple facts which are often required. We assume that all products
occurring here are defined in the sense that the orders of the matrices make them
compatibleformultiplication.
R.B.Bapat,LinearAlgebraandLinearModels,Universitext, 1
DOI10.1007/978-1-4471-2739-0_1,©Springer-VerlagLondonLimited2012