Table Of Content

Matrix Algebra From a Statistician's Perspective David A. Harville Springer Preface Matrix algebra plays a very important role in statistics and in many other disci- plines.Inmanyareasofstatistics,ithasbecomeroutinetousematrixalgebrain thepresentationandthederivationorverificationofresults.Onesuchareaislinear statisticalmodels;anotherismultivariateanalysis.Intheseareas,aknowledgeof matrixalgebraisneededinapplyingimportantconcepts,aswellasinstudyingthe underlying theory, and is even needed to use various software packages (if they aretobeusedwithconfidenceandcompetence). On many occasions, I have taught graduate-level courses in linear statistical models.Typically,theprerequisitesforsuchcoursesincludeanintroductory(un- dergraduate) course in matrix (or linear) algebra. Also typically, the preparation providedbythisprerequisitecourseisnotfullyadequate.Thereareseveralrea- sonsforthis.Thelevelofabstractionorgeneralityinthematrix(orlinear)algebra coursemayhavebeensohighthatitdidnotleadtoa“workingknowledge”ofthe subject,or,attheotherextreme,thecoursemayhaveemphasizedcomputationsat theexpenseoffundamentalconcepts.Further,thecontentofintroductorycourses onmatrix(orlinear)algebravarieswidelyfrominstitutiontoinstitutionandfrom instructortoinstructor.Topicssuchasquadraticforms,partitionedmatrices,and generalized inverses that play an important role in the study of linear statistical modelsmaybecoveredinadequatelyifatall.Anadditionaldifficultyisthatsev- eral years may have elapsed between the completion of the prerequisite course onmatrix(orlinear)algebraandthebeginningofthecourseonlinearstatistical models. Thisbookisaboutmatrixalgebra.Adistinguishingfeatureisthatthecontent, theorderingoftopics,andthelevelofgeneralityareonesthatIconsiderappro- priate for someone with an interest in linear statistical models and perhaps also vi forsomeonewithaninterestinanotherareaofstatisticsorinarelateddiscipline. I have tried to keep the presentation at a level that is suitable for anyone who has had an introductory course in matrix (or linear) algebra. In fact, the book is essentiallyself-contained,anditishopedthatmuch,ifnotall,ofthematerialmay becomprehensibletoadeterminedreaderwithrelativelylittlepreviousexposure tomatrixalgebra.Tomakethematerialreadableforasbroadanaudienceaspos- sible,Ihaveavoidedtheuseofabbreviationsandacronymsandhavesometimes adoptedterminologyandnotationthatmayseemmoremeaningfulandfamiliarto thenon-mathematicianthanthosefavoredbymathematicians.Proofsareprovided foressentiallyalloftheresultsinthebook.Thebookincludesanumberofresults and proofs that are not readily available from standard sources and many others thatcanbefoundonlyinrelativelyhigh-levelbooksorinjournalarticles. The book can be used as a companion to the textbook in a course on linear statistical models or on a related topic through- it can be used to supplement whateverresultsonmatricesmaybeincludedinthetextbookandasasourceof proofs.And,itcanbeusedasaprimaryorsupplementarytextinasecondcourse onmatrices,includingacoursedesignedtoenhancethepreparationofthestudents foracourseorcoursesonlinearstatisticalmodelsand/orrelatedtopics.Aboveall, itcanserveasaconvenientreferencebookforstatisticiansandforvariousother professionals. Whilethemotivationforthewritingofthebookcamefromthestatisticalapplica- tionsofmatrixalgebra,thebookitselfdoesnotincludeanyappreciablediscussion of statistical applications. It is assumed that the book is being read because the readerisawareoftheapplications(oratleastofthepotentialforapplications)or becausethematerialisofintrinsicinterestthrough-thisassumptionisconsistent withtheusesdiscussedinthepreviousparagraph.(Inanycase,Ihavefoundthat thediscussionsofapplicationsthataresometimesinterjectedintotreatisesonma- trixalgebratendtobemeaningfulonlytothosewhoarealreadyknowledgeable abouttheapplicationsandcanbemoreofadistractionthanahelp.) Thebookhasanumberoffeaturesthatcombinetosetitapartfromthemore traditionalbooksonmatrixalgebrathrough-italsodiffersinsignificantrespects from those matrix-algebra books that share its (statistical) orientation, such as the books of Searle (1982), Graybill (1983), and Basilevsky (1983). The cover- ageisrestrictedtorealmatrices(i.e.,matriceswhoseelementsarerealnumbers) through-complexmatrices(i.e.,matriceswhoseelementsarecomplexnumbers) aretypicallynotencounteredinstatisticalapplications,andtheirexclusionleads tosimplificationsinterminology,notation,andresults.Thecoverageincludeslin- ear spaces, but only linear spaces whose members are (real) matrices through- theinclusionoflinearspacesfacilitatesadeeperunderstandingofvariousmatrix concepts(e.g.,rank)thatareveryrelevantinapplicationstolinearstatisticalmod- els,whiletherestrictiontolinearspaceswhosemembersarematricesmakesthe presentationmoreappropriatefortheintendedaudience. The book features an extensive discussion of generalized inverses and makes heavyuseofgeneralizedinversesinthediscussionofsuchstandardtopicsasthe solutionoflinearsystemsandtherankofamatrix.Thediscussionofeigenvalues vii andeigenvectorsisdeferreduntilthenext-to-lastchapterofthebookthrough-I havefounditunnecessarytouseresultsoneigenvaluesandeigenvectorsinteaching a first course on linear statistical models and, in any case, find it aesthetically displeasingtouseresultsoneigenvaluesandeigenvectorstoprovemoreelementary matrixresults.And,thediscussionoflineartransformationsisdeferreduntilthe verylastchapterthrough-inmoreadvancedpresentations,matricesareregarded assubservienttolineartransformations. The book provides rather extensive coverage of some nonstandard topics that haveimportantapplicationsinstatisticsandinmanyotherdisciplines.Thesein- cludematrixdifferentiation(Chapter15),thevecandvechoperators(Chapter16), theminimizationofasecond-degreepolynomial(innvariables)subjecttolinear constraints(Chapter19),andtheranks,determinants,andordinaryandgeneral- izedinversesofpartitionedmatricesandofsumsofmatrices(Chapter18andparts ofChapters8,9,1316,17,and19).Anattempthasbeenmadetowritethebook insuchawaythatthepresentationiscoherentandnonredundantbut,atthesame time,isconducivetousingthevariouspartsofthebookselectively. With the obvious exception of certain of their parts, Chapters 12 through 22 (whichcompriseapproximatelythree-quartersofthebook’spages)canbereadin arbitraryorder.TheorderingofChapters1through11(bothrelativetoeachother andrelativetoChapters12through22)ismuchmorecritical.Nevertheless,even Chapters 1 through 11 include sections or subsections that are prerequisites for onlyasmallpartofthesubsequentmaterial.Moreoftenthannot,thelessessential sectionsorsubsectionsaredeferreduntiltheendofthechapterorsection. Thebookdoesnotaddressthecomputationalaspectsofmatrixalgebrainany systematic way, however it does include descriptions and discussion of certain computationalstrategiesandcoversanumberofresultsthatcanbeusefulindealing withcomputationalissues.Matrixnormsarediscussed,butonlytoalimitedextent. In particular, the coverage of matrix norms is restricted to those norms that are definedintermsofinnerproducts. Inwritingthebook,IwasinfluencedtoaconsiderableextentbyHalmos’s(1958) bookonfinite-dimensionalvectorspaces,byMarsagliaandStyan’s(1974)paper on ranks, by Henderson and Searle’s (1979, 1981b) papers on the vec and vech operators,byMagnusandNeudecker’s(1988)bookonmatrixdifferentialcalculus, andbyRaoandMitra’s(1971)bookongeneralizedinverses.And,Ibenefitedfrom conversations with Oscar Kempthorne and from reading some notes (on linear systems,determinants,matrices,andquadraticforms)thathehadpreparedfora course(onlinearstatisticalmodels)atIowaStateUniversity.Ialsobenefitedfrom reading the first two chapters (pertaining to linear algebra) of notes prepared by JustusF.Seelyforacourse(onlinearstatisticalmodels)atOregonStateUniversity. Thebookcontainsmanynumberedexercises.Theexercisesarelocatedat(or near)theendsofthechaptersandaregroupedbysectionthrough-someexercises mayrequiretheuseofresultscoveredinprevioussections,chapters,orexercises. Manyoftheexercisesconsistofverifyingresultssupplementarytothoseincluded inthebodyofthechapter.Bybreakingsomeofthemoredifficultexercisesinto partsand/orbyprovidinghints,Ihaveattemptedtomakealloftheexercisesappro- viii priatefortheintendedaudience.Ihavepreparedsolutionstoalloftheexercises, anditismyintentiontomakethemavailableonatleastalimitedbasis. Theoriginandhistoricaldevelopmentofmanyoftheresultscoveredinthebook are difficult (if not impossible) to discern, and I have not made any systematic attempt to do so. However, each of Chapters 15 through 21 ends with a short sectionentitledBibliographicandSupplementaryNotes.SourcesthatIhavedrawn onmore-or-lessdirectlyforanextensiveamountofmaterialareidentifiedinthat section.Sourcesthattracethehistoricaldevelopmentofvariousideas,results,and terminologyarealsoidentified.And,forcertainofthesectionsinachapter,some indicationmaybegivenofwhetherthatsectionisaprerequisiteforvariousother sections(orviceversa). Thebookisdividedinto(22)numberedchapters,thechaptersintonumbered sections, and (in some cases) the sections into lettered subsections. Sections are identified by two numbers (chapter and section within chapter) separated by a decimalpointthrough-thus,thethirdsectionofChapter9isreferredtoasSection 9.3.Withinasection,asubsectionisreferredtobyletteralone.Asubsectionin a different chapter or in a different section of the same chapter is referred to by referringtothesectionandbyappendingalettertothesectionnumberthrough- forexample,inSection9.3,SubsectionbofSection9.1isreferredtoasSection 9.1b.Anexerciseinadifferentchapterisreferredtobythenumberobtainedby insertingthechapternumber(andadecimalpoint)infrontoftheexercisenumber. Certainofthedisplayed“equations”arenumbered.Anequationnumbercom- prises two parts (corresponding to section within chapter and equation within section)separatedbyadecimalpoint(andisenclosedinparentheses).Anequa- tion in a different chapter is referred to by the “number” obtained by starting withthechapternumberandappendingadecimalpointandtheequationnumber through-forexample,inChapter6,result(2.5)ofChapter5isreferredtoasresult (5.2.5).Forpurposesofnumbering(andreferringto)equationsintheexercises,the exercisesineachchapteraretoberegardedasformingSectionEofthatchapter. Preliminaryworkonthebookdatesbacktothe1982–1983academicyear,which IspentasavisitingprofessorintheDepartmentofMathematicsattheUniversity of Texas at Austin (on a faculty improvement leave from my then position as a professorofstatisticsatIowaStateUniversity).Theactualwritingbeganaftermy returntoIowaStateandcontinuedonasporadicbasis(astimepermitted)untilmy departureinDecember1995.Theworkwascompletedduringthefirstpartofmy tenure in the Mathematical Sciences Department of the IBM Thomas J. Watson ResearchCenter. IamindebtedtoBettyFlehinger,EmmanuelYashchin,ClaudeGreengard,and BillPulleyblank(allofwhomareorweremanagersattheResearchCenter)forthe timeandsupporttheyprovidedforthisactivity.Themostvaluableofthatsupport (byfar)cameintheformofthesecretarialhelpofPeggyCargiulo,whoentered a thelastsixchaptersofthebookinLT Xandwasofimmensehelpingettingthe E manuscript into final form. I am also indebted to Darlene Wicks (of Iowa State a University),whoenteredChapters1through16inLT X. E IwishtothankJohnKimmel,whohasbeenmyeditoratSpringer-Verlag.He ix has been everything an author could hope for. I also wish to thank Paul Nikolai (formerlyoftheAirForceFlightDynamicsLaboratoryofWright-PattersonAir Force Base, Ohio) and Dale Zimmerman (of the Department of Statistics and ActuarialScienceoftheUniversityofIowa),whosecarefulreadingandmarking ofthemanuscriptledtoanumberofcorrectionsandimprovements.Thesechanges were in addition to ones stimulated by the earlier comments of two anonymous reviewers(andbythecommentsoftheeditor).