Linear Algebra [draft]

371 Pages·2015·1.147 MB·English

by  Henry C. Pinkham

Henry C. Pinkham
Linear Algebra
July10,2015
Springer

Preface
Thisisatextbookforatwo-semestercourseonLinearAlgebra.Althoughthepre-
requisitesforthisbookareasemesterofmultivariablecalculus,inrealityeverything
isdevelopedfromscratchandmathematicalmaturityistherealprerequisite.Tradi-
tionallylinearalgebraisthefirstcourseinthemathcurriculumwherestudentsare
askedtounderstandproofs,andthisbookemphasizesthispoint:itgivestheback-
groundtohelpstudentsunderstandproofsandgivesfullproofsforallthetheorems
inthebook.
Whywriteatextbookforatwosemestercourse?Firstsemestertextbookstend
to focus exclusively on matrices and matrix manipulation, while second semester
textbookstendtodismissmatricesasinferiortools.Thissegregationofmatrixtech-
niques on one hand, and linear transformations of the other tends to obscure the
intimaterelationshipbetweenthetwo.
Students can enjoy the book without understanding all the proofs, as many nu-
mericallyexamplesillustratealltheconcepts.
As is the case for most elementary textbooks on linear algebra, we only study
finitedimensionalvectorspacesandrestrictthescalarstorealorcomplexnumbers.
Weemphasizecomplexnumbersandhermitianmatrices,sincethecomplexcaseis
essential in understanding the real case. However, whenever possible, rather than
writing one proof for the hermitian case that also works for the real symmetric
case, they are treated in separate sections, so the student who is mainly interested
intherealcase,andknowslittleaboutcomplexnumbers,canreadon,skippingthe
sectionsdevotedtothecomplexcase.
Wespend moretimethat usualinstudying systemsoflinear equationswithout
using the matrix technology. This allows for flexibility that one loses when using
matrices. We take advantage of this work to study families of linear inequalities,
whichisusefulfortheoptionalchapteronconvexityandoptimizationattheendof
thebook.
Inthesecondchapter,westudymatricesandGaussianeliminationintheusual
way,whilecomparingwitheliminationinsystemsofequationsfromthefirstchap-
ter. We also spend more time than usual on matrix multiplication: the rest of the
bookshowshowessentialitistounderstandinglinearalgebra.
v
vi
Then we study vector spaces and linear maps. We give the classical definition
oftherankofamatrix:thelargestsizeofanon-singularsquaresubmatrix,aswell
asthestandardones.Wealsoproveotherclassicresultsonmatricesthatareoften
omitted in recent textbooks. We give a complete change of basis presentation in
Chapter5.
In a portion of the book that can be omitted on first reading, we study duality
andgeneralbilinearforms.Thenwestudyinner-productspaces:vectorspaceswith
apositivedefinitescalar(orhermitian)product),intheusualway.Weintroducethe
inner product late, because it is an additional piece of structure on a vector space.
Wereplaceitbydualityintheearlyargumentswhereitcanbeused.
Nextwestudylinearoperatorsoninnerproductspace,alinearoperatorbeinga
lineartransformationfromavectorspacetoitself,westudyimportantspeciallinear
operators:symmetric,hermitian,orthogonalandunitaryoperatrps,dealingwiththe
realandthecomplexoperatorsseparatelyFinallywedefinenormaloperators.
Thenwiththegoalofclassifyinglinearoperatorswedeveloptheimportantno-
tionofpolynomialsofmatrices.Theelementarytheoryofpolynomialsinonevari-
able, that most students will have already seen, is reviewed in an appendix. This
leadsustotheminimalpolynomialofalinearoperator,whichallowsustoestablish
theJordannormalforminboththecomplexandrealcase.
Onlythendoweturntodeterminants.Thisbookshowshowmuchoftheelemen-
tarytheorycanbedonewithoutdeterminants,justusingtherankandothersimilar
tools.Ourpresentationofdeterminantsisbuiltonpermutations,andourdefinition
istheLeibnitzformulaintermsofpermutations.Wethenestablishallthefamiliar
theoremsondeterminants,butgoalittlefurther:westudytheadjugatematrixand
provetheclassicCauchy-Binettheorem.
Next we study the characteristic polynomial of a linear operator, and prove the
Cayley-Hamilton theorem. We establish the classic meaning of all the coefficients
ofthecharacteristicpolynomial,notjustthedeterminantandthetrace.
We conclude with the Spectral Theorem, the most important theorem of linear
algebra. We have a few things to say about the importance of the computations of
eigenvaluesandeigenvectors.Wederivealltheclassictestsforpositivedefiniteand
positivesemidefinitematrices.
Nextthereisanoptionalchapteronpolytopes,polyhedraandconvexity,anatural
outgrowth of our study of inequalities in the first chapter. This only involves real
linearalgebra.
Finally, there is a chapter on the usefulness of linear algebra in the study of
difference equations and linear ordinary differential equations. This only uses real
linearalgebra.
There are three appendices. the first is the summary of the notation used in the
boof;thesecondgivessomemathematicalbackgroundthatoccasionallyprovesuse-
ful,especiallythereviewofcomplexnumbers.Thelastappendixonpolynomialsis
veryimportantifyouhavenotseenthematerialinitbefore.Extensiveuseofitis
madeinthestudyoftheminimalpolynomial.
Leitfaden
Thereareseveralpathwaysthroughthebook.
vii
1. Many readers with have seen the material of the first three sections of Chapter
1; Chapters 2, 3, 4 and 5 form the core of the book and should be read care-
fully by everyone. I especially recommend a careful reading of the material on
matrixmultiplicationinChapter2,sincemanyoftheargumentslaterondepend
essentiallyonagoodknowledgeofit.
2. Chapter6onduality,andChapter7onbilinearformsformanindependentsec-
tionthatcanbeskippedinaonesemestercourse.
3. Chapter 8 studies what we call inner-product spaces: either real vector spaces
with a positive definite scalar product or complex vector spaces with a positive
definitehermitianproduct.Thisbeginsourstudyofvectorspacesequippedwith
a new bit of structure: an inner product. Chapter 9 studies operators on an in-
ner product space. First it shows how to write all of them, and then it studies
thosethathaveaspecialstructurewithrespecttotheinnerproduct.Asalready
mentioned,thematerialforrealvectorspacesispresentedindependentlyforthe
readerwhowantstofocusonrealvectorspaces.Thesetwochapterareessential.
4. In Chapter 9, we go back to the study of vector spaces without an inner prod-
uct.Thegoalistounderstandalloperators,soinfactlogicallythiscouldcome
before the material on operators on inner product spaces. After an introduction
of the goals of the chapter, the theory of polynomials of matrices is developed.
My goal is to convince the reader that there is nothing difficult here. The key
result is the existence of the minimal polynomial of an operator. Then we can
provetheprimarydecompositionandtheJordancanonicalform,whichallowus
to decompose any linear operator into smaller building blocks that are easy to
analyze.
5. Finally we approach the second main objective of linear algebra: the study of
theeigenvaluesandeigenvectorsofalinearoperator.Thisisdoneinthreesteps.
FirstthedeterminantinChapter11,thenthecharacteristicpolynomialinChapter
12,andfinallythespectraltheoreminChapter13.Inthechapterconcerningthe
spectral theorem we use the results on inner products and special operators of
chapters 8 and 9 for the first time. It is essential to get to this material in a one
semestercourse, whichmay requireskipping items2 and4.Some applications
showtheimportanceofeigenvectorcomputation.
6. Chapter13coversthemethodofleastsquares,oneofthemostimportantappli-
cationsoflinearalgebra.Thisisoptionalforaone-semestercourse.
7. Chapter14,anotheroptionalchapterconsidersfirstanobviousgeneralizationof
linearalgebra:affinegeometry.Thisisusefulindevelopingthetheoryofiinear
inequalities.Fromthereisanasmallsteptogettothebeautifultheoryofconvex-
ity,withanemphasisonthecomplexbodiesthatcomefromlinearinequalities:
polyhedraandpolytopes.Thisisidealforthesecondsemesterofalinearalgebra
course,orforaone-semestercoursethatonlystudiesreallinearalgebra.
8. Finally the material on systems of differential equations forms a good applica-
tionsforstudentswhoarefamiliarwithmultivariablecalculus.
9. Therearethreeappendices:firstacatalogofthenotationsystemused,thenabrief
reviewofsomemathematics,includingcomplexnumbers,andwhatismostim-
viii
portantforus,therootsofpolynomialswithrealorcomplexcoefficients.Finally
thelastappendixcarefullyreviewspolynomialsinonevariable.
RecommendedBooks
Likegenerationsofwritersoflinearalgebratextbooksbeforeme,Imustdisclaim
anyoriginalityintheestablishmentoftheresultsofthisbook,mostofwhichareat
least a century old. Here is a list of texts that I have found very helpful in writing
thisbookandthatIrecommend.
• Onthematrixside,Irecommendthreebooks:
Gantmacher’sclassictwovolumetext[8],verythoroughandperhapssomewhat
hardtoread;
Franklin’sconciseandclearbook[6].
DenisSerre’sbeautifulbook[24],veryconciseandelegant.
HornandJohnson’sencyclopedictreatmentofofmatrices[13],whichalsoshows
howmatricesandanalysiscanbeinterwoven.
• OnthelinearalgebrasideanexcellentexampleofanoldertextbookisMinsky.
Morerecentlythereis[12]-verycomplete.
• TheclassictextbookontheabstractsideisHalmos’sbook[10].Forthosewho
wanttogoevenfurtherinseeinghowlinearalgebraisthefirststepinstudying
“abstractalgebra”,MichaelArtin’stext[1]isrecommended,sinceheuseslinear
algebraasthefirstbuildingblocktoabstractalgebra.
• Linearalgebraisveryusefulinstudyingadvancedgeometry.Anexcellentbook
that quite unusually combines the linear algebra with the geometry is Shafare-
vich.EvenmoreadvancedisManin’sbook.
• Therearetwogoodself-described“secondsemester”linearalgebratexts:Serge
Lang’sbook[15]whichsuffersfromitsseparationfromhismoreelementarytext
thatdevelopsthematrixtechniques,andthenSheldonAxler’sbeautifullywritten
book[2].
• Finallytherearebooksthatfocusonthecomputationalside.Itisbecauselinear
algebraalgorithmscanbeimplementedoncomputersisacentralreasonthatlin-
earalgebrahascometooccupyacentralpositioninthemathematicscurriculum.
Wedonotdomuchofthatinthisbook.TheclassictextisGolub-VanLoan[9].
Therearebookscompletelydevotedtothecomputationofeigenvectors.
Comments,corrections,andothersuggestionsforimprovingthesenotesarewel-
[email protected].
HENRYC.PINKHAM
NewYork,NY
DraftofJuly10,2015
Contents
1 LinearEquations .............................................. 1
1.1 LinearEquations ........................................... 1
1.2 GeometryInterpretation ..................................... 5
1.3 Elimination ............................................... 6
1.4 ExamplesofElimination .................................... 10
1.5 ConsequencesofLinearSystems ............................. 15
1.6 DiagonallyDominantSystems ............................... 16
1.7 History ................................................... 17
2 Matrices ...................................................... 21
2.1 Matrices .................................................. 21
2.2 MatrixMultiplication ....................................... 24
2.3 SquareMatrices............................................ 28
2.4 Submatrices ............................................... 32
2.5 GaussianEliminationinMatrixNotation....................... 33
2.6 ReducedRow–EchelonForm................................. 37
2.7 SolvingLinearSystemsofEquations .......................... 38
2.8 ElementaryMatrices........................................ 40
2.9 BlockDecompositionofMatrices............................. 45
2.10 ColumnOperations......................................... 49
3 VectorSpaces.................................................. 51
3.1 Scalars ................................................... 51
3.2 VectorSpaces.............................................. 52
3.3 Subspaces................................................. 55
3.4 Bases..................................................... 58
3.5 Dimension ................................................ 61
3.6 ProductsandDirectSums ................................... 63
ix
x Contents
4 LinearMaps................................................... 65
4.1 LinearMaps............................................... 65
4.2 TheNullspaceandtheRangeofaLinearMap .................. 68
4.3 CompositionofLinearMaps ................................. 72
4.4 LinearOperators ........................................... 75
4.5 InvertibleLinearMaps ...................................... 76
4.6 Projections ................................................ 77
5 RepresentingLinearMapsbyMatrices........................... 81
5.1 TheMatrixofaLinearMap.................................. 81
5.2 TheLinearMapofaMatrix.................................. 83
5.3 ChangeofBasis............................................ 84
5.4 EquivalentLinearMaps ..................................... 87
5.5 EquivalentLinearOperators ................................. 88
5.6 TheRankofaMatrix ....................................... 90
5.7 MoreonLinearEquations ................................... 93
5.8 RealandComplexLinearMaps .............................. 95
5.9 NilpotentOperators......................................... 98
5.10 TheRankviaSubmatrices ...................................101
6 Duality........................................................107
6.1 TheDualSpace ............................................107
6.2 Application:LagrangeInterpolation ...........................109
6.3 BilinearForms:theGeneralCase .............................112
6.4 Annihilators ...............................................113
6.5 TheDoubleDual...........................................115
6.6 Duality ...................................................117
7 BilinearForms.................................................121
7.1 BilinearForms.............................................121
7.2 QuadraticForms ...........................................125
7.3 DecompositionofaSymmetricBilinearForm...................127
7.4 DiagonalizationofSymmetricBilinearForms...................129
7.5 Lagrange’sDiagonalizationAlgorithm.........................130
7.6 SkewSymmetricLinearForms ...............................132
7.7 SylvestersLawofInertia ....................................133
7.8 HermitianForms ...........................................138
7.9 DiagonalizationofHermitianForms...........................141
8 InnerProductSpaces...........................................143
8.1 ScalarProducts ............................................143
8.2 TheGeometryofEuclideanSpaces............................146
8.3 Gram-SchmidtOrthogonalization .............................149
8.4 OrthogonalProjectioninEuclideanSpaces .....................154
8.5 SolvingtheInconsistentInhomogeneousSystem ................156
8.6 HermitianProducts .........................................159
Contents xi
8.7 TheGeometryofHermitianSpaces ...........................160
8.8 ScalarProductonSpacesofMatrices..........................162
9 OperatorsonInnerProductSpaces ..............................167
9.1 AdjointsonRealSpacesandSymmetricMatrices ...............167
9.2 AdjointsforHermitianProductsandHermitianMatrices..........170
9.3 PositiveDefiniteOperatorsandMatrices .......................173
9.4 OrthogonalOperators .......................................174
9.5 UnitaryOperators ..........................................176
9.6 NormalOperators ..........................................177
9.7 TheFourSubspaces ........................................178
10 TheMinimalPolynomial........................................181
10.1 LinearOperators:theProblem................................181
10.2 PolynomialsofMatrices.....................................184
10.3 TheMinimalPolynomial ....................................186
10.4 CyclicVectors .............................................189
10.5 ThePrimaryDecompositionTheorem .........................191
10.6 TheJordanCanonicalForm..................................193
10.7 UniquenessoftheJordanForm ...............................195
10.8 TheJordanFormovertheRealNumbers.......................197
10.9 AnApplicationoftheJordanCanonicalForm...................197
11 TheDeterminant...............................................199
11.1 Permutations ..............................................199
11.2 PermutationMatrices .......................................202
11.3 PermutationsandtheDeterminant.............................206
11.4 PropertiesoftheDeterminant ................................210
11.5 TheLaplaceExpansion......................................213
11.6 Cramer’sRule .............................................217
11.7 TheAdjugateMatrix........................................218
11.8 TheCauchy-BinetTheorem..................................219
11.9 GaussianEliminationviaDeterminants ........................221
11.10DeterminantsandVolumes...................................224
11.11TheBirkhoff-KoenigTheorem ...............................225
12 TheCharacteristicPolynomial...................................227
12.1 TheCharacteristicPolynomial................................227
12.2 TheMultiplicityofEigenvalues...............................230
12.3 TheTraceandtheDeterminant ...............................231
12.4 TheCayley-HamiltonTheorem...............................232
12.5 TheSchurUnitaryTriangularizationTheorem ..................233
12.6 TheCharacteristicPolynomialoftheCompanionMatrix..........235
12.7 TheMinorsofaSquareMatrix ...............................237
12.8 ComputationofEigenvectors.................................238
12.9 TheBigPicture ............................................238
xii Contents
12.10TheCoefficientsoftheCharacteristicPolynomial ...............238
13 TheSpectralTheorem ..........................................243
13.1 TriangulationofComplexOperators...........................243
13.2 TheRayleighQuotient ......................................244
13.3 TheSpectralTheorem.......................................246
13.4 TheSpectralTheoremforSelf-AdjointOperators................249
13.5 PositiveDefiniteMatrices ...................................250
13.6 TheSpectralTheoremforUnitaryOperators....................257
13.7 TheSpectralTheoremforOrthogonalOperators.................257
13.8 TheSpectralTheoremforNormalOperators....................259
13.9 ThePolarDecomposition....................................261
13.10TheSingularValueDecomposition............................262
14 TheMethodofLeastSquares....................................265
14.1 TheMethodofLeastSquares ................................265
14.2 FittingtoaLine............................................266
14.3 ConnectiontoStatistics .....................................269
14.4 OrthogonalLeastSquares....................................273
14.5 ComputationalTechniquesinLeastSquares ....................275
15 LinearInequalitiesandPolyhedra ...............................277
15.1 AffineGeometry ...........................................277
15.2 SystemsofLinearInequalitiesandPolyhedra ...................283
15.3 ConvexSets ...............................................290
15.4 PolyhedraandPolytopes ....................................297
15.5 Carathe´odory’sTheorem ....................................300
15.6 Minkowski’sTheorem ......................................301
15.7 PolarityforConvexSets.....................................304
16 LinearDifferentialEquations ...................................309
16.1 DifferentialCalculusReview.................................309
16.2 Examples .................................................310
16.3 TheGeneralCase ..........................................313
16.4 SystemsofFirstOrderDifferentialEquations ...................314
16.5 EigenvectorComputationsforLinearODE .....................315
16.6 DifferenceEquations .......................................315
A Notation ......................................................317
A.1 Generalities ...............................................317
A.2 RealandComplexVectorSpaces .............................317
A.3 Matrices ..................................................318
A.4 LinearTransformations......................................319