Linear Algebra, Theory And Applications [Lecture notes]

492 Pages·2015·2.539 MB·English

by  Kenneth L. Kuttler

Linear Algebra, Theory And Applications
Kenneth Kuttler
April 10, 2015
2
Contents
1 Preliminaries 11
1.1 Sets And Set Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3 The Number Line And Algebra Of The Real Numbers . . . . . . . . . . . . . 12
1.4 Ordered fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.5 The Complex Numbers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.7 Completeness of R . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.8 Well Ordering And Archimedean Property . . . . . . . . . . . . . . . . . . . . 22
1.9 Division . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.10 Systems Of Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.12 Fn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
1.13 Algebra in Fn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
1.14 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.15 The Inner Product In Fn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.16 What Is Linear Algebra? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
1.17 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2 Linear Transformations 39
2.1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.1.1 The ijth Entry Of A Product . . . . . . . . . . . . . . . . . . . . . . . 44
2.1.2 Digraphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.1.3 Properties Of Matrix Multiplication . . . . . . . . . . . . . . . . . . . 48
2.1.4 Finding The Inverse Of A Matrix . . . . . . . . . . . . . . . . . . . . . 51
2.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3 Linear Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.4 Some Geometrically Defined Linear Transformations . . . . . . . . . . . . . . 58
2.5 The Null Space Of A Linear Transformation . . . . . . . . . . . . . . . . . . . 61
2.6 Subspaces And Spans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.7 An Application To Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
2.8 Matrices And Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
2.8.1 The Coriolis Acceleration . . . . . . . . . . . . . . . . . . . . . . . . . 69
2.8.2 The Coriolis Acceleration On The Rotating Earth . . . . . . . . . . . 73
2.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3
4 CONTENTS
3 Determinants 85
3.1 Basic Techniques And Properties . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.2 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.3 The Mathematical Theory Of Determinants . . . . . . . . . . . . . . . . . . . 91
3.3.1 The Function sgn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
3.3.2 The Definition Of The Determinant . . . . . . . . . . . . . . . . . . . 93
3.3.3 A Symmetric Definition . . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.3.4 Basic Properties Of The Determinant . . . . . . . . . . . . . . . . . . 96
3.3.5 Expansion Using Cofactors . . . . . . . . . . . . . . . . . . . . . . . . 97
3.3.6 A Formula For The Inverse . . . . . . . . . . . . . . . . . . . . . . . . 99
3.3.7 Rank Of A Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
3.3.8 Summary Of Determinants . . . . . . . . . . . . . . . . . . . . . . . . 103
3.4 The Cayley Hamilton Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 104
3.5 Block Multiplication Of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 106
3.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4 Row Operations 113
4.1 Elementary Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.2 The Rank Of A Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.3 The Row Reduced Echelon Form . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.4 Rank And Existence Of Solutions To Linear Systems . . . . . . . . . . . . . . 124
4.5 Fredholm Alternative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5 Some Factorizations 131
5.1 LU Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
5.2 Finding An LU Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
5.3 Solving Linear Systems Using An LU Factorization . . . . . . . . . . . . . . . 133
5.4 The PLU Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.5 Justification For The Multiplier Method . . . . . . . . . . . . . . . . . . . . . 136
5.6 Existence For The PLU Factorization . . . . . . . . . . . . . . . . . . . . . . 137
5.7 The QR Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
5.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
6 Spectral Theory 145
6.1 Eigenvalues And Eigenvectors Of A Matrix . . . . . . . . . . . . . . . . . . . 145
6.2 Some Applications Of Eigenvalues And Eigenvectors . . . . . . . . . . . . . . 153
6.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.4 Schur’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
6.5 Trace And Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.6 Quadratic Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.7 Second Derivative Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.8 The Estimation Of Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.9 Advanced Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
6.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
7 Vector Spaces And Fields 189
7.1 Vector Space Axioms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.2 Subspaces And Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
7.2.1 Basic Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190
7.2.2 A Fundamental Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 191
7.2.3 The Basis Of A Subspace . . . . . . . . . . . . . . . . . . . . . . . . . 195
CONTENTS 5
7.3 Lots Of Fields. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.3.1 Irreducible Polynomials . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.3.2 Polynomials And Fields . . . . . . . . . . . . . . . . . . . . . . . . . . 200
7.3.3 The Algebraic Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . 206
7.3.4 The Lindemannn Weierstrass Theorem And Vector Spaces. . . . . . . 209
7.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
8 Linear Transformations 215
8.1 Matrix Multiplication As A Linear Transformation . . . . . . . . . . . . . . . 215
8.2 L(V,W) As A Vector Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
8.3 The Matrix Of A Linear Transformation . . . . . . . . . . . . . . . . . . . . . 217
8.3.1 Rotations About A Given Vector . . . . . . . . . . . . . . . . . . . . . 224
8.3.2 The Euler Angles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
8.4 Eigenvalues And Eigenvectors Of Linear Transformations . . . . . . . . . . . 228
8.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
9 Canonical Forms 233
9.1 A Theorem Of Sylvester, Direct Sums . . . . . . . . . . . . . . . . . . . . . . 233
9.2 Direct Sums, Block Diagonal Matrices . . . . . . . . . . . . . . . . . . . . . . 236
9.3 Cyclic Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
9.4 Nilpotent Transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
9.5 The Jordan Canonical Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
9.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
9.7 The Rational Canonical Form . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
9.8 Uniqueness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
9.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
10 Markov Processes 263
10.1 Regular Markov Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
10.2 Migration Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
10.3 Absorbing States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
10.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
11 Inner Product Spaces 273
11.1 General Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
11.2 The Gram Schmidt Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
11.3 Riesz Representation Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 278
11.4 The Tensor Product Of Two Vectors . . . . . . . . . . . . . . . . . . . . . . . 281
11.5 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
11.6 Fredholm Alternative Again . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
11.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
11.8 The Determinant And Volume . . . . . . . . . . . . . . . . . . . . . . . . . . 289
11.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
12 Self Adjoint Operators 293
12.1 Simultaneous Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
12.2 Schur’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
12.3 Spectral Theory Of Self Adjoint Operators. . . . . . . . . . . . . . . . . . . . 298
12.4 Positive And Negative Linear Transformations . . . . . . . . . . . . . . . . . 302
12.5 The Square Root . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
12.6 Fractional Powers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
12.7 Square Roots And Polar Decompositions . . . . . . . . . . . . . . . . . . . . . 306
6 CONTENTS
12.8 An Application To Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
12.9 The Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . 311
12.10Approximation In The Frobenius Norm . . . . . . . . . . . . . . . . . . . . . 313
12.11Least Squares And Singular Value Decomposition . . . . . . . . . . . . . . . . 315
12.12The Moore Penrose Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
12.13Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
13 Norms 323
13.1 The p Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
13.2 The Condition Number . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
13.3 The Spectral Radius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
13.4 Series And Sequences Of Linear Operators . . . . . . . . . . . . . . . . . . . . 336
13.5 Iterative Methods For Linear Systems . . . . . . . . . . . . . . . . . . . . . . 340
13.6 Theory Of Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
13.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348
14 Numerical Methods, Eigenvalues 357
14.1 The Power Method For Eigenvalues. . . . . . . . . . . . . . . . . . . . . . . . 357
14.1.1 The Shifted Inverse Power Method . . . . . . . . . . . . . . . . . . . . 360
14.1.2 The Explicit Description Of The Method . . . . . . . . . . . . . . . . 361
14.1.3 Complex Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366
14.1.4 Rayleigh Quotients And Estimates for Eigenvalues . . . . . . . . . . . 367
14.2 The QR Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371
14.2.1 Basic Properties And Definition . . . . . . . . . . . . . . . . . . . . . 371
14.2.2 The Case Of Real Eigenvalues . . . . . . . . . . . . . . . . . . . . . . 374
14.2.3 The QR Algorithm In The General Case. . . . . . . . . . . . . . . . . 378
14.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
A Matrix Calculator On The Web 387
A.1 Use Of Matrix Calculator On Web . . . . . . . . . . . . . . . . . . . . . . . . 387
B Positive Matrices 389
C Functions Of Matrices 397
D Differential Equations 403
D.1 Theory Of Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . 403
D.2 Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
D.3 Local Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
D.4 First Order Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
D.5 Geometric Theory Of Autonomous Systems . . . . . . . . . . . . . . . . . . . 414
D.6 General Geometric Theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
D.7 The Stable Manifold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420
E Compactness And Completeness 425
E.1 The Nested Interval Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
E.2 Convergent Sequences, Sequential Compactness . . . . . . . . . . . . . . . . . 426
F Fundamental Theorem Of Algebra 429
CONTENTS 7
G Fields And Field Extensions 431
G.1 The Symmetric Polynomial Theorem . . . . . . . . . . . . . . . . . . . . . . . 431
G.2 The Fundamental Theorem Of Algebra. . . . . . . . . . . . . . . . . . . . . . 433
G.3 Transcendental Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
G.4 More On Algebraic Field Extensions . . . . . . . . . . . . . . . . . . . . . . . 445
G.5 The Galois Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 450
G.6 Normal Subgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
G.7 Normal Extensions And Normal Subgroups . . . . . . . . . . . . . . . . . . . 456
G.8 Conditions For Separability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458
G.9 Permutations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462
G.10Solvable Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
G.11Solvability By Radicals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
H Selected Exercises 481
Copyright ⃝c 2012,
8 CONTENTS
Preface
This is a book on linear algebra and matrix theory. While it is self contained, it will work
bestforthosewhohavealreadyhadsomeexposuretolinearalgebra. Itisalsoassumedthat
the reader has had calculus. Some optional topics require more analysis than this, however.
I think that the subject of linear algebra is likely the most significant topic discussed in
undergraduate mathematics courses. Part of the reason for this is its usefulness in unifying
so many different topics. Linear algebra is essential in analysis, applied math, and even in
theoretical mathematics. This is the point of view of this book, more than a presentation
of linear algebra for its own sake. This is why there are numerous applications, some fairly
unusual.
This book features an ugly, elementary, and complete treatment of determinants early
in the book. Thus it might be considered as Linear algebra done wrong. I have done this
because of the usefulness of determinants. However, all major topics are also presented in
an alternative manner which is independent of determinants.
The book has an introduction to various numerical methods used in linear algebra.
This is done because of the interesting nature of these methods. The presentation here
emphasizes the reasons why they work. It does not discuss many important numerical
considerations necessary to use the methods effectively. These considerations are found in
numerical analysis texts.
In the exercises, you may occasionally see ↑ at the beginning. This means you ought to
have a look at the exercise above it. Some exercises develop a topic sequentially. There are
also a few exercises whichappear more than once in the book. I havedone this deliberately
becauseIthinkthattheseillustrateexceptionallyimportanttopicsandbecausesomepeople
don’treadthewholebookfromstarttofinishbutinsteadjumpintothemiddlesomewhere.
ThereisoneonatheoremofSylvesterwhichappearsnofewerthan3times. Thenitisalso
proved in the text. There are multiple proofs of the Cayley Hamilton theorem, some in the
exercises. Someexercisesalsoareincludedforthesakeofemphasizingsomethingwhichhas
been done in the preceding chapter.
9
10 CONTENTS