Matrix Differential Calculus with Applications in - Jan Magnus

468 Pages·2008·2.21 MB·English

Matrix Differential Calculus
with Applications in Statistics
and Econometrics
WILEY SERIES IN PROBABILITY AND STATISTICS
Established by WALTER E. SHEWHART AND SAMUEL S. WILKS
Editors: Vic Barnett, Noel A. C. Cressie, Nicholas, I. Fisher,
Iain M. Johnstone, J. B. Kadane, David, G. Kendall, David W. Scott,
Bernard W. Silverman, Adrian F. M. Smith, Jozef L. Teugels
Editors Emeritus: Ralph A. Bradley, J. Stuart Hunter
A complete list of the titles in this series appears at the end of this volume
Matrix
Differential
Calculus
with Applications
in Statistics
and Econometrics
Third Edition
JAN R. MAGNUS
CentER, Tilburg University
and
HEINZ NEUDECKER
Cesaro, Schagen
JOHN WILEY & SONS
Chichester • New York • Weinheim • Brisbane • Singapore • Toronto
Copyright(cid:13)c1988,1999JohnWiley&SonsLtd,
BaffinsLane,Chichester,
WestSussexPO191UD,England
National01243779777
International ([phone]
Copyright(cid:13)c1999oftheEnglishandRussianLATEXfileCentER,TilburgUniversity,
P.O.Box90153,5000LETilburg,TheNetherlands
Copyright(cid:13)c2007oftheThirdEditionJanMagnusandHeinzNeudecker.Allrightsreserved.
Publication dataforthe second (revised) edition
LibraryofCongressCataloging in PublicationData
Magnus,JanR.
Matrixdifferentialcalculuswithapplicationsinstatisticsand
econometrics/J.R.MagnusandH.Neudecker—Rev.ed. p. cm.
Includesbibliographicalreferencesandindex.
ISBN0-471-98632-1(alk.paper);ISBN0-471-98633-X(pbk:alk.paper)
1.Matrices. 2.DifferentialCalculus. 3.Statistics.
4.Econometrics. I.Neudecker,Heinz. II.Title.
QA188.M345 1999
512.9′434—dc21 98-53556
CIP
British LibraryCataloguing inPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
ISBN0-471-98632-1; 0-471-98633-X(pbk)
Publication dataforthe third edition
Thisisversion07/01.
Lastupdate:16January2007.
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Part One — Matrices
1 Basic properties of vectors and matrices 3
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 Matrices: addition and multiplication . . . . . . . . . . . . . . . 4
4 The transpose of a matrix . . . . . . . . . . . . . . . . . . . . . 6
5 Square matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
6 Linear forms and quadratic forms . . . . . . . . . . . . . . . . . 7
7 The rank of a matrix . . . . . . . . . . . . . . . . . . . . . . . . 8
8 The inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
9 The determinant . . . . . . . . . . . . . . . . . . . . . . . . . . 10
10 The trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
11 Partitioned matrices . . . . . . . . . . . . . . . . . . . . . . . . 11
12 Complex matrices . . . . . . . . . . . . . . . . . . . . . . . . . 13
13 Eigenvalues and eigenvectors . . . . . . . . . . . . . . . . . . . 14
14 Schur’s decomposition theorem . . . . . . . . . . . . . . . . . . 17
15 The Jordan decomposition . . . . . . . . . . . . . . . . . . . . . 18
16 The singular-value decomposition . . . . . . . . . . . . . . . . . 19
17 Further results concerning eigenvalues . . . . . . . . . . . . . . 20
18 Positive (semi)definite matrices . . . . . . . . . . . . . . . . . . 23
19 Three further results for positive definite matrices . . . . . . . 25
20 A useful result . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 27
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2 Kronecker products, the vec operator and the Moore-Penrose inverse 31
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2 The Kronecker product . . . . . . . . . . . . . . . . . . . . . . 31
3 Eigenvalues of a Kronecker product . . . . . . . . . . . . . . . . 33
4 The vec operator . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5 The Moore-Penrose(MP) inverse . . . . . . . . . . . . . . . . . 36
6 Existence and uniqueness of the MP inverse . . . . . . . . . . . 37
v
vi Contents
7 Some properties of the MP inverse . . . . . . . . . . . . . . . . 38
8 Further properties . . . . . . . . . . . . . . . . . . . . . . . . . 39
9 The solution of linear equation systems . . . . . . . . . . . . . 41
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 43
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3 Miscellaneous matrix results 47
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2 The adjoint matrix . . . . . . . . . . . . . . . . . . . . . . . . . 47
3 Proof of Theorem 1. . . . . . . . . . . . . . . . . . . . . . . . . 49
4 Bordered determinants . . . . . . . . . . . . . . . . . . . . . . . 51
5 The matrix equation AX =0 . . . . . . . . . . . . . . . . . . . 51
6 The Hadamard product . . . . . . . . . . . . . . . . . . . . . . 53
7 The commutation matrix K . . . . . . . . . . . . . . . . . . 54
mn
8 The duplication matrix D . . . . . . . . . . . . . . . . . . . . 56
n
9 Relationship between D and D , I . . . . . . . . . . . . . . 58
n+1 n
10 Relationship between D and D , II . . . . . . . . . . . . . . 60
n+1 n
11 Conditions for a quadratic form to be positive (negative) sub-
ject to linear constraints . . . . . . . . . . . . . . . . . . . . . . 61
12 Necessary and sufficient conditions for r(A:B)=r(A)+r(B) 64
13 The bordered Gramian matrix . . . . . . . . . . . . . . . . . . 66
14 The equations X A+X B′ =G ,X B =G . . . . . . . . . . 68
1 2 1 1 2
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 71
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Part Two — Differentials: the theory
4 Mathematical preliminaries 75
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
2 Interior points and accumulation points . . . . . . . . . . . . . 75
3 Open and closed sets . . . . . . . . . . . . . . . . . . . . . . . . 76
4 The Bolzano-Weierstrasstheorem . . . . . . . . . . . . . . . . . 79
5 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6 The limit of a function . . . . . . . . . . . . . . . . . . . . . . . 81
7 Continuous functions and compactness . . . . . . . . . . . . . . 82
8 Convex sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
9 Convex and concave functions . . . . . . . . . . . . . . . . . . . 85
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5 Differentials and differentiability 89
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
2 Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3 Differentiability and linear approximation . . . . . . . . . . . . 91
4 The differential of a vector function. . . . . . . . . . . . . . . . 93
5 Uniqueness of the differential . . . . . . . . . . . . . . . . . . . 95
6 Continuity of differentiable functions . . . . . . . . . . . . . . . 96
7 Partial derivatives . . . . . . . . . . . . . . . . . . . . . . . . . 97
Contents vii
8 The first identification theorem . . . . . . . . . . . . . . . . . . 98
9 Existence of the differential, I . . . . . . . . . . . . . . . . . . . 99
10 Existence of the differential, II . . . . . . . . . . . . . . . . . . 101
11 Continuous differentiability . . . . . . . . . . . . . . . . . . . . 103
12 The chain rule . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
13 Cauchy invariance . . . . . . . . . . . . . . . . . . . . . . . . . 105
14 The mean-value theorem for real-valued functions . . . . . . . . 106
15 Matrix functions . . . . . . . . . . . . . . . . . . . . . . . . . . 107
16 Some remarks on notation . . . . . . . . . . . . . . . . . . . . . 109
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 110
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6 The second differential 113
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
2 Second-order partial derivatives . . . . . . . . . . . . . . . . . . 113
3 The Hessian matrix. . . . . . . . . . . . . . . . . . . . . . . . . 114
4 Twice differentiability and second-order approximation,I . . . 115
5 Definition of twice differentiability . . . . . . . . . . . . . . . . 116
6 The second differential . . . . . . . . . . . . . . . . . . . . . . . 118
7 (Column) symmetry of the Hessian matrix . . . . . . . . . . . . 120
8 The second identification theorem . . . . . . . . . . . . . . . . 122
9 Twice differentiability and second-order approximation,II . . . 123
10 Chain rule for Hessian matrices . . . . . . . . . . . . . . . . . . 125
11 The analogue for second differentials . . . . . . . . . . . . . . . 126
12 Taylor’s theorem for real-valued functions . . . . . . . . . . . . 128
13 Higher-order differentials. . . . . . . . . . . . . . . . . . . . . . 129
14 Matrix functions . . . . . . . . . . . . . . . . . . . . . . . . . . 129
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
7 Static optimization 133
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2 Unconstrained optimization . . . . . . . . . . . . . . . . . . . . 134
3 The existence of absolute extrema . . . . . . . . . . . . . . . . 135
4 Necessary conditions for a local minimum . . . . . . . . . . . . 137
5 Sufficient conditions for a local minimum: first-derivative test . 138
6 Sufficient conditions for a local minimum: second-derivative test140
7 Characterizationof differentiable convex functions . . . . . . . 142
8 Characterizationof twice differentiable convex functions . . . . 145
9 Sufficient conditions for an absolute minimum . . . . . . . . . . 147
10 Monotonic transformations . . . . . . . . . . . . . . . . . . . . 147
11 Optimization subject to constraints . . . . . . . . . . . . . . . . 148
12 Necessary conditions for a local minimum under constraints . . 149
13 Sufficient conditions for a local minimum under constraints . . 154
14 Sufficient conditions for an absolute minimum under constraints158
15 A note on constraints in matrix form . . . . . . . . . . . . . . . 159
16 Economic interpretation of Lagrange multipliers. . . . . . . . . 160
Appendix: the implicit function theorem . . . . . . . . . . . . . . . . 162
viii Contents
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
Part Three — Differentials: the practice
8 Some important differentials 167
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
2 Fundamental rules of differential calculus . . . . . . . . . . . . 167
3 The differential of a determinant . . . . . . . . . . . . . . . . . 169
4 The differential of an inverse . . . . . . . . . . . . . . . . . . . 171
5 Differential of the Moore-Penroseinverse . . . . . . . . . . . . . 172
6 The differential of the adjoint matrix . . . . . . . . . . . . . . . 175
7 On differentiating eigenvalues and eigenvectors . . . . . . . . . 177
8 The differential of eigenvalues and eigenvectors:symmetric case 179
9 The differential of eigenvalues and eigenvectors:complex case . 182
10 Two alternative expressions for dλ . . . . . . . . . . . . . . . . 185
11 Second differential of the eigenvalue function . . . . . . . . . . 188
12 Multiple eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . 189
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 189
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
9 First-order differentials and Jacobian matrices 193
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
2 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
3 Bad notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
4 Good notation . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
5 Identification of Jacobian matrices . . . . . . . . . . . . . . . . 198
6 The first identification table . . . . . . . . . . . . . . . . . . . . 198
7 Partitioning of the derivative . . . . . . . . . . . . . . . . . . . 199
8 Scalar functions of a vector . . . . . . . . . . . . . . . . . . . . 200
9 Scalar functions of a matrix, I: trace . . . . . . . . . . . . . . . 200
10 Scalar functions of a matrix, II: determinant. . . . . . . . . . . 202
11 Scalar functions of a matrix, III: eigenvalue . . . . . . . . . . . 204
12 Two examples of vector functions . . . . . . . . . . . . . . . . . 204
13 Matrix functions . . . . . . . . . . . . . . . . . . . . . . . . . . 205
14 Kronecker products . . . . . . . . . . . . . . . . . . . . . . . . . 208
15 Some other problems . . . . . . . . . . . . . . . . . . . . . . . . 210
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
10 Second-order differentials and Hessian matrices 213
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
2 The Hessian matrix of a matrix function . . . . . . . . . . . . . 213
3 Identification of Hessian matrices . . . . . . . . . . . . . . . . . 214
4 The second identification table . . . . . . . . . . . . . . . . . . 215
5 An explicit formula for the Hessian matrix . . . . . . . . . . . . 217
6 Scalar functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
7 Vector functions . . . . . . . . . . . . . . . . . . . . . . . . . . 219
8 Matrix functions, I . . . . . . . . . . . . . . . . . . . . . . . . . 220
Contents ix
9 Matrix functions, II . . . . . . . . . . . . . . . . . . . . . . . . 221
Part Four — Inequalities
11 Inequalities 225
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
2 The Cauchy-Schwarzinequality . . . . . . . . . . . . . . . . . . 225
3 Matrix analogues of the Cauchy-Schwarz inequality . . . . . . . 227
4 The theorem of the arithmetic and geometric means . . . . . . 228
5 The Rayleigh quotient . . . . . . . . . . . . . . . . . . . . . . . 230
6 Concavity of λ , convexity of λ . . . . . . . . . . . . . . . . . 231
1 n
7 Variational description of eigenvalues . . . . . . . . . . . . . . . 232
8 Fischer’s min-max theorem . . . . . . . . . . . . . . . . . . . . 233
9 Monotonicity of the eigenvalues . . . . . . . . . . . . . . . . . . 235
10 The Poincar´eseparation theorem . . . . . . . . . . . . . . . . . 236
11 Two corollaries of Poincar´e’s theorem . . . . . . . . . . . . . . 237
12 Further consequences of the Poincar´e theorem . . . . . . . . . . 238
13 Multiplicative version . . . . . . . . . . . . . . . . . . . . . . . 239
14 The maximum of a bilinear form . . . . . . . . . . . . . . . . . 241
15 Hadamard’s inequality . . . . . . . . . . . . . . . . . . . . . . . 242
16 An interlude: Karamata’s inequality . . . . . . . . . . . . . . . 243
17 Karamata’s inequality applied to eigenvalues . . . . . . . . . . 245
18 An inequality concerning positive semidefinite matrices. . . . . 245
19 A representation theorem for ( ap)1/p . . . . . . . . . . . . . 246
i
20 A representation theorem for (trAp)1/p . . . . . . . . . . . . . . 248
P
21 H¨older’s inequality . . . . . . . . . . . . . . . . . . . . . . . . . 249
22 Concavity of logA . . . . . . . . . . . . . . . . . . . . . . . . . 250
| |
23 Minkowski’s inequality . . . . . . . . . . . . . . . . . . . . . . . 252
24 Quasilinear representation of A1/n . . . . . . . . . . . . . . . . 254
| |
25 Minkowski’s determinant theorem. . . . . . . . . . . . . . . . . 256
26 Weighted means of order p. . . . . . . . . . . . . . . . . . . . . 256
27 Schl¨omilch’s inequality . . . . . . . . . . . . . . . . . . . . . . . 259
28 Curvature properties of M (x,a) . . . . . . . . . . . . . . . . . 260
p
29 Least squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
30 Generalized least squares . . . . . . . . . . . . . . . . . . . . . 263
31 Restricted least squares . . . . . . . . . . . . . . . . . . . . . . 263
32 Restricted least squares: matrix version . . . . . . . . . . . . . 265
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 266
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
Part Five — The linear model
12 Statistical preliminaries 275
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
2 The cumulative distribution function . . . . . . . . . . . . . . . 275
3 The joint density function . . . . . . . . . . . . . . . . . . . . . 276
4 Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
x Contents
5 Variance and covariance . . . . . . . . . . . . . . . . . . . . . . 277
6 Independence of two random variables . . . . . . . . . . . . . . 279
7 Independence of n random variables . . . . . . . . . . . . . . . 281
8 Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
9 The one-dimensional normal distribution . . . . . . . . . . . . . 281
10 The multivariate normal distribution . . . . . . . . . . . . . . . 282
11 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 285
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
13 The linear regression model 287
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
2 Affine minimum-trace unbiased estimation . . . . . . . . . . . . 288
3 The Gauss-Markovtheorem . . . . . . . . . . . . . . . . . . . . 289
4 The method of least squares . . . . . . . . . . . . . . . . . . . . 292
5 Aitken’s theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 293
6 Multicollinearity . . . . . . . . . . . . . . . . . . . . . . . . . . 295
7 Estimable functions . . . . . . . . . . . . . . . . . . . . . . . . 297
8 Linear constraints: the case (R′) (X′) . . . . . . . . . . 299
M ⊂M
9 Linear constraints: the general case . . . . . . . . . . . . . . . . 302
10 Linear constraints: the case (R′) (X′)= 0 . . . . . . . 305
M ∩M { }
11 A singular variance matrix: the case (X) (V) . . . . . . 306
12 A singular variance matrix: the case Mr(X′V+⊂XM)=r(X) . . . . 308
13 A singular variance matrix: the general case, I . . . . . . . . . . 309
14 Explicit and implicit linear constraints . . . . . . . . . . . . . . 310
15 The general linear model, I . . . . . . . . . . . . . . . . . . . . 313
16 A singular variance matrix: the general case, II . . . . . . . . . 314
17 The general linear model, II . . . . . . . . . . . . . . . . . . . . 317
18 Generalized least squares . . . . . . . . . . . . . . . . . . . . . 318
19 Restricted least squares . . . . . . . . . . . . . . . . . . . . . . 319
Miscellaneous exercises. . . . . . . . . . . . . . . . . . . . . . . . . . 321
Bibliographical notes . . . . . . . . . . . . . . . . . . . . . . . . . . . 322
14 Further topics in the linear model 323
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
2 Best quadratic unbiased estimation of σ2 . . . . . . . . . . . . 323
3 The best quadratic and positive unbiased estimator of σ2 . . . 324
4 The best quadratic unbiased estimator of σ2 . . . . . . . . . . . 326
5 Best quadratic invariant estimation of σ2 . . . . . . . . . . . . 329
6 The best quadratic and positive invariant estimator of σ2 . . . 330
7 The best quadratic invariant estimator of σ2 . . . . . . . . . . . 331
8 Best quadratic unbiased estimation: multivariate normal case . 332
9 Bounds for the bias of the least squares estimator of σ2, I . . . 335
10 Bounds for the bias of the least squares estimator of σ2, II . . . 336
11 The prediction of disturbances . . . . . . . . . . . . . . . . . . 338
12 Best linear unbiased predictors with scalar variance matrix . . 339
13 Best linear unbiased predictors with fixed variance matrix, I . . 341