Tensor Spaces and Numerical Tensor Calculus

2019·3.12 MB·English

by  Wolfgang Hackbusch

Preface Page: vii
Contents Page: ix
About the Author Page: xix
List of Symbols and Abbreviations Page: xxi
Symbols Page: xxi
Greek Letters Page: xxiii
Latin Letters Page: xxiv
Abbreviations and Algorithms Page: xxviii
Part I: Algebraic Tensors Page: 1
Chapter 1: Introduction Page: 3
1.1 What are Tensors? Page: 3
1.1.1 Tensor Product of Vectors Page: 3
1.1.2 Tensor Product of Matrices, Kronecker Product Page: 5
1.1.3 Tensor Product of Functions Page: 7
1.2 Where do Tensors Appear? Page: 8
1.2.1 Tensors as Coefficients Page: 8
1.2.2 Tensor Decomposition for Inverse Problems Page: 9
1.2.3 Tensor Spaces in Functional Analysis Page: 10
1.2.4 Large-Sized Tensors in Analysis Applications Page: 10
1.2.5 Tensors in Quantum Chemistry Page: 13
1.3 Tensor Calculus Page: 13
1.4 Preview Page: 14
1.4.1 Part I: Algebraic Properties Page: 14
1.4.2 Part II: Functional Analysis of Tensors Page: 15
1.4.3 Part III: Numerical Treatment Page: 16
1.4.4 Topics Outside the Scope of the Monograph Page: 17
1.5 Software Page: 18
1.6 Comments about the Early History of Tensors Page: 18
1.7 Notations Page: 19
Chapter 2: Matrix Tools Page: 23
2.1 Matrix Notations Page: 23
2.2 Matrix Rank Page: 25
2.3 Matrix Norms Page: 27
2.4 Semidefinite Matrices Page: 29
2.5 Matrix Decompositions Page: 30
2.5.1 Cholesky Decomposition Page: 30
2.5.2 QR Decomposition Page: 31
2.5.3 Singular-Value Decomposition Page: 33
2.6 Low-Rank Approximation Page: 39
2.7 Linear Algebra Procedures Page: 41
2.8 Dominant Columns Page: 44
Chapter 3: Algebraic Foundations of Tensor Spaces Page: 49
3.1 Vector Spaces Page: 49
3.1.1 Basic Facts Page: 49
3.1.2 Free Vector Space over a Set Page: 50
3.1.3 Quotient Vector Space Page: 52
3.1.4 (Multi-)Linear Maps, Algebraic Dual, Basis Transformation Page: 53
3.2 Tensor Product Page: 54
3.2.1 Constructive Definition Page: 54
3.2.2 Characteristic Properties Page: 56
3.2.3 Isomorphism to Matrices for d = 2 Page: 58
3.2.4 Tensors of Order d > 3 Page: 60
3.2.5 Different Types of Isomorphisms Page: 63
3.2.6 Rr and Tensor Rank Page: 65
3.3 Linear and Multilinear Mappings Page: 76
3.3.1 Definition on the Set of Elementary Tensors Page: 76
3.3.2 Embeddings Page: 77
3.4 Tensor Spaces with Algebra Structure Page: 85
3.5 Symmetric and Antisymmetric Tensor Spaces Page: 88
3.5.1 Basic Definitions Page: 88
3.5.2 Quantics Page: 91
3.5.3 Determinants Page: 92
3.5.4 Application of Functionals Page: 93
Part II: Functional Analysis of Tensor Spaces Page: 95
Chapter 4: Banach Tensor Spaces Page: 97
4.1 Banach Spaces Page: 97
4.1.1 Norms Page: 97
4.1.2 Basic Facts about Banach Spaces Page: 98
4.1.3 Examples Page: 100
4.1.4 Operators Page: 101
4.1.5 Dual Spaces Page: 104
4.1.6 Examples Page: 106
4.1.7 Weak Convergence Page: 106
4.1.8 Continuous Multilinear Mappings Page: 108
4.2 Topological Tensor Spaces Page: 108
4.2.1 Notations Page: 108
4.2.2 Continuity of the Tensor Product, Crossnorms Page: 110
4.2.3 Projective Norm ||.||^(V;W) Page: 116
4.2.4 Duals and Injective Norm ||.|| (V;W) Page: 120
4.2.5 Embedding of V* into L(V W;W) Page: 127
4.2.6 Reasonable Crossnorms Page: 128
4.2.7 Reflexivity Page: 132
4.2.8 Uniform Crossnorms Page: 133
4.2.9 Nuclear and Compact Operators Page: 136
4.3 Tensor Spaces of Order d Page: 137
4.3.1 Continuity, Crossnorms Page: 137
4.3.2 Recursive Definition of the Topological Tensor Space Page: 140
4.3.3 Proofs Page: 143
4.3.4 Embedding into Embedding into L(V; Vj) and L(V;V�) Page: 147
4.3.5 Intersections of Banach Tensor Spaces Page: 151
4.3.6 Tensor Space of Operators Page: 154
4.4 Hilbert Spaces Page: 155
4.4.1 Scalar Product Page: 155
4.4.2 Basic Facts about Hilbert Spaces Page: 155
4.4.3 Operators on Hilbert Spaces Page: 157
4.4.4 Orthogonal Projections Page: 159
4.5 Tensor Products of Hilbert Spaces Page: 161
4.5.1 Induced Scalar Product Page: 161
4.5.2 Crossnorms Page: 163
4.5.3 Tensor Products of L(Vj ; Vj) Page: 164
4.5.4 Gagliardo–Nirenberg Inequality Page: 165
4.5.5 Partial Scalar Products Page: 170
4.6 Tensor Operations Page: 171
4.6.1 Vector Operations Page: 171
4.6.2 Matrix-Vector Multiplication Page: 172
4.6.3 Matrix-Matrix Operations Page: 172
4.6.4 Hadamard Multiplication Page: 174
4.6.5 Convolution Page: 174
4.6.6 Function of a Matrix Page: 176
4.7 Symmetric and Antisymmetric Tensor Spaces Page: 179
4.7.1 Hilbert Structure Page: 179
4.7.2 Banach Spaces and Dual Spaces Page: 180
Chapter 5: General Techniques Page: 183
5.1 Vectorisation Page: 183
5.1.1 Tensors as Vectors Page: 183
5.1.2 Kronecker Tensors Page: 185
5.2 Matricisation Page: 187
5.2.1 General Case Page: 187
5.2.2 Finite-Dimensional Case Page: 189
5.2.3 Hilbert Structure Page: 194
5.2.4 Matricisation of a Family of Tensors Page: 198
5.3 Tensorisation Page: 198
Chapter 6: Minimal Subspaces Page: 201
6.1 Statement of the Problem, Notations Page: 201
6.2 Tensors of Order Two Page: 202
6.2.1 Existence of Minimal Subspaces Page: 202
6.2.2 Use of the Singular-Value Decomposition Page: 205
6.2.3 Minimal Subspaces for a Family of Tensors Page: 206
6.3 Minimal Subspaces of Tensors of Higher Order Page: 207
6.4 Hierarchies of Minimal Subspaces and Page: 210
6.5 Sequences of Minimal Subspaces Page: 213
6.6 Minimal Subspaces of Topological Tensors Page: 218
6.6.1 Setting of the Problem Page: 218
6.6.2 First Approach Page: 218
6.6.3 Second Approach Page: 221
6.7 Minimal Subspaces for Intersection Spaces Page: 224
6.7.1 Algebraic Tensor Space Page: 224
6.7.2 Topological Tensor Space Page: 225
6.8 Linear Constraints and Regularity Properties Page: 226
6.9 Minimal Subspaces for (Anti-)Symmetric Tensors Page: 229
Part III: Numerical Treatment Page: 231
Chapter 7: r-Term Representation Page: 233
7.1 Representations in General Page: 234
7.1.1 Concept Page: 234
7.1.2 Computational and Memory Cost Page: 235
7.1.3 Tensor Representation versus Tensor Decomposition Page: 236
7.2 Full and Sparse Representation Page: 237
7.3 r-Term Representation Page: 238
7.4 Tangent Space and Sensitivity Page: 241
7.4.1 Tangent Space Page: 241
7.4.2 Sensitivity Page: 242
7.5 Representation of Vj Page: 244
7.6 Conversions between Formats Page: 247
7.6.1 From Full Representation into r-Term Format Page: 247
7.6.2 From r-Term Format into Full Representation Page: 248
7.6.3 From r-Term into N-Term Format for r>N Page: 248
7.6.4 Sparse-Grid Approach Page: 249
7.6.5 From Sparse Format into Page: 251
7.7 Representation of (Anti-)Symmetric Tensors Page: 253
7.7.1 Sums of Symmetric Rank-1 Tensors Page: 254
7.7.2 Indirect Representation Page: 254
7.8 Modifications Page: 256
Chapter 8: Tensor Subspace Representation Page: 257
8.1 The Set Tr Page: 257
8.2 Tensor Subspace Formats Page: 261
8.2.1 General Frame or Basis Page: 261
8.2.2 Transformations Page: 264
8.2.3 Tensors in KI Page: 265
8.2.4 Orthonormal Basis Page: 266
8.2.5 Summary of the Formats Page: 270
8.2.6 Hybrid Format Page: 271
8.3 Higher-Order Singular-Value Decomposition (HOSVD) Page: 273
8.3.1 Definitions Page: 273
8.3.2 Examples Page: 275
8.3.3 Computation and Computational Cost Page: 277
8.4 Tangent Space and Sensitivity Page: 283
8.4.1 Uniqueness Page: 283
8.4.2 Tangent Space Page: 284
8.4.3 Sensitivity Page: 285
8.5 Conversions between Different Formats Page: 287
8.5.1 Conversion from Full Representation into Tensor Subspace Format Page: 287
8.5.2 Conversion from Rr to Tr Page: 287
8.5.3 Conversion from Tr to Rr Page: 291
8.5.4 A Comparison of Both Representations Page: 292
8.5.5 r-Term Format for Large r > N Page: 293
8.6 Joining two Tensor Subspace Representation Systems Page: 293
8.6.1 Trivial Joining of Frames Page: 293
8.6.2 Common Bases Page: 294
Chapter 9: r-Term Approximation Page: 297
9.1 Approximation of a Tensor Page: 297
9.2 Discussion for r = 1 Page: 299
9.3 Discussion in the Matrix Case d = 2 Page: 301
9.4 Discussion in the Tensor Case d > 3 Page: 303
9.4.1 Nonclosedness of Rr Page: 303
9.4.2 Border Rank Page: 304
9.4.3 Stable and Unstable Sequences Page: 306
9.4.4 A Greedy Algorithm Page: 308
9.5 General Statements on Nonclosed Formats Page: 309
9.5.1 Definitions Page: 309
9.5.2 Nonclosed Formats Page: 311
9.5.3 Discussion of F = Rr Page: 312
9.5.4 General Case Page: 312
9.5.5 On the Strength of Divergence Page: 313
9.5.6 Uniform Strength of Divergence Page: 314
9.5.7 Extension to Vector Spaces of Larger Dimension Page: 317
9.6 Numerical Approaches for the r-Term Approximation Page: 318
9.6.1 Use of the Hybrid Format Page: 318
9.6.2 Alternating Least-Squares Method Page: 320
9.6.3 Stabilised Approximation Problem Page: 329
9.6.4 Newton’s Approach Page: 330
9.7 Generalisations Page: 332
9.8 Analytical Approaches for the r-Term Approximation Page: 333
9.8.1 Quadrature Page: 334
9.8.2 Approximation by Exponential Sums Page: 335
9.8.3 Sparse Grids Page: 346
Chapter 10: Tensor Subspace Approximation Page: 347
10.1 Truncation to Tr Page: 347
10.1.1 HOSVD Projection Page: 348
10.1.2 Successive HOSVD Projection Page: 350
10.1.3 Examples Page: 352
10.1.4 Other Truncations Page: 354
10.1.5 L Estimate of the Truncation Error Page: 355
10.2 Best Approximation in the Tensor Subspace Format Page: 358
10.2.1 General Setting Page: 358
10.2.2 Approximation with Fixed Format Page: 359
10.2.3 Properties Page: 361
10.3 Alternating Least-Squares Method (ALS) Page: 362
10.3.1 Algorithm Page: 362
10.3.2 ALS for Different Formats Page: 364
10.3.3 Approximation with Fixed Accuracy Page: 367
10.4 Analytical Approaches for the Tensor Subspace Approximation Page: 369
10.4.1 Linear Interpolation Techniques Page: 369
10.4.2 Polynomial Approximation Page: 372
10.4.3 Polynomial Interpolation Page: 374
10.4.4 Sinc Approximations Page: 376
10.5 Simultaneous Approximation Page: 383
10.6 Resume Page: 385
Chapter 11: Hierarchical Tensor Representation Page: 387
11.1 Introduction Page: 387
11.1.1 Hierarchical Structure Page: 387
11.1.2 Properties Page: 390
11.1.3 Historical Comments Page: 391
11.2 Basic Definitions Page: 392
11.2.1 Dimension Partition Tree Page: 392
11.2.2 Algebraic Characterisation, Hierarchical Subspace Family Page: 394
11.2.3 Minimal Subspaces Page: 395
11.2.4 Conversions Page: 398
11.3 Construction of Bases Page: 400
11.3.1 Hierarchical Basis Representation Page: 400
11.3.2 Orthonormal Bases Page: 410
11.3.3 HOSVD Bases Page: 415
11.3.4 Tangent Space and Sensitivity Page: 422
11.3.5 Sensitivity Page: 422
11.3.6 Conversion from Rr to Hr Revisited Page: 429
11.4 Approximations in Hr Page: 431
11.4.1 Best Approximation in Hr Page: 431
11.4.2 HOSVD Truncation to Hr Page: 433
11.5 Joining two Hierarchical Tensor Representation Systems Page: 446
11.5.1 Setting of the Problem Page: 446
11.5.2 Trivial Joining of Frames Page: 447
11.5.3 Common Bases Page: 447
Chapter 12: Matrix Product Systems Page: 453
12.1 Basic TT Representation Page: 453
12.2 Function Case Page: 456
12.3 TT Format as Hierarchical Format Page: 456
12.3.1 Related Subspaces Page: 456
12.3.2 From Subspaces to TT Coefficients Page: 457
12.3.3 From Hierarchical Format to TT Format Page: 458
12.3.4 Construction with Minimal pj Page: 460
12.3.5 Extended TT Representation Page: 460
12.3.6 Properties Page: 461
12.3.7 HOSVD Bases and Truncation Page: 462
12.4 Conversions Rr to Tp Page: 463
12.4.1 Conversion from Rr to Tp Page: 463
12.4.2 Conversion from Tp to Hr with a General Tree Page: 463
12.4.3 Conversion from Hr to Tp Page: 465
12.5 Cyclic Matrix Products and Tensor Network States Page: 467
12.5.1 Cyclic Matrix Product Representation Page: 467
12.5.2 Site-Independent Representation Page: 470
12.5.3 Tensor Network Page: 471
12.6 Representation of Symmetric and Antisymmetric Tensors Page: 472
Chapter 13: Tensor Operations Page: 473
13.1 Addition Page: 474
13.1.1 Full Representation Page: 474
13.1.2 r-Term Representation Page: 475
13.1.3 Tensor Subspace Representation Page: 475
13.1.4 Hierarchical Representation Page: 477
13.2 Entry-wise Evaluation Page: 477
13.2.1 r-Term Representation Page: 478
13.2.2 Tensor Subspace Representation Page: 478
13.2.3 Hierarchical Representation Page: 479
13.2.4 Matrix Product Representation Page: 480
13.3 Scalar Product Page: 480
13.3.1 Full Representation Page: 481
13.3.2 r-Term Representation Page: 481
13.3.3 Tensor Subspace Representation Page: 482
13.3.4 Hybrid Format Page: 484
13.3.5 Hierarchical Representation Page: 485
13.3.6 Orthonormalisation Page: 489
13.4 Change of Bases Page: 490
13.4.1 Full Representation Page: 490
13.4.2 Hybrid r-Term Representation Page: 490
13.4.3 Tensor Subspace Representation Page: 491
13.4.4 Hierarchical Representation Page: 491
13.5 General Binary Operation Page: 492
13.5.1 r-Term Representation Page: 493
13.5.2 Tensor Subspace Representation Page: 493
13.5.3 Hierarchical Representation Page: 494
13.6 Hadamard Product of Tensors Page: 495
13.7 Convolution of Tensors Page: 496
13.8 Matrix-Matrix Multiplication Page: 496
13.9 Matrix-Vector Multiplication Page: 497
13.9.1 Identical Formats Page: 498
13.9.2 Separable Form (13.25a) Page: 498
13.9.3 Elementary Kronecker Tensor (13.25b) Page: 499
13.9.4 Matrix in p-Term Format (13.25c) Page: 500
13.10 Functions of Tensors, Fixed-Point Iterations Page: 501
13.11 Example: Operations in Quantum Chemistry Applications Page: 503
Chapter 14: Tensorisation Page: 507
14.1 Basics Page: 507
14.1.1 Notations, Choice for TD Page: 507
14.1.2 Format Htens Page: 509
14.1.3 Operations with Tensorised Vectors Page: 510
14.1.4 Application to Representations by Other Formats Page: 512
14.1.5 Matricisation Page: 513
14.1.6 Generalisation to Matrices Page: 514
14.2 Approximation of Grid Functions Page: 515
14.2.1 Grid Functions Page: 515
14.2.2 Exponential Sums Page: 516
14.2.3 Polynomials Page: 516
14.2.4 Multiscale Feature and Conclusion Page: 520
14.2.5 Local Grid Refinement Page: 520
14.3 Convolution Page: 521
14.3.1 Notation Page: 521
14.3.2 Separable Operations Page: 523
14.3.3 Tensor Algebra A(l0) Page: 524
14.3.4 Algorithm Page: 531
14.4 Fast Fourier Transform Page: 534
14.4.1 FFT for Cn Vectors Page: 534
14.4.2 FFT for Tensorised Vectors Page: 535
14.5 Tensorisation of Functions Page: 537
14.5.1 Isomorphism Fn Page: 537
14.5.2 Scalar Products Page: 538
14.5.3 Convolution Page: 539
14.5.4 Continuous Functions Page: 539
Chapter 15: Multivariate Cross Approximation Page: 541
15.1 Approximation of General Tensors Page: 541
15.1.1 Approximation of Multivariate Functions Page: 542
15.1.2 Multiparametric Boundary-Value Problem and PDE with Stochastic Coefficients Page: 543
15.1.3 Function of a Tensor Page: 545
15.2 Notations Page: 546
15.3 Properties in the Matrix Case Page: 548
15.4 Case Page: 551
15.4.1 Matricisation Page: 551
15.4.2 Nestedness Page: 553
15.4.3 Algorithm Page: 555
Chapter 16: Applications to Elliptic Partial Differential Equations Page: 559
16.1 General Discretisation Strategy Page: 559
16.2 Solution of Elliptic Boundary-Value Problems Page: 560
16.2.1 Separable Differential Operator Page: 561
16.2.2 Discretisation Page: 561
16.2.3 Solution of the Linear System Page: 563
16.2.4 Accuracy Controlled Solution Page: 565
16.3 Solution of Elliptic Eigenvalue Problems Page: 566
16.3.1 Regularity of Eigensolutions Page: 566
16.3.2 Iterative Computation Page: 568
16.3.3 Alternative Approaches Page: 569
16.4 On Other Types of PDEs Page: 569
Chapter 17: Miscellaneous Topics Page: 571
17.1 Minimisation Problems on Page: 571
17.1.1 Algorithm Page: 571
17.1.2 Convergence Page: 572
17.2 Solution of Optimisation Problems Involving Tensor Formats Page: 573
17.2.1 Formulation of the Problem Page: 574
17.2.2 Reformulation, Derivatives, and Iterative Treatment Page: 575
17.3 Ordinary Differential Equations Page: 576
17.3.1 Tangent Space Page: 576
17.3.2 Dirac–Frenkel Discretisation Page: 576
17.3.3 Tensor Subspace Format Tr Page: 577
17.3.4 Hierarchical Format Hr Page: 579
17.4 ANOVA Page: 581
17.4.1 Definitions Page: 581
17.4.2 Properties Page: 582
17.4.3 Combination with Tensor Representations Page: 584
17.4.4 Symmetric Tensors Page: 584
References Page: 585
Index Page: 599