Low-Rank Approximation

280 Pages·2019·3.86 MB·English

by  Ivan Markovsky

Communications and Control Engineering
Ivan Markovsky
Low-Rank
Approximation
Algorithms, Implementation,
Applications
Second Edition
Communications and Control Engineering
Series editors
Alberto Isidori, Roma, Italy
Jan H. van Schuppen, Amsterdam, The Netherlands
Eduardo D. Sontag, Boston, USA
Miroslav Krstic, La Jolla, USA
Communications and Control Engineering is a high-level academic monograph
series publishing research in control and systems theory, control engineering and
communications. It has worldwide distribution to engineers, researchers, educators
(severalofthetitlesinthisseriesfinduseasadvancedtextbooksalthoughthatisnot
their primary purpose), and libraries.
Theseriesreflectsthemajortechnologicalandmathematicaladvancesthathave
a great impact in the fields of communication and control. The range of areas to
which control and systems theory is applied is broadening rapidly with particular
growth being noticeable in the fields of finance and biologically-inspired control.
Books in this series generally pull together many related research threads in more
matureareasofthesubjectthanthehighly-specialisedvolumesofLectureNotesin
ControlandInformationSciences.Thisseries’smathematicalandcontrol-theoretic
emphasis is complemented by Advances in Industrial Control which provides a
much more applied, engineering-oriented outlook.
Publishing Ethics: Researchers should conduct their research from research
proposaltopublication inlinewithbestpracticesandcodesofconductofrelevant
professional bodies and/or national and international regulatory bodies. For more
details on individual ethics matters please see:
https://www.springer.com/gp/authors-editors/journal-author/journal-author-
helpdesk/publishing-ethics/14214
More information about this series at http://www.springer.com/series/61
Ivan Markovsky
Low-Rank Approximation
Algorithms, Implementation, Applications
Second Edition
123
IvanMarkovsky
Department ELEC
Vrije Universiteit Brussel
Brussels, Belgium
Additional material tothis book,including lecture slides, canbedownloaded from http://
extras.springer.com.
ISSN 0178-5354 ISSN 2197-7119 (electronic)
Communications andControl Engineering
ISBN978-3-319-89619-9 ISBN978-3-319-89620-5 (eBook)
https://doi.org/10.1007/978-3-319-89620-5
LibraryofCongressControlNumber:2018939004
MATLAB® is a registered trademark of The MathWorks, Inc., 3 Apple Hill Drive, Natick, MA
01760-2098,USA,http://www.mathworks.com.
MathematicsSubjectClassification(2010): 93A30,93B30,93B40,37M10,41A29,62J12,49N30
1stedition:©Springer-VerlagLondonLimited2012
2ndedition:©SpringerInternationalPublishingAG,partofSpringerNature2019
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Simple linear models are commonly used in engineering despite of the fact that
the real world is often nonlinear. At the same time as being simple, however, the
models have to be accurate. Mathematical models are obtained from first princi-
ples(naturallaws,interconnection,etc.)andexperimentaldata.Modelingfromfirst
principlesaimsatexactmodels.Thisapproachisthedefaultoneinnaturalsciences.
Modeling from data, on the other hand, allows us to tune complexity vs accuracy.
Thisapproachiscommoninengineering,whereexperimentaldataisavailableand
asimpleapproximatemodelispreferredtoacomplicatedexactone.Forexample,
optimal control of a complex (high-order, nonlinear, time-varying) system is cur-
rentlyintractable,however,usingsimple(low-order,linear,time-invariant)approxi-
matemodelsandrobustcontrolmethodsmayachievesufficientlyhighperformance.
Thetopicofthebookisdatamodelingbyreducedcomplexitymathematicalmodels.
A unifying theme of the book is low-rank approximation: a prototypical data
modeling problem. The rank of a matrix constructed from the data corresponds to
thecomplexityofalinearmodelthatfitsthedataexactly.Thedatamatrixbeingfull
rankimpliesthatthereisnoexactlowcomplexitylinearmodelforthatdata.Inthis
case,theaimistofindanapproximatemodel.Theapproximatemodelingapproach
considered in the book is to find small (in some specified sense) modification of
the data that renders the modified data exact. The exact model for the modified
dataisanoptimal(inthespecifiedsense)approximatemodelfortheoriginaldata.
The corresponding computational problem is low-rank approximation of the data
matrix.Therankoftheapproximationallowsustotrade-offaccuracyvscomplexity.
Apart from the choice of the rank, the book covers two other user choices: the
approximationcriterionandthematrixstructure.Thesechoicescorrespondtoprior
knowledgeabouttheaccuracyofthedataandthemodelclass,respectively.Anex-
ampleofamatrixstructure,calledHankelstructure,correspondstothelineartime-
invariant model class. The book presents local optimization, subspace and convex
relaxation-basedheuristicmethodsforaHankelstructuredlow-rankapproximation.
v
vi Preface
Low-rank approximation is a core problem in applications. Generic examples
in systems and control are model reduction and system identification. Low-rank
approximationisequivalenttotheprincipalcomponentanalysismethodinmachine
learning.Indeed,dimensionalityreduction,classification,andinformationretrieval
problemscanbeposedandsolvedasparticularlow-rankapproximationproblems.
Sylvester structured low-rank approximation has applications in computer algebra
forthedecoupling,factorization,andcommondivisorcomputationofpolynomials.
The book covers two complementary aspects of data modeling: stochastic esti-
mation and deterministic approximation. The former aims to find from noisy data
that is generated by a low-complexity system an estimate of that data generating
system. The latter aims to find from exact data that is generated by a high com-
plexity system a low-complexity approximation of the data generating system. In
applications,boththestochasticestimationanddeterministicapproximationaspects
arepresent:thedataisimpreciseduetomeasurementerrorsandispossiblygener-
atedbyacomplicatedphenomenonthatisnotexactlyrepresentablebyamodelin
theconsideredmodelclass.Thedevelopmentofdatamodelingmethodsinsystem
identificationandsignalprocessing,however,hasbeendominatedbythestochastic
estimationpointofview.Theapproximationerrorisrepresentedinthemainstream
datamodelingliteratureasarandomprocess.However,theapproximationerroris
deterministic,sothatitisnotnaturaltotreatitasarandomprocess.Moreover,the
approximationerrordoesnotsatisfystochasticregularityconditionssuchasstation-
arity,ergodicity,andGaussianity.Theseaspectscomplicatethestochasticapproach.
An exception to the stochastic paradigm in data modeling is the behavioral ap-
proach,initiatedbyJ.C.Willemsinthemid80’s.Althoughthebehavioralapproach
ismotivatedbythedeterministicapproximationaspectofdatamodeling,itdoesnot
excludethestochasticestimationapproach.Thisbookusesthebehavioralapproach
asaunifyinglanguageindefiningmodelingproblemsandpresentingtheirsolutions.
Thetheoryandmethodsdevelopedinthebookleadtoalgorithms,whichareim-
plementedinsoftware.Thealgorithmsclarifytheideasandviceversethesoftware
implementation clarifies the algorithms. Indeed, the software is the ultimate un-
ambiguousdescriptionofhowthetheoryandmethodsareappliedinpractice.The
softwareallowsthereadertoreproduceandtomodifytheexamplesinthebook.The
expositionreflectsthesequence:theory algorithms implementation.Corre-
7→ 7→
spondingly,thetextisinterwovenwithcodethatgeneratesthenumericalexamples
being discussed. The reader can try out these methods on their own problems and
data.Experimentingwiththemethodsandworkingontheexerciseswouldleadyou
toadeeperunderstandingofthetheoryandhands-onexperiencewiththemethods.
Leuven, Belgium IvanMarkovsky
January2018
Acknowledgements
Anumberofindividualsandfundingagenciessupportedmeduringthepreparation
of the book. Oliver Jackson—Springer’s editor engineering—encouraged me to
embarkontheprojectandtopreparethissecondeditionofthebook.Mycolleagues
at ESAT/STADIUS (formally SISTA), K. U. Leuven, ECS/VLC (formerly ISIS),
UniversityofSouthampton,andtheDepartmentELEC,VrijeUniversiteitBrussels
createdtherightenvironmentfordevelopingtheideasinthebook.Inparticular,Iam
indebttoJanC.Willemsforhispersonalguidanceandexampleofcriticalthinking.
ThebehavioralapproachthatJaninitiatedintheearly1980sispresentinthisbook.
M. De Vos, D. Sima, K. Usevich, and J. C. Willems proofread chapters of the
first edition and suggested improvements. I gratefully acknowledge funding from:
(cid:129) the European Research Council (ERC) under the European Union’s Seventh
FrameworkProgramme(FP7/2007–2013)/ERCGrantagreementnumber258581
“Structuredlow-rankapproximation:Theory,algorithms,andapplications”;
(cid:129) theFundforScientificResearch (FWO-Vlaanderen),FWO projects G028015N
“Decoupling multivariate polynomials in nonlinear system identification” and
G090117N “Block-oriented nonlinear identification using Volterra series”;
(cid:129) the Belgian Science Policy Office Interuniversity Attraction Poles Programme,
network on “Dynamical Systems, Control, and Optimization” (DYSCO); and
(cid:129) theFWO/FNRSExcellenceofScienceproject“Structuredlow-rankmatrix/tensor
approximation:numericaloptimization-basedalgorithmsandapplications”.
vii
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Classical and Behavioral Paradigms for Data Modeling . . . . . . . . 1
1.2 Motivating Example for Low-Rank approximation . . . . . . . . . . . . 3
1.3 Overview of Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Overview of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Part I Linear Modeling Problems
2 From Data to Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.1 Static Model Representations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.2 Dynamic Model Representations . . . . . . . . . . . . . . . . . . . . . . . . . 46
2.3 Stochastic Model Representation . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.4 Exact and Approximate Data Modeling . . . . . . . . . . . . . . . . . . . . 59
2.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3 Exact Modeling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.1 Kung’s Realization Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2 Impulse Response Computation. . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.3 Stochastic System Identification. . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.4 Missing Data Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4 Approximate Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.1 Unstructured Low-Rank Approximation. . . . . . . . . . . . . . . . . . . . 100
4.2 Structured Low-Rank Approximation. . . . . . . . . . . . . . . . . . . . . . 109
4.3 Nuclear Norm Heuristic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4.4 Missing Data Estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
ix
x Contents
Part II Applications and Generalizations
5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.1 Model Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
5.2 System Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
5.3 Approximate Common Factor of Two Polynomials . . . . . . . . . . . 149
5.4 Pole Placement by a Low-Order Controller . . . . . . . . . . . . . . . . . 154
5.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6 Data-Driven Filtering and Control . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.1 Model-Based Versus Data-Driven Paradigms . . . . . . . . . . . . . . . . 162
6.2 Missing Data Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
6.3 Estimation and Control Examples . . . . . . . . . . . . . . . . . . . . . . . . 164
6.4 Solution via Matrix Completion . . . . . . . . . . . . . . . . . . . . . . . . . 166
6.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
7 Nonlinear Modeling Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
7.1 A Framework for Nonlinear Data Modeling. . . . . . . . . . . . . . . . . 174
7.2 Nonlinear Low-Rank Approximation . . . . . . . . . . . . . . . . . . . . . . 178
7.3 Computational Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
7.4 Identification of Polynomial Time-Invariant Systems . . . . . . . . . . 190
7.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
8 Dealing with Prior Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
8.1 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
8.2 Approximate Low-Rank Factorization . . . . . . . . . . . . . . . . . . . . . 206
8.3 Complex Least Squares with Constrained Phase. . . . . . . . . . . . . . 211
8.4 Blind Identification with Deterministic Input Model . . . . . . . . . . . 217
8.5 Notes and References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Appendix A: Total Least Squares .. .... .... .... .... .... ..... .... 225
Appendix B: Solutions to the Exercises.. .... .... .... .... ..... .... 233
Appendix C: Proofs .... .... ..... .... .... .... .... .... ..... .... 259
Index .... .... .... .... .... ..... .... .... .... .... .... ..... .... 269