Low Rank Approximation: Algorithms, Implementation, Applications

260 Pages·2012·3.41 MB·English

by  Ivan Markovsky (auth.)

Communications and Control Engineering
Forfurthervolumes:
www.springer.com/series/61
SeriesEditors
(cid:2) (cid:2) (cid:2) (cid:2)
A.Isidori J.H.vanSchuppen E.D.Sontag M.Thoma M.Krstic
Publishedtitlesinclude:
StabilityandStabilizationofInfiniteDimensional SwitchedLinearSystems
SystemswithApplications ZhendongSunandShuzhiS.Ge
Zheng-HuaLuo,Bao-ZhuGuoandOmerMorgul
SubspaceMethodsforSystemIdentification
NonsmoothMechanics(Secondedition) TohruKatayama
BernardBrogliato
DigitalControlSystems
NonlinearControlSystemsII IoanD.LandauandGianlucaZito
AlbertoIsidori
MultivariableComputer-controlledSystems
L2-GainandPassivityTechniquesinNonlinearControl EfimN.RosenwasserandBernhardP.Lampe
ArjanvanderSchaft
DissipativeSystemsAnalysisandControl
ControlofLinearSystemswithRegulationandInput (Secondedition)
Constraints BernardBrogliato,RogelioLozano,BernhardMaschke
AliSaberi,AntonA.StoorvogelandPeddapullaiah andOlavEgeland
Sannuti
AlgebraicMethodsforNonlinearControlSystems
RobustandH∞Control GiuseppeConte,ClaudeH.MoogandAnnaM.Perdon
BenM.Chen
PolynomialandRationalMatrices
ComputerControlledSystems TadeuszKaczorek
EfimN.RosenwasserandBernhardP.Lampe
Simulation-basedAlgorithmsforMarkovDecision
ControlofComplexandUncertainSystems Processes
StanislavV.EmelyanovandSergeyK.Korovin HyeongSooChang,MichaelC.Fu,JiaqiaoHuand
StevenI.Marcus
RobustControlDesignUsingH∞Methods
IanR.Petersen,ValeryA.Ugrinovskiand IterativeLearningControl
AndreyV.Savkin Hyo-SungAhn,KevinL.MooreandYangQuanChen
ModelReductionforControlSystemDesign DistributedConsensusinMulti-vehicleCooperative
GoroObinataandBrianD.O.Anderson Control
WeiRenandRandalW.Beard
ControlTheoryforLinearSystems
HarryL.Trentelman,AntonStoorvogelandMaloHautus ControlofSingularSystemswithRandomAbrupt
Changes
FunctionalAdaptiveControl
El-KébirBoukas
SimonG.FabriandVisakanKadirkamanathan
NonlinearandAdaptiveControlwithApplications
Positive1Dand2DSystems
AlessandroAstolfi,DimitriosKaragiannisandRomeo
TadeuszKaczorek
Ortega
IdentificationandControlUsingVolterraModels
Stabilization,OptimalandRobustControl
FrancisJ.DoyleIII,RonaldK.PearsonandBabatunde
AzizBelmiloudi
A.Ogunnaike
ControlofNonlinearDynamicalSystems
Non-linearControlforUnderactuatedMechanical
FelixL.Chernous’ko,IgorM.AnanievskiandSergey
Systems
A.Reshmin
IsabelleFantoniandRogelioLozano
PeriodicSystems
RobustControl(Secondedition)
SergioBittantiandPatrizioColaneri
JürgenAckermann
DiscontinuousSystems
FlowControlbyFeedback
YuryV.Orlov
OleMortenAamoandMiroslavKrstic
ConstructionsofStrictLyapunovFunctions
LearningandGeneralization(Secondedition)
MichaelMalisoffandFrédéricMazenc
MathukumalliVidyasagar
ControllingChaos
ConstrainedControlandEstimation
HuaguangZhang,DerongLiuandZhiliangWang
GrahamC.Goodwin,MariaM.Seronand
JoséA.DeDoná StabilizationofNavier-StokesFlows
ViorelBarbu
RandomizedAlgorithmsforAnalysisandControl
ofUncertainSystems DistributedControlofMulti-agentNetworks
RobertoTempo,GiuseppeCalafioreandFabrizio WeiRenandYongcanCao
Dabbene
Ivan Markovsky
Low Rank
Approximation
Algorithms, Implementation,
Applications
IvanMarkovsky
SchoolofElectronics&ComputerScience
UniversityofSouthampton
Southampton,UK
[e-mail]
Additionalmaterialtothisbookcanbedownloadedfromhttp://extras.springer.com
ISSN0178-5354 CommunicationsandControlEngineering
ISBN978-1-4471-2226-5 e-ISBN978-1-4471-2227-2
DOI10.1007/978-1-4471-2227-2
SpringerLondonDordrechtHeidelbergNewYork
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressControlNumber:2011942476
©Springer-VerlagLondonLimited2012
Apartfromanyfairdealingforthepurposesofresearchorprivatestudy,orcriticismorreview,asper-
mittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,
storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublish-
ers,orinthecaseofreprographicreproductioninaccordancewiththetermsoflicensesissuedbythe
CopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethosetermsshouldbesentto
thepublishers.
Theuseofregisterednames,trademarks,etc.,inthispublicationdoesnotimply,evenintheabsenceofa
specificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandthereforefree
forgeneraluse.
Thepublishermakesnorepresentation,expressorimplied,withregardtotheaccuracyoftheinformation
containedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsoromissions
thatmaybemade.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Mathematicalmodelsareobtainedfromfirstprinciples(naturallaws,interconnec-
tion, etc.) and experimental data. Modeling from first principles is common in
naturalsciences,whilemodelingfromdataiscommoninengineering.Inengineer-
ing, often experimental data are available and a simple approximate model is pre-
ferredtoacomplicateddetailedone.Indeed,althoughoptimalpredictionandcon-
trolofacomplex(high-order,nonlinear,time-varying)systemiscurrentlydifficult
toachieve,robust analysisanddesignmethods,basedona simple(low-order,lin-
ear,time-invariant)approximatemodel,mayachievesufficientlyhighperformance.
Thisbookaddressestheproblemofdataapproximationbylow-complexitymodels.
A unifying theme of the book is low rank approximation: a prototypical data
modelingproblem. The rank of a matrix constructed from the data corresponds to
thecomplexityofalinearmodelthatfitsthedataexactly.Thedatamatrixbeingfull
rankimpliesthatthereisnoexactlowcomplexitylinearmodelforthatdata.Inthis
case,theaimistofindanapproximatemodel.Oneapproachforapproximatemod-
eling,consideredinthebook,istofindsmall(insomespecifiedsense)modification
of the data thatrenders the modifieddata exact.The exactmodel for the modified
dataisanoptimal(inthespecifiedsense)approximatemodelfortheoriginaldata.
Thecorrespondingcomputationalproblemislowrankapproximation.Itallowsthe
usertotradeoffaccuracyvs.complexitybyvaryingtherankoftheapproximation.
Thedistancemeasureforthedatamodificationisauserchoicethatspecifiesthe
desiredapproximationcriterionorreflectspriorknowledgeabouttheaccuracyofthe
data.Inaddition,theusermayhavepriorknowledgeaboutthesystemthatgenerates
thedata.Suchknowledgecanbeincorporatedinthemodelingproblembyimposing
constraintsonthemodel.Forexample,ifthemodelisknown(orpostulated)tobe
alineartime-invariantdynamicalsystem,thedatamatrixhasHankelstructureand
the approximating matrix should have the same structure. This leads to a Hankel
structuredlowrankapproximationproblem.
A tenet of the book is: the estimation accuracy of the basic low rank approx-
imation method can be improved by exploiting prior knowledge, i.e., by adding
constraintsthatareknowntoholdforthedatageneratingsystem.Thispathofde-
velopment leads to weighted, structured, and other constrained low rank approxi-
v
vi Preface
mationproblems.The theory andalgorithmsof these newclasses of problemsare
interestingintheirownrightandbeingapplicationdrivenarepracticallyrelevant.
Stochastic estimation and deterministic approximation are two complementary
aspectsofdatamodeling.Theformeraimstofindfromnoisydata,generatedbya
low-complexitysystem,anestimateofthatdatageneratingsystem.Thelatteraims
tofindfromexactdata,generatedbyahighcomplexitysystem,alow-complexity
approximation of the data generating system. In applications both the stochastic
estimation and deterministic approximation aspects are likely to be present. The
dataarelikelytobeimpreciseduetomeasurementerrorsandislikelytobegener-
atedbyacomplicatedphenomenonthatisnotexactlyrepresentablebyamodelin
theconsideredmodelclass.Thedevelopmentofdatamodelingmethodsinsystem
identification and signal processing, however, has been dominated by the stochas-
tic estimation point of view. If considered, the approximation error is represented
inthemainstreamdatamodelingliteratureasarandomprocess.Thisisnotnatural
because the approximation error is by definition deterministic and even if consid-
ered as a random process, it is not likely to satisfy standard stochastic regularity
conditionssuchaszeromean,stationarity,ergodicity,andGaussianity.
An exception to the stochastic paradigm in data modeling is the behavioral ap-
proach, initiated by J.C. Willems in the mid-1980s. Although the behavioral ap-
proachismotivatedbythedeterministicapproximationaspectofdatamodeling,it
doesnotexcludethestochasticestimationapproach.Inthisbook,weusethebehav-
ioralapproachasalanguagefordefiningdifferentmodelingproblemsandpresent-
ingtheirsolutions.Weemphasizetheimportanceofdeterministicapproximationin
datamodeling,however,weformulateandsolvestochasticestimationproblemsas
lowrankapproximationproblems.
Manywellknownconceptsandproblemsfromsystemsandcontrol,signalpro-
cessing, and machine learning reduce to low rank approximation. Generic exam-
plesinsystemtheoryaremodelreductionandsystemidentification.Theprincipal
componentanalysismethodinmachinelearningisequivalenttolowrankapprox-
imation, which suggests that related dimensionality reduction, classification, and
informationretrievalproblemscanbephrasedaslowrankapproximationproblems.
Sylvesterstructuredlowrankapproximationhasapplicationsincomputationswith
polynomialsandisrelatedtomethodsfromcomputeralgebra.
The developed ideas lead to algorithms, which are implemented in software.
The algorithms clarify the ideas and the software implementation clarifies the al-
gorithms.Indeed,thesoftwareistheultimateunambiguousdescriptionofhowthe
ideas are put to work. In addition, the provided software allows the reader to re-
producetheexamplesinthebookandtomodifythem.Theexpositionreflectsthe
sequence
theory(cid:3)→algorithms(cid:3)→implementation.
Correspondingly,thetextisinterwovenwithcodethatgeneratesthenumericalex-
amplesbeingdiscussed.
Preface vii
Prerequisitesand practiceproblems
Acommonfeatureofthecurrentresearchactivityinallareasofscienceandengi-
neeringisthenarrowspecialization.Inthisbook,wepickapplicationsinthebroad
area of data modeling, posing and solving them as low rank approximation prob-
lems.Thisunifiesseeminglyunrelatedapplicationsandsolutiontechniquesbyem-
phasisingtheircommonaspects(e.g.,complexity–accuracytrade-off)andabstract-
ing from the application specific details, terminology, and implementation details.
Despiteofthefactthatapplicationsinsystemsandcontrol,signalprocessing,ma-
chinelearning,andcomputervisionareusedasexamples,theonlyrealprerequisites
forfollowingthepresentationisknowledgeoflinearalgebra.
Thebookisintendedtobeusedforselfstudybyresearchersintheareaofdata
modelingandbyadvancedundergraduate/graduatelevelstudentsasacomplemen-
tary text for a course on system identification or machine learning. In either case,
theexpectedknowledgeisundergraduatelevellinearalgebra.Inaddition,MATLAB
codeisused,sothatfamiliaritywithMATLABprogramminglanguageisrequired.
Passivereadingofthebookgivesabroadperspectiveonthesubject.Deeperun-
derstanding,however,requiresactiveinvolvement,suchassupplyingmissingjusti-
ficationofstatementsandspecificexamplesofthegeneralconcepts,applicationand
modificationofpresentedideas,andsolutionoftheprovidedexercisesandpractice
problems. There are two types of practice problem: analytical, asking for a proof
ofastatementclarifyingorexpandingthematerialinthebook,andcomputational,
askingforexperimentswithrealorsimulateddataofspecificapplications.Mostof
theproblemsareeasytomediumdifficulty.Afewproblems(markedwithstars)can
beusedassmallresearchprojects.
Thecodeinthebook,availablefrom
http://extra.springer.com/
hasbeentestedwithMATLAB7.9,runningunderLinux,andusestheOptimization
Toolbox4.3,ControlSystemToolbox8.4,andSymbolicMathToolbox5.3.Aver-
sion of the code that is compatible with Octave (a free alternative to MATLAB) is
alsoavailablefromthebook’swebpage.
Acknowledgements
AnumberofindividualsandtheEuropeanResearchCouncilcontributedandsup-
ported me during the preparation of the book. Oliver Jackson—Springer’s edi-
tor (engineering)—encouraged me to embark on the project. My colleagues in
ESAT/SISTA,K.U.LeuvenandECS/ISIS,Southampton,UKcreatedtherighten-
vironmentfordevelopingtheideasinthebook.Inparticular,IamindebttoJanC.
Willems (SISTA) for his personal guidance and example of critical thinking. The
behavioralapproachthatJaninitiatedintheearly1980’sispresentinthisbook.
Maarten De Vos, Diana Sima, Konstantin Usevich, and Jan Willems proofread
chaptersofthebookandsuggestedimprovements.Igratefullyacknowledgefunding
viii Preface
fromtheEuropeanResearchCouncilundertheEuropeanUnion’sSeventhFrame-
workProgramme(FP7/2007–2013)/ERCGrantagreementnumber258581“Struc-
turedlow-rankapproximation:Theory,algorithms,andapplications”.
Southampton,UK IvanMarkovsky
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 ClassicalandBehavioralParadigmsforDataModeling . . . . . . 1
1.2 MotivatingExampleforLowRankApproximation . . . . . . . . . 3
1.3 OverviewofApplications . . . . . . . . . . . . . . . . . . . . . . 7
1.4 OverviewofAlgorithms . . . . . . . . . . . . . . . . . . . . . . . 21
1.5 LiterateProgramming . . . . . . . . . . . . . . . . . . . . . . . . 24
1.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
PartI LinearModelingProblems
2 FromDatatoModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.1 LinearStaticModelRepresentations . . . . . . . . . . . . . . . . 35
2.2 LinearTime-InvariantModelRepresentations . . . . . . . . . . . 45
2.3 ExactandApproximateDataModeling . . . . . . . . . . . . . . . 52
2.4 UnstructuredLowRankApproximation . . . . . . . . . . . . . . 60
2.5 StructuredLowRankApproximation . . . . . . . . . . . . . . . . 67
2.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.1 SubspaceMethods . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.2 AlgorithmsBasedonLocalOptimization . . . . . . . . . . . . . . 81
3.3 DataModelingUsingtheNuclearNormHeuristic . . . . . . . . . 96
3.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4 ApplicationsinSystem,Control,andSignalProcessing . . . . . . . . 107
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.2 ModelReduction. . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.3 SystemIdentification . . . . . . . . . . . . . . . . . . . . . . . . 115
4.4 AnalysisandSynthesis . . . . . . . . . . . . . . . . . . . . . . . 122
ix
x Contents
4.5 SimulationExamples . . . . . . . . . . . . . . . . . . . . . . . . 125
4.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
PartII MiscellaneousGeneralizations
5 MissingData,Centering,andConstraints . . . . . . . . . . . . . . . 135
5.1 WeightedLowRankApproximationwithMissingData . . . . . . 135
5.2 AffineDataModeling . . . . . . . . . . . . . . . . . . . . . . . . 147
5.3 ComplexLeastSquaresProblemwithConstrainedPhase . . . . . 156
5.4 ApproximateLowRankFactorizationwithStructuredFactors . . . 163
5.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6 NonlinearStaticDataModeling . . . . . . . . . . . . . . . . . . . . . 179
6.1 AFrameworkforNonlinearStaticDataModeling . . . . . . . . . 179
6.2 NonlinearLowRankApproximation . . . . . . . . . . . . . . . . 182
6.3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
6.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
6.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
7 FastMeasurementsofSlowProcesses. . . . . . . . . . . . . . . . . . 199
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
7.2 EstimationwithKnownMeasurementProcessDynamics . . . . . 202
7.3 EstimationwithUnknownMeasurementProcessDynamics . . . . 204
7.4 ExamplesandReal-LifeTesting . . . . . . . . . . . . . . . . . . . 212
7.5 AuxiliaryFunctions . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
AppendixA ApproximateSolutionof anOverdeterminedSystem
ofEquations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
AppendixB Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
AppendixP Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
Notation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
ListofCodeChunks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
FunctionsandScriptsIndex . . . . . . . . . . . . . . . . . . . . . . . . . 251
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253