Low Rank Approximation: Algorithms, Implementation, Applications PDF
Preview Low Rank Approximation: Algorithms, Implementation, Applications
Communications and Control Engineering Forfurthervolumes: www.springer.com/series/61 SeriesEditors (cid:2) (cid:2) (cid:2) (cid:2) A.Isidori J.H.vanSchuppen E.D.Sontag M.Thoma M.Krstic Publishedtitlesinclude: StabilityandStabilizationofInfiniteDimensional SwitchedLinearSystems SystemswithApplications ZhendongSunandShuzhiS.Ge Zheng-HuaLuo,Bao-ZhuGuoandOmerMorgul SubspaceMethodsforSystemIdentification NonsmoothMechanics(Secondedition) TohruKatayama BernardBrogliato DigitalControlSystems NonlinearControlSystemsII IoanD.LandauandGianlucaZito AlbertoIsidori MultivariableComputer-controlledSystems L2-GainandPassivityTechniquesinNonlinearControl EfimN.RosenwasserandBernhardP.Lampe ArjanvanderSchaft DissipativeSystemsAnalysisandControl ControlofLinearSystemswithRegulationandInput (Secondedition) Constraints BernardBrogliato,RogelioLozano,BernhardMaschke AliSaberi,AntonA.StoorvogelandPeddapullaiah andOlavEgeland Sannuti AlgebraicMethodsforNonlinearControlSystems RobustandH∞Control GiuseppeConte,ClaudeH.MoogandAnnaM.Perdon BenM.Chen PolynomialandRationalMatrices ComputerControlledSystems TadeuszKaczorek EfimN.RosenwasserandBernhardP.Lampe Simulation-basedAlgorithmsforMarkovDecision ControlofComplexandUncertainSystems Processes StanislavV.EmelyanovandSergeyK.Korovin HyeongSooChang,MichaelC.Fu,JiaqiaoHuand StevenI.Marcus RobustControlDesignUsingH∞Methods IanR.Petersen,ValeryA.Ugrinovskiand IterativeLearningControl AndreyV.Savkin Hyo-SungAhn,KevinL.MooreandYangQuanChen ModelReductionforControlSystemDesign DistributedConsensusinMulti-vehicleCooperative GoroObinataandBrianD.O.Anderson Control WeiRenandRandalW.Beard ControlTheoryforLinearSystems HarryL.Trentelman,AntonStoorvogelandMaloHautus ControlofSingularSystemswithRandomAbrupt Changes FunctionalAdaptiveControl El-KébirBoukas SimonG.FabriandVisakanKadirkamanathan NonlinearandAdaptiveControlwithApplications Positive1Dand2DSystems AlessandroAstolfi,DimitriosKaragiannisandRomeo TadeuszKaczorek Ortega IdentificationandControlUsingVolterraModels Stabilization,OptimalandRobustControl FrancisJ.DoyleIII,RonaldK.PearsonandBabatunde AzizBelmiloudi A.Ogunnaike ControlofNonlinearDynamicalSystems Non-linearControlforUnderactuatedMechanical FelixL.Chernous’ko,IgorM.AnanievskiandSergey Systems A.Reshmin IsabelleFantoniandRogelioLozano PeriodicSystems RobustControl(Secondedition) SergioBittantiandPatrizioColaneri JürgenAckermann DiscontinuousSystems FlowControlbyFeedback YuryV.Orlov OleMortenAamoandMiroslavKrstic ConstructionsofStrictLyapunovFunctions LearningandGeneralization(Secondedition) MichaelMalisoffandFrédéricMazenc MathukumalliVidyasagar ControllingChaos ConstrainedControlandEstimation HuaguangZhang,DerongLiuandZhiliangWang GrahamC.Goodwin,MariaM.Seronand JoséA.DeDoná StabilizationofNavier-StokesFlows ViorelBarbu RandomizedAlgorithmsforAnalysisandControl ofUncertainSystems DistributedControlofMulti-agentNetworks RobertoTempo,GiuseppeCalafioreandFabrizio WeiRenandYongcanCao Dabbene Ivan Markovsky Low Rank Approximation Algorithms, Implementation, Applications IvanMarkovsky SchoolofElectronics&ComputerScience UniversityofSouthampton Southampton,UK [email protected] Additionalmaterialtothisbookcanbedownloadedfromhttp://extras.springer.com ISSN0178-5354 CommunicationsandControlEngineering ISBN978-1-4471-2226-5 e-ISBN978-1-4471-2227-2 DOI10.1007/978-1-4471-2227-2 SpringerLondonDordrechtHeidelbergNewYork BritishLibraryCataloguinginPublicationData AcataloguerecordforthisbookisavailablefromtheBritishLibrary LibraryofCongressControlNumber:2011942476 ©Springer-VerlagLondonLimited2012 Apartfromanyfairdealingforthepurposesofresearchorprivatestudy,orcriticismorreview,asper- mittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced, storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublish- ers,orinthecaseofreprographicreproductioninaccordancewiththetermsoflicensesissuedbythe CopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethosetermsshouldbesentto thepublishers. Theuseofregisterednames,trademarks,etc.,inthispublicationdoesnotimply,evenintheabsenceofa specificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandthereforefree forgeneraluse. Thepublishermakesnorepresentation,expressorimplied,withregardtotheaccuracyoftheinformation containedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsoromissions thatmaybemade. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) Preface Mathematicalmodelsareobtainedfromfirstprinciples(naturallaws,interconnec- tion, etc.) and experimental data. Modeling from first principles is common in naturalsciences,whilemodelingfromdataiscommoninengineering.Inengineer- ing, often experimental data are available and a simple approximate model is pre- ferredtoacomplicateddetailedone.Indeed,althoughoptimalpredictionandcon- trolofacomplex(high-order,nonlinear,time-varying)systemiscurrentlydifficult toachieve,robust analysisanddesignmethods,basedona simple(low-order,lin- ear,time-invariant)approximatemodel,mayachievesufficientlyhighperformance. Thisbookaddressestheproblemofdataapproximationbylow-complexitymodels. A unifying theme of the book is low rank approximation: a prototypical data modelingproblem. The rank of a matrix constructed from the data corresponds to thecomplexityofalinearmodelthatfitsthedataexactly.Thedatamatrixbeingfull rankimpliesthatthereisnoexactlowcomplexitylinearmodelforthatdata.Inthis case,theaimistofindanapproximatemodel.Oneapproachforapproximatemod- eling,consideredinthebook,istofindsmall(insomespecifiedsense)modification of the data thatrenders the modifieddata exact.The exactmodel for the modified dataisanoptimal(inthespecifiedsense)approximatemodelfortheoriginaldata. Thecorrespondingcomputationalproblemislowrankapproximation.Itallowsthe usertotradeoffaccuracyvs.complexitybyvaryingtherankoftheapproximation. Thedistancemeasureforthedatamodificationisauserchoicethatspecifiesthe desiredapproximationcriterionorreflectspriorknowledgeabouttheaccuracyofthe data.Inaddition,theusermayhavepriorknowledgeaboutthesystemthatgenerates thedata.Suchknowledgecanbeincorporatedinthemodelingproblembyimposing constraintsonthemodel.Forexample,ifthemodelisknown(orpostulated)tobe alineartime-invariantdynamicalsystem,thedatamatrixhasHankelstructureand the approximating matrix should have the same structure. This leads to a Hankel structuredlowrankapproximationproblem. A tenet of the book is: the estimation accuracy of the basic low rank approx- imation method can be improved by exploiting prior knowledge, i.e., by adding constraintsthatareknowntoholdforthedatageneratingsystem.Thispathofde- velopment leads to weighted, structured, and other constrained low rank approxi- v vi Preface mationproblems.The theory andalgorithmsof these newclasses of problemsare interestingintheirownrightandbeingapplicationdrivenarepracticallyrelevant. Stochastic estimation and deterministic approximation are two complementary aspectsofdatamodeling.Theformeraimstofindfromnoisydata,generatedbya low-complexitysystem,anestimateofthatdatageneratingsystem.Thelatteraims tofindfromexactdata,generatedbyahighcomplexitysystem,alow-complexity approximation of the data generating system. In applications both the stochastic estimation and deterministic approximation aspects are likely to be present. The dataarelikelytobeimpreciseduetomeasurementerrorsandislikelytobegener- atedbyacomplicatedphenomenonthatisnotexactlyrepresentablebyamodelin theconsideredmodelclass.Thedevelopmentofdatamodelingmethodsinsystem identification and signal processing, however, has been dominated by the stochas- tic estimation point of view. If considered, the approximation error is represented inthemainstreamdatamodelingliteratureasarandomprocess.Thisisnotnatural because the approximation error is by definition deterministic and even if consid- ered as a random process, it is not likely to satisfy standard stochastic regularity conditionssuchaszeromean,stationarity,ergodicity,andGaussianity. An exception to the stochastic paradigm in data modeling is the behavioral ap- proach, initiated by J.C. Willems in the mid-1980s. Although the behavioral ap- proachismotivatedbythedeterministicapproximationaspectofdatamodeling,it doesnotexcludethestochasticestimationapproach.Inthisbook,weusethebehav- ioralapproachasalanguagefordefiningdifferentmodelingproblemsandpresent- ingtheirsolutions.Weemphasizetheimportanceofdeterministicapproximationin datamodeling,however,weformulateandsolvestochasticestimationproblemsas lowrankapproximationproblems. Manywellknownconceptsandproblemsfromsystemsandcontrol,signalpro- cessing, and machine learning reduce to low rank approximation. Generic exam- plesinsystemtheoryaremodelreductionandsystemidentification.Theprincipal componentanalysismethodinmachinelearningisequivalenttolowrankapprox- imation, which suggests that related dimensionality reduction, classification, and informationretrievalproblemscanbephrasedaslowrankapproximationproblems. Sylvesterstructuredlowrankapproximationhasapplicationsincomputationswith polynomialsandisrelatedtomethodsfromcomputeralgebra. The developed ideas lead to algorithms, which are implemented in software. The algorithms clarify the ideas and the software implementation clarifies the al- gorithms.Indeed,thesoftwareistheultimateunambiguousdescriptionofhowthe ideas are put to work. In addition, the provided software allows the reader to re- producetheexamplesinthebookandtomodifythem.Theexpositionreflectsthe sequence theory(cid:3)→algorithms(cid:3)→implementation. Correspondingly,thetextisinterwovenwithcodethatgeneratesthenumericalex- amplesbeingdiscussed. Preface vii Prerequisitesand practiceproblems Acommonfeatureofthecurrentresearchactivityinallareasofscienceandengi- neeringisthenarrowspecialization.Inthisbook,wepickapplicationsinthebroad area of data modeling, posing and solving them as low rank approximation prob- lems.Thisunifiesseeminglyunrelatedapplicationsandsolutiontechniquesbyem- phasisingtheircommonaspects(e.g.,complexity–accuracytrade-off)andabstract- ing from the application specific details, terminology, and implementation details. Despiteofthefactthatapplicationsinsystemsandcontrol,signalprocessing,ma- chinelearning,andcomputervisionareusedasexamples,theonlyrealprerequisites forfollowingthepresentationisknowledgeoflinearalgebra. Thebookisintendedtobeusedforselfstudybyresearchersintheareaofdata modelingandbyadvancedundergraduate/graduatelevelstudentsasacomplemen- tary text for a course on system identification or machine learning. In either case, theexpectedknowledgeisundergraduatelevellinearalgebra.Inaddition,MATLAB codeisused,sothatfamiliaritywithMATLABprogramminglanguageisrequired. Passivereadingofthebookgivesabroadperspectiveonthesubject.Deeperun- derstanding,however,requiresactiveinvolvement,suchassupplyingmissingjusti- ficationofstatementsandspecificexamplesofthegeneralconcepts,applicationand modificationofpresentedideas,andsolutionoftheprovidedexercisesandpractice problems. There are two types of practice problem: analytical, asking for a proof ofastatementclarifyingorexpandingthematerialinthebook,andcomputational, askingforexperimentswithrealorsimulateddataofspecificapplications.Mostof theproblemsareeasytomediumdifficulty.Afewproblems(markedwithstars)can beusedassmallresearchprojects. Thecodeinthebook,availablefrom http://extra.springer.com/ hasbeentestedwithMATLAB7.9,runningunderLinux,andusestheOptimization Toolbox4.3,ControlSystemToolbox8.4,andSymbolicMathToolbox5.3.Aver- sion of the code that is compatible with Octave (a free alternative to MATLAB) is alsoavailablefromthebook’swebpage. Acknowledgements AnumberofindividualsandtheEuropeanResearchCouncilcontributedandsup- ported me during the preparation of the book. Oliver Jackson—Springer’s edi- tor (engineering)—encouraged me to embark on the project. My colleagues in ESAT/SISTA,K.U.LeuvenandECS/ISIS,Southampton,UKcreatedtherighten- vironmentfordevelopingtheideasinthebook.Inparticular,IamindebttoJanC. Willems (SISTA) for his personal guidance and example of critical thinking. The behavioralapproachthatJaninitiatedintheearly1980’sispresentinthisbook. Maarten De Vos, Diana Sima, Konstantin Usevich, and Jan Willems proofread chaptersofthebookandsuggestedimprovements.Igratefullyacknowledgefunding viii Preface fromtheEuropeanResearchCouncilundertheEuropeanUnion’sSeventhFrame- workProgramme(FP7/2007–2013)/ERCGrantagreementnumber258581“Struc- turedlow-rankapproximation:Theory,algorithms,andapplications”. Southampton,UK IvanMarkovsky Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 ClassicalandBehavioralParadigmsforDataModeling . . . . . . 1 1.2 MotivatingExampleforLowRankApproximation . . . . . . . . . 3 1.3 OverviewofApplications . . . . . . . . . . . . . . . . . . . . . . 7 1.4 OverviewofAlgorithms . . . . . . . . . . . . . . . . . . . . . . . 21 1.5 LiterateProgramming . . . . . . . . . . . . . . . . . . . . . . . . 24 1.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 PartI LinearModelingProblems 2 FromDatatoModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.1 LinearStaticModelRepresentations . . . . . . . . . . . . . . . . 35 2.2 LinearTime-InvariantModelRepresentations . . . . . . . . . . . 45 2.3 ExactandApproximateDataModeling . . . . . . . . . . . . . . . 52 2.4 UnstructuredLowRankApproximation . . . . . . . . . . . . . . 60 2.5 StructuredLowRankApproximation . . . . . . . . . . . . . . . . 67 2.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.1 SubspaceMethods . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.2 AlgorithmsBasedonLocalOptimization . . . . . . . . . . . . . . 81 3.3 DataModelingUsingtheNuclearNormHeuristic . . . . . . . . . 96 3.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4 ApplicationsinSystem,Control,andSignalProcessing . . . . . . . . 107 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.2 ModelReduction. . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.3 SystemIdentification . . . . . . . . . . . . . . . . . . . . . . . . 115 4.4 AnalysisandSynthesis . . . . . . . . . . . . . . . . . . . . . . . 122 ix x Contents 4.5 SimulationExamples . . . . . . . . . . . . . . . . . . . . . . . . 125 4.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 PartII MiscellaneousGeneralizations 5 MissingData,Centering,andConstraints . . . . . . . . . . . . . . . 135 5.1 WeightedLowRankApproximationwithMissingData . . . . . . 135 5.2 AffineDataModeling . . . . . . . . . . . . . . . . . . . . . . . . 147 5.3 ComplexLeastSquaresProblemwithConstrainedPhase . . . . . 156 5.4 ApproximateLowRankFactorizationwithStructuredFactors . . . 163 5.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 6 NonlinearStaticDataModeling . . . . . . . . . . . . . . . . . . . . . 179 6.1 AFrameworkforNonlinearStaticDataModeling . . . . . . . . . 179 6.2 NonlinearLowRankApproximation . . . . . . . . . . . . . . . . 182 6.3 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 6.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 6.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 7 FastMeasurementsofSlowProcesses. . . . . . . . . . . . . . . . . . 199 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 7.2 EstimationwithKnownMeasurementProcessDynamics . . . . . 202 7.3 EstimationwithUnknownMeasurementProcessDynamics . . . . 204 7.4 ExamplesandReal-LifeTesting . . . . . . . . . . . . . . . . . . . 212 7.5 AuxiliaryFunctions . . . . . . . . . . . . . . . . . . . . . . . . . 222 7.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 AppendixA ApproximateSolutionof anOverdeterminedSystem ofEquations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 AppendixB Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 AppendixP Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 Notation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 ListofCodeChunks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 FunctionsandScriptsIndex . . . . . . . . . . . . . . . . . . . . . . . . . 251 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
Description:The list of books you might like
