Table Of Content

Yun Fu Editor Low-Rank and Sparse Modeling for Visual Analysis Low-Rank and Sparse Modeling for Visual Analysis Yun Fu Editor Low-Rank and Sparse Modeling for Visual Analysis 123 Editor YunFu Northeastern University Boston,MA USA ISBN 978-3-319-11999-1 ISBN 978-3-319-12000-3 (eBook) DOI 10.1007/978-3-319-12000-3 LibraryofCongressControlNumber:2014951660 SpringerChamHeidelbergNewYorkDordrechtLondon ©SpringerInternationalPublishingSwitzerland2014 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifically for the purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe work. Duplication of this publication or parts thereof is permitted only under the provisions of theCopyrightLawofthePublisher’slocation,initscurrentversion,andpermissionforusemustalways beobtainedfromSpringer.PermissionsforusemaybeobtainedthroughRightsLinkattheCopyright ClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. While the advice and information in this book are believed to be true and accurate at the date of publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with respecttothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) Preface Visualanalysishasbeenprevalentlatelysince large-scaledataaregeneratedevery dayfrom advance vision andimaging devices.Low-rankandsparsemodellingare emerging mathematical tools dealing with uncertainties of real-world visual data. This book provides a unique view of low-rank and sparse computing, especially approximation, recovery, representation, scaling, coding, embedding, and learning among unconstrained visual data. These techniques will significantly advance existing methodologies of image and video analysis and understanding by taking advantage of low-rank and sparse modeling. Visual analysis under uncertainty stands at the core of numerous real-world applications which bring broad impacts andgeneratesignificantacademicandindustrialvalues.Asaprofessionalreference book and research monograph, this book, through several chapters covers popular research topics from several fields such as Pattern Recognition, Computer Vision, Big Data, Social Media, Image and Video Classification, and Machine Learning. These chapters, contributed by the editor, top experts, and practitioners, comple- menteachotherfromvariousperspectiveandcomposeasolidoverviewofthelow- rank and sparse modelling techniques for visual analysis. Readers from different backgrounds may all benefit from the well-balanced contents for both theoretical analysis and real-world applications. The book is composed of ten chapters in a coherent manner. “Nonlinearly StructuredLow-RankApproximation”presentsanAdjustedLeast-SquaresEstimation method for polynomially structured low-rank approximation, which is computation- allycheapandstatisticallyconsistent;“LatentLow-RankRepresentation”presentsthe formulationoftheLatentLow-RankRepresentation(LatLRR),whichconstructsthe dictionaryusingbothobservedandunobserved,hiddendata,andseamlesslyintegrates subspace clustering and feature extraction into a unified framework; “Scalable Low-Rank Representation” addresses problem of solving nuclear norm regularized optimization problems for Low-Rank Representation under large-scale settings throughatransformation,whichisachievedbyfactoringthelarge-sizesolutionmatrix into the product of a small-size orthonormal matrix (active subspace) and another small-sizematrix;“Low-RankandSparseDictionaryLearning”introduceslow-rank andsparsedictionarylearningmethods,which learn discriminativedictionarieswith v vi Preface low-rank and sparse constraints for modeling; “Low-Rank Transfer Learning” discussestransferlearninginageneralizedsubspacewhereeachtargetsamplecanbe represented by some combination of source samples under a low-rank constraint; “Sparse Manifold Subspace Learning” presents a linear dimensionality reduction algorithm called Sparse Manifold Subspace Learning, based on sparse eigendecom- position, by considering the locality of samples and their neighbors; “Low Rank TensorManifoldLearning” presentsa supervisedlowranktensormanifoldlearning modeltolearntheintrinsicstructureanddimensionalityofthetensorsembeddedina high-dimensional Euclidean space; “Low-Rank and Sparse Multi-task Learning” proposes to correlate multiple tasks using a low-rank representation and formulate multi-tasklearningapproachesasmathematicaloptimizationproblemsofminimizing theempiricallossregularizedbythelow-rankstructureandaseparatesparsestructure; “Low-RankOutlierDetection”presentsalow-rankoutlierdetectionapproach,which incorporates a low-rank constraint into the support vector data description model; “Low-Rank Online Metric Learning” presents an online metric learning model considering the low-rank constraint to address the online image classification/scene recognitionproblemviaadaptivesimilaritymeasurement. This book aims at broad groups of audience, such as professional researchers, graduatestudents, university faculties.Specifically,this bookcanbe usedbythese audiences in the background of computer science/engineering, statistics, and mathematics.Otherpotentialaudiencescanbeattractedfrombroadfieldsofscience andengineeringsincethistopicisinterdisciplinaryandthetopicscoveredsynergize cross-domain knowledge. I would like to sincerely thank all the contributors of this book for presenting their most recent research advances in an easily accessible manner. I would also sincerelythankeditorsBrettKurzman,RebeccaR.Hytowitz,andMaryJamesfrom Springer for support to this book project. Boston, MA Yun Fu Contents Nonlinearly Structured Low-Rank Approximation . . . . . . . . . . . . . . . 1 Ivan Markovsky and Konstantin Usevich Latent Low-Rank Representation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Guangcan Liu and Shuicheng Yan Scalable Low-Rank Representation. . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Guangcan Liu and Shuicheng Yan Low-Rank and Sparse Dictionary Learning . . . . . . . . . . . . . . . . . . . . 61 Sheng Li, Liangyue Li and Yun Fu Low-Rank Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Ming Shao, Dmitry Kit and Yun Fu Sparse Manifold Subspace Learning. . . . . . . . . . . . . . . . . . . . . . . . . . 117 Ming Shao, Mingbo Ma and Yun Fu Low Rank Tensor Manifold Learning . . . . . . . . . . . . . . . . . . . . . . . . 133 Guoqiang Zhong and Mohamed Cheriet Low-Rank and Sparse Multi-task Learning . . . . . . . . . . . . . . . . . . . . 151 Jianhui Chen, Jiayu Zhou and Jieping Ye Low-Rank Outlier Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 Sheng Li, Ming Shao and Yun Fu Low-Rank Online Metric Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Yang Cong, Ji Liu, Junsong Yuan and Jiebo Luo Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 vii Nonlinearly Structured Low-Rank Approximation IvanMarkovskyandKonstantinUsevich Abstract Polynomiallystructuredlow-rankapproximationproblemsoccurin • algebraiccurvefitting,e.g.,conicsectionfitting, • subspaceclustering(generalizedprincipalcomponentanalysis),and • nonlinearandparameter-varyingsystemidentification. The maximum likelihood estimation principle applied to these nonlinear models leads to nonconvex optimization problems and yields inconsistent estimators in theerrors-in-variables(measurementerrors)setting.Weproposeacomputationally cheap and statistically consistent estimator based on a bias correction procedure, calledAdjustedLeast-SquaresEstimation.Themethodissuccessfullyusedforconic sectionfittingandwasrecentlygeneralizedtoalgebraiccurvefitting.Thecontribu- tionofthisbook’schapteristheapplicationofthepolynomiallystructuredlow-rank approximationproblemand,inparticular,theadjustedleast-squaresmethodtosub- space clustering, nonlinear and parameter-varying system identification. The clas- sicalinsystemidentificationinput-outputnotionofadynamicalmodelisreplaced by the behavioral definition of a model as a set, represented by implicit nonlinear differenceequations. · · Keywords Structured low-rank approximation Conic section fitting Subspace · clustering Nonlinearsystemidentification B I.Markovsky( )·K.Usevich DepartmentELEC,VrijeUniversiteitBrussel, Pleinlaan2,BuildingK,B-1050Brussels,Belgium e-mail:[email protected] K.Usevich e-mail:[email protected] ©SpringerInternationalPublishingSwitzerland2014 1 Y.Fu(ed.),Low-RankandSparseModelingforVisualAnalysis, DOI10.1007/978-3-319-12000-3_1 2 I.MarkovskyandK.Usevich 1 Introduction Datamodeling,missingdataestimation,anddimensionalityreductionproblemsare closely related to the problem of approximating a given matrix by another matrix of reduced rank. Apart from the approximation criterion and the desired rank, the low-rankapproximationprobleminvolvesadditionalconstraintsthatrepresentprior knowledgeabouttheto-be-estimated“true”datageneratingsystem.Commonexam- ples are non-negativity and structure (e.g., Hankel, Toeplitz, and Sylvester) of the approximationmatrix. The reduced rank of the approximation matrix corresponds to the reduction of dimensionalityaswellastothereductionofthemodelcomplexityindatamodel- ing.Inlineartime-invariantsystemidentification,forexample,therankofthedata matrixisrelatedtotheorderofthemodel.BytheEckart-Young-Mirskytheorem[6], unstructuredoptimalinspectralandFrobeniusnormreducedrankapproximationis obtainedfromthetruncatedsingularvaluedecompositionofthematrix.Withafew exceptions,thisresulthasnotbeengeneralizedtostructuredapproximationproblems andweightedapproximationcriteria.Forstructuredweightedapproximationprob- lemsconvexrelaxationsaswellaslocaloptimizationmethodshavebeendeveloped, see[18]. Inthisbook’schapter,weconsiderthelow-rankapproximationproblemwiththe constraint that the rank deficient matrix is polynomially structured. Formally, the polynomiallystructuredlow-rankapproximationproblemisdefinedasfollows. Given a data matrix D, an approximation criterion (cid:2)D − D(cid:2)(cid:2), a polynomial mappingΦ : D(cid:2)(cid:3)→ D(cid:2) ,andanupperboundontherankr, ext minimize overD(cid:2) (cid:2)D−D(cid:2)(cid:2) (cid:3) (cid:4) (1) subjectto rank Φ(D(cid:2)) ≤r. The polynomially structured low-rank approximation problem (1) has applica- tionsin • curvefitting[18,Chap.6], • manifoldlearning[16, 31], • subspaceclustering[28],and • nonlinearsystemidentification[26, 27]. The simplest special case of nonlinear curve fitting is conic section fitting, which leads to low-rank approximation with quadratic structure constraint, see Sects.2.1 and 3. More involved is the application to subspace clustering, which is low-rank approximationwithVeronesestructureoftheapproximationandanadditional(fac- torizability)conditiononthekernel. NonlinearlyStructuredLow-RankApproximation 3 Asanoptimizationproblem,(1)isnonconvex.Contrarytoaffinestructuredlow- rankapproximationproblems(see[17, 19]foranoverviewofrecentresultsonthis (cid:2) problem),(1)doesnotallowtheapproximationmatrix D tobeeliminatedanalyti- callyviathevariableprojectionsmethod[9].Therefore,thenumberofoptimization variablesisoftheorderofmagnitudeofthenumberofdatapoints.Thismakesthe useoflocaloptimizationmethodsinfeasibleformediumtolargescalepolynomially structuredlow-rankapproximationproblems. Theprincipalcomponentanalysismethod[11, 12]iscloselyrelatedtolow-rank approximation. Principal component analysis gives a stochastic interpretation of thedeterministiclow-rankapproximation.Viceverse,low-rankapproximationisa deterministicoptimizationproblemresultingfromtheprincipalcomponentanalysis method.Nonlinearlystructuredlow-rankapproximationproblemsareconsideredin theprincipalcomponentanalysiscontextunderthenamesofprincipalcurves[10] and kernel principal component analysis [1, Chap.12], [23]. The kernel principal component analysis method is unstructured low-rank approximation of the matrix Φ(D),i.e.,itdoesnotimposethepolynomialstructureoftheapproximatingmatrix. Weadopttheerrors-in-variablesstochasticmodel,i.e.,thegivendataisobtained fromtruedatathatsatisfiesatruedatageneratingmodelplusadditivenoise,see[3]. Thefollowingnoisepropertiesareassumedthroughoutthechapter: • zeromean, • independentsamples, • Gaussiandistribution, • knowncovariancematrixuptoascalingfactor. Thesolutionofthepolynomiallystructuredlow-rankapproximationproblem(1)is amaximumlikelihoodestimatorintheerrors-in-variablesetting.Itiswellknown, see, e.g., [15], that the maximum likelihood estimator is inconsistent in nonlinear errors-invariablesestimationproblems. The method proposed in this book’s chapter is a generalization of the adjusted least squares method of [14, 20] developed for ellipsoid fitting. The adjustment procedure ismotivated fromtheidea ofcorrecting forthebias oftheunstructured low-rank approximation method. The bias correction is explicitly given in terms of the noise variance and a procedure for the estimation of the noise variance is proposed.Ageneralizationoftheadjustedleastsquaresmethodtoalgebraiccurve fittingisdescribedin[18,Chap.6].Inthiscontribution,weshowthatpolynomially structuredlow-rankapproximationproblemsappearnaturallyinsubspaceclustering andnonlinearsystemidentification,sothattheadjustedleastsquaresalgorithmisa promisingestimationmethodalsointheseapplicationareas. 1.1 Outline In Sect.2, we start with an overview of the application of polynomially structured low-rankapproximationtoconicsectionfitting,subspaceclustering,andnonlinear