Numerical methods for the best low multilinear rank approximation of higher-order tensors

166 Pages·2009·1.48 MB·English

ArenbergDoctoralSchoolofScience,Engineering&Technology
FacultyofEngineering
DepartmentofElectricalEngineering(ESAT)
Numerical methods for the best low multilinear
rank approximation of higher-order tensors
Mariya Ishteva
Promotors: Dissertationpresentedinpartial
Prof. dr. ir. SabineVanHuffel fulfillmentoftherequirementsfor
Prof. dr. ir. LievenDeLathauwer thedegreeofDoctor
inEngineering
December2009

Numerical methods for the best low multilinear
rank approximation of higher-order tensors
Mariya Ishteva
Jury: Dissertationpresentedinpartial
Prof. dr. ir. AnnHaegemans,president fulfillmentoftherequirementsfor
Prof. dr. ir. SabineVanHuffel,promotor thedegreeofDoctor
Prof. dr. ir. LievenDeLathauwer,promotor inEngineering
Prof. dr. ir. JoosVandewalle
Prof. dr. ir. MarcVanBarel
Prof. dr. ir. P.-A.Absil(UCL)
Prof. dr. ir. RodolpheSepulchre(ULg)
Prof. dr. ir. LarsEldén(LinköpingUniv.,Sweden)
December2009
©KatholiekeUniversiteitLeuven–FacultyofEngineering
KasteelparkArenberg1/2200,B-3001Leuven(Belgium)
Alle rechten voorbehouden. Niets uit deze uitgave mag worden vermenigvuldigd en/of
openbaargemaaktwordendoormiddelvandruk,fotocopie,microfilm,elektronischofop
welkeanderewijzeookzondervoorafgaandeschriftelijketoestemmingvandeuitgever.
All rights reserved. No part of the publication may be reproduced in any form by print,
photoprint,microfilmoranyothermeanswithoutwrittenpermissionfromthepublisher.
D/2009/7515/143
ISBN978-94-6018-163-4
Foreword
TheresearchworkpresentedinthisthesiswascarriedoutattheDepartmentofElectrical
Engineering(ESAT)oftheKatholiekeUniversiteitLeuveninthepresenceofcompetent
andenthusiasticprofessorsandcolleagues.ThisPhDwouldhavebeenimpossiblewithout
theirprofessionalandmoralsupport.
In particular, first and foremost, I would like to thank my supervisors Prof. Sabine Van
HuffelandProf. LievenDeLathauwer. ThankyouSabineforbelievinginme,foryour
advice,guidance,exampleandforthenicesocialeventsinyourhouse. Lieven,thankyou
foryourcompetence,foralwaysshowingmetherightdirection,answeringmynumerous
questions, letting me represent you at important conferences en dat je altijd Nederlands
hebtgesprokentegenmij.
I am particularly grateful to Prof. P.-A. Absil for agreeing being in my advisory
committee, for being an understanding co-author, for his manifold suggestions and
improvements and for believing in me. Merci. My thanks go also to Prof. Marc Van
Barelforhisavailability, constructivesuggestions, adviceandforthecarefulreadingof
thethesis.IamhonoredthatProf.JoosVandewalleandProf.RodolpheSepulchreagreed
to take part in the examination committee and am grateful for their valuable feedback.
Thanks to Prof. Ann Haegemans for acting as a chairman and the flexibility in solving
administrative issues. Special thanks to Prof. Lars Eldén, not only for coming from
abroad for my preliminary defence and for the critical reading of the thesis but also for
the invitation to visit the Linköping University in the autumn of 2007. This is greatly
appreciated.
TherearesomemorepeoplethatIwouldliketothankpersonally:Prof.PetkoPetkovwho
introducedmetoSabine;DianaSimaforherhelpfuladviceonbothscientificandpersonal
level; Berkant Savas for the nice stay in Linköping and Bart Vandereycken and Michel
Journée for the interesting conversations and updates at the DYSCO meetings. Thanks
also to all people I met at the conferences for their interest, time and useful discussions
andthankstoalltheothersIforgottomentioninthislist.
i
ii
My current and former colleagues from SCD and in particular BioMed, including our
temporary visitors from other universities, provided a wonderful working atmosphere.
Thisisespeciallytrueformyofficematesthroughtheyears,thefrequent“guests”from
other offices and the everyday company for lunch. The list is too long but I need to
mention at least Maria Isabel and Anca. Thanks for all meaningful and meaningless
discussionsthatleadtoacontinuousgoodmood.Iwouldalsoliketothankoursecretaries
andtechnicalstafffortheirconstantprofessionalhelp.
Specialthanksandwarmgreetingstoallmyfriendsfromcloseandfaraway,especially
toAylinandIliafortheirlong-termfriendshipandtoDylanforhislanguagecorrections,
adviceonhowtodoaPhDbutmostlyforthelateevenings’conversationsandforbeing
there.
I nakra(cid:31), no sъvsem ne na posledno m(cid:31)sto, speciani blagodarnosti na
Nina (mama), Kamen (tate), Saxo (brat mi) i Nikola (polovinkata) za
neprestannata podkrepa, v(cid:31)ra, l(cid:24)bov, vnimanie i nasъrqeni(cid:31) ot blizo
i daleq, kakto i za mnogo drugi newa, za koito ne vъrvi da pixa tuk,
no we vi predam liqno. Pozdravi i celuvki ;)
Abstract
In multilinear algebra, the basic quantities are generalizations of vectors and matrices,
calledhigher-ordertensors.Theyareusedinmanyapplicationfields,suchashigher-order
statistics,signalprocessingandscientificcomputing. Efficientandreliablealgorithmsfor
manipulatingthesestructuresarethushighlyappreciated.
Matricesaresecond-ordertensorswithwell-studiedproperties. Thematrixrankisawell-
understood concept. In particular, the low-rank approximation of a matrix is essential
forvariousresultsandalgorithms. Thesolutiontothelow-rankapproximationproblem
isknownandgivenbythetruncatedsingularvaluedecomposition(SVD).However,the
matrix rank and its properties are not easily or uniquely generalizable to higher-order
tensors.
Thisthesisisdevotedtoageneralizationofthematrixcolumnandrowrank,namelythe
multilinear rank. We focus on the best low multilinear rank approximation of higher-
ordertensors. Givenahigher-ordertensor,wearelookingforanothertensor,ascloseas
possibletotheoriginaloneandwithmultilinearrankboundedbyprespecifiednumbers.
Thisapproximationisusedfordimensionalityreductionandsignalsubspaceestimation.
Higher-order generalizations of SVD exist but their truncation results in a suboptimal
solution of the problem. A refinement by iterative algorithms is required. The higher-
orderorthogonaliterationisonesuchalgorithmwithlinearconvergencespeed.
We aim for conceptually faster algorithms. However, standard optimization algorithms
face a difficulty caused by unwanted symmetry property of the cost function. Namely,
there are infinitely many equivalent solutions whereas numerical algorithms have nice
convergence properties if the solutions are isolated. We remove the symmetry problem
by working on quotient matrix manifolds, a concept studied in the field of optimization
onmanifolds. Wedevelopthreenewalgorithms,basedonNewton’smethod,trust-region
schemeandconjugategradients. Wealsodiscusstheissueoflocalminimaandconsider
aparticularapplicationofthealgorithms.
iii

Notation
Listofsymbols
a,b,...,a ,b ,... scalars
a,b,... vectors
A,B,... matrices
A,B,... tensors
aijk,(A)ijk elementofA atposition(i,j,k)
Ai=a subtensorofA obtainedbyfixingtheindexi
A mode-nmatrixrepresentationofatensorA
(n)
I identitymatrix
I unittensor(diagonaltensorwithonesonitsdiagonal)
H Hankelmatrix
U,V,W singularvectormatrices
I,I ,I ,I ,N,R,... specialscalarsandupperbounds
1 2 3
exp exponentialfunction
{·}
j √ 1
−
p 1) 3.141592653589793or2)thequotientmap
Aˆ low≈multilinearrankapproximationofA
xˆ estimateofx
s ithsingularvalue
i
s (n) ithmode-nsingularvalue
i
grad gradient
Hess Hessian
M manifold
x ,h update/tangentvectors
VX verticalspaceatX
HX horizontalspaceatX
x horizontalliftofx
Ph orthogonalprojectionontothehorizontalspaceatX
X
v
vi
Listofsets
R thesetofrealnumbers
C thesetofcomplexnumbers
Rm n real(m n)-matrix
×
RI1×I2×I3 realI1 ×I2 I3tensor
Rn×p theset×ofal×lfull-rank(n p)-matrices,n p,
∗ Rn×p= X Rn×p:det(×XTX)=0,n p≥
O th∗eortho{gon∈algroup(thesetof6allorth≥ogo}nal(p p)-matrices),
p
O = X Rp p:XTX=XXT =I ×
p ×
{ ∈ }
St(p,n) theStiefelmanifold(thesetofallcolumn-wiseorthonormal(n p)-matrices),
St(p,n)= X Rn p:XTX=I,n p ×
×
{ ∈ ≥ }
Ssym(p) thesetofallsymmetricmatricesoforder p,
Ssym(p)= S Rp×p:ST =S
{ ∈ }
Sskew(p) thesetofallskew-symmetric(orantisymmetric)matricesoforder p,
Sskew(p)= W Rp×p:W T = W
{ ∈ − }
Basicoperations
AT transposeofthematrixA
A 1 inverseofA
−
[ab] matrixwithcolumnsaandb
vec(A) vectorrepresentationofA(allcolumnsofAstackedaftereachother)
a absolutevalueofa
| |
, inner/scalarproduct
h· ·i
A Frobeniusnorm, trace(ATA)
k k
det(A) determinantofA
p
diag(v) squarediagonalmatrixwithvectorvasdiagonal
trace(A) traceofthematrixA,(cid:229) a
ii
sym(B) (B+BT)/2
skew(B) (B BT)/2
−
rank() rank
·
rankn(A) mode-nrankofA
im(f) imageof f
col(Y) thecolumnspaceofY, Ya:a Rp
{ ∈ }
[Y] theequivalenceclassofY
S sum
(cid:213) product
Cartesianproduct,A B:= (a,b )
i j
× × { }
Kroneckerproduct,A B:=[a B]
ij
⊗ ⊗
a b outerproductofvectorsaandb
◦
A nM mode-nproductofatensorA andamatrixM
•
a b aisequivalenttob
∼
a b aisapproximatelyequaltob
≈
belongsto
∈