DTIC ADA606967: Minimal Krylov Subspaces for Dimension Reduction

4.5 MB·English

by  Defense Technical Information Center

Minimal Krylov Subspaces for Dimension Reduction

by Alexander M. Breuer

ARL-RP-0480 May 2014

A reprint from ProQuest/UMI, Indiana University, 2013; ATT 3552604

Approved for public release; distribution is unlimited.
NOTICES

Disclaimers

The findings in this report are not to be construed as an official Department of the Army position unless
so designated by other authorized documents.

Citation of manufacturer’s or trade names does not constitute an official endorsement or approval of the
use thereof.

Destroy this report when it is no longer needed. Do not return it to the originator.
Army Research Laboratory
Aberdeen Proving Ground, MD 21005-5068

ARL-RP-0480 May 2014

Minimal Krylov Subspaces for Dimension Reduction

Alexander M. Breuer
Survivability/Lethality Analysis Directorate, ARL

A reprint from ProQuest/UMI, Indiana University, 2013; ATT 3552604

Approved for public release; distribution is unlimited.
REPORT DOCUMENTATION PAGE Form Approved
OMB No. 0704-0188
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering
and maintaining the data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing the burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson
Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to
comply with a collection of information if it does not display a currently valid OMB control number.
PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.
1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To)
May 2014 Reprint 1 August 2008–31 December 2012
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
Minimal Krylov Subspaces for Dimension Reduction
5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER
Alexander M. Breuer
5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
REPORT NUMBER
U.S. Army Research Laboratory
ATTN: RDRL-SLB-W ARL-RP-0480
Aberdeen Proving Ground, MD 21005-5068
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR'S REPORT
NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution is unlimited.
13. SUPPLEMENTARY NOTES
A reprint from ProQuest/UMI, Indiana University, 2013; ATT 3552604

14. ABSTRACT
Krylov subspaces may be used as an alternative to singular vector spaces or eigenvector spaces for projective dimension
reduction for low-rank matrix approximation. Though the truncated spectral or singular value decomposition is optimal for
minimizing Frobenius norm error of a low-rank approximation, substituting a Krylov subspace projection may result in marked
compute time savings. Previous efforts to apply Krylov subspaces to low-rank approximation problems are extended to block
Krylov subspaces. Closely related random projection methods are compared to block Krylov subspaces, and new hybrid
approaches are developed. Hybrid random projection Krylov subspace methods offer faster compute times than random
projection methods and lower approximation errors when sufficient conditions are met. A novel adaptively blocked Krylov
subspace algorithm is developed that offers superior compute times to random projection methods. Stationary inner iteration is
considered for accelerating convergence of Krylov subspaces and applied to the low-rank approximation problem; a
generalization of eigenvalue approximation bounds is presented for Krylov subspaces augmented with inner iteration. All
aforementioned methods are evaluated in terms offloating-point operations and applied to numerous problems.
15. SUBJECT TERMS
Krylov subspaces, random projections, dimension reduction, low-rank approximation, iterative methods
17. LIMITATION 18. NUMBER 19a. NAME OF RESPONSIBLE PERSON
16. SECURITY CLASSIFICATION OF: OF ABSTRACT OF PAGES Alexander M. Breuer

a. REPORT b. ABSTRACT c. THIS PAGE 19b. TELEPHONE NUMBER (Include area code)
Unclassified Unclassified Unclassified UU 200 410-278-9157
Standard Form 298 (Rev. 8/98)
Prescribed by ANSI Std. Z39.18
MINIMAL KRYLOV SUBSPACES FOR
DIMENSION REDUCTION
Alex Breuer
Submitted to the faculty of the University Graduate School
in partial fulfillment of the requirements
for the degree
Doctor of Philosophy
in the Division of Computer Science
of the School of Informatics and Computing,
Indiana University
January 2013
AcceptedbytheGraduateFaculty,IndianaUniversity,inpartialfulfillmentoftherequirementsfor
thedegreeofDoctorofPhilosophy.
Doctoral Committee
Andrew LUMSDAINE,Ph.D.
Gonzalo ARCE,Ph.D.
David CRANDALL,Ph.D.
Esfandiar HAGHVERDI,Ph.D.
Predrag RADIVOJAC,Ph.D.
December13,2012
ii
Acknowledgments
I would like to gratefully acknowledge my research advisor, Professor Andrew Lumsdaine, for his
support throughout my graduate study. Without his patience and accommodation, this would not
have been possible. I would also like to express my appreciation for my entire graduate committee,
ProfessorsGonzaloArce,DavidCrandall,EsfandiarHaghverdiandPredragRadivojac. Iwouldalso
liketorecognizetheinvaluablesupportofLisaRoach,ChiefoftheWarfighterSurvivabilityBranch
in the U. S. Army Research Laboratory, as well as Director Dr. Paul Tanenbaum and Division Chief
Robert Bowen for their encouragement and support during this process. I would also like to thank
mywife,StephanieBreuer,whosetirelessfaithinmehasbeenaconstantblessing.
iii
AlexBreuer
MinimalKrylovSubspacesforDimensionReduction
Krylov subspaces may be used as an alternative to singular vector spaces or eigenvector spaces
for projective dimension reduction for low-rank matrix approximation. Though the truncated spec-
tral or singular value decomposition is optimal for minimizing Frobenius norm error of a low-rank
approximation, substituting a Krylov subspace projection may result in marked compute time sav-
ings. Previous efforts to apply Krylov subspaces to low-rank approximation problems are extended
toblockKrylovsubspaces. Closely-relatedrandomprojectionmethodsarecomparedtoblockKrylov
subspaces, and new hybrid approaches are developed. Hybrid random-projection Krylov subspace
methods offer faster compute times than random projection methods, and lower approximation er-
rors when sufficient conditions are met. A novel adaptively-blocked Krylov subspace algorithm is
developed that offers superior compute times to random projection methods. Stationary inner it-
eration is considered for accelerating convergence of Krylov subspaces and applied to the low-rank
approximationproblem;ageneralizationofeigenvalueapproximationboundsispresentedforKrylov
subspaces augmented with inner iteration. All aforementioned methods are evaluated in terms of
floating-pointoperationsandappliedtonumerousproblems.
iv
Table of Contents
1 Introduction 1
1.1 Thelow-rankapproximationproblem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 ApproximationoftheSVDforlow-rankapproximation . . . . . . . . . . . . . . . . . . . . 7
2 Low-rankapproximationbackground 12
2.1 Genericdimensionreductionbackground . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Krylovsubspacebackground . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 ConvergenceofeigenvaluesinKrylovsubspaces . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Low-rankapproximationerrorandeigenvalueapproximations . . . . . . . . . . . . . . . 18
2.5 Krylovsubspacealgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.6 Randomprojectionmethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3 ShortblockKrylovsubspacesforlow-rankapproximation 25
3.1 Randomprojectionsforlow-rankmatrixapproximation . . . . . . . . . . . . . . . . . . . 29
3.2 BlockLanczosforlow-rankapproximation . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.1 BlockLanczosforwithrefinementforimprovedstability . . . . . . . . . . . . . . 32
3.2.2 Theshrink-and-iterateapproach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2.3 Thehybridrandomprojection-Krylovsubspacemethod . . . . . . . . . . . . . . . 35
3.3 ConvergenceAnalyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.1 ReviewofexistingboundsforrandomprojectionsandKrylovsubspaces . . . . . 38
3.3.2 Derivationofnewbounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
v
3.3.3 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3.4 Convergenceexample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.4 StabilityAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4.1 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4 ApplicationsofshortblockKrylovsubspaces 58
4.1 ExperimentswiththeYalefacedata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 Low-rankapproximationinInformationRetrieval . . . . . . . . . . . . . . . . . . . . . . . 64
4.3 ExperimentswiththeColoradoroadnetworkLaplacian . . . . . . . . . . . . . . . . . . . 70
4.4 Stiffnessmatrixexperiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5 GrABL:AGreedyAdaptiveBlockLanczos
method 85
5.1 Typicalspectraofgenericdimensionreductionproblems . . . . . . . . . . . . . . . . . . . 87
5.2 ClassicblockLanczos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
5.3 ABLE:adaptivelyblockedLanczosfortheeigenproblem . . . . . . . . . . . . . . . . . . . 88
5.3.1 ConvergenceofeigenvaluesinblockKrylovsubspaces . . . . . . . . . . . . . . . . 89
5.3.2 Convergenceviablocksizeversusconvergenceviaiterations . . . . . . . . . . . . 91
5.4 GreedyAdaptively-blockedLanczos. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.5 RandomprojectionsandorthogonalLanczositeration. . . . . . . . . . . . . . . . . . . . . 92
5.5.1 Choosing mautomatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.6 TheGrABLalgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
6 ApplicationsofGrABL 104
6.1 ExperimentswiththeYalefacedata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.2 ExperimentswiththeColoradoroadnetworkLaplacian . . . . . . . . . . . . . . . . . . . 110
6.3 ExperimentswiththeBagofWordsterm-documentmatrix . . . . . . . . . . . . . . . . . 116
vi