Table Of Content

Springer Series in Computational Mathematics 60 Tomohiro Sogabe Krylov Subspace Methods for Linear Systems Principles of Algorithms Springer Series in Computational Mathematics Volume 60 SeriesEditors RandolphE.Bank,DepartmentofMathematics,UniversityofCalifornia SanDiego,LaJolla,CA,USA WolfgangHackbusch,Max-Planck-InstitutfürMathematikinden Naturwissenschaften,Leipzig,Germany JosefStoer,InstitutfürMathematik,UniversityofWürzburg,Würzburg,Germany HarryYserentant,InstitutfürMathematik,TechnischeUniversitätBerlin,Berlin, Germany This is basically a numerical analysis series in which high-level monographs are published. Wedevelopthisseriesaimingathavingmorepublicationsinitwhicharecloser to applications. There are several volumes in the series which are linked to some mathematicalsoftware. Thisisalistofalltitlespublishedinthisseries:https://www.springer.com/series/ 797?detailsPage=titles Tomohiro Sogabe Krylov Subspace Methods for Linear Systems Principles of Algorithms TomohiroSogabe DepartmentofAppliedPhysics NagoyaUniversity Nagoya,Japan ISSN 0179-3632 ISSN 2198-3712 (electronic) SpringerSeriesinComputationalMathematics ISBN 978-981-19-8531-7 ISBN 978-981-19-8532-4 (eBook) https://doi.org/10.1007/978-981-19-8532-4 ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature SingaporePteLtd.2022 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar ordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore Preface In many fields of scientific computing and data science, we frequently face the problem of solving large and sparse linear systems of the form Ax = b, which is one of the most time-consuming parts of all computations. From this fact, many researchers have devoted themselves to developing efficient numerical algorithms forsolvingthelinearsystems,andKrylovsubspacemethodsarenowadayspopular numericalalgorithmsandareknownasoneofthetoptenalgorithmsofthetwentieth century,othersincludingfastFouriertransformandQuickSort[39].Thoughthebasic theorywasestablishedinthetwentiethcentury,Krylovsubspacemethodshavebeen developedbymathematicians,engineers,physicists,andmanyothers. TherearemanyexcellentbooksonKrylovsubspacemethods,includingthoseby: • OweAxelsson,1994[10], • RichardBarrettetal.,1994[18], • WolfgangHackbusch,1994[92], • AreMagnusBruaset,1995[29], • RüdigerWeiss,1996[203], • AnneGreenbaum,1997[81], • YousefSaad,2003[151], • HenkA.vanderVorst,2003[196], • JörgLiesenandZdeneˇkStrakoš,2012[122], • GérardMeurantandJurjenDuintjerTebbens,2020[129]. In [129], detailed historical notes of Krylov subspace methods are described, whichformamasterpiece(around700pages)ofKrylovsubspacemethodsfornon- Hermitianlinearsystems.Thefeaturesofthisbookarelistedasfollows: (1) Many applications of linear systems from computational science and data science; (2) Krylov subspace methods for complex symmetric linear systems such as the COCGmethodandtheCOCRmethod; (3) Krylovsubspacemethodsfornon-HermitianlinearsystemssuchastheBiCR method,theGPBiCGmethod,andthe(block)IDR(s)method; v vi Preface (4) KrylovsubspacemethodsforshiftedlinearsystemssuchastheshiftedIDR(s) method; (5) Matrixfunctionsasapplicationsofshiftedlinearsystems. Feature (1) corresponds to Chap. 2, and linear systems are derived from various applications: partial differential equations (finite difference discretization methodsandthefiniteelementmethod);computationalphysics:condensedmatter physics (computation of Green’s function) and lattice quantum chromodynamics (Wilsonfermionmatrix);machine learning(least-squaresproblems);matrixequa- tions(Sylvester-typematrixequations);optimization(HessianmatrixoverEuclidean spaceandRiemannianmanifoldusingtensorcomputationnotations). Features (2) and (3) correspond to Chap. 3. In this chapter, Krylov subspace methods are classified into three groups: Hermitian linear systems, complex symmetriclinearsystems,andnon-Hermitianlinearsystems.ForHermitianlinear systems,theCGmethodisderivedfromthematrixformoftheLanczosprocess,the CRmethodisderivedfromtheCGmethod,andtheMINRESmethodisderivedfrom theLanczosprocess.Forcomplexsymmetriclinearsystems,theCOCGmethod,the COCRmethod,andtheQMR_SYMmethodaredescribedasextensionsoftheCG method,theCRmethod,andtheMINRESmethod,respectively.Fornon-Hermitian linear systems, the BiCG method, the BiCR method, and the QMR method are describedasextensionsoftheCGmethod,theCRmethod,andtheMINRESmethod, respectively.ThedetailedderivationsoftheGPBiCGmethodandthe(block)IDR(s) method,oneofthefeaturesofthisbook,aredescribedinthischapter.Inaddition, somepreconditioningtechniquesarebrieflydescribed. Feature(4)correspondstoChap.4.Inthischapter,Krylovsubspacemethodsfor shiftedlinearsystemsareclassifiedintothreegroups:Hermitian,complexsymmetric, andnon-Hermitianlinearsystems.Thedetailedderivationsofthesealgorithmsare describedsystematically. Feature (5) corresponds to Chap. 5. If one needs a large matrix function, then KrylovsubspacemethodsandKrylovsubspacemethodsforshiftedlinearsystems aremethodsofchoicesincethesealgorithmscanproduceanyelementofthematrix function.Thedefinitionsofmatrixfunctionsandwell-knownalgorithmsformatrix functionsarealsodescribed. Anadditionalfeatureofthisbookisthattherearenonumericalexperimentsexcept sometypicalnumericalexamplesforfurtherunderstandingtheconvergencebehavior ofKrylovsubspacemethods.TheconvergenceofKrylovsubspacemethodsdepends highly on the coefficient matrix, and the best algorithm changes if the coefficient matrix changes. So, if the reader wants to solve linear systems, I recommend the readertoapplyseveralKrylovsubspacemethods(includingtheBiCGSTABmethod andtheGMRESmethod)totheirproblemandchoosethebestoneamongthem. This book is suitable for anyone who studied linear algebra and needs to solve largeandsparselinearsystems.Ihopethisbookishelpfulforthereadertounderstand Preface vii theprinciplesandpropertiesofKrylovsubspacemethodsandtocorrectlyuseKrylov subspacemethodstosolvetheirproblems. Nagoya,Japan TomohiroSogabe September2022 Acknowledgements IwouldliketothankProf.Dr.LeiDuatDalianUniversityofTechnologyforkindly sending me the LaTeX code of his block IDR(s) method. I appreciate Masayuki Nakamura’s work. I could raise the priority of writing this book from his timely reminders,togetherwithhiskindseasonalgreetings.IthankLokeshwaranManick- avasagam and Poojitha Ravichandran for their help with the production of this book. I am grateful to three anonymous reviewers for their careful reading of the manuscript and highly constructive comments that enhanced the quality of the manuscript. I would like to express my sincere thanks to three editors from the Springer Series in Computational Mathematics (SSCM) editorial board: Prof. Dr. RandolphE.Bank,Prof.Dr.WolfgangHackbusch,Prof.Dr.JosefStoer,andProf. Dr. Harry Yserentant for providing the author with fruitful comments from broad anddistinguishedprofessionalviewpoints. Finally,IappreciateProf.Dr.Shao-LiangZhangwhointroducedmetothefield ofKrylovsubspacemethodswhenIwasaPh.D.studentattheUniversityofTokyo about20yearsago,andIwishtothankmylovingwife,YuanLi,forherday-to-day supportandherkindness.Sherecommendedmetowritethisbookonweekends,and shewastakingcareoftwoactivesons:KazumaandTakuma,andheroldfather. ix Contents 1 IntroductiontoNumericalMethodsforSolvingLinearSystems .... 1 1.1 LinearSystems ............................................. 1 1.1.1 VectorNorm ....................................... 2 1.1.2 MatrixNorm ....................................... 2 1.2 ConditionNumber .......................................... 4 1.3 DirectMethods ............................................ 4 1.3.1 LUDecomposition .................................. 5 1.3.2 LUDecompositionwithPivoting ...................... 6 1.3.3 IterativeRefinement ................................. 8 1.4 DirectMethodsforSymmetricLinearSystems ................. 9 1.4.1 CholeskyDecomposition ............................. 9 1.4.2 LDL(cid:2)Decomposition ............................... 10 1.5 DirectMethodsforLargeandSparseLinearSystems ............ 12 1.6 StationaryIterativeMethods ................................. 13 1.6.1 TheJacobiMethod .................................. 14 1.6.2 TheGauss–SeidelMethod ........................... 15 1.6.3 TheSORMethod ................................... 15 1.6.4 ConvergenceoftheStationaryIterativeMethods ........ 16 1.7 MultigridMethods .......................................... 18 1.8 KrylovSubspaceMethods ................................... 19 1.9 OrthogonalizationMethodsforKrylovSubspaces ............... 23 1.9.1 TheArnoldiProcess ................................. 23 1.9.2 TheBi-LanczosProcess ............................. 26 1.9.3 TheComplexSymmetricLanczosProcess .............. 27 1.9.4 TheLanczosProcess ................................ 28 2 SomeApplicationstoComputationalScienceandDataScience ..... 31 2.1 PartialDifferentialEquations ................................. 31 2.1.1 FiniteDifferenceMethods ............................ 31 2.1.2 TheFiniteElementMethod .......................... 42 2.1.3 WeakForm ........................................ 43 xi