Table Of Content

Springer Undergraduate Mathematics Series AdvisoryBoard M.A.J.Chaplain,UniversityofSt.Andrews A.MacIntyre,QueenMaryUniversityofLondon S.Scott,King’sCollegeLondon N.Snashall,UniversityofLeicester E.Süli,UniversityofOxford M.R.Tehranchi,UniversityofCambridge J.F.Toland,UniversityofCambridge Moreinformationaboutthis series at http://www.springernature.com/series/3423 Folkmar Bornemann Numerical Linear Algebra A Concise Introduction with MATLAB and Julia Translated by Walter Simson Folkmar Bornemann Center for Mathematical Sciences Technical University of Munich Garching, Germany Translated by Walter Simson, Munich, Germany ISSN 1615-2085 ISSN 2197-4144 (electronic) Springer Undergraduate Mathematics Series ISBN 978-3-319-74221-2 ISBN 978-3-319-74222-9 (eBook) https://doi.org/978-3-319-74222-9 Library of Congress Control Number: 2018930015 © Springer International Publishing AG 2018 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Printed on acid-free paper This Springer imprint is published by Springer Nature The registered company is Springer International Publishing AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Thetroublewithpeopleisnotthattheydon’t knowbutthattheyknowsomuchthatain’tso. (JoshBillings1874) Iwouldhaveunderstoodmanythings ifno-onehadexplainedthemtome. (StanisławJerzyLec1957) Anexpertissomeonewhohasmadeallthe mistakeswhichcanbemadeinanarrowfield. (NielsBohr1954) lowerandadjointuppertriangularsystemfromthemanuscript†ofA.-L.Cholesky(1910),cf.§8.3 †«Surlarésolutionnumériquedessystèmesd’équationslinéaires»,FondsAndré-LouisCholesky(1875-1918), courtesyoftheArchivesdel’EcolePolytechnique,Palaiseau,France. Preface Thisbookwasdevelopedfromthelecturenotesofanundergraduatelevelcourse forstudentsofmathematicsattheTechnicalUniversityofMunich,consistingof twolecturesperweek.Itsgoalistopresentandpassonaskillsetofalgorithmic andnumericalthinkingbasedonthefundamentalproblemsetofnumericallinear algebra.Limitingthescopetolinearalgebracreatesastrongerthematiccoherency in the course material than would be found in other introductory courses on numerical analysis. Beyond the didactic advantages, numerical linear algebra representsthebasisforthefieldofnumericalanalysis,andshouldthereforebe learnedandmasteredasearlyaspossible. Thisexpositionwillemphasizetheviabilityofpartitioningvectorsandmatrices block-wiseascomparedtoamoreclassic,component-by-componentapproach.In thisway,weachievenotonlyamorelucidnotationandshorteralgorithms,but alsoasignificantimprovementintheexecutiontimeofcalculationsthankstothe ubiquityofmodernvectorprocessorsandhierarchicalmemoryarchitectures. Themottoforthisbookwillthereforebe: Ahigherlevelofabstractionisactuallyanasset. Duringourdiscussionoferroranalysiswewillbedivinguncompromisinglydeep into the relevantconcepts in orderto gain the greatestpossible understanding ofassessingnumericalprocesses.Moreshallowapproaches(e.g.,theinfamous “rulesofthumb”)onlyresultinunreliable,expensiveandsometimesdownright dangerousoutcomes. Thealgorithmsandaccompanyingnumericalexampleswillbeprovidedinthe programmingenvironmentMATLAB,whichisneartoubiquitousatuniversities aroundtheworld.Additionally,onecanfindthesameexamplesprogrammedin thetrailblazingnumericallanguageJuliafromMITinAppendixB.Myhopeis thatthefollowingpassageswillnotonlypassontheintendedknowledge,but alsoinspirefurthercomputerexperiments. Theaccompanyinge-bookofferstheabilitytoclickthroughlinksinthepassages. Linkstoreferenceselsewhereinthebookwillappearblue,whileexternallinks willappearred.Thelatterwillleadtoexplanationsoftermsandnomenclature whichareassumedtobepreviousknowledge,ortosupplementarymaterialsuch asweb-basedcomputations,historicalinformationandsourcesofthereferences. Munich,October2017 FolkmarBornemann [email protected] vii viii Preface Student’s Laboratory Inordertoenhancethelearningexperiencewhenreadingthisbook,Irecommendcreating one’sownlaboratory,andoutfittingitwiththefollowing“tools”. Tool1:ProgrammingEnvironment Duetothecurrentprevalenceofthenumerical developmentenvironmentMATLABbyMathWorksinbothacademicandindustrialfields, wewillbeusingitasour“go-to”scriptinglanguageinthisbook.Nevertheless,Iwould adviseeveryreadertolookintotheprogramminglanguageJuliafromMIT. Thisinge- nious,forward-lookinglanguageisgrowingpopularity,makingitalanguagetolearn.All programmingexamplesinthisbookhavebeenrewritteninJuliainAppendixB. Tool2:CalculationWorkHorse Iwillbedevotingmostofthepagesofthisbookto theideasandconceptsofnumericallinearalgebra,andwillavoidgettingcaughtupwith tediouscalculations.Sincemanyofthesecalculationsaremechanicalinnature,Iencourage everyreadertofindasuitable“calculationworkhorse”toaccomplishthesetasksforthem. SomeconvenientoptionsincludecomputeralgebrasystemssuchasMapleorMathematica; thelatteroffersafree“one-liner”versiononlineintheformofWolframAlpha.Several examplescanbefoundasexternallinksin§14. Tool3:TextbookX Inordertogainfurtherperspectiveonthesubjectmatterathand, Irecommendalwayshavingasecondopinion,or“TextbookX”,withinreach.BelowIhave listedafewexcellentoptions: PeterDeuflhard,AndreasHohmann:NumericalAnalysisinModernScientificComputing,2nded., (cid:15) Springer-Verlag,NewYork,2003. Arefreshingbook;accordingtothepreface,myyouthfulenthusiasmhelpedformthepresentationoferror analysis. LloydN.Trefethen,DavidBau:NumericalLinearAlgebra,SocietyofIndustrialandApplied (cid:15) Mathematics,Philadelphia,1997. Aclassicandanalltimebestsellerofthepublisher,writteninalivelyvoice. JamesW.Demmel:AppliedNumericalLinearAlgebra,SocietyofIndustrialandAppliedMathe- (cid:15) matics,Philadelphia,1997. DeeperandmoredetailedthanTrefethen–Bau,aclassicaswell. Tool 4: Reference Material Forevengreaterimmersionandasastartingpointfor furtherresearch,Istronglyrecommendthefollowingworks: GeneH.Golub,CharlesF.VanLoan:MatrixComputations,4thed.,TheJohnsHopkinsUniversity (cid:15) Press,Baltimore,2013. The“Bible”onthetopic. NicholasJ.Higham:AccuracyandStabilityofNumericalAlgorithms,2nded.,SocietyofIndustrial (cid:15) andAppliedMathematics,Philadelphia,2002. Thethoroughmodernstandardreferenceforerroranalysis(withouteigenvalueproblems,though). RogerA. Horn,CharlesR. Johnson: MatrixAnalysis,2nded.,CambridgeUniversityPress, (cid:15) Cambridge,2012. Aclassiconthetopicofmatrixtheory;verythoroughanddetailed,amust-havereference. Contents Preface vii Student’sLaboratory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii I ComputingwithMatrices 1 1 WhatisNumericalAnalysis? . . . . . . . . . . . . . . . . . . . . . . 1 2 MatrixCalculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3 MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 4 ExecutionTimes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5 TriangularMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6 UnitaryMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 II MatrixFactorization 21 7 TriangularDecomposition . . . . . . . . . . . . . . . . . . . . . . . . 21 8 CholeskyDecomposition. . . . . . . . . . . . . . . . . . . . . . . . . 28 9 QRDecomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 III ErrorAnalysis 39 10 ErrorMeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 11 ConditioningofaProblem. . . . . . . . . . . . . . . . . . . . . . . . 41 12 MachineNumbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 13 StabilityofanAlgorithm. . . . . . . . . . . . . . . . . . . . . . . . . 50 14 ThreeExemplaryErrorAnalyses . . . . . . . . . . . . . . . . . . . . 54 15 ErrorAnalysisofLinearSystemsofEquations . . . . . . . . . . . . 60 IV LeastSquares 69 16 NormalEquation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 17 Orthogonalization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 V EigenvalueProblems 75 18 BasicConcepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 19 PerturbationTheory . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 20 PowerIteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 21 QRAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Appendix 99 A MATLAB:AVeryShortIntroduction . . . . . . . . . . . . . . . . . 99 ix x Contents B Julia:AModernAlternativetoMATLAB . . . . . . . . . . . . . . . 105 C Norms:RecapandSupplement . . . . . . . . . . . . . . . . . . . . . 119 D TheHouseholderMethodforQRDecomposition . . . . . . . . . . 123 E FortheCurious,theConnoisseur,andtheCapable . . . . . . . . . 125 ModelBackwardsAnalysisofIterativeRefinement . . . . . . . 125 GlobalConvergenceoftheQRAlgorithmwithoutShifts . . . 126 LocalConvergenceoftheQRAlgorithmwithShifts . . . . . . 129 StochasticUpperBoundoftheSpectralNorm . . . . . . . . . 132 F MoreExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Notation 147 Index 149