Numerical Linear Algebra: A Concise Introduction with MATLAB and Julia

157 Pages·2018·3.226 MB·English

by  Folkmar Bornemann, Walter Simson

Springer Undergraduate Mathematics Series
AdvisoryBoard
M.A.J.Chaplain,UniversityofSt.Andrews
A.MacIntyre,QueenMaryUniversityofLondon
S.Scott,King’sCollegeLondon
N.Snashall,UniversityofLeicester
E.Süli,UniversityofOxford
M.R.Tehranchi,UniversityofCambridge
J.F.Toland,UniversityofCambridge
Moreinformationaboutthis series at http://www.springernature.com/series/3423
Folkmar Bornemann
Numerical Linear Algebra
A Concise Introduction
with MATLAB and Julia
Translated by Walter Simson
Folkmar Bornemann
Center for Mathematical Sciences
Technical University of Munich
Garching, Germany
Translated by Walter Simson, Munich, Germany
ISSN 1615-2085 ISSN 2197-4144 (electronic)
Springer Undergraduate Mathematics Series
ISBN 978-3-319-74221-2 ISBN 978-3-319-74222-9 (eBook)
https://doi.org/978-3-319-74222-9

Library of Congress Control Number: 2018930015

© Springer International Publishing AG 2018
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology
now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the
relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, express or implied, with respect to the material contained herein
or for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.

Printed on acid-free paper

This Springer imprint is published by Springer Nature
The registered company is Springer International Publishing AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Thetroublewithpeopleisnotthattheydon’t
knowbutthattheyknowsomuchthatain’tso.
(JoshBillings1874)
Iwouldhaveunderstoodmanythings
ifno-onehadexplainedthemtome.
(StanisławJerzyLec1957)
Anexpertissomeonewhohasmadeallthe
mistakeswhichcanbemadeinanarrowfield.
(NielsBohr1954)
lowerandadjointuppertriangularsystemfromthemanuscript†ofA.-L.Cholesky(1910),cf.§8.3
†«Surlarésolutionnumériquedessystèmesd’équationslinéaires»,FondsAndré-LouisCholesky(1875-1918),
courtesyoftheArchivesdel’EcolePolytechnique,Palaiseau,France.
Preface
Thisbookwasdevelopedfromthelecturenotesofanundergraduatelevelcourse
forstudentsofmathematicsattheTechnicalUniversityofMunich,consistingof
twolecturesperweek.Itsgoalistopresentandpassonaskillsetofalgorithmic
andnumericalthinkingbasedonthefundamentalproblemsetofnumericallinear
algebra.Limitingthescopetolinearalgebracreatesastrongerthematiccoherency
in the course material than would be found in other introductory courses on
numerical analysis. Beyond the didactic advantages, numerical linear algebra
representsthebasisforthefieldofnumericalanalysis,andshouldthereforebe
learnedandmasteredasearlyaspossible.
Thisexpositionwillemphasizetheviabilityofpartitioningvectorsandmatrices
block-wiseascomparedtoamoreclassic,component-by-componentapproach.In
thisway,weachievenotonlyamorelucidnotationandshorteralgorithms,but
alsoasignificantimprovementintheexecutiontimeofcalculationsthankstothe
ubiquityofmodernvectorprocessorsandhierarchicalmemoryarchitectures.
Themottoforthisbookwillthereforebe:
Ahigherlevelofabstractionisactuallyanasset.
Duringourdiscussionoferroranalysiswewillbedivinguncompromisinglydeep
into the relevantconcepts in orderto gain the greatestpossible understanding
ofassessingnumericalprocesses.Moreshallowapproaches(e.g.,theinfamous
“rulesofthumb”)onlyresultinunreliable,expensiveandsometimesdownright
dangerousoutcomes.
Thealgorithmsandaccompanyingnumericalexampleswillbeprovidedinthe
programmingenvironmentMATLAB,whichisneartoubiquitousatuniversities
aroundtheworld.Additionally,onecanfindthesameexamplesprogrammedin
thetrailblazingnumericallanguageJuliafromMITinAppendixB.Myhopeis
thatthefollowingpassageswillnotonlypassontheintendedknowledge,but
alsoinspirefurthercomputerexperiments.
Theaccompanyinge-bookofferstheabilitytoclickthroughlinksinthepassages.
Linkstoreferenceselsewhereinthebookwillappearblue,whileexternallinks
willappearred.Thelatterwillleadtoexplanationsoftermsandnomenclature
whichareassumedtobepreviousknowledge,ortosupplementarymaterialsuch
asweb-basedcomputations,historicalinformationandsourcesofthereferences.
Munich,October2017 FolkmarBornemann
[email protected]
vii
viii Preface
Student’s Laboratory
Inordertoenhancethelearningexperiencewhenreadingthisbook,Irecommendcreating
one’sownlaboratory,andoutfittingitwiththefollowing“tools”.
Tool1:ProgrammingEnvironment Duetothecurrentprevalenceofthenumerical
developmentenvironmentMATLABbyMathWorksinbothacademicandindustrialfields,
wewillbeusingitasour“go-to”scriptinglanguageinthisbook.Nevertheless,Iwould
adviseeveryreadertolookintotheprogramminglanguageJuliafromMIT. Thisinge-
nious,forward-lookinglanguageisgrowingpopularity,makingitalanguagetolearn.All
programmingexamplesinthisbookhavebeenrewritteninJuliainAppendixB.
Tool2:CalculationWorkHorse Iwillbedevotingmostofthepagesofthisbookto
theideasandconceptsofnumericallinearalgebra,andwillavoidgettingcaughtupwith
tediouscalculations.Sincemanyofthesecalculationsaremechanicalinnature,Iencourage
everyreadertofindasuitable“calculationworkhorse”toaccomplishthesetasksforthem.
SomeconvenientoptionsincludecomputeralgebrasystemssuchasMapleorMathematica;
thelatteroffersafree“one-liner”versiononlineintheformofWolframAlpha.Several
examplescanbefoundasexternallinksin§14.
Tool3:TextbookX Inordertogainfurtherperspectiveonthesubjectmatterathand,
Irecommendalwayshavingasecondopinion,or“TextbookX”,withinreach.BelowIhave
listedafewexcellentoptions:
PeterDeuflhard,AndreasHohmann:NumericalAnalysisinModernScientificComputing,2nded.,
(cid:15)
Springer-Verlag,NewYork,2003.
Arefreshingbook;accordingtothepreface,myyouthfulenthusiasmhelpedformthepresentationoferror
analysis.
LloydN.Trefethen,DavidBau:NumericalLinearAlgebra,SocietyofIndustrialandApplied
(cid:15)
Mathematics,Philadelphia,1997.
Aclassicandanalltimebestsellerofthepublisher,writteninalivelyvoice.
JamesW.Demmel:AppliedNumericalLinearAlgebra,SocietyofIndustrialandAppliedMathe-
(cid:15)
matics,Philadelphia,1997.
DeeperandmoredetailedthanTrefethen–Bau,aclassicaswell.
Tool 4: Reference Material Forevengreaterimmersionandasastartingpointfor
furtherresearch,Istronglyrecommendthefollowingworks:
GeneH.Golub,CharlesF.VanLoan:MatrixComputations,4thed.,TheJohnsHopkinsUniversity
(cid:15)
Press,Baltimore,2013.
The“Bible”onthetopic.
NicholasJ.Higham:AccuracyandStabilityofNumericalAlgorithms,2nded.,SocietyofIndustrial
(cid:15)
andAppliedMathematics,Philadelphia,2002.
Thethoroughmodernstandardreferenceforerroranalysis(withouteigenvalueproblems,though).
RogerA. Horn,CharlesR. Johnson: MatrixAnalysis,2nded.,CambridgeUniversityPress,
(cid:15)
Cambridge,2012.
Aclassiconthetopicofmatrixtheory;verythoroughanddetailed,amust-havereference.
Contents
Preface vii
Student’sLaboratory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
I ComputingwithMatrices 1
1 WhatisNumericalAnalysis? . . . . . . . . . . . . . . . . . . . . . . 1
2 MatrixCalculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3 MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4 ExecutionTimes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
5 TriangularMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
6 UnitaryMatrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
II MatrixFactorization 21
7 TriangularDecomposition . . . . . . . . . . . . . . . . . . . . . . . . 21
8 CholeskyDecomposition. . . . . . . . . . . . . . . . . . . . . . . . . 28
9 QRDecomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
III ErrorAnalysis 39
10 ErrorMeasures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
11 ConditioningofaProblem. . . . . . . . . . . . . . . . . . . . . . . . 41
12 MachineNumbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
13 StabilityofanAlgorithm. . . . . . . . . . . . . . . . . . . . . . . . . 50
14 ThreeExemplaryErrorAnalyses . . . . . . . . . . . . . . . . . . . . 54
15 ErrorAnalysisofLinearSystemsofEquations . . . . . . . . . . . . 60
IV LeastSquares 69
16 NormalEquation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
17 Orthogonalization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
V EigenvalueProblems 75
18 BasicConcepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
19 PerturbationTheory . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
20 PowerIteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
21 QRAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Appendix 99
A MATLAB:AVeryShortIntroduction . . . . . . . . . . . . . . . . . 99
ix
x Contents
B Julia:AModernAlternativetoMATLAB . . . . . . . . . . . . . . . 105
C Norms:RecapandSupplement . . . . . . . . . . . . . . . . . . . . . 119
D TheHouseholderMethodforQRDecomposition . . . . . . . . . . 123
E FortheCurious,theConnoisseur,andtheCapable . . . . . . . . . 125
ModelBackwardsAnalysisofIterativeRefinement . . . . . . . 125
GlobalConvergenceoftheQRAlgorithmwithoutShifts . . . 126
LocalConvergenceoftheQRAlgorithmwithShifts . . . . . . 129
StochasticUpperBoundoftheSpectralNorm . . . . . . . . . 132
F MoreExercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Notation 147
Index 149