Probabilistic Numerics: Computation as Machine Learning PDF
Preview Probabilistic Numerics: Computation as Machine Learning
ProbabilisticNumerics Probabilistic numerical computation formalises the connection between machine learning and applied mathematics. Numerical algorithms approximate intractable quantities from computable ones.Theyestimateintegralsfromevaluationsoftheintegrand,orthepathofadynamicalsystem described by differential equations from evaluations of the vector field. In other words, they infer a latent quantity from data. This book shows that it is thus formally possible to think of computationalroutinesaslearningmachines,andtousethenotionofBayesianinferencetobuild moreflexible,efficient,orcustomisedalgorithmsforcomputation. The text caters for Masters’ and PhD students, as well as postgraduate researchers in artificial intelligence,computerscience,statistics,andappliedmathematics.Extensivebackgroundmaterial is provided along with a wealth of figures, worked examples, and exercises (with solutions) to developintuition. Philipp Hennig holds the Chair for the Methods of Machine Learning at the University of Tu¨bingen, and an adjunct position at the Max Planck Institute for Intelligent Systems. He has dedicated most of his career to the development of Probabilistic Numerical Methods. Hennig’s research has been supported by Emmy Noether, Max Planck and ERC fellowships. He is a co- Director of the Research Program for the Theory, Algorithms and Computations of Learning MachinesattheEuropeanLaboratoryforLearningandIntelligentSystems(ELLIS). Michael A. Osborne is Professor of Machine Learning at the University of Oxford, and a co- Founder of Mind Foundry Ltd. His research has attracted £10.6M of research funding and has beencitedover15,000times.Heisvery,veryBayesian. Hans P. Kersting is a postdoctoral researcher at INRIA and E´cole Normale Supe´rieure in Paris, working in machine learning with expertise in Bayesian inference, dynamical systems, and optimisation. Published online by Cambridge University Press ‘Thisimpressivetextrethinksnumericalproblemsthroughthelensofprobabilisticinferenceand decisionmaking.Thisfreshperspectiveopensupanewchapterinthisfield,andsuggestsnewand highlyefficientmethods.Alandmarkachievement!’ –ZoubinGhahramani,UniversityofCambridge ‘Inthisstunningandcomprehensivenewbook,earlydevelopmentsfromKacandLarkinhavebeen comprehensively built upon, formalised, and extended by including modern-day machine learn- ing, numerical analysis, and the formal Bayesian statistical methodology. Probabilistic numerical methodologyisofenormousimportanceforthisageofdata-centricscienceandHennig,Osborne, andKerstingaretobecongratulatedinprovidinguswiththisdefinitivevolume.’ –MarkGirolami,UniversityofCambridgeandTheAlanTuringInstitute ‘This book presents an in-depth overview of both the past and present of the newly emerging areaofprobabilisticnumerics,whererecentadvancesinprobabilisticmachinelearningareusedto developprincipledimprovementswhicharebothfasterandmoreaccuratethanclassicalnumerical analysisalgorithms.Amust-readforeveryalgorithmdeveloperandpractitionerinoptimization!’ –RalfHerbrich,HassoPlattnerInstitute ‘Probabilistic numerics spans from the intellectual fireworks of the dawn of a new field to its practicalalgorithmicconsequences.Itisprecisebutaccessibleandrichinwide-ranging,principled examples. This convergence of ideas from diverse fields in lucid style is the very fabric of good science.’ –CarlEdwardRasmussen,UniversityofCambridge ‘An important read for anyone who has thought about uncertainty in numerical methods; an essentialreadforanyonewhohasn’t’ –JohnCunningham,ColumbiaUniversity ‘This is a rare example of a textbook that essentially founds a new field, re-casting numerics on stronger,moregeneralfoundations.Atourdeforce.’ –DavidDuvenaud,UniversityofToronto ‘Theauthorssucceedindemonstratingthepotentialofprobabilisticnumericstotransformtheway wethinkaboutcomputationitself.’ –ThoreGraepel,SeniorVicePresident,AltosLabs Published online by Cambridge University Press P H I L I P P H E N N I G Eberhard–Karls–Universität Tübingen, Germany M I C H A E L A . O S B O R N E University of Oxford H A N S P. K E R S T I N G École Normale Supérieure, Paris P R O B A B I L I S T I C N U M E R I C S COMPUTATION AS MACHINE LEARNING Published online by Cambridge University Press UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom OneLibertyPlaza,20thFloor,NewYork,NY10006,USA 477WilliamstownRoad,PortMelbourne,VIC3207,Australia 314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,NewDelhi–110025,India 103PenangRoad,#05-06/07,VisioncrestCommercial,Singapore238467 CambridgeUniversityPressispartoftheUniversityofCambridge. ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof education,learning,andresearchatthehighestinternationallevelsofexcellence. www.cambridge.org Informationonthistitle:www.cambridge.org/9781107163447 DOI:10.1017/9781316681411 ©PhilippHennig,MichaelA.OsborneandHansP.Kersting2022 Thispublicationisincopyright.Subjecttostatutoryexception andtotheprovisionsofrelevantcollectivelicensingagreements, noreproductionofanypartmaytakeplacewithoutthewritten permissionofCambridgeUniversityPress. Firstpublished2022 PrintedintheUnitedKingdombyTJBooksLimited,PadstowCornwall AcataloguerecordforthispublicationisavailablefromtheBritishLibrary. ISBN978-1-107-16344-7Hardback CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyof URLsforexternalorthird-partyinternetwebsitesreferredtointhispublication anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain, accurateorappropriate. Published online by Cambridge University Press Toourfamilies. Published online by Cambridge University Press Measurement owes its existence to Earth Estimation of quantity to Measurement Calculation to Estimation of quantity Balancing of chances to Calculation and Victory to Balancing of chances. Sun Tzu – The Art of War §4.18: Tactical Dispositions Translation by Lionel Giles, 1910 Published online by Cambridge University Press Contents Acknowledgements pageix SymbolsandNotation xi Introduction 1 I Mathematical Background 17 1 KeyPoints 19 2 ProbabilisticInference 21 3 GaussianAlgebra 23 4 Regression 27 5 Gauss–MarkovProcesses:FilteringandSDEs 41 6 HierarchicalInferenceinGaussianModels 55 7 SummaryofPartI 61 II Integration 63 8 KeyPoints 65 9 Introduction 69 10 BayesianQuadrature 75 11 LinkstoClassicalQuadrature 87 12 ProbabilisticNumericalLessonsfromIntegration 107 13 SummaryofPartIIandFurtherReading 119 III Linear Algebra 123 14 KeyPoints 125 15 RequiredBackground 127 16 Introduction 131 17 EvaluationStrategies 137 18 AReviewofSomeClassicSolvers 143 19 ProbabilisticLinearSolvers:AlgorithmicScaffold 149 20 ComputationalConstraints 169 21 UncertaintyCalibration 175 Published online by Cambridge University Press viii Contents 22 Proofs 183 23 SummaryofPartIII 193 IV Local Optimisation 195 24 KeyPoints 197 25 ProblemSetting 199 26 Step-SizeSelection–aCaseStudy 203 27 ControllingOptimisationbyProbabilisticEstimates 221 28 First-andSecond-OrderMethods 229 V Global Optimisation 243 29 KeyPoints 245 30 Introduction 247 31 BayesianOptimisation 251 32 ValueLoss 259 33 OtherAcquisitionFunctions 267 34 FurtherTopics 275 VI Solving Ordinary Differential Equations 279 35 KeyPoints 281 36 Introduction 285 37 ClassicalODESolversasRegressionMethods 289 38 ODEFiltersandSmoothers 295 39 TheoryofODEFiltersandSmoothers 317 40 PerturbativeSolvers 331 41 FurtherTopics 339 VII The Frontier 349 42 SoWhat? 351 VIII Solutions toExercises 357 References 369 Index 395 Published online by Cambridge University Press Acknowledgements Many people helped in the preparation of this book. We, the authors,extendourgratitudetothefollowingpeople,without whomthisbookwouldhavebeenimpossible. WeareparticularlygratefultoMarkGirolamiforhisinvolve- mentduringtheearlystagesofthisbookasaproject.Though hecouldnotjoinasanauthorintheend,heprovidedalotof supportandmotivationtomakethisbookareality. Wewouldliketodeeplythankthemanypeoplewhooffered detailed and thoughtful comments on drafts of the book: On- drej Bajgar, Nathanael Bosch, Jon Cockayne, Michael Cohen, Paul Duckworth, Nina Effenberger, Carl Henrik Ek, Giacomo Garegnani, Roman Garnett, Alexandra Gessner, Saad Hamid, Marius Hobbhahn, Toni Karvonen, Nicholas Krämer, Emilia Magnani, Chris Oates, Jonathan Schmidt, Sebastian Schulze, ThomasSchön,ArnoSolin,TimJ.Sullivan,SimoSärkkä,Filip Tronarp,EdWagstaff,XingchenWan,andRichardWilkinson. PhilippHennig Iwouldliketothankmyresearchgroup,notjustforthorough proof-reading,butforanintenseresearcheffortthatcontributed substantiallytotheresultspresentedinthisbook.And,above all,forwageringsomeoftheirprimeyears,andtheircareer,on me,andontheideaofprobabilisticnumerics: Edgar Klenske, Maren Mahsereci, Michael Schober, Simon Bartels,LukasBalles,AlexandraGessner,FilipdeRoos,Frank Schneider,EmiliaMagnani,NiklasWahlandHans-PeterWieser, FelixDangel,FrederikKünstner,JonathanWenger,Agustinus Kristiadi, Nicholas Krämer, Nathanael Bosch, Lukas Tatzel, ThomasGläßle,JuliaGrosse,KatharinaOtt,MariusHobbhahn, MotonobuKanagawa,FilipTronarp,RobinSchmidt,Jonathan Schmidt, Marvin Pförtner, Nina Effenberger, and Franziska Weiler. I am particularly grateful to those among this group who https://doi.org/10.1017/9781316681411.001 Published online by Cambridge University Press
The list of books you might like
