Algorithms with JULIA: Optimization, Machine Learning, and Differential Equations Using the JULIA Language

447 Pages·2022·3.069 MB·

by  Clemens Heitzinger

Clemens Heitzinger
Algorithms
with JULIA
Optimization, Machine Learning, and
Differential Equations Using the JULIA
Language
Algorithms with JULIA
Clemens Heitzinger
Algorithms with JULIA
Optimization, Machine Learning,
and Differential Equations Using the JULIA
Language
123
Clemens Heitzinger
Centerfor Artificial Intelligence
andMachineLearning (CAIML)
and
Department ofMathematics
andGeoinformation
Technische UniversitätWien
Vienna,Austria
ISBN978-3-031-16559-7 ISBN978-3-031-16560-3 (eBook)
https://doi.org/10.1007/978-3-031-16560-3
MathematicsSubjectClassification: 65-XX,34K28,65Mxx,65M06,65M08,65M60,65Kxx,65Yxx,
62M45,68T05
©SpringerNatureSwitzerlandAG2022
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
ToM.S.
Foreword
Students of applied mathematics are often confronted with textbooks that ei-
ther cover the mathematical principles and concepts of mathematical models
orwithtextbookswhichintroducethebasiclanguagestructuresofaprogram-
minglanguage.Manyauthorsfailtocovertheunderlyingmathematicaltheory
ofmodels,whichiscrucialinunderstandingtheapplicabilityofmodels–and
especiallytheirlimitations–torealworldproblems.Ontheotherhand,many
textbooksandmonographsfailtoaddressthecrucialstepfromthealgorithmic
formulationofanappliedproblemtotheactualimplementationandsolutionin
theformofanexecutableprogram.Thisbookbrilliantlycombinesthesetwoas-
pectsusingahighlevelopensourcecomputerlanguageandcoversmanyareasof
continuummodelbasedareasofnaturalandsocialsciences,appliedmathemat-
icsandengineering.Juliaisahigh-level,high-performance,dynamicprogram-
minglanguagewhichcanbeusedtowriteanyapplicationinnumericalanalysis
andcomputationalscience.ClemensHeitzingerhasgonethroughgreatlengths
toorganizethisbookintosequencesthatmakesenseforthebeginneraswellas
fortheexpertinoneparticularfield.
Theappliedtopicsarecarefullychosen,fromthemostrelevantstandardareas
likeordinaryandpartialdifferentialequationsandoptimizationtomorerecent
fields of interest like machine learning and neural networks. The chapters on
ordinaryandpartialdifferentialequationsincludeexamplesofhowtouseexist-
ingpackagesincludedintheJuliasoftware.Inthechapteraboutoptimization
themethodsforstandardlocaloptimizationarenicelyexplained.However,this
bookalsocontainsaveryrelevantchapteraboutglobaloptimization,including
methodssuchassimulatedannealingandagentbasedoptimizationalgorithms.
Allthisisnotsomethingusuallyfoundinthesamebook.Again,theglobalop-
timizationtheory,asfarasthegeneraltheoryexists,iswellpresentedandthe
applicationexamples(and,mostimportantly,thebenchmarkproblems)arewell
chosen.Onechapter–concernedwiththecurrentlymaybemostrelevantarea
–introducespracticalproblemsolvinginthefieldofmachinelearning.Theau-
thorcoversthebasicapproachoflearningviaartificialneuralnetworksaswell
asprobabilisticmethodsbasedonBayesiantheory.Again,thetopicsandexam-
vii
viii Foreword
plesarewellchosen,theunderlyingtheoryiswellexplained,andthesolutions
ofthechosenapplicationproblemsareimmediatelyimplementableinJulia.
ClemensHeitzingerhasbeeninvolvedinsomeofthemostimpressiveapplica-
tionsinengineeringandtheappliedphysicalsciences,coveringmicroelectron-
ics,sensorsandbiomedicalapplications.Thisbookcoversboththetheoretical
andpracticalaspectsofthispartofmodernscience.Theapproachtakeninthis
bookisnovelinthesensethatitgoesintoquitesomedetailinthetheoretical
background,whileatthesametimebeingbasedonamoderncomputingplat-
form.Inasense,thisworkservestheroleoftwobooks.Itismuchmorethana
cookbookfor“howtosolveproblemswithJulia,”butalsoagoodintroduction
tothemostrelevantproblemsincontinuummodelbasedscienceandengineer-
ing.Atthesametime,itgivesthenoviceinJuliaprogrammingagoodintroduc-
tiononhowtousethishigherlevelprogramminglanguage.Itcanthereforebe
usedasatextforstudentsinanadvancedgraduatelevelcourseaswellasamono-
graphbytheresearcherplanningtosolveactualproblemsbyprogrammingin
Julia.
Tempe,April2022 ChristianRinghofer
Preface
Why computation? The middle of the last century marks the beginning of
a new era in the history of mathematics. Although calculators and computers
hadbeenenvisionedcenturiesbeforeandmechanicalcalculatorsorcalculating
machineswereinwidespreadusealreadyinthenineteenthcentury,onlythein-
ventionofpurelyelectroniccomputersmadeitpossibletoperformcalculations
onincreasinglylargescales.Thereasonisquitesimple:mechanicalandelectro-
mechanicalcalculators(seeFig.0.1)areseverelylimitedbyfriction.
The advent of electronic computers and later the rise of the integrated cir-
cuithaveresultedinportabledevicesofastoundingcomputationalpowerattiny
power consumption (see Fig. 0.2). Computations that were unthinkable a few
decadesagocannowbeperformedatlowcostandatgreatspeed.Thesedevel-
opmentsinthephysicalrealmhaveresultedinthebirthofnewmathematical
disciplines.Computeralgebra,scientificcomputing,machinelearning,artificial
intelligence,andrelatedareasareconcernedwithsolvingabstractmathematical
problemsaswellasscientificanddata-scienceproblemscorrectly,precisely,and
efficiently.
Althoughlotsofcomputationalpowerareavailabletoday,fundamentalques-
tionswillalwayshavetobeanswered.Howshouldthecomputationsbestruc-
tured?Whataretheadvantagesanddisadvantagesofvariousalgorithms?How
accurate will the results be? How can we best take advantage of the computa-
tional resources available to us? These are fundamental questions that lead to
newandfascinatingmathematicalproblems.Inthissense,theinventionofelec-
tronic computers has had and will have a twofold influence on mathematics:
computersarebothanenablingtechnologyandasourceofnewmathematical
problems.
There is no doubt that computers and mathematical algorithms have im-
pactedourlivesinmanyways.Inmanyengineeringdisciplines,ithasbecome
common to perform simulations for the rational design and the optimization
of all kinds of devices and processes. Simulations can be much cheaper than
performingmanyexperimentsandtheyprovidetheoreticalandquantitativein-
sights.Examplesareairplanes,combustionengines,antennas,andtheconstruc-
ix
x Preface
Fig. 0.1 A Brunsviga 15 mechanical calculator, produced by Brunsviga-Maschinenwerke
Grimme,Natalis&Co.AG,Braunschweig,between1934and1947.Theshroudsareremoved
torevealtheinternalmechanic.(PhotobyCEphoto,UweAranas,nochanges,licenseCC-BY-
SA-3.0,https://creativecommons.org/licenses/by-sa/3.0/.)
Fig. 0.2 Motorola 68000 cpu. (Photo by Pauli Rautakorpi, no changes, license CC-BY-3.0,
https://creativecommons.org/licenses/by/3.0/.)
tion of bridges and other buildings. Large-scale computations are also behind
searchengines,financialservices,andotherdata-intensiveindustries.Thereis
probablynotasinglehourinourdailyliveswhenwedonotuseaserviceorade-
vicethathasonlybecomepossiblebycomputersandmathematicalalgorithms
orthathasbeenmuchimprovedbythem.
Preface xi
Usingthisbook,youwilllearnamodern,general-purpose,andefficientpro-
gramminglanguage,namelyJulia,aswellassomeofthemostimportantmeth-
odsinoptimization,machinelearning,anddifferentialequationsandhowthey
work.Thesethreefields,optimization,machinelearning,anddifferentialequa-
tions,havebeenchosenbecausetheycoverawiderangeofcomputationaltasks
inscience,engineering,andindustry.
Methodsandalgorithmsintheseareaswillbediscussedinsufficientdetailto
arriveatacompleteunderstanding.Youwillunderstandhowthecomputational
approachesworkstartingfromthebasicmathematicaltheory.Importantresults
andproofswillbegivenineachchapter(andcanbeskippedonfirstreading).
Basedonthesefoundations,youwillbeprovidedwiththeknowledgetoimple-
mentthealgorithmsandyourownvariants.Tothisend,sampleprogramsand
hintsforimplementationinJuliaareprovided.
Theultimatepurposeofthisbookistoprovidethereaderbothwithaworking
knowledge of the Julia programming languages as well as with more than a
superficial understanding of modern topics in three important computational
fields.Usingthealgorithmsandthesamplecodesforleadingproblems,youwill
be able to translate the theory into working knowledge in order to solve your
scientific,engineering,mathematical,orindustrialproblems.
How is this book unique? This book strivesto providea modern, practical,
andwellfoundedperspectiveonalgorithmsinoptimization,machine-learning,
and differential equations. Hence there are two points how the present book
differsfromotherbooksinthisarea.
First,thetopicswereselectedwithamodernviewofcomputationinmind.
Asmathematicsandcomputationevolve,weareabletosolvemoreandmore
difficultproblems.Theseadvancesarereflectedinthematerialinthisbook.For
example,topicssuchasartificialneuralnetworks,computationalBayesianesti-
mation,andpartialdifferentialequationsarediscussed,butnumericallysolving
systemsoflinearequationsisnot,sinceyouwillmostlikelynotwriteyourown
programtodosoduetheavailabilityofwelltestedlibraries(alsoimmediately
availableinJulia).
Optimizationisofgreatvalueinalmostalldisciplines.Differentialequations
areofgreatutilityinmanycaseswherefundamentalrelationshipsbetweenthe
known and unknown variables exist, for example in physics, chemistry, and
many engineering disciplines. Furthermore, machine learning in particular is
anareathathasbenefitedfromincreasesincomputationalpowerandavailable
memoryandthatisofutmostimportancewhenlargeamountsofdataareavail-
able,butfundamentalrelationshipsareunknown.
Second, the Julia language, a rather young language designed with scien-
tificandtechnicalcomputinginmind,isusedtoimplementthealgorithms.Its
implementation includes a compiler and a type system that leads to fast com-
piled code. It builds on modern and general programming concepts so that it
isusableformanydifferentpurposes.Itcomeswithlinear-algebraalgorithms,
sparsematrices,andapackagesystem.Itisopensourceanditssyntaxiseasy