Principles of Parallel Scientific Computing

302 Pages·2022·19.998 MB·English

by  Tobias Weinzierl

Undergraduate Topics in Computer Science
Tobias Weinzierl
Principles of
Parallel Scientific
Computing
A First Guide to Numerical Concepts
and Programming Methods
Undergraduate Topics in Computer
Science
Series Editor
Ian Mackie, University of Sussex, Brighton, UK
Advisory Editors
Samson Abramsky , Department of Computer Science, University of Oxford,
Oxford, UK
ChrisHankin ,DepartmentofComputing,ImperialCollegeLondon,London,UK
Mike Hinchey , Lero – The Irish Software Research Centre, University of
Limerick, Limerick, Ireland
Dexter C. Kozen, Department of Computer Science, Cornell University, Ithaca,
NY, USA
Andrew Pitts , Department of Computer Science and Technology, University of
Cambridge, Cambridge, UK
HanneRiisNielson ,DepartmentofAppliedMathematicsandComputerScience,
Technical University of Denmark, Kongens Lyngby, Denmark
StevenS.Skiena,DepartmentofComputerScience,StonyBrookUniversity,Stony
Brook, NY, USA
Iain Stewart , Department of Computer Science, Durham University, Durham,
UK
‘Undergraduate Topics in Computer Science’ (UTiCS) delivers high-quality
instructional content for undergraduates studying in all areas of computing and
information science. From core foundational and theoretical material to final-year
topics and applications, UTiCS books take a fresh, concise, and modern approach
and are ideal for self-study or for a one- or two-semester course. The texts are all
authoredbyestablishedexpertsintheirfields,reviewedbyaninternationaladvisory
board,andcontainnumerousexamplesandproblems,manyofwhichincludefully
worked solutions.
TheUTiCSconceptreliesonhigh-quality,concisebooksinsoftbackformat,and
generally a maximum of 275–300 pages. For undergraduate textbooks that are
likelytobelonger,moreexpository,Springercontinuestoofferthehighlyregarded
Texts in Computer Science series, to which we refer potential authors.
More information about this series at https://link.springer.com/bookseries/7592
Tobias Weinzierl
Principles of Parallel
Scientific Computing
A First Guide to Numerical Concepts
and Programming Methods
123
TobiasWeinzierl
Department ofComputer Science
Durham University
Durham, UK
ISSN 1863-7310 ISSN 2197-1781 (electronic)
Undergraduate Topics inComputer Science
ISBN978-3-030-76193-6 ISBN978-3-030-76194-3 (eBook)
https://doi.org/10.1007/978-3-030-76194-3
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
SwitzerlandAG2021
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseof
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregard
tojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Why This Book
IstartedtothinkaboutthismanuscriptwhenIgotappointedasAssociateProfessor
atDurhamUniversityandhadbeentoldtoreadnumericalalgorithmsforcomputer
scientists plus theparallel programming submodule. InDurham, we teach thisfirst
submodule on numerics in the third year of the computer science degree. It spans
only ten hours. Ten hours is ridiculously short, but I think it is enough to spark
interest in simulation codes and the simulation business. Thinking about it, it is
great that our numerics teaching runs parallel to a parallel programming course.
Numerical simulation without parallelism today is barely imaginable, so this
arrangement gives us the great opportunity to use examples from one area in the
other one. We can co-teach the two things: numerics basics and parallel coding.
When we later designed our Master in Scientific Computing and Data Analysis
(MISCADA), itbecameclearthatco-teaching thetwofieldswouldbeagoodidea
for this course, too. With a very heterogeneous student intake—both by means of
qualification, cultural background, educational system, and discipline interest—we
need to run introductory courses that bring all students quickly to a decent level,
while numerical principles as well as parallel programming are omnipresent.
Obviously,theM.Sc.-level teaching progressesatadifferent paceand,inDurham,
also covers more lectures. Yet, some contentoverlapswith thethirdyear’scourse.
There seems to be a limited amount of literature that discusses parallel pro-
gramming and (the basics of) numerical techniques in one go and still remains
easily accessible. There are many books that cover scientific computing from a
mathematics point of view. Many scientists (including me) insist that a formal
approach with a strong mathematical background is imperative for research in
scientificcomputing.However,wehavetoappreciatethatmanyofourundergrads
are enrolled in courses that are lighter on the mathematics side or focus, for
example,moreondiscretemaths.Somemathsthusmightbealittlebitrustywhen
studentsentertheirthirdyear.Ourmathematicaltextbooksruntheriskthatwelose
outonstudentsiftheyorbitaroundtheprofoundmathsfirst.Theymightgetbored
beforetheyreachthepointwheretheycanprogramafirstnumericalcodethatdoes
“something useful”.
v
vi Preface
Theothertypeofteachingliteraturethatisouttherecombinesthefirstcoursein
numericswithaprogrammingtutorial.AlthoughalotofthematerialIhaveseenis
really good, this type of approach is ill-suited for our students. Most of them are
decent programmers already.
To make a long story short, I think that there is a need for courses on numerics
witha“rollingupyoursleeves”and“gettheroughideaandmakeitworkincode”
attitude.Somestudentsliketogettheirhandsdirtyandwritesomething“thatdoes
something” before they start to dive into more details—for which excellent liter-
ature does exist. I am also strongly convinced that it makes sense to teach intro-
ductoryscientificcomputingincombinationwithparallelisation.Afterall,wewant
to have future Ph.D. students and colleagues who can help us with the big simu-
lation codes.
Mission Statement
Since we try to keep the theory light and hands-on, this book is not a compre-
hensive overview of the vast field of numerical algorithms and analysis, or even
only a certain subfield. It neither is a rock solid introduction to the difficulties,
pitfalls and depths of programming in high performance computing (HPC). It is
(cid:129) a rush through some basic numerical concepts that are omnipresent when we
writesimulationsoftware.Wewanttogetanideaofhowsomenumericalthings
work—so we can eventually look them up in a “real” maths book.
(cid:129) a numerics primer with implementation glasses on: For all of the concepts, we
wanttohaveanideaofhowtorealisethemonarealmachineorwhichimpact
they have on our codes.
(cid:129) a hands-on text. The best way to understand numerical concepts is to program
them or, even better, to see your own code fail and then find a theoretical
explanation (together with the fix) for the break down. And it is close to
impossible to learn parallel programming solely from textbooks anyway.
(cid:129) a guide that makes students start to write simulation codes running on parallel
computers.
Structure and Style
Themanuscriptjumpsfromparallelprogrammingtonumericsandback.Isprinkle
insomelight-touchrevisionofthingsthatIconsidertobeimportantbuthavebeen
told by my students that many computer science curricula do not cover them
anymore. Even if taught at school before, students might have forgotten them.
Wherepossible,ItrytomakeallcontentfeedintoonesimpleN-bodysolver.That’s
the application leitmotif throughout the manuscript. If readers translate it into a
working code, they will end up with an N-body solver with simple gravity, will
Preface vii
havelearnedfundamentalideasbehindthesolversuchasnumericalterminologyor
challengesfacedbyintegratorsforordinarydifferentialequations,andwillhavethe
skill set to make this solver run efficiently on a shared memory machine or some
GPUs.There’snoneedforareader(orteacher)tosticktomyconceptortoreadthe
chapters one by one. Feel free to skip chapters, pick chapters, or use a different
application example.
Mathematicians will be horrified by my hands-on explanations and lack of
formalism. I am fine with this. This book is not yet another maths course. It is a
bookthatshall bequickandeasytoread andmotivatestudents tostudy themaths
behind the scenes later, oncethe first generation oftheir simulation codeisup and
running.Thisbookisalsonotacoursewherestudentsreallylearnsomethingabout
N-bodycodesforaparticularapplicationarea.SimpleN-bodycodesjustseemedto
be a convenient toy problem that we can understand with A-levels maths.
ExpertsdeeplyfamiliarwiththedetailsofOpenMP,e.g.,alsowillbehorrifiedby
theinaccuraciesinthetext.Forexample,thevisibilityofavariableintheOpenMP
standard is determined through a sequence of rules. These rules are important, but
most students quickly will be lost in them. Also, most programmers don’t have to
memorise them. They have toknow what happens in 99% ofthe cases.
As I want to push students into the field of scientific computing plus HPC, I
madeallexamplesinthetextrelyonC/C++.Iamwellawarethatmanycodesstill
are written in Fortran, and I am well aware that people nowadays can write inter-
esting and fast software with Python, e.g. But C still is very dominant and, fortu-
nately, many computer science students are still exposed to C in one form or the
other throughout their undergraduate studies. While A-levels maths is sufficient to
digest the book’s content, I hence expect students to bring along C programming
skills.1
Maybe unusual for a student-facing book, this text does not contain any exer-
cises.Therearetworeasonsforthis:Ontheonehand,thereareplentyofbooks,in
particularinnumerics,thatarefullofexercises;andmoreandmorewebpageswith
(interactive) examples follow. On the other hand, the prime way to exercising
scientific computing is to write codes, and I think that there is a big difference
between writing small, well-defined example calculations or a bigger piece of
software. With this text, I want to encourage students to do the latter.
Acknowledgements
Thanksareduetonumerousstudentswhoprovidedfeedback.Particularthanksare
duetomycolleaguesFlorianRuppandPhilippNeumann.Theyfoundtonsofbugs
andhelpedmetofixalotofphrases.ThanksarealsoduetoLawrenceMitchellfor
proofreadingsomeofthechapterswithanHPCtouch.HolgerSchulzworkedwith
1The UTICS series hosts, for example, the Pitt-Francis and Whiteley book “Guide to Scientific
ComputinginC++”whereChaps.1–5provideareasonablebaseforthepresentbook.
viii Preface
me on the ExCALIBUR/ExaClaw project throughout the write-up and helped me
quite a lot with the OpenMP-focussed chapters. Marion Weinzierl finally was
maybe the most critical reader, but eventually it had been her comments and her
support over all the years that made writing this manuscript possible.
Learning and Teaching Mode
MajorpartsofthismanuscripthavebeenwrittenduringthefirstCoronalockdown.
Iamsurethatthisunexpectedsituationhasforcedmanylecturerstorevise,rethink
and re-evaluate their academic teaching style. I had to. Though I do not know
(yet)whetherthisbookisofanyuseorhelpforthebrandnewworldofblendedand
virtual teaching, the manuscript style is a result of my personal “rethink your
teaching” exercise.
TheappendixhostssomeideasonhowIusethebook’scontenttostructuremy
classes. The text is written in a language such that students can smoothly work
through the content themselves: If I have to make a choice, I run for the sloppy
jargon rather than ultra-formal, dryish phrases. Furthermore, I tried to use illus-
trationsfortechnicaldetails,asIammyselfavisuallearner.Keylearningoutcomes
areextractedandorganisedintoboxes,andtheappendixhostssomechecklistsand
cheatsheetsforstudents’coursework.Allofthisallowsmetousethetextasabasis
for courses in a “flipped classroom”-style. Obviously, this style can be comple-
mented by online/synchronous sessions, videos, exercises and code studies; and,
obviously, reading lists that dive deeper into the subject.
Enjoy the text, and I hope that it is useful.
Durham, UK Tobias Weinzierl
January 2022
Contents
Part I Introduction: Why to Study the Subject
1 The Pillars of Science . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1 A Third Pillar? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Computational X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3 About the Style of the Text and Some Shortcomings . . . . . . . . 9
2 Moore Myths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Moore’s Law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Dennard Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 The Three Layers of Parallelism . . . . . . . . . . . . . . . . . . . . . . . 15
3 Our Model Problem (Our First Encounter with the Explicit
Euler) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.1 The N-Body Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2 Time Discretisation: Running a Movie. . . . . . . . . . . . . . . . . . . 23
3.3 Numerical Versus Analytical Solutions. . . . . . . . . . . . . . . . . . . 25
Wrap-up
Part II How the Machine Works
4 Floating Point Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.1 Fixed Point Formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.2 Floating Point Numbers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.2.1 Normalisation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.2.2 The IEEE Format. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2.3 Realising Floating-Point Support. . . . . . . . . . . . . . . . . 40
4.3 Machine Precision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
4.4 Programming Guidelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
ix