Numerical Solution of Partial Differential Equations on Parallel Computers

491 Pages·2005·7.467 MB·English

by  Hans Petter Langtangen

Editors
TimothyJ.Barth
MichaelGriebel
DavidE.Keyes
RistoM.Nieminen
DirkRoose
TamarSchlick
Are Magnus Bruaset Aslak Tveito (Eds.)
Numerical Solution
of Partial Differential
Equations on Parallel
Computers
With201Figuresand42Tables
ABC
Editors
AreMagnusBruaset
Aslak Tveito
SimulaResearchLaboratory
P.O.Box134
1325Lysaker,Fornebu,Norway
email:[email protected]
[email protected]
The second editor of this book has received financial support from the NFF – Norsk faglitterær
forfatter-ogoversetterforening
LibraryofCongressControlNumber:2005934453
MathematicsSubjectClassification:
Primary:65M06,65M50,65M55,65M60,65Y05,65Y10
Secondary:65N06,65N30,65N50,65N55,65F10,65F50
ISBN-10 3-540-29076-1SpringerBerlinHeidelbergNewYork
ISBN-13 978-3-540-29076-6SpringerBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting,
reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,
1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violationsare
liableforprosecutionundertheGermanCopyrightLaw.
SpringerisapartofSpringerScience+BusinessMedia
springer.com
(cid:1)c Springer-VerlagBerlinHeidelberg2006
PrintedinTheNetherlands
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnotimply,
evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotectivelaws
andregulationsandthereforefreeforgeneraluse.
Typesetting:bytheauthorsandTechBooksusingaSpringerLATEXmacropackage
Coverdesign:design&productionGmbH,Heidelberg
Printedonacid-freepaper SPIN:11548843 46/TechBooks 543210
Preface
Since the dawn of computing, the quest for a better understanding of Nature has
been a driving force for technological development. Groundbreaking achievements
bygreatscientistshavepavedthewayfromtheabacustothesupercomputingpower
oftoday.WhentryingtoreplicateNatureinthecomputer’ssilicontesttube,thereis
needforpreciseandcomputableprocessdescriptions.ThescientificfieldsofMath-
ematics and Physics provide a powerful vehicle for such descriptions in terms of
PartialDifferentialEquations(PDEs).Formulatedassuchequations,physicallaws
can become subject to computational and analytical studies. In the computational
setting,theequationscanbediscreti edforefficientsolutiononacomputer,leading
tovaluabletoolsforsimulationofnaturalandman-madeprocesses.Numericalsolu-
tionofPDE-based mathematical models hasbeenanimportant researchtopicover
centuries,andwillremainsoforcenturiestocome.
Inthecontextofcomputer-basedsimulations,thequalityofthecomputedresults
isdirectlyconnectedtothemodel’scomplexityandthenumberofdatapointsused
forthecomputations.Therefore,computationalscientiststendtofilleventhelargest
and most powerful computers they can get access to, either by increasing the si e
ofthedatasets,orbyintroducingnewmodeltermsthatmakethesimulationsmore
realistic,oracombinationofboth.Today,manyimportantsimulationproblemscan
notbesolvedbyonesinglecomputer,butcallsforparallelcomputing.Whetherbe-
ingadedicatedmulti-processorsupercomputeroralooselycoupledclusterofoffice
workstations,theconceptofparallelismoffersincreaseddatastorageandincreased
computing power. In theory, one gets access to the grand total of the resources of-
feredbytheindividualunitsthatmakeupthemulti-processorenvironment.Inprac-
tice,thingsaremorecomplicated,andtheneedfordatacommunicationbetweenthe
differentcomputationalunitsconsumespartsofthetheoreticalgainofpower.
Summingupthebitsandpiecesthatgointoalarge-scaleparallelcomputation,
thereareaspects of hardware, systemsoftware, communication protocols, memory
management,andsolutionalgorithmsthathavetobeaddressed.However,overtime
efficient ways of addressing these issues have emerged, better software tools have
becomeavailable,andthecostofhardwarehasfallenconsiderably.Today,compu-
tational clusters made from commodity parts can be set up within the budget of a
VI Preface
typical research department, either as a turn-key solution or as a do-it-yourself
project.Supercomputinghasbecomeaffordableandaccessible.
Aboutthisbook
This book addresses the major topics involved in numerical simulations on paral-
lel computers, where the underlying mathematical models are formulated in terms
of PDEs. Most of the chapters dealing with the technological components of par-
allelcomputingarewritteninasurveystyleandwillprovideacomprehensive,but
stillreadable,introductionforstudentsandresearchers.Otherchaptersaremorespe-
cialized,forinstancefocusingonaspecificapplicationthatcandemonstratepracti-
calproblemsandsolutionsassociatedwithparallelcomputations.Aseditorsweare
proudtoputtogetheravolumeofhigh-qualityandusefulcontributions,writtenby
internationallyacknowledgedexpertsonhigh-performancecomputing.
The first part of the book addresses fundamental parts of parallel computing in
terms of hardware and system software. These issues are vital to all types of par-
allel computing, not only in the context of numerical solution of PDEs. To start
with,RickyKendallandco-authorsdiscusstheprogrammingmodelsthataremost
commonlyusedforparallelapplications,inenvironmentsrangingfromasimplede-
partmentalclusterofworkstationstosomeofthemostpowerfulcomputersavailable
today.Theirdiscussioncoversmodelsformessagepassingandsharedmemorypro-
gramming,aswellassomefutureprogrammingmodels.Inacloselyrelatedchapter,
JimTerescoetal.lookathowdatashouldbepartitionedbetweentheprocessorsin
aparallelcomputingenvironment,suchthatthecomputationalresourcesareutilized
as efficient as possible. In a similar spirit, the contribution by Martin Rumpf and
RobertStrzodkaalsoaimsatimprovedutilizationoftheavailablecomputationalre-
sources. However, their approach is somewhat unconventional, looking at ways to
benefit from the considerable power available in graphics processors, not only for
visualization purposes but also for numerical PDE solvers. Given the low cost and
easy access of such commodity processors, one might imagine future cluster solu-
tionswithreallyimpressiveprice-performanceratios.
Oncethecomputationalinfrastructureisinplace,oneshouldconcentrateonhow
the PDE problems can be solved in an efficient manner. This is the topic of the
second part of the book, which is dedicated to parallel algorithms that are vital to
numerical PDE solution. Luca Formaggia and co-authors present parallel domain
decompositionmethods.Inparticular,theygiveanoverviewofalgebraicdomainde-
compositiontechniques,andintroducesophisticatedpreconditionersbasedonamul-
tilevelapproximativeSchurcomplementsystemandaSchwarz-typedecomposition,
respectively. As Schwarz-type methods call for a coarse level correction, the paper
alsoproposesastrategyforconstructingcoarseoperatorsdirectlyfromthealgebraic
problemformulation,therebyhandlingunstructuredmeshesforwhichacoarsegrid
canbedifficulttodefine.Complementingthismultilevelapproach,FrankHu¨lsemann
etal.discusshowanotherimportantfamilyofveryefficientPDEsolvers,geometric
multigrid, can be implemented on parallel computers. Like domain decomposition
methods, multigrid algorithms are potentially capable of being order-optimal such
Preface VII
that the solution time scales linearly with the number of unknowns. However, this
paper demonstrates that in order to maintain high computational performance the
constructionofaparallelmultigridsolveriscertainlyproblem-dependent.Inthefol-
lowing chapter, Ulrike Meier Yang addresses parallel algebraic multigrid methods.
In contrast to the geometric multigrid variants, these algorithms work only on the
algebraic system arising from the discretization of the PDE, rather than on a mul-
tiresolutiondiscretizationofthecomputationaldomain.Endingthesectiononparal-
lel algorithms, Nikos Chrisochoides surveys methods for parallel mesh generation.
MeshingproceduresareanimportantpartofthediscretizationofaPDE,eitherused
asapreprocessingsteppriortothesolutionphase,orincaseofachanginggeometry,
as repeated steps in course of the simulation. This contribution concludes that it is
possibletodevelopparallelmeshingsoftwareusingoff-the-shelfsequentialcodesas
buildingblockswithoutsacrificingthequalityoftheconstructedmesh.
Making advanced algorithms work in practice calls for development of sophis-
ticated software. This is especially important in the context of parallel computing,
asthecomplexityofthesoftwaredevelopmenttendstobesignificantlyhigherthan
for its sequential counterparts. For this reason, it is desirable to have access to a
widerangeofsoftwaretoolsthatcanhelpmakeparallelcomputingaccessible.One
wayofaddressingthisneedistosupplyhigh-qualitysoftwarelibrariesthatprovide
parallel computing power to the application developer, straight out of the box. The
hyprelibrarypresentedbyRobertD.Falgoutetal.doesexactlythisbyofferingpar-
allelhigh-performancepreconditioners.Theirpaperconcentratesontheconceptual
interfaces in this package, how these are implemented for parallel computers, and
how they are used in applications. As an alternative, or complement, to the library
approach,onemightlookforprogramminglanguagesthattriestoeasetheprocess
ofparallelcoding.Ingeneral,thisisaquiteopenissue,butXingCaiandHansPet-
ter Langtangen contribute to this discussion by considering whether the high-level
languagePythoncanbeusedtodevelopefficientparallelPDEsolvers.Theyaddress
this topic from two different angles, looking at the performance of parallel PDE
solvers mainly based on Python code and native data structures, and through the
use of Python to parallelize existing sequential PDE solvers written in a compiled
languagelikeFORTRAN,CorC++.Thelatterapproachalsoopensforthepossibil-
ity of combining different codes in order to address a multi-model or multiphysics
problem. This is exactly the concern of Lois Curfman McInnes and her co-authors
whentheydiscusstheuseoftheCommonComponentArchitecture(CCA)forparal-
lelPDE-basedsimulations.TheirpapergivesanintroductiontoCCAandhighlights
several parallel applications for which this component technology is used, ranging
fromclimatemodelingtosimulationofaccidentalfiresandexplosions.
To communicate experiences gained from work on some complete simulators,
selected parallel applications are discussed in the latter part of the book. Xing Cai
and Glenn Terje Lines present work on a full-scale parallel simulation of the elec-
trophysiology of the human heart. This is a computationally challenging problem,
whichduetoamultiscalenaturerequiresalargeamountofunknowns thathaveto
beresolvedforsmalltimesteps.Itcanbearguedthatfull-scalesimulationsofthis
problem can not be done without parallel computers. Another challenging geody-
VIII Preface
namicsproblem,modelingthemagmagenesisinsubductionzones,isdiscussedby
Matthew G. Knepley et al. They have ported an existing geodynamics code to use
PETSc, thereby making it parallel and extending its functionality. Simulations per-
formedwiththeresultingapplicationconfirmsphysicalobservationsofthethermal
propertiesinsubductionzones,whichuntilrecentlywerenotpredictedbycomputa-
tions. Finally, in the last chapter of the book, Carolin Ko¨rner et al. present parallel
Lattice Boltzmann Methods (LBMs) that are applicable to problems in Computa-
tionalFluidDynamics.AlthoughnotbeingaPDE-basedmodel,theLBMapproach
canbeanattractivealternative,especiallyintermsofcomputationalefficiency.The
powerofthemethodisdemonstratedthroughcomputationof3Dfreesurfaceflow,
asintheinteractionandgrowingofgasbubblesinamelt.
Acknowledgements
We wish to thank all the chapter authors, who have written very informative and
thoroughcontributionsthatwethinkwillservethecomputationalcommunitywell.
Theirenthusiasmhasbeencrucialforthequalityoftheresultingbook.
Moreover,wewishtoexpressourgratitudetoallreviewers,whohaveputtime
and energy into this project. Their expert advice on the individual papers has been
usefultoeditorsandcontributorsalike.WearealsoindebtedtoDr.MartinPetersat
Springer-Verlagformanyinterestingandusefuldiscussions,andforencouragingthe
publicationofthisvolume.
Fornebu AreMagnusBruaset
September,2005 AslakTveito
Contents
PartI ParallelComputing
1 Parallel Programming Models Applicable to Cluster Computing
andBeyond
RickyA.Kendall,MashaSosonkina,WilliamD.Gropp,RobertW.Numrich,
ThomasSterling .................................................. 3
1.1 Introduction ................................................ 3
1.2 Message-PassingInterface .................................... 7
1.3 Shared-MemoryProgrammingwithOpenMP .................... 20
1.4 DistributedShared-MemoryProgrammingModels................ 36
1.5 FutureProgrammingModels .................................. 42
1.6 FinalThoughts.............................................. 49
References ...................................................... 50
2 PartitioningandDynamicLoadBalancingfortheNumericalSolution
ofPartialDifferentialEquations
JamesD.Teresco,KarenD.Devine,JosephE.Flaherty .................. 55
2.1 ThePartitioningandDynamicLoadBalancingProblems .......... 56
2.2 PartitioningandDynamicLoadBalancingTaxonomy ............. 60
2.3 AlgorithmComparisons ...................................... 69
2.4 Software................................................... 71
2.5 CurrentChallenges .......................................... 74
References ...................................................... 81
3 GraphicsProcessorUnits:NewProspectsforParallelComputing
MartinRumpf,RobertStrzodka ...................................... 89
3.1 Introduction ................................................ 89
3.2 Theory .................................................... 97
3.3 Practice.................................................... 103
3.4 Prospects .................................................. 118
3.5 Appendix:GraphicsProcessorUnits(GPUs)In-Depth ............ 121
X Contents
References ...................................................... 131
PartII ParallelAlgorithms
4 DomainDecompositionTechniques
LucaFormaggia,MarzioSala,FaustoSaleri............................135
4.1 Introduction ................................................ 135
4.2 TheSchurComplementSystem................................ 138
4.3 TheSchurComplementSystemUsedasaPreconditioner.......... 146
4.4 TheSchwarzPreconditioner .................................. 147
4.5 Applications................................................ 152
4.6 Conclusions ................................................ 159
References ...................................................... 162
5 ParallelGeometricMultigrid
FrankHu¨lsemann,MarkusKowarschik,MarcusMohr,UlrichRu¨de .........165
5.1 Overview .................................................. 165
5.2 IntroductiontoMultigrid ..................................... 166
5.3 ElementaryParallelMultigrid ................................. 177
5.4 ParallelMultigridforUnstructuredGridApplications ............. 189
5.5 Single-NodePerformance .................................... 193
5.6 AdvancedParallelMultigrid .................................. 195
5.7 Conclusions ................................................ 204
References ...................................................... 205
6 Parallel Algebraic Multigrid Methods – High
PerformancePreconditioners
UlrikeMeierYang.................................................209
6.1 Introduction ................................................ 209
6.2 AlgebraicMultigrid-ConceptandDescription................... 210
6.3 CoarseGridSelection........................................ 212
6.4 Interpolation................................................ 220
6.5 Smoothing ................................................. 223
6.6 NumericalResults........................................... 225
6.7 SoftwarePackages........................................... 230
6.8 ConclusionsandFutureWork ................................. 232
References ...................................................... 233
7 ParallelMeshGeneration
NikosChrisochoides ...............................................237
7.1 Introduction ................................................ 237
7.2 DomainDecompositionApproaches............................ 238
7.3 ParallelMeshGenerationMethods ............................. 240
7.4 Taxonomy ................................................. 255
7.5 Implementation ............................................. 255