Table Of ContentAdvances in Numerical Mathematics
Gerhard Zumbusch
Parallel Multilevel Methods
Advances in Numerical Mathematics
Editors
Hans Georg Bock
Wolfgang Hackbusch
Mitchell Luskin
Rolf Rannacher
Gerhard Zumbusch
Parallel Multilevel
Methods
Adaptive Mesh Refinement
and loadbalancing
Teubner
B. G. Teubner Stuttgart· Leipzig· Wiesbaden
Bibliografische Information der Deutschen Bibliothek
Die Deutsche Bibliothek verzeichnet diese Publikation in der Deutschen Nationalbibliographie;
detaillierte bibliografische Daten sind im Internet Ober <http://dnb.ddb.de> abrufbar.
Prof. Dr. Gerhard Zumbusch
Geboren 1968 in MOnster. Studium der Mathematik 1987-1992 an der TU MOnchen, Diplom. Von
1993 bis 1995 Konrad-Zuse-Zentrum fOr Informationstechnik Berlin, Promotion 1995 FU Berlin, an
schlieBend SINTEF Anvendt Matematikk Oslo 1996. Danach Universitat Bonn 1997-2002, Habilitation
2001, Privat-Dozent 2002. Seit 2002 Professor an der Friedrich-Schiller-Universitat Jena, Lehrstuhl
fOr Wissenschaftliches Rechnen/Numerische Mathematik, Direktor des Instituts fOr Angewandte
Mathematik.
1 . Auflage November 2003
Aile Rechte vorbehalten
© B. G. Teubner Verlag I GWV Fachverlage GmbH, Wiesbaden 2003
Der B. G. Teubner Verlag ist ein Unternehmen von Springer Scienc€+Business Media.
www.teubner.de
Das Werk einschlieBlich aller seiner Teile ist urheberrechtlich geschOtzt. Jede Verwertung
auBerhalb der engen Grenzen des Urheberrechtsgesetzes ist ohne Zustimmung des Ver
lags unzulassig und strafbar. Das gilt insbesondere fOr Vervielfaltigungen, Obersetzun
gen, Mikroverfilmungen und die Einspeicherung und Verarbeitung in elektronischen
Systemen.
Die Wiedergabe von Gebrauchsnamen, Handelsnamen, Warenbezeichnungen usw. in diesem Werk
berechtigt auch ohne besondere Kennzeichnung nicht zu der Annahme, dass solche Namen im Sinne
der Warenzeichen- und Markenschutz-Gesetzgebung als frei zu betrachten waren und daher von
jedermann benutzt werden dOrften.
Umschlaggestaltung: Ulrike Weigel, www.CorporateDesignGroup.de
Gedruckt auf saurefreiem und chlorfrei gebleichtem Papier.
ISBN-13:978-3-519-00451-6 e-ISBN-13:978-3-322-80063-3
001: 10.1007/978-3-322-80063-3
Preface
Numerical simulation promises new insight in science and engineering. In ad
dition to the traditional ways to perform research in science, that is laboratory
experiments and theoretical work, a third way is being established: numerical
simulation. It is based on both mathematical models and experiments con
ducted on a computer. The discipline of scientific computing combines all
aspects of numerical simulation. The typical approach in scientific computing
includes modelling, numerics and simulation, see Figure l.
Quite a lot of phenomena in science and engineering can be modelled by
partial differential equations (PDEs). In order to produce accurate results,
complex models and high resolution simulations are needed. While it is easy
to increase the precision of a simulation, the computational cost of doing so is
often prohibitive. Highly efficient simulation methods are needed to overcome
this problem. This includes three building blocks for computational efficiency,
discretisation, solver and computer.
Adaptive mesh refinement, high order and sparse grid methods lead to
discretisations of partial differential equations with a low number of degrees of
freedom. Multilevel iterative solvers decrease the amount of work per degree
of freedom for the solution of discretised equation systems. Massively parallel
computers increase the computational power available for a single simulation.
However, parallel computers require parallel algorithms and special methods
to code them including data distribution and communication, which poses a
severe problem for adaptive mesh refinement. Furthermore multilevel solvers
have to be specifically tailored so that they can be applied to the adaptive
discretisation. Even the efficient implementation of multilevel methods for
sequential and parallel computers poses a severe problem. These aspects will
be covered in detail in the following chapters.
Last but not least, let me thank all who supported the present work in one
way or another. To name but a few, let me begin with my supervisor Prof.
M. Griebel, who supported my research over many years and created a re
search environment which was probably unique at a mathematics department.
6 Preface
Figure 1. Three ingredients of scientific computing: a mathematical model, a
numerical method and the simulation on a computer.
This enabled the combination of ideas from fields as diverse as approximation
theory and molecular dynamics, multilevel methods and high speed network
ing. Furthermore, the leading edge equipment allowed for many projects years
before it became close being mainstream. However, he also contributed the
basic idea of the present work, namely the idea of applying space-filling curve
techniques from astrophysical particle methods to parallel adaptive multigrid
methods, a topic I worked on ten years ago at TU Miinchen then with his
support and supervised by Prof. R. Hoppe.
Of course I have to thank the whole group Scientific Computing and Numer
ical Simulation, members of the Institute for Applied Mathematics and mem
bers of the SFB 256 (Sonderforschungsbereich) Non-linear Partial Differential
Equations. Let me name some of them individually, e.g. M. A. Schweitzer for
the collaboration on the construction of our cluster computing resources and
discussions on multigrid methods and parallelisation in general. The sparse
grid and wavelet parts were influenced by F. Koster and T. Schiekofer, who
calculated some wavelet coefficients for the best approximation results and laid
the algorithmic foundations of the finite difference sparse grid discretisation
respectively. Some research related to space-filling curves was done by M. Eller
brake and G. Spahn, who created the pictures of the tetrahedron meshes. The
calculations on the T3E at Cray Inc. were supervised by M. Arndt. Further
more I want to thank Prof. P. Oswald (Lucent) and Prof. H.-J. Bungartz
(Stuttgart) for useful discussions on sparse grids and space-filling curves re
spectively. P. Anderson, M. Arndt, M. Bader, F. Kiefer and M. A. Schweitzer
did some proof reading. Thanks also to the referees of the original thesis text
Preface 7
and the editors of the book series for their effort of reviewing, their patience,
and their comments. I also wish to express my gratitude to Teubner-Verlag
for their friendly cooperation.
Finally I have to thank SFB 256 at Universitiit Bonn for the financial sup
port, the Institute for Scientific Computing Research (ISCR) and members
of the Center for Applied Scientific Computing (CASe) at Lawrence Liver
more National Laboratory for the opportunity to stay there as a guest and
to access their computer resources, namely the ASCI Blue Pacific computer
and several smaller systems. Furthermore I have to thank Cray Inc. and
NIC (Forschungszentrum Ji1lich) for the computing time on their Cray T3E
systems.
Jena, August 2003 Gerhard Zumbusch
Contents
1 Introduction 11
2 Multilevel Iterative Solvers 19
2.1 Direct and Iterative Solvers . . 20
2.2 Subspace Correction Schemes . 28
2.3 Multigrid and Multilevel Methods 36
2.4 Domain Decomposition Methods 40
2.5 Sparse Grid Solvers ...... 46
3 Adaptively Refined Meshes 59
3.1 The Galerkin Method, Finite Elements and Finite Differences 60
3.2 Error Estimation and Adaptive Mesh Refinement 67
3.3 Data Structures for Adaptively Refined Meshes 75
4 Space-Filling Curves 90
4.1 Definition and Construction 93
4.2 Partitioning................ .110
4.3 Partitions of Adaptively Refined Meshes · 120
4.4 Partitions of Sparse Grids . . . . . . . . · 138
5 Adaptive Parallel Multilevel Methods 144
5.1 Multigrid on Adaptively Refined Meshes · 144
5.2 Parallel Multilevel Methods · 150
5.3 Parallel Adaptive Methods · 160
6 Numerical Applications 169
6.1 Parallel Multigrid for a Poisson Problem · 172
6.2 Parallel Multigrid for Linear Elasticity . · 180
6.3 Parallel Solvers for Sparse Grid Discretisations · 187
Concluding Remarks and Outlook 194
Bibliography 197
Index 215
Chapter 1
Introduction
In a short example we want to illustrate some of the concepts this book is
about. Let us consider the two dimensional Poisson problem as a homogeneous
Dirichlet boundary value problem
-flu = f in 0,
(1.1)
U = 0 on a~,
where flu = a2u/ax2 +a2u/ay2 is the Laplace operator, 0 is a bounded, open
domain like [0,1]2 whose boundary is dentoed by a~. We are looking for a
solution U as a function 0 t-----t R for a given right hand side function f : 0 t-----t R
The finite difference approximation of the problem is based on a discretisation
of U at grid points Xi,j defined by Xi,j = (ih, j h) with 0 ::; i, j ::; N and mesh
size h = l/N. We denote the discretised solution Uh at the grid pionts by
Ui,j = Uh(Xi,j), the right hand side accordingly by Aj = f(Xi,j) and write the
finite difference stencil as
.h.!2.. (4u·t ,). - Ut· +l ,). - Ut· -l ,). - U·t ,)· +1 - Ut·, )· -1) - ft·,·) for 0 < i, j < N,
U· . - 0 else.
t,)
(1.2)
Putting the Taylor expansion of Uh like
Uh(Xi±l,j) = Uh(Xi,j) ± h8~1 Uh(Xi,j) + ~2 ~Uh(Xi,j)
± ~ ~Uh(Xi,j) + ~: ~Uh(()
1 1
with some point ( into (1.2) it turns out that the centered finite difference
stencil does indeed approximate the Laplace operator second order accurate
for sufficiently smooth U and Uh. The conditions (1.2) can be rewritten as an
equation system
(1.3)
G. Zumbusch, Parallel Multilevel Methods
© B. G. Teubner Verlag / GWV Fachverlage GmbH, Wiesbaden 2003
12 1. Introduction
with vectors Uh, fh E jRn, matrix Ah = (ak,l) E jRnxn, and n = (N - 1)2. The
matrix entries with an enumeration of the unknowns by k = i + j (N - 1) are
given by
k = l,
1 { 4 if
akl = - -1 if k -l = 1, -1, N -1, or - N + 1,
, h2 a
else.
The matrix Ah is sparse in the sense that it contains only O(n) non-zero en
tries. However, the solution of the equation system (1.3) by standard Gaussian
elimination requires O(n3) arithmetic operations.
We conclude that a solution of accuracy E = O(h2) = O(N-2) = O(l/n)
requires O(n3) = O(N6) arithmetic operations. This means roughly eight-fold
operations in order to reduce the approximation error be one half. For this
simple example there are several way to improve this ratio of work to accuracy.
First of all, we can use solvers of the equation system (1.3) which exploit
the structure of the matrix Ah or even properties of the solution u. This leads
us to multilevel and multigrid methods which reduce the arithmetic operations
to O(n). Next we can improve the discretisation scheme (1.2). Higher order
schemes usually show higher accuracy than our second order central differ
ences. However, in the presence of non-smooth solutions we do not even get
U
E = O(l/n) and we can turn to adaptive mesh refinement. The goal is to min
imise the number of unknowns n for a given accuracy E and a given problem.
Another concept are sparse grids, where one reduces n for a given accuracy
E
a priori. As a third way to reduce computing time we can use parallel com
puters. The arithmetic operations are distributed to several independently
operating processors such that the time to perform n operations reduces to
values below O(n). Using p processors some parallel algorithms require only
O(n/p + logp + logn) time.
Of course we are interested in a combination of the effects which means
multilevel methods on adaptively refined meshes, parallel multilevel methods,
parallel adaptive mesh refinement, and finally the combination of all three, see
Figure 1.1. We will discuss these aspects in detail in the following chapters.
Finite-Element, Finite-Volume and Finite-Difference methods for the so
lution of partial differential equations are based on meshes. The solution is
represented by degrees of freedom attached to certain locations on the mesh.
Numerical algorithms operate on these degrees of freedom during steps like
the assembly of a linear equation system or the solution of an equation sys
tem. A natural way of porting algorithms to a parallel computer is the data
parallel approach. The mesh with attached degrees of freedom is decomposed
