Opportunities and Constraints of Parallel Computing

152 Pages·1989·7.47 MB·English

by  Alok Aggarwal (auth.)

Opportunities and Constraints
of Parallel Computing
Jorge L. C. Sanz
Editor
Opportunities and Constraints
of Parallel Computing
Springer-Verlag
New York Berlin Heidelberg
London Paris Tokyo Hong Kong
Jorge L.C. Sanz
Computer Science Department
IBM Almaden Research Center
San Jose, CA 95120-6099
USA
With 7 illustrations
Library of Congress Cataloging-in-Publication Data
Opportunities and constraints of parallel computing / Jorge L.c. Sanz,
editor.
p. cm.
Paper presented at a workshop, Dec. 5-6, 1988, at the IBM Almaden
Research Center, San Jose, Calif., sponsored by the center and the
National Science Foundation.
I. Parallel processing (Electronic computers )-Congresses.
I. Sanz, J.L.c. (Jorge L.c.), 1955- II. Almaden Research
Center (IBM Research) III. National Science Foundation (U.S.)
004' .35-dc20 89-21575
Printed on acid-free paper.
© 1989 by Springer-Verlag New York Inc.
Softcover reprint of the hardcover 1s t edition 1989
All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher
(Springer-Verlag, 175 Fifth Avenue, New York, NY 10010, USA), except for brief excerpts in connection with reviews or scholarly
analysis. Use in connection with any form of information storage and retrieval, electronic adaption, computer software, or by similar
or dissimiliar methodology now known or hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc. in this publication, even if the former are not especially identified,
is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act, may accordingly be
used freely by anyone.
Camera-Ready copy provided by the author.
987654321
ISBN-13: 978-1-4613-9670-3 e-ISBN-13: 978-1-4613-9668-0
DOT: 10.1007/978-1-4613-9668-0
Foreword
At the initiative of the IBM Almaden Research Center and the National Science Foundation, a
workshop on "Opportunities and Constraints of Parallel Computing" was held in San Jose,
California, on December 5-6, 1988. The Steering Committee of the workshop consisted of Prof.
R. Karp (University of California at Berkeley), Prof. L. Snyder (University of Washington at
Seattle), and Dr. J. L. C. Sanz (IBM Almaden Research Center).
This workshop was intended to provide a vehicle for interaction for people in the technical
community actively engaged in research on parallel computing. One major focus of the workshop
was massive parallelism, covering theory and models of computing, algorithm design and
analysis, routing architectures and interconnection networks, languages, and application
requirements. More conventional issues involving the design and use of parallel computers with a
few dozen processors were not addressed at the meeting.
A driving force behind the realization of this workshop was the need for interaction between
theoreticians and practitioners of parallel computation. Therefore, a group of selected participants
from the theory community was invited to attend, together with well-known colleagues actively
involved in parallelism from national laboratories, government agencies, and industry.
The two-day workshop was organized around a series of technical presentations. Each of the two
days was closed with a round table. The topics of discussion were "Theory and Applications",
and "Architectures and Programming". The technical presentations were the source of many
exciting discussions, and the round tables covered a number of controversial issues involving the
practical role of massively parallel algorithms, shared-memory and message-passing models of
parallel computing, routing methodologies for interconnection networks, the characteristics of
numerical computing applications, the need for adequate software, and several others.
Each participant to the workshop was asked to submit a short "position paper". These position
papers cover topics of high interest in parallel computing in an informal manner and with a
possibly controversial character. This volume presents the position papers contributed by the
attendees to the workshop. These short papers are a milestone in the field since it is probably the
first time researchers in both the theory and the practice of parallel computation have gathered
together in a meeting. Some papers have a unique value in describing the state-of-the-art in an area
of massively parallel computing, while others indicate directions for further research in the field. I
hope this material will be a good motivation for students, and of great interest to other colleagues.
I would like to thank my colleagues on the Steering Committee for their participation and help in
organizing the meeting. In addition, I would like to thank the participants of the workshop for
their infinite energy and enthusiastic response. The two extremely demanding days of the
workshop found all our colleagues with endless dedication to accomplish our goals. I would also
like to thank the managers of the IBM Almaden Research Center for their kind support of the
meeting, including the hosting of the event at the Center. Finally, my indebted thanks to the
Workshop Assistant, Mrs. Wendy Clayton, for her many hours of dedication to make the meeting
a real success.
Dr. Jorge L. C. Sanz
Computer Science Department
IBM Almaden Research Center
San Jose, California
Table of Contents
The following papers were presented at the "Opportunities and Constraints of
Parallel Computing" workshop, December 5-6, 1988 at IBM Almaden Research
Center, San Jose, California.
Foreword ............................................................................... v
Workshop Participants ............................................................... IX
Alok Aggarwal ........................................................................ 1
David H. Bailey........................ ................. ............. ................ 5
Alan Baratz and Kevin McAuliffe ................................................. 9
Gianfranco Bilardi .................................................................... 11
Tony F. Chan .......................................................................... 15
K. Mani Chandy ...................................................................... 21
Richard Cole .......................................................................... 25
Robert Cypher ......................................................................... 29
Jack J. Dongarra and Danny C. Sorensen ........................................ 33
Jerome A. Feldman .................................................................. 37
Dennis Gannon ........................................................................ 39
Kourosh Gharachorloo ............................................................... 49
Phillip B. Gibbons ................................................................... 55
Susanne E. Hambrusch .............................................................. 59
Michael T. Heath ..................................................................... 63
John Hennessy . . . . . . . . .. . . . . . . . . . ... . . . . . . . . . ... . . . . . . . . . . .... . . . . . . . . . . . .. . . . . . . . . . . . 67
Leah H. Jamieson .................................................................... 69
Richard M. Karp ..................................................................... 73
Clyde P. Kruskal ...................................................................... 77
Tom Leighton .......................................................................... 81
Bruce Maggs .......................................................................... 83
Gary L. Miller ........................................................................ 85
K. Mohiuddin ......................................................................... 87
John Y. Ngai and Charles L. Seitz ................................................ 89
Rishiyur S. Nikhil .................... ............ ............... ..................... 93
Abhiram Ranade ...................................................................... 97
John H. Reif and Sandeep Sen ..................................................... 101
Arnold L. Rosenberg ................................................................. 107
Jorge L.C. Sanz ....................................................................... 111
Robert B. Schnabel ................................................................... 117
Charles L. Seitz ....................................................................... 119
Alan Siegel ............................................................................ 123
Barbara Simons ....................................................................... 131
Stephen Skedzielewski ............................................................... 135
Burton J. Smith ....................................................................... 137
Marc Snir .............................................................................. 139
Lawrence Snyder ...................................................................... 147
Alan Sussman ......................................................................... 151
L.G. Valiant ........................................................................... 155
Andre M. van Tilborg ............................................................... 159
Uzi Vishkin ............................................................................ 161
Robert G. Voigt ....................................................................... 165
Workshop Participants
Alok Aggarwal Dennis Gannon
IBM T. J. Watson Research Center Indiana University
Department 420/36-245 Computer Science Department
P.O. Box 218 I 01 Lindley Hall
Yorktown Heights, NY 10598 Bloomington, IN 47405-4101
David H. Bailey Kourosh Gharachorloo
NASA Ames Research Center Electrical Engineering Department
Mail Stop 258-5 Stanford University
Moffet Field, CA 94035 Stanford, CA 94305
Alan Baratz Phillip B. Gibbons
IBM T. J. Watson Research Center mM Almaden Research Center
Department 530/H4-D58 Department K53/802
P. O. Box 704 650 Harry Road
Yorktown Heights, NY 10598 San Jose, CA 95120
Gianfranco Bilardi Susanne E. Hambrusch
Cornell University Department of Computer Sciences
Department of Computer Science Purdue University
4130 Upson Hall West Lafayette, IN 47907-0501
Ithaca, NY 14853-7501
Michael T. Heath
Tony F. Chan Oak Ridge National Laboratory
Department of Mathematics P.O. Box 2009
University of California Oak Ridge, TN 37831-8083
Los Angeles, CA 90024
John Hennessy
K. Mani Chandy Center for Integrated Systems
California Institute of Technology Stanford University
256-80 Stanford, CA 94305
Pasadena, CA 91125
Jean-Paul Jacob
Richard Cole mM Almaden Research Center
Courant Institute Department KO 1/802
New York University 650 Harry Road
New York, NY 10012 San Jose, CA 95120
Robert Cypher Leah H. Jamieson
Department of Computer Science, FR-35 Department of Electrical Engineering
University of Washington Purdue University
Seattle, W A 98195 West Lafayette, IN 47907-0501
Jack J. Dongarra Richard M. Karp
Argonne National Laboratory Computer Science Division
9700 S. Cass Avenue University of California
Argonne, IL 60439 Berkeley, CA 94720
Jerome A. Feldman Clyde P. Kruskal
International Computer Science Institute Department of Computer Science
1947 Center Street, Suite 600 University of Maryland
Berkeley, CA 94704 College Park, MD 20742
Tom Leighton Arnold L. Rosenberg
Laboratory for Computer Science Department of Computer & Information Science
Massachusetts Institute of Technology University of Massachusetts
545 Technology Square Amherst, MA 01003
Cambridge, MA 02139
Gerhard Rossbach
Bruce Maggs Springer Verlag
Laboratory for Computer Science 815 De La Vina Street
Massachusetts Institute of Technology SantaBarbara, CA 93101
545 Technology Square
Cambridge, MA 02139 Jorge L. C. Sanz
llM Almaden Research Center
Juri Matisoo Department K53/802
llM Almaden Research Center 650 Harry Road
KOI/802 San Jose, CA 95120
650 Harry Road
San Jose, CA 95120 Robert B. Schnabel
Department of Computer Science
Kevin McAuliffe University of Colorado at Boulder
IBM T. J. Watson Research Center ECOT 7-7 Engineering Center
Department 533/H2-BS2 Campus Box 430
P.O. Box 704 Boulder, CO 80309-0430
Yorktown Heights, NY 10598
Charles L. Seitz
Gary L. Miller Department of Computer Science
Department of Computer Science California Institute of Technology
Carnegie-Mellon University Pasadena, CA 91125
Pittsburgh, PA 15212-3890
Sandeep Sen
K. Moidin Mohiuddin Duke University
llM Almaden Research Center Department of Computer Science
Department KO 1/802 Durham, NC 27706
650 Harry Road
San Jose, CA 95120 Alan Siegel
Courant Institute
John Y. Ngai New York University
California Institute of Technology 251 Mercer Street
Department of Computer Science New York, NY 10012
Pasadena, CA 91125
Barbara Simons
Rishiyur S. Nikhil llM Almaden Research Center
Massachusetts Institute of Technology Department K53/802
Laboratory for Computer Science 650 Harry Road
545 Technology Square San Jose, CA 95120
Cambridge, MA 02139
Abhiram Ranade Stephen Skedzielewski
Division of Computer Science Lawrence Livermore National Laboratories
University of California P.O. Box 808, L-306
Berkeley, CA 94720 Livermore, CA 94550
John H. Reif Burton J. Smith
Department of Computer Science Tera Computer Company
Duke University 400 North 34th Street, Suite 300
North Building Seattle, W A 98103
Durham, NC 27706
lC
Marc Snir IrvTraiger
IBM T. J. Watson Research Center IBM Almaden Research Center
Department 420/36-241 Department K51/802
P. O. Box 218 650 Harry Road
Yorktown Heights, NY 10598 San Jose, CA 95120
Lawrence Snyder L. G. Valiant
Department of Computer Science, FR-35 Harvard University
University of Washingon Aiken Computational Laboratory
Seattle, W A 98195 33 Oxford Street
Cambridge, MA 02138
Danny C. Sorensen
Argonne National Laboratory Uzi Vishkin
9700 S. Cass Avenue Institute for Advanced Computer Studies
Argonne, IL 60439 University of Maryland
College Park, MD 20742-3251
Alan Sussman
Computer Science Department Robert G. Voigt
Carnegie-Mellon University Institute for Computer Applications
Pittsburgh, PA 15213 in Science and Engineering
NASA Langley Research Center
Andre M. van Tilborg Hampton, VA 23665-5225
Office of Naval Research, C- 1133
800 N. Quincy Street Ben Wah
Arlington, VA 22217 - 5000 MIPS Division, Room 414
National Science Foundation
1800 G. Street NW
Washington, D.C. 20050
xi
A Critique of the PRAM Model of Computation
Alok Aggarwal
IBM T. J. Watson Research Center
In theoretical computer science, parallel computation has been traditionally
studied by investigating time, processor, and space complexities of various prob
lems in a model of parallel random access machine called the PRAM model. The
PRAM model assumes that there are some number of processors and a global
memory into which the data is stored. In one step, a processor can read or write
one wore! from the global memory into its registers or it can perform some simple
operations such as adding two numbers or comparing them. Unfortunately, this
model is too simplistic and does not capture some important aspects of parallel
computation that are observed in practice. In particular, real computers are
connected to each other by an interconnection network.; This imposes the fol
lowing constraints that are not taken into account by the PRAM model: com
munication latency and local versus global communication, block transfers,
memory and network conflicts, and the bandwidth of the interconnection net
work. We give below a brief description of these constraints and indicate some
recent work that has been done in order to obtain more realistic models of parallel
computation.
Commu1licatio1l Late1lcy: In most parallel machines that are available today, ac
cess to global memory takes much longer than performing an operation on two
words that are present in the registers of a processor. This difference is partic
ularly acute in message passing systems that have communication latency of
hundreds to thousands of cycles [K87]; for such systems, much of this time is
consumed in software overhead. (In practice, the programmer attempts to mini
mize the effect of communication latency by a judicious algorithm design.)
However, unfortunately, the PRAM model does not account for the communi
cation latency. Recently, some models have been proposed that modify the
PRAM in order to explicitly account for the communication overhead
[PU87,LM88,AC87J. Papadimitriou and Ullman [PU87] and Aggarwal and
Chandra [AC87] account for communication latency by assuming that process
ors have local memory in addition to the global memory and that the communi
cation steps (in which the processors access clements from the global memory) are
different from the computation steps (in which the processors either access ele
ments from the local memory or perform operations on the elements present in
their registers). On the other hand, Leiserson and Maggs [LM77] evaluate the
communication requirements of parallel algorithms by providing a Distributed
Random Access Machine (DRAM) model in which memory accesses are imple
mented by routing messages through a communication network.
Block Transfers: In practice, message transfer time is usually dominated by a
fixed startup overhead that is typically consumed by the software. Consequently,
if s denotes th~ communication latency and c, a proper constant, then as a first
approximation, it can be assumed that a processor takes s + cm time steps to ac
cess a block of m elements that are stored in contiguous locations of the global
memory. Unfortunately, this notion of block transfers is neither taken into ac
count by the PRAM model, nor by the models given by [PU87,LM88,AC88].
Aggarwal, Chandra, and Snir [ACS88] recently gave a parallel model of com
putation that explicitly takes into account the constraints placed by actual block
transfers.
Memory and Network Conflicts During Routing: Since there is no global memory
in reality, whenever two or more processors access the memory of other process
ors simultaneously, they often cause conflicts. These conflicts can either occur
because two or more processors want to access the same location in the memory
of the third processor (this phenomenan can occur even when the processors are
connected to form a complete graph) or these conflicts can occur because two or
more processors want to simultaneously send different messages across the same
link. Although, conflicts are not taken into account by the PRAM model, both
memory conflicts and network conflicts have been studied to a certain extent in
the recent literature [UW86,VW83,Ra87,LMR88].
Bandwith of Interconnection Networks: Since the underlying interconnection net
work is typically not a complete graph but rather a sparse network (such as the
BBN Butterfly network or the Boolean hypercube), congestion usually takes place
when the number of messages across a cut exceeds its bandwidth by a substantial
amount. In the PRAM model, the issue of communication bandwidth does not
arise since memory accesses are assumed to take unit time. The Distributed
Random Access Machine (DRAM) model given by Leiserson and Maggs
[LM88] explicitly models the congestion in the network; however, besides this
paper, not much is known regarding the bandwidth constraint.
In view of the above discussion, it is clear that the PRAM model does
not reflect the reality very well. This obviously raises many questions, for exam
ple, should the PRAM model be modifed so as to incorporate all of the above
(and, may be, more) constraints? The answer to this question certainly depends
upon a person's viewpoint. It would definitely be useful to incorporate all these
constraints but the resulting model may no longer be clean or simple; for example,
such a model is likely to have a lot of parameters associated with it and the
interplay among these parameters may tend to obscure some important aspects
of parallel computation. Consequently, the choice of the right model (or the right
models) remains unresolved to a great extent.
References
[AC88] A. Aggarwal and A.K. Chandra, "Communication complexity of
PRAMs," Proc. of the 15th Int. CoIl. on Automata, Lannguages and Program
ming, 1988, pp. 1-18.
2