Table Of ContentAlgorithms,
Software and Hardware
of Parallel Computers
Edited by
J. Mildosko and V. E. Kotov
With Contributions by J. Chudik G. David
V. E. Kotov J. Mildosko N. N. Mirenkov J. Ondas
I. Plander V. A. V alkovskii
With 181 Figures
Springer-Verlag
Berlin Heidelberg GmbH 1984
Jozef Miklosko
Institute of Technical Cybernetics,
Slovak Academy of Sciences
Dtibravska 9
X42 37 Bratislava
Czechoslovakia
Vadim Evgenich Kotov
Computer Centre,
Sibirian Branch of the Academy of Sciences of the USSR
Akad. Lavrentev 6
630 090 Novosibirsk - 90
USSR
ISBN 978-3-662-11108-6 ISBN 978-3-662-11106-2 (eBook)
DOI 10.1007/978-3-662-11106-2
© Springer-Verlag Berlin Heide1berg 1984
Originally published by Springer-Verlag Berlin Heide1berg New York Tokyo in 1984
Softcover reprint of the hardcover 1s t edition 1984
AII righls reserved. No part of this publication may be reproduced. stored in a retrieval system. or
transmitted in any form or by any means. electronics. mechanical photocopying. recording or otherwise.
without the preceding written permission of the publishers.
Contents
Preface 9
1. SYNTHESIS OF PARALLEL NUMERICAL ALGORITHMS (J. Miklosko) . . . . 13
1.1 Parallelism in numerical algorithms . . . . . . . . . . . . . . . . . . 13
1.2 Transformation of an algorithm with unbounded parallelism into an algorithm with
bounded parallelism . . . . . . . . . . . . . . . . . . 19
1.3 Differences between serial and parallel algorithms and their implementation . . . . 22
1.4 Methods for creating parallel algorithms . . . . . . . . . . . . . . . . 27
Appendix: An improved application of the second Buneman's variant for parallel computa-
tion of the semidirect process (M. Vajtersic) . . . . . . . . . . . . . . . 35
2. COMPLEXITY OF PARALLEL ALGORITHMS (J. Miklosko) 45
2.1 Parallel computation of x" . . . . 45
2.2 Bounded and unbounded parallelism 4 7
2.3 Analysis of parallel algorithms 49
2.4 Speedup of parallel algorithms . . 52
Appendix: A parallel algorithm for solving band systems and matrix inversion (L. Hal ada) 58
3. AUTOMATIC CONSTRUCTION OF PARALLEL PROGRAMS (V. E. Kotov
and V. A. Valkovskii) . . . . 65
3.1 Expression transformation 67
3.2 Parallel cycle execution. . 78
3.3 Parallelization of program schemata 91
4. FORMAL MODELS OF PARALLEL COMPUTATIONS (V. E. Kotov) 109
4.1 Program schemes . . . . 109
4.2 Karp-Miller schemes . . . . . 111
4.3 Schemes with limited controls 114
4.4 Subclasses of Karp-Miller schemes 116
4.5 Data-flowschemes . . . . . . 118
4.6 Abstract parallel structures 123
4. 7 Semantic issues of parallel programs 133
5. ON PARALLEL LANGUAGES (V. E. Kotov) 143
5.1 Parallelism in programs and computations 143
5.2 Parallelism of interacting processes . . . 145
5.3 Languages for matrix and vector computers 154
5.4 The problem of basic parallelism . . . . 158
5.5 Asynchronousprogramming . . . . . 159
5.6 Theuniversalfamilyofparallellanguages. 164
6
Appendix: High level of programming language and indeterminancy: SETL (D. Ya. Levin
and A. S. Narinyani) . . . . . . . . . . . . . . . . . . . . 173
6. PROVING CORRECfNESS AND AUTOMATIC SYNTHESIS OF PARALLEL
PROGRAMS (G. David) . . . 179
6.1 Introduction and motivation . . 179
6.2 Structurelogic(SL) . . . . . 181
6.3 Description of parallel architectures 188
6.4 Specification and synthesis of parallel programs . 198
Appendix: Function in SL, the language of architectures (I. Losonczi and A. Sarkozy) 204
7. OPERATING SYSTEMS FOR MODULAR PARTIALLY RECONFIGURABLE
MULTIPROCESSOR-SYSTEMS(N. N.Mirenkov). 209
7.1 Introduction . . . . . . . . . . 209
7.2 Basic notions . . . . . . . . . . . . 211
7.3 Design philosophy and some OS properties . . 212
7.4 Decision-making levels and organizational hierarchy . 216
7.5 Subsystem operational modes . . . . . . . . 218
7.6 General remarks about planning and control algorithms 219
7. 7 Concept of parallel algorithm . . . . 220
7.8 Individual system interaction statements 222
7. 9 Collective system interaction statements 223
8. ALGORITHMS FOR SCHEDULING HOMOGENEOUS MULTIPROCESSOR
COMPUTERS (J. Ondas) 227
8.1 Introduction . . . . . . . . . 227
8.2 Models and basic concepts . . . . . . . . . . . . . . . . . . . . . 228
8.3 A survey of optimal and heuristic algorithms for scheduling homogeneous multiproces-
sor computers . . . . . . . . 230
8.3.1 Non-preemptive scheduling . . . . . . . . 231
8.3 .2 Preemptive scheduling . . . . . . . . . . 233
8.3.3 Heuristic scheduling algorithms and their properties 235
8.4 Algorithms for scheduling resources of special multiprocessor computers 239
8.4.1 Independent, equally long tasks 240
8.4.2 Independent tasks . . 241
8.4.3 Arbitrary task system . . . . 244
8.4.4 Preemptive scheduling . . . 246
Appendix: Uniform computer system: scheduling of jobs with unknown execution time
(V. V. lgnatushchenko) . . . . . . . . . . . . . . . . . . . . . 249
9. ALGORITHMS FOR SCHEDULING INHOMOGENEOUS MULTIPROCESSOR
COMPUTERS (J. Ondas) . . . . . 259
9.1 Introduction . . . . . . . . 259
9.2 Scheduling with minimum schedule duration . 261
9.2.1 Non-preemptive scheduling 261
9.2.2 Preemptivescheduling 265
9.3 Somespecialproblems . . . . . 268
9.3.1 Flow-shopproblem . . . 268
9.3.2 SchedulingwithminimalWMT 269
7
10. PARALLEL PROCESSORS AND MULTICOMPUTER SYSTEMS (I. Plander) 273
10.1 Introduction . . . . . . . . . . . . . . . 273
10.2 Classification of architectures and definition of processors 274
10 .2.1 Sequential SISD processors . . . . . . . 274
10 .2.2 The pipeline MISD processor . . . . . . . 276
10 .2.3 Array and vector processors with SIMD architecture 285
10 .2.4 Associative array processor. 289
10.2.5 Orthogonal array processor 299
10 .2.6 Pipeline array processor . 300
10 .2. 7 MIMD multiprocessors 300
10.3 Centralizedprocessorsystems 306
10.4 Distributed computer systems 306
10.5 Fault-tolerant centralized multiprocessor and distributed multicomputer systems 309
10.6 Switching and permutation networks 312
10.6.1 Omeganetworks . . 315
10 .6.2 Omega network control . . 316
11. DATA FLOW COMPUTER ARCHITECTURE (J. Chudik). 323
11.1 Introduction . . . . . . . . 3 23
11.2 The data flow model of computation 325
11.3 Data flow programming . . . . 331
11.4 Basic models of data flow systems . 332
11.4.1 Data flow architecture of search mode type 333
11.4.2 Data flow architecture of interconnection mode 335
11.5 Some architectures of DF computers . . . . . . 336
11.5.1 Architecture of the data flow computer of Dennis and Misunas 336
11.5.2 Architecture of the data flow Rumbaugh computer . . . . 339
11.5 .3 Architecture of the Arvind and Gostelow D F computer 342
11.5 .4 Architecture of the data flow computer of Gurd and Treleaven 346
11.5.5 Architecture ofthe LAU data flow computer . 349
12. CORRELATION OF ALGORITHMS, SOFTWARE AND HARDWARE OF
PARALLEL COMPUTERS (J. Miklosko) 359
12.1 Parallelcomputers 360
12.2 Arrayprocessors. . . 363
12.3 Pipelineprocessors . . 370
12.4 Multiprocessor computers. 376
12.5 Specialized parallel computers 383
Appendix: Generalized perfect shuffle (0. Sykora) 390
Preface
Both algorithms and the software .and hardware of automatic computers have
gone through a rapid development in the past 35 years. The dominant factor in this
development was the advance in computer technology. Computer parameters were
systematically improved through electron tubes, transistors and integrated circuits
of ever-increasing integration density, which also influenced the development of
new algorithms and programming methods. Some years ago the situation in
computers development was that no additional enhancement of their performance
could be achieved by increasing the speed of their logical elements, due to the
physical barrier of the maximum transfer speed of electric signals. Another
enhancement of computer performance has been achieved by parallelism, which
makes it possible by a suitable organization of n processors to obtain a perform
ance increase of up to n times.
Research into parallel computations has been carried out for several years in
many countries and many results of fundamental importance have been obtained.
Many parallel computers have been designed and their algorithmic and program
ming systems built. Such computers include ILLIAC IV, DAP, STARAN, OMEN,
STAR-100, TEXAS INSTRUMENTS ASC, CRAY-1, C mmp, CM*, CLIP-3,
PEPE.
This trend is supported by the fact that:
a) many algorithms and programs are highly parallel in their structure,
b) the new LSI and VLSI technologies have allowed processors to be combined
into large parallel structures,
c) greater and greater demands for speed and reliability of computers are made.
This research into parallel computations has brought with it new problems both
in the creation of algorithms and programs, and in computer architecture design.
These problems cannot be solved using the methods that were applied in classical
serial computers. Moreover, parallel algorithms and programs are closely as
sociated with the architecture of parallel computers so that their creation, analysis
and implementation cannot be examined isolated from problems of computer
architecture. In this book we examine both parallel algorithms and parallel
programming, and the hardware of parallel computers. To our knowledge, there
has not been published a book that covers the entire set of problems of parallel
computations to this depth so far.
The book has been compiled in cooperation between members of the Institute of
10
Technical Cybernetics of the Slovak Academy of Sciences in Bratislava and the
Computation Centre of the Siberian Branch of the Academy of Sciences of the
USSR in Novosibirsk. In preparing it the editors chose the topics of the chapters so
as to cover as far as possible the problems of parallel computations, and afterwards
they asked some outstanding specialists in the USSR, CSSR and Hungary for their
active participation in this project. This book is the result of their effort.
It consists of 12 chapters. The individual chapters are mostly surveys, though
some include also the results of the authors. Chapters 1, 2, 4, 5, 6, 8 and 12 contain
also short appendices, in which original solutions of some problems described in the
chapter are given.
Each chapter or appendix introduces a special notation and numbering of
formulae and definitions. References are always given at the end of each chapter or
appendix. Although the individual chapters are related and supplement each other,
the reader can study each chapter separately, too.
This are the authors of the single sections of the book: J. Chudik, J. Miklosko,
I. Plander, L. Halada, 0. Sykora, M. Vajtersic, Institute of Technical Cybernetics,
Slovak Academy of Sciences, Bratislava, Czechoslovakia; J. Ondas, Institute of
Control and Computational Technique, Kosice, Czechoslovakia; V. E. Kotov,
V. A. Valkovskii, D. Ya. Levin, A. S. Narinyani, Computer Centre, Siberian
Branch of the Academy of Sciences of the USSR, Novosibirsk, USSR;
N. N. Mirenkov, Mathematical Institute, Siberian Branch of the Academy of
Sciences of the USSR, Novosibirsk, USSR; V. V. lgnatushchenko, Institute of
Control Sciences, Moscow, USSR; and G. David, I. Losonczi, A. Sarkozy, Com
puter and Automation Institute, Hungarian Academy of Sciences, Budapest,
Hungary.
The structure of the book is given by its name : algorithms, software and
hardware of parallel computers. Chapters 1 to 3 are devoted to the creation and
analysis of parallel algorithms and programs. The first chapter (by J. Miklosko)
describes the fundamental principles of creating effective numerical algorithms ; its
appendix (by M. Vajter8ic) describes a new fast parallel algorithm for solving
biharmonic equations in the square. In the second chapter (by J. Miklosko ),
a survey is given of the main results of the theory of complexity of parallel
algorithms. Its appendix (by L. Halada) contains results on the upper bounds of the
number of steps and processors for some parallel algorithms of linear algebry. The
third chapter (by V. E. Kotov and V. A. Valkovskii) is devoted to algorithms and
methods for the automatic generation of parallel programs. Chapters 4 to 6 deal
with the theoretical and practical problems of parallel programming. The fourth
chapter (by V. E. Kotov) gives a survey of the best-known formal models of
parallel computations. Chapter 5 (by V. E. Kotov) examines parallel programming
languages; in its appendix (by D. Ya. Levin and A. S. Narinyani) the facilities of
the high-level language SETL are described. Chapter 6 (by G. David) and its
11
appendix (by I. Losonczi and A. Sarkozy) discusses correctness of parallel pro
grams and their automatic synthesis. Chapters 7 to 9 deal with problems of
operating systems for parallel computers. The seventh chapter (by N. N. Miren
kov) is devoted to operating systems of multiprocessor computers. In Chapter 8 (by
J. Ondas) and in Chapter 9 (by J. Ondas) some optimal and heuristic algorithms
for scheduling devices in homogeneous and inhomogeneous multiprocessor com
puter systems are described. The appendix of Chapter 8 (by V. V. Ignatushchenko)
describes an algorithm for task scheduling with unknown execution time. The last
three chapters (Chapters 10 to 12) examine hardware problems of parallel
computers. Chapter 10 (by I. Plander) contains a survey of various parallel
computers architectures. Chapter 11 (by J. Chudik) describes the architecture of
data-flow computers, and Chapter 12 (by J. Miklosko) discusses the interrelation
ship of algorithm and software with the architecture of the parallel computer. Its
appendix (by 0. Sykora) describes the interprocessor communication network
called the generalized perfect shuffle and its possible applications.
With the demand for faster and more powerful computing resources and plans
for new generations of computers, research into all aspects of parallel computation
is so active that any substantial survey of results is already bound to be outdated
in some respects by the time it is published. Our aim has been to provide a survey
of approaches to the central problems in the area, attempting to emphasize those
contributions which have had, or can be expected to have, a lasting impact upon
its development. To avoid the risk of straying too far from this aim, we chose 1978
as our- admittedly somewhat arbitrary- cut-off date for research eligible for
inclussion. More recent research, in VLSI, parallel algorithm design, distributed
processes, programming languages for concurrent processes, and the theory of
parallelism can be found in an ever-growing number of journals and conference
proceedings. In so far as we have achieved our aim, the book is recommended as
a source and a reference especially to students of computer science, to members
of computation centres, universities and research institutes and to interested spe
cialists and students in the fields of natural, engineering and economic sciences.
In conclusion, we would like to express our acknowledgement to the scientific
editor Prof. Dr. A. Huta, CSc. and the reviewers Assoc. Prof. lng. S. Neuschl, CSc.,
Assoc. Prof. Dr. L. Molnar, CSc. and Dr. J. Sajda, CSc., for their constructive
criticism of the manuscripts; to Dr. J. Nichta and Dr. M. Kasner for translating the
texts from Slovak and Russian into English; to the members of VEDA, Publishing
House of the Slovak Academy of Sciences, for their understanding in solving the
many problems which have arisen in the course of preparing this book, and to the
management of the Institute of Technical Cybernetics of the Slovak Academy of
Sciences in Bratislava as well as to the Computer Centre of the Siberian Branch of
the Academy of Sciences of the USSR in Novosibirsk for having created good
working conditions which have also contributed to the implementation of this
ambitious project.
J. Miklosko, V. E. Kotov
Editors
