Advanced Computer Architecture: Parallelism, Scalability, Programmability

749 Pages·2008·82.2 MB·English

by  Kai Hwang

n»!|lv:Gm\-PHiifiompwvm "
ADVANCED IIDHPIITIEII ARCHITECTURE
Parallelism, Scalability, mgranunamuw
Second Edition
About the Authors
Kai Hwang is a Proiessor ofElectrical Engineering and Computer Science at the University ofSouthern
California. Prior to joining USC, he was a faculty at Purdue University ibr I'D years. He received his
tmdergraduate education at the National Taiwan University in China and eamed his PhD degree from
the University of California at Berkeley. Dr Hwang has been engaged in research and teaching on
computer architecture, parallel processing and network-based computing for well over 3-D years. He
has authored or coauthored live books and 120 journal and conference papers in the Computer Science
and Engineering areas. He is the founding Cocditor-in-Chiefof the .J'oumrn' ofP.nr.r1Hef rmd Distributed
Conn-mring.
He has served as the founding Director of the USC Computer Research Institute and a Distinguished
Visitor of the IEEE Computer Society. He has chaired several international computer conferences and
lecturedworldwideonadvancedcomputcrtopics. His researcheshave been supportedby NSF,IBM,AT&T,
AFOSR, ONR, DOT,Alliant, and Intel. Hehasbeenaconsultant ibrIBM, .IPL, Fujitsu,Japan's ETL,GMD
i11Germany,and ITRIandAcademia Sinica inChi11a. He is alsoamemberofthcadvisoryboards ofseweral
internationaljournalsandresearchorganizations.
The Institute of Electrical and Electronics Engineers elected him as an IEEE Fellow in 1986 for his
contributions in computerarchitectures, digitalarithmetic, andparallelprocessing. Hewas the holderofthe
DistinguishedCDCVisiting Chair Professorship inComputer ScienceattheUniversityofMinnesotaduring
the spring quarterof I939. He has guided overa dozen PhD students at Purdue and USC. At present, he
headsa sponsoredresearchprojectonGrid Security at USC. Hiscurrentresearch interestsarei11 theareasof
network-basedcomputing, Intcmetsecurity,andclusteredsystems.Clvertheyears,hehasreceivednumerous
awards ibroutstanding teachingand research, anddelivered invitedand keynote lectures inmanycountries.
Naresh Jotwani is presently serving as Director, School of Solar Energy, Pandit Doendayal Petroleum
University,Gandhinagar. Earlier, hehasservedas Professorand Dean('R&D)atDA-IICT,Gandhinagar,and
as Principal atG H PatelCollegeofEngineeringandTechnology, VallabhVidyanagar.
Dr Jotwani obtained his BTech degree in Electrical Engineering from IIT Bombay, a.nd Doctorate in
Computer Science from Rice University, Houston. His teaching career has spalmed overtwenty-five years,
in India, Singapore and the US. He has also worked in the IT indusuy for about five years, in India and
Singapore,with briefstints inthe US. Intheearly I9BD‘s, heworkedon system software development fora
64-bit multiprocessorsystemwith microcoded instructionsforinter-processcommunication.
Dr Jotwani has carried out several consultancy assignments, written four books and several research
publications, and delivered numerous invited lecttu'es. His textbook Computer .5:1-‘.S‘I£"J'fl Organisation was
publishedbyTataMefiraw-Hill. Hiscurrentresearch interestsare inthefieldofsolarphotovoltaicdevices.
1'?»iiilrfimu-HriiCcinoflilri l _.
ADVANCED CQMPUTER I-\RC|Il'l'|§CTIIR|i
Parallelism, Scalability, Prugrainmaliilitii
Second Edition
Kai Hwang
ProfessorofEiectricai Engineeringand ComputerScience
UniversityofSoutiiem Caiifomia, USA
Naresh Jotwani
Director, Schooi ofSoiarEnergy
PanciitDee-ndayaiF-‘etroietrrn University
Ganizininagar, Giiiarat
Tata McC-iraw Hill Education Private Limited
NEW DELHI
M‘cGr¢aW-Hiii Offices
New Delhi New York St Louis San Francisco Auckland Bogota Caracas
Koala Lumpur l_lS-IJDFI London II-ibdrid Meiiuoofiity Milan Montreal
San Juan Santiago Singapore Sydney Tokyo Toronto
Fr-1-Mrfiraw HJ'lft'=>-rm.--im-. _
It 1
'Tata llllcfiraw-Hill
PublishedbytheTata l'vIcCraw llill EducationPrivate Limited__
Twest Patel Hagar, New Delihi IIDDDB.
Advanced ComputerArchltecltlre,2e
Copyright-E1‘.10!I.2000,byTataMtflraw llillEducation PrivateLimited.
No part ofthis publication may be reproducedor distributed in any form or by any means, electronic, mechanicai,
photocopying, recording,or otherwiseorstoredinadatabase orretrievalsystemwithoutthepriorwrittenpermissionof
thepublishers.‘Theprogramlistings(ifany)maybeentered,storedandexecutedinacomputersystem, ltutthey maynot
bereproducedforpublication.
This editioncanbeexportedfromIndiaonlyby thepublishers,
TataMcflmwIlillEducationPrivateLimited.
L?BN{I3digit]-L9TB-DJTTJTTUZID-3
ISBN{IOdigit}:D-D?-DTDZID-I
Vice PresidentandManagingDirector hrlcfiraw-HillEducation:Asia Pacific Region:.=t_;'a_v.'H1';tr.i\'.ft.r
liead lliglterEducationPublishingandMarketing: Pibha.'Hahq,iun
Manager:Sponsoring SEM&TechEd: She-JiniJim
Asst Sponsoring Editor. SumbhiShulda
DeveboprnentEditor:Surfi-hiStrman
Executive EditorialServices:.S'ohlniMulcire-q'ee
JrManager-- Production: Aqjufiflurdun
DyMarketingManager:SEM&TechEd: EliaGa-nesan
GeneralManager Production:RqjendcrPG.h'an.te'fa
Asst General Manager Production:BLDogrsr
Information contained in thiswork has been obtained by Tata McGraw-Hill, front sources beiieved to be reliable.
Howeter, neitherTataMcGraw-I-lill noritsauthorsguarantee theaccuracy orcompletenessofauty informationpub-
Eishedherein,andneitherTataMcfiraw-Hillnoritsauthorsshallberesponsibleforanyerrors,omissions,ordamages
arising outofuseofthis information. Thisworkispublishedwith the understandingthat TataMctlraw-llilland its
authorsaresupplying informationbut arenotattemptingtorenderengineeringorotherprofessional services. Ifsuch
servicesare required,theassistanceofanappropriateprofemionaishouldbesought.
Typeset atTejComposers, W2 3‘£I'l, Madipur,New Deflti IIDU63and printedatPashupati Printers Pvt Ltd., l»'42'J'.!l-Ei,
GaliNo. I,FriendsColony, IndustrialArea,GIT. Road,Shahdara_ Deihi IIDUQS
CoverPrinter:SDRPrinters
RYACRRQZDRXAA
TheMCG."flW'Hl'"Companies __ :
Fr-1-Mrfirow HJ'iic'|--r.-n.---in-~ _
This hook is dedicatedto those who areeager to learn in a rapidiy changing woriafl
to those who teachandshareknowiedgewithoutdiscrimination, andto thosewho are
determinedto moire acontribution through creative work.
Kai Hwang
Dedicatedto G, R, I/andD.
Naresh Jotwani
Contents
ForewordtotheFirstEdition .1"v
Ptetitceto the.S'eco.ndEdition .1"vii
Prefaceto theFirstEdition xtiii
Part I Theory ofParallelism 1
3
1. Parallel ComputerModels
1.1 The State oflfomputittg 3
1.1.1 ComputerDevelopment Milestones 3
1.1.2 Elementsoflvlo-dern Computers 6
1.1.3 EvolutionofC'omputcr.-“architecture 8
1.1.4 SystemAttributes to Performance iE
1.2 Multiptoeessotsand Multicomputcrs II-'
1.2.1 Shared-MemoryMultiprocessors H’
1.2.2 Distributed-Memory Multicomputers
l'~.a1'»:
1.2.3 ATaxonomyofMIMDComputers 24
1.3 Multiveclorand SIMD Computers E5
1.3.1 Vectorflupcrcomputers 25
1.3.2 STMDSupercomputers 2?
1.4 PRAM and VLSI Models 29
1.4.1 Parallel Random-Access Machines 3|‘?
1.4.2 VLSI Complexity Model 33
1.5 Architectural DevelopmentTracks 36
1.5.1 Multiple-ProcessorTracks 36
1.5.2 Multivectorand SIMDTracks 38
1.5.3 MultithrcadcdandDatafiowTracks 39
.Sitnuti.ort-‘ 4|’)
Exercises 4!
FhrrulffiffllliH“Pl'r>¢rIq|r_.\.I|n*\ ‘I _
viii i Contems
2. Program andNetwork Properties 44
2.1 ConditionsofParallelism 44
2.1.1 DataandResourceDcpcndenccs 44
2.1.2 HardwareandSoflwzwcParallelism 49
2.1.3 TheRoleofCompilers 52
2.2 ProgramPartitioningandScheduling 52
2.2.1 Grain SizesandLatency 52
2.2.2 Grain PackingandScheduling 55
2.2.3 Static MultiprocessorScheduling 38
2.3 ProgramFlowMechanisms 6!
2.3.1 Control Flow Versus DataFlow fit
2.3.2 Demand-Driven Mechanisms 65
2.3.3 Comparison ofFlow Mechanisms 65
2.4 System InterconnectArchitectures 66
2.4.1 Network Properties andRouting 6F’
2.4.2 Static ConnectionNetworks ?r'?
2.4.3 Dynamic ConnectionNetworks ??
Srrrrimrtrr-' 83
E.t'ercises 84
3. PrinciplesofScalable Performance 89
3.1 Performance Metrics and Measures 89'
3.1.1 ParallelismProfileinPrograms 39
3.1.2 Mean Performance 92
3.1.3 Efiiciency.Utilization, andQuality 93
3.1.4 Benchmarksand Performance Measures 9?
3.2 ParallelProcessingApplications 99
3.2.1 MassiveParallelism forGrandChallenges 99
3.2.2 ApplicationModelsofParallel Computers I11?.’
3.2.3 ScalabilityofParallelttlgorithms i04
3.3 Speedup Performance Laws 108
3.3.1 Amr:lahl'sLaw fora Fixed Workload IDS
3.3.2 Cn.rstafson‘s LawforScaledProblems Hi
3.3.3 Memory-Bounded Speedup Model HI.’
3.4 ScalabilityAnalysisandApproaches H6
3.4.l Scalability Metrics andGoals H6
3.4.2 EvolutionofScalableComputers I20
3.4.3 Research Issuesand Solutions IE3
SIt.l'fl.l'flflt'__1-' i.?.'i
E.\'ercises i25
Thu‘Ml.'I;Ifllb'HI"l'n¢r.q|r_.u|»rs -
CD’-“E,-"3 —
Part H Hardware Technologies
4. Processors and Memory Ilierarchy
4.1 Advanced ProcessorTechnology I33
4.1.1 Design SpaceofProcessors 1'33
4.1.2 Instruction-SetArehitectures I3?
4.1.3 CISC ScalarProcessors 139
4.1.4 RISC ScalarProcessors 143
4.2 SuperscalarandVectorProcessors I50
4.2.1 SuperscalarProcessors I50
4.2.2 TheVLIWArchitecture 154’
4.2.3 Vectorand Symbolic Processors 156
4.3 Memory HierarchyTechnology 1'60
4.3.1 Hierarchical MemoryTechnology 161'?
4.3.2 Inclusion,Coherence,andLocality rs:
4.3.3 Memory Capacity Planning I65
4.4 Virtual Memory Technology .16?
4.4.1 Virtual Memory Models I6?
4.4.2 TLB, Paging, and Segmentation I69
4.4.3 Memory ReplacementPolicies II-74
Smrmmrg-' II’?
Exercises I5'3
5. Bus. Cache. and SharedMemory
5.1 Bus Systems I82‘
5.1.1 BackplaneBusSpecification I82
5.1.2 AddressingandTimingProtocols I84
5.1.3 Arbitration, Transaction,and Interrupt 186
5.1.4 IEEEFutureb|.|.s+andother Standards I89
5.2 CacheMemoryOrganizations I92
5.2.1 CacheAddressingModels I93
5.2.2 DirectMappingandAssociativeCaches 195
5.2.3 Set-AssociativeandSectorCaches I98
5.2.4 CachePerformance Issues 202
5.3 Shared-MemoryOrganizations 205
5.3.1 Interleaved Memory Organization 205
5.3.2 BandwidthandFaultTolerance 208
5.3.3 Memory Allocation Schemes EH?
5.4 Sequential and‘WeakConsistency Models 213
5.4.1 Atomicity and EventOrdering 213
5.4.2 Sequential ConsistencyModel El3-‘
Fr‘:-rMelirowrrrtrr-...¢-,.,..t.¢. '
x — _ Cements
5.4.3 Weak Consistency Models 213
Surrrmrrrt-' 3.?!
E.rr3'er'.s'es' 33.?
6. Pip-eliningand Sup-erscalarTechniques 22'?
6.1 Linear Pipeline Processors 22?
6.1.1 Asynchronousand Synchronous Models 2.73?
6.1.2 ClockingandTimingControl 329
6.1.3 Speedup,Efficienc-y,andThroughput 229
6.2 NonlinearPipelineProcessors 232
6.2.1 Reservationand LatencyAnalysis 232
6.2.2 Collision-FreeScheduling 235
6.2.3 PipelineScheduleDptimization 23?
6.3 Instruction Pipeline Design 240
6.3.1 Instruction Execution Phases .240
6.3.2 Men-hanistns forInstruction Pipelining E43
15.3.3 Dynamic instruction Scheduling 24F
15.3.4 Branch HandlingTechniques 351?
6.4 Arithmetic Pipeline Design 255
6.4.1 ComputerAritlurtetic Principles E55
6.4.2 StaticArithmetic Pipelines 25F
15.4.3 MultiiirnctionalArithmetic Pipelines 363
6.5 SuperscalarPipeline Design J66
Srrrmrrrrry EF3
Exerer‘ses J2'4
Part ITI Parallel and Scalable Architectures 279
zsr
7. Mulfiprocessors and llrlulticomputers
T.1 MultiprocessorSystem Interconnects 281'
'1".1.1 Hierarchical BusSystems .7382
7.1.2 CrossbarSwitchandMultiportMemory 236
7.1.3 MultistageandCombining Networks 391'?
12 CacheCoherenceand Synchronimtion Mechanisms 396
7.2.1 TheCacheCoherenceProblem .396
7.2.2 Snoopy Bus Protocols 299
7.2.3 Directory-BasedProtocols 303
7.2.4 HardwareSynchronization Mechanisms 308
'F.3 ThreeGenerations ofMulticomputers 31.’
'i".3.1 Design Choices inthePast 3.1.7.?
,,,,,,,,,,, rs.-or o rtm- ;- _- 3 ._,
xi
2.3.2 Present andFutureDevelopment 314
7.3.3 Thelntel Paragon System 316
'l'.4 Message-Passing Mechanisms 313
']".4.l Message-Routing Schemes 319
T.-4.2 DeadlockVirtualChannels 32.?
11.4.3 FlowControl Strategies 324
7.4.4 Multicast RoutingAlgorithrrts 329
Srrrrrrrrort-‘ 334
Exercises 335
8. Multivector and SIMDComputers 341
8.l Vector Processing Principles 341’
8.1.l Vectorlttstructiorn Types 341'
8.1.2 Vector-AccessMemory Schemes 345
8.1.3 Early Supercomputers 34?
8.2 MultivectorMultiproeessors 352
8.2.1 Perforrnance-DirectedDesign Rules 35.?
8.2.2 CrayY-MP; C-90, andMPP 356
8.2.3 FujitsuVPZGUDandVPPSUD 36.73
8.2.4 Mainframesand Minisupercornputers 365
8.3 CompoundVectorProcessing 3Fl’
8.3.l CompoundVectorOperations 3F2
8.3.2 VectorLoops andChaining _i?4
8.3.3 MultipipelineNetworking 3F3
8.4 SIMD ComputerOrganizations 332
8.4.l lmplementation Models 333
8.4.2 TheCM-2Architecture 385
8.4.3 TheMasPar MP-1 Architecture 333
8.5 The Connection MachineCM-5 39.?
8.5.1 ASynchronized MIMD Machine 39.?
8.5.2 TheCM-5 NetworkArchitecture 395
8.5.3 Control Processors and Processing Nodes 39?
8.5.4 lntetprocessorCommunications 399
Surrrmort-' 403
E.t'r:r'er’ses 404
9. Scalable,Multithreaded, and DataflowArchitectures 408
9.l Latency-HidingTechniques 403
9.1.1 Shared Virtual Memory 4173
9.1.2 PrefetchingTechniques 412
9.1.3 DistributedCoherentCaches 413