Table Of Content

n»!|lv:Gm\-PHiifiompwvm " ADVANCED IIDHPIITIEII ARCHITECTURE Parallelism, Scalability, mgranunamuw Second Edition About the Authors Kai Hwang is a Proiessor ofElectrical Engineering and Computer Science at the University ofSouthern California. Prior to joining USC, he was a faculty at Purdue University ibr I'D years. He received his tmdergraduate education at the National Taiwan University in China and eamed his PhD degree from the University of California at Berkeley. Dr Hwang has been engaged in research and teaching on computer architecture, parallel processing and network-based computing for well over 3-D years. He has authored or coauthored live books and 120 journal and conference papers in the Computer Science and Engineering areas. He is the founding Cocditor-in-Chiefof the .J'oumrn' ofP.nr.r1Hef rmd Distributed Conn-mring. He has served as the founding Director of the USC Computer Research Institute and a Distinguished Visitor of the IEEE Computer Society. He has chaired several international computer conferences and lecturedworldwideonadvancedcomputcrtopics. His researcheshave been supportedby NSF,IBM,AT&T, AFOSR, ONR, DOT,Alliant, and Intel. Hehasbeenaconsultant ibrIBM, .IPL, Fujitsu,Japan's ETL,GMD i11Germany,and ITRIandAcademia Sinica inChi11a. He is alsoamemberofthcadvisoryboards ofseweral internationaljournalsandresearchorganizations. The Institute of Electrical and Electronics Engineers elected him as an IEEE Fellow in 1986 for his contributions in computerarchitectures, digitalarithmetic, andparallelprocessing. Hewas the holderofthe DistinguishedCDCVisiting Chair Professorship inComputer ScienceattheUniversityofMinnesotaduring the spring quarterof I939. He has guided overa dozen PhD students at Purdue and USC. At present, he headsa sponsoredresearchprojectonGrid Security at USC. Hiscurrentresearch interestsarei11 theareasof network-basedcomputing, Intcmetsecurity,andclusteredsystems.Clvertheyears,hehasreceivednumerous awards ibroutstanding teachingand research, anddelivered invitedand keynote lectures inmanycountries. Naresh Jotwani is presently serving as Director, School of Solar Energy, Pandit Doendayal Petroleum University,Gandhinagar. Earlier, hehasservedas Professorand Dean('R&D)atDA-IICT,Gandhinagar,and as Principal atG H PatelCollegeofEngineeringandTechnology, VallabhVidyanagar. Dr Jotwani obtained his BTech degree in Electrical Engineering from IIT Bombay, a.nd Doctorate in Computer Science from Rice University, Houston. His teaching career has spalmed overtwenty-five years, in India, Singapore and the US. He has also worked in the IT indusuy for about five years, in India and Singapore,with briefstints inthe US. Intheearly I9BD‘s, heworkedon system software development fora 64-bit multiprocessorsystemwith microcoded instructionsforinter-processcommunication. Dr Jotwani has carried out several consultancy assignments, written four books and several research publications, and delivered numerous invited lecttu'es. His textbook Computer .5:1-‘.S‘I£"J'fl Organisation was publishedbyTataMefiraw-Hill. Hiscurrentresearch interestsare inthefieldofsolarphotovoltaicdevices. 1'?»iiilrfimu-HriiCcinoflilri l _. ADVANCED CQMPUTER I-\RC|Il'l'|§CTIIR|i Parallelism, Scalability, Prugrainmaliilitii Second Edition Kai Hwang ProfessorofEiectricai Engineeringand ComputerScience UniversityofSoutiiem Caiifomia, USA Naresh Jotwani Director, Schooi ofSoiarEnergy PanciitDee-ndayaiF-‘etroietrrn University Ganizininagar, Giiiarat Tata McC-iraw Hill Education Private Limited NEW DELHI M‘cGr¢aW-Hiii Offices New Delhi New York St Louis San Francisco Auckland Bogota Caracas Koala Lumpur l_lS-IJDFI London II-ibdrid Meiiuoofiity Milan Montreal San Juan Santiago Singapore Sydney Tokyo Toronto Fr-1-Mrfiraw HJ'lft'=>-rm.--im-. _ It 1 'Tata llllcfiraw-Hill PublishedbytheTata l'vIcCraw llill EducationPrivate Limited__ Twest Patel Hagar, New Delihi IIDDDB. Advanced ComputerArchltecltlre,2e Copyright-E1‘.10!I.2000,byTataMtflraw llillEducation PrivateLimited. No part ofthis publication may be reproducedor distributed in any form or by any means, electronic, mechanicai, photocopying, recording,or otherwiseorstoredinadatabase orretrievalsystemwithoutthepriorwrittenpermissionof thepublishers.‘Theprogramlistings(ifany)maybeentered,storedandexecutedinacomputersystem, ltutthey maynot bereproducedforpublication. This editioncanbeexportedfromIndiaonlyby thepublishers, TataMcflmwIlillEducationPrivateLimited. L?BN{I3digit]-L9TB-DJTTJTTUZID-3 ISBN{IOdigit}:D-D?-DTDZID-I Vice PresidentandManagingDirector hrlcfiraw-HillEducation:Asia Pacific Region:.=t_;'a_v.'H1';tr.i\'.ft.r liead lliglterEducationPublishingandMarketing: Pibha.'Hahq,iun Manager:Sponsoring SEM&TechEd: She-JiniJim Asst Sponsoring Editor. SumbhiShulda DeveboprnentEditor:Surfi-hiStrman Executive EditorialServices:.S'ohlniMulcire-q'ee JrManager-- Production: Aqjufiflurdun DyMarketingManager:SEM&TechEd: EliaGa-nesan GeneralManager Production:RqjendcrPG.h'an.te'fa Asst General Manager Production:BLDogrsr Information contained in thiswork has been obtained by Tata McGraw-Hill, front sources beiieved to be reliable. Howeter, neitherTataMcGraw-I-lill noritsauthorsguarantee theaccuracy orcompletenessofauty informationpub- Eishedherein,andneitherTataMcfiraw-Hillnoritsauthorsshallberesponsibleforanyerrors,omissions,ordamages arising outofuseofthis information. Thisworkispublishedwith the understandingthat TataMctlraw-llilland its authorsaresupplying informationbut arenotattemptingtorenderengineeringorotherprofessional services. Ifsuch servicesare required,theassistanceofanappropriateprofemionaishouldbesought. Typeset atTejComposers, W2 3‘£I'l, Madipur,New Deflti IIDU63and printedatPashupati Printers Pvt Ltd., l»'42'J'.!l-Ei, GaliNo. I,FriendsColony, IndustrialArea,GIT. Road,Shahdara_ Deihi IIDUQS CoverPrinter:SDRPrinters RYACRRQZDRXAA TheMCG."flW'Hl'"Companies __ : Fr-1-Mrfirow HJ'iic'|--r.-n.---in-~ _ This hook is dedicatedto those who areeager to learn in a rapidiy changing woriafl to those who teachandshareknowiedgewithoutdiscrimination, andto thosewho are determinedto moire acontribution through creative work. Kai Hwang Dedicatedto G, R, I/andD. Naresh Jotwani Contents ForewordtotheFirstEdition .1"v Ptetitceto the.S'eco.ndEdition .1"vii Prefaceto theFirstEdition xtiii Part I Theory ofParallelism 1 3 1. Parallel ComputerModels 1.1 The State oflfomputittg 3 1.1.1 ComputerDevelopment Milestones 3 1.1.2 Elementsoflvlo-dern Computers 6 1.1.3 EvolutionofC'omputcr.-“architecture 8 1.1.4 SystemAttributes to Performance iE 1.2 Multiptoeessotsand Multicomputcrs II-' 1.2.1 Shared-MemoryMultiprocessors H’ 1.2.2 Distributed-Memory Multicomputers l'~.a1'»: 1.2.3 ATaxonomyofMIMDComputers 24 1.3 Multiveclorand SIMD Computers E5 1.3.1 Vectorflupcrcomputers 25 1.3.2 STMDSupercomputers 2? 1.4 PRAM and VLSI Models 29 1.4.1 Parallel Random-Access Machines 3|‘? 1.4.2 VLSI Complexity Model 33 1.5 Architectural DevelopmentTracks 36 1.5.1 Multiple-ProcessorTracks 36 1.5.2 Multivectorand SIMDTracks 38 1.5.3 MultithrcadcdandDatafiowTracks 39 .Sitnuti.ort-‘ 4|’) Exercises 4! FhrrulffiffllliH“Pl'r>¢rIq|r_.\.I|n*\ ‘I _ viii i Contems 2. Program andNetwork Properties 44 2.1 ConditionsofParallelism 44 2.1.1 DataandResourceDcpcndenccs 44 2.1.2 HardwareandSoflwzwcParallelism 49 2.1.3 TheRoleofCompilers 52 2.2 ProgramPartitioningandScheduling 52 2.2.1 Grain SizesandLatency 52 2.2.2 Grain PackingandScheduling 55 2.2.3 Static MultiprocessorScheduling 38 2.3 ProgramFlowMechanisms 6! 2.3.1 Control Flow Versus DataFlow fit 2.3.2 Demand-Driven Mechanisms 65 2.3.3 Comparison ofFlow Mechanisms 65 2.4 System InterconnectArchitectures 66 2.4.1 Network Properties andRouting 6F’ 2.4.2 Static ConnectionNetworks ?r'? 2.4.3 Dynamic ConnectionNetworks ?? Srrrrimrtrr-' 83 E.t'ercises 84 3. PrinciplesofScalable Performance 89 3.1 Performance Metrics and Measures 89' 3.1.1 ParallelismProfileinPrograms 39 3.1.2 Mean Performance 92 3.1.3 Efiiciency.Utilization, andQuality 93 3.1.4 Benchmarksand Performance Measures 9? 3.2 ParallelProcessingApplications 99 3.2.1 MassiveParallelism forGrandChallenges 99 3.2.2 ApplicationModelsofParallel Computers I11?.’ 3.2.3 ScalabilityofParallelttlgorithms i04 3.3 Speedup Performance Laws 108 3.3.1 Amr:lahl'sLaw fora Fixed Workload IDS 3.3.2 Cn.rstafson‘s LawforScaledProblems Hi 3.3.3 Memory-Bounded Speedup Model HI.’ 3.4 ScalabilityAnalysisandApproaches H6 3.4.l Scalability Metrics andGoals H6 3.4.2 EvolutionofScalableComputers I20 3.4.3 Research Issuesand Solutions IE3 SIt.l'fl.l'flflt'__1-' i.?.'i E.\'ercises i25 Thu‘Ml.'I;Ifllb'HI"l'n¢r.q|r_.u|»rs - CD’-“E,-"3 — Part H Hardware Technologies 4. Processors and Memory Ilierarchy 4.1 Advanced ProcessorTechnology I33 4.1.1 Design SpaceofProcessors 1'33 4.1.2 Instruction-SetArehitectures I3? 4.1.3 CISC ScalarProcessors 139 4.1.4 RISC ScalarProcessors 143 4.2 SuperscalarandVectorProcessors I50 4.2.1 SuperscalarProcessors I50 4.2.2 TheVLIWArchitecture 154’ 4.2.3 Vectorand Symbolic Processors 156 4.3 Memory HierarchyTechnology 1'60 4.3.1 Hierarchical MemoryTechnology 161'? 4.3.2 Inclusion,Coherence,andLocality rs: 4.3.3 Memory Capacity Planning I65 4.4 Virtual Memory Technology .16? 4.4.1 Virtual Memory Models I6? 4.4.2 TLB, Paging, and Segmentation I69 4.4.3 Memory ReplacementPolicies II-74 Smrmmrg-' II’? Exercises I5'3 5. Bus. Cache. and SharedMemory 5.1 Bus Systems I82‘ 5.1.1 BackplaneBusSpecification I82 5.1.2 AddressingandTimingProtocols I84 5.1.3 Arbitration, Transaction,and Interrupt 186 5.1.4 IEEEFutureb|.|.s+andother Standards I89 5.2 CacheMemoryOrganizations I92 5.2.1 CacheAddressingModels I93 5.2.2 DirectMappingandAssociativeCaches 195 5.2.3 Set-AssociativeandSectorCaches I98 5.2.4 CachePerformance Issues 202 5.3 Shared-MemoryOrganizations 205 5.3.1 Interleaved Memory Organization 205 5.3.2 BandwidthandFaultTolerance 208 5.3.3 Memory Allocation Schemes EH? 5.4 Sequential and‘WeakConsistency Models 213 5.4.1 Atomicity and EventOrdering 213 5.4.2 Sequential ConsistencyModel El3-‘ Fr‘:-rMelirowrrrtrr-...¢-,.,..t.¢. ' x — _ Cements 5.4.3 Weak Consistency Models 213 Surrrmrrrt-' 3.?! E.rr3'er'.s'es' 33.? 6. Pip-eliningand Sup-erscalarTechniques 22'? 6.1 Linear Pipeline Processors 22? 6.1.1 Asynchronousand Synchronous Models 2.73? 6.1.2 ClockingandTimingControl 329 6.1.3 Speedup,Efficienc-y,andThroughput 229 6.2 NonlinearPipelineProcessors 232 6.2.1 Reservationand LatencyAnalysis 232 6.2.2 Collision-FreeScheduling 235 6.2.3 PipelineScheduleDptimization 23? 6.3 Instruction Pipeline Design 240 6.3.1 Instruction Execution Phases .240 6.3.2 Men-hanistns forInstruction Pipelining E43 15.3.3 Dynamic instruction Scheduling 24F 15.3.4 Branch HandlingTechniques 351? 6.4 Arithmetic Pipeline Design 255 6.4.1 ComputerAritlurtetic Principles E55 6.4.2 StaticArithmetic Pipelines 25F 15.4.3 MultiiirnctionalArithmetic Pipelines 363 6.5 SuperscalarPipeline Design J66 Srrrmrrrrry EF3 Exerer‘ses J2'4 Part ITI Parallel and Scalable Architectures 279 zsr 7. Mulfiprocessors and llrlulticomputers T.1 MultiprocessorSystem Interconnects 281' '1".1.1 Hierarchical BusSystems .7382 7.1.2 CrossbarSwitchandMultiportMemory 236 7.1.3 MultistageandCombining Networks 391'? 12 CacheCoherenceand Synchronimtion Mechanisms 396 7.2.1 TheCacheCoherenceProblem .396 7.2.2 Snoopy Bus Protocols 299 7.2.3 Directory-BasedProtocols 303 7.2.4 HardwareSynchronization Mechanisms 308 'F.3 ThreeGenerations ofMulticomputers 31.’ 'i".3.1 Design Choices inthePast 3.1.7.? ,,,,,,,,,,, rs.-or o rtm- ;- _- 3 ._, xi 2.3.2 Present andFutureDevelopment 314 7.3.3 Thelntel Paragon System 316 'l'.4 Message-Passing Mechanisms 313 ']".4.l Message-Routing Schemes 319 T.-4.2 DeadlockVirtualChannels 32.? 11.4.3 FlowControl Strategies 324 7.4.4 Multicast RoutingAlgorithrrts 329 Srrrrrrrrort-‘ 334 Exercises 335 8. Multivector and SIMDComputers 341 8.l Vector Processing Principles 341’ 8.1.l Vectorlttstructiorn Types 341' 8.1.2 Vector-AccessMemory Schemes 345 8.1.3 Early Supercomputers 34? 8.2 MultivectorMultiproeessors 352 8.2.1 Perforrnance-DirectedDesign Rules 35.? 8.2.2 CrayY-MP; C-90, andMPP 356 8.2.3 FujitsuVPZGUDandVPPSUD 36.73 8.2.4 Mainframesand Minisupercornputers 365 8.3 CompoundVectorProcessing 3Fl’ 8.3.l CompoundVectorOperations 3F2 8.3.2 VectorLoops andChaining _i?4 8.3.3 MultipipelineNetworking 3F3 8.4 SIMD ComputerOrganizations 332 8.4.l lmplementation Models 333 8.4.2 TheCM-2Architecture 385 8.4.3 TheMasPar MP-1 Architecture 333 8.5 The Connection MachineCM-5 39.? 8.5.1 ASynchronized MIMD Machine 39.? 8.5.2 TheCM-5 NetworkArchitecture 395 8.5.3 Control Processors and Processing Nodes 39? 8.5.4 lntetprocessorCommunications 399 Surrrmort-' 403 E.t'r:r'er’ses 404 9. Scalable,Multithreaded, and DataflowArchitectures 408 9.l Latency-HidingTechniques 403 9.1.1 Shared Virtual Memory 4173 9.1.2 PrefetchingTechniques 412 9.1.3 DistributedCoherentCaches 413