Parallel Computer Organization and Design

562 Pages·2012·7.017 MB·English

by  Michel Dubois, Murali Annavaram, Per Stenström

Parallel Computer Organization and Design
Teachingfundamentaldesignconceptsandthechallengesofemergingtechnology,thistextbook
preparesstudentsforacareerdesigningthecomputersystemsofthefuture.In-depthcoverage
of complexity, power, reliability, and performance, coupled with treatment of parallelism at
alllevels,includingILPandTLP,providesthestate-of-the-arttrainingthatstudentsneed.The
wholegamutofparallelarchitecturedesignoptionsisexplained,fromcoremicroarchitectureto
chipmultiprocessorstolarge-scalemultiprocessorsystems.Allthechaptersareself-contained,
yetconciseenoughthatthematerialcanbetaughtinacoupleofsemesters,makingitperfect
for use in senior undergraduate and graduate computer architecture courses. The book is also
teemingwithpracticalexamplestoaidthelearningprocess,showingconcreteapplicationsof
definitions.Withsimplemodelsandcodesusedthroughout,allmaterialisaccessibletoabroad
rangeofcomputerengineering/sciencestudentswithonlyabasicknowledgeofhardwareand
software.
Michel Dubois is a Professor in the Ming Hsieh Department of Electrical Engineering at the
University of Southern California (USC) and part of the Computer Engineering Directorate.
BeforejoiningUSCin1984,hewasaresearchengineerattheCentralResearchLaboratoryof
Thomson-CSFinOrsay,France.Hehaspublishedmorethan150technicalpapersoncomputer
architectureandhaseditedtwobooks.HeisaFellowoftheIEEEandoftheACM.
Murali Annavaram is an Associate Professor and Robert G. and Mary G. Lane Early Career
ChairintheMingHsiehDepartmentofElectricalEngineeringatUSCandpartoftheComputer
Engineering Directorate, where he has developed and taught advanced computer architecture
courses. Prior to USC, he spent 6 years at Intel researching various aspects of future CMP
designs.
Per Stenstro¨misaProfessorofComputerEngineeringatChalmersUniversityofTechnology,
Sweden.Hehaspublishedtwotextbooksandover100technicalpapers.Hehasbeenavisiting
scientist at Carnegie-Mellon, Stanford, and USC, and also was engaged in research at Sun
Microsystems on its chip multi-threading technology. He is a Fellow of the IEEE and of the
ACM, and is a member of the Royal Swedish Academy of Engineering Sciences and the
AcademiaEuropaea.
“Parallelcomputersandmulticorearchitecturesarerapidlygainingimportancebecausetheperformance
ofasinglecoreisnotimprovingatthesamehistoricallevel.ProfessorsDubois,Annavaram,andStenstrom
havecreatedaneasilyreadablebookontheintricaciesofparallelarchitecturedesignthatacademicians
andpractitionersalikewillfindextremelyuseful.”
ShubuMukherjee,Cavium,Inc.
“Thebookcanhelpthereaderstounderstandtheprinciplesofparallelsystemscrystallyclear.Anecessary
booktoreadforthedesignersofparallelsystems.”
YunjiChen,InstituteofComputingTechnology,ChineseAcademyofSciences
“Allfutureelectronicsystemswillcompriseofabuilt-inmicroprocessor,consequentlytheimportance
of computer architecture will surge. This book provides an excellent tutorial of computer architecture
fundamentalsfromthebasictechnologyviaprocessorandmemoryarchitecturetochipmultiprocessors.
Ifoundthebookveryeducationallyflowandreadable–anexcellentinstructivebookworthusing.”
UriWeiser,Technion
“Thisbookreallyfulfilstheneedtounderstandthebasictechnologicalon-chipfeaturesandconstraints
inconnectionwiththeirimpactoncomputerarchitecturedesignchoices.Allcomputingsystemsstudents
anddevelopersshouldfirstmasterthesesingleandmulticorefoundationsinaplatformindependentway,
asthiscomprehensivetextdoes.”
MateoValero,BSC
“After the drastic shift towards multi-cores that processor architecture has experienced in the past few
years,thedomainwasindireneedofacomprehensiveandup-to-datebookonthetopic.Michel,Murali,
and Per have crafted an excellent textbook which can serve both as an introduction to multi-core and
parallelarchitectures,aswellasareferenceforengineersandresearchers.”
OlivierTemam,INRIA,France
“ParallelComputerOrganizationandDesignfillsanurgentneedforacomprehensiveandauthoritative
yet approachable tutorial and reference text for advanced computer architecture topics. All of the key
principlesandconceptscoveredinWisconsin’sthree-coursecomputerarchitecturesequenceareaddressed
inawell-organized,thoughtful,andpedagogicallyappealingmanner,withoutoverwhelmingthereader
withdistractingtriviaoranexcessofquantitativedata.Inparticular,thecoverageofchipmultiprocessors
inChapter8 is fully upto datewith thestate ofthe artinindustry practice, while the finalchapteron
quantitativeevaluation–atruegem!–isauniqueandvaluableassetthatwillclearlysetthisbookapart
fromitscompetition.”
MikkoLipasti,UniversityofWisonsin-Madison
“Thebookcontainsin-depthcoverageofalltheaspectsofthecomputersystems.Itiscomprehensive,
systematic, and in sync with the latest development in the field. The skillfully organized book uses
self-containedchapterstoallowthereadersgetacompleteunderstandingofatopicwithoutwandering
through the whole book. Its content is rich, coherent and clear. Its questions are crafted to stimulate
creativethinking.Irecommendthebookasamustreadtoallgraduatestudentsandyoungresearchers
andengineersdesigningthecomputers.”
LixinZhang,InstituteofComputingTechnology,ChineseAcademyofSciences
“...parallel architectures are the key for high performance and high efficiency computing systems.
Thisbooktellsthestoryofparallelarchitectureatalllevels–fromthesingletransistortothefullblown
CMP–anunforgettablejourney!”
RonnyRonen,Intel
“Multicore chips have made parallel architectures ubiquitous and their understanding a necessity. This
textprovidesacomprehensivetreatmentofparallelsystemarchitectureandthefundamentalsofcache
coherence and memory consistency in the most compact form to date. This is a perfect text for a one
semestergraduatecourse.”
LawrenceRauchwerger,TexasA&MUniversity
“Itisthebestoftoday’sbooksonthesubject,andIplantouseitinmyclass.Itisanup-to-datepicture
ofparallelcomputingthatiswritteninastylethatisclearandaccessible.”
TrevorMudge,BredtFamilyProfessorofComputerEngineering,UniversityofMichigan
“Parallelism,atmultiplelevelsandinmanydifferentforms,isnowanecessityforallfuturecomputer
systems,andthenewgenerationofcomputerscientistsandengineershavetomasterit.Tounderstandthe
complexinteractionsamongthehundredsofexistingideas,options,andchoices,onehastocategorize
them, put them in order, and then synthesize them. That is precisely what Dubois, Annavaram, and
Stenstro¨mdo,inamagnificentway,inthisextremelycontemporaryandtimelybook.Iwanttoparticularly
stress the uniquely clear way in which the authors explain the hardest among these topics: coherence,
synchronization,andmemoryconsistency.”
ManolisKatevenis,ProfessorofComputerScience,UniversityofCrete
“This book is a truly comprehensive treatment of parallel computers, from some of the top experts in
thefield.Wellgroundedintechnologyyetremainingveryaccessible,italsoincludesimportantbutoften
overlookedtopicssuchreliability,power,andsimulation.”
NormJouppi,HP
“Thistexttakesafreshcutattraditionalcomputerarchitecturetopicsandconsidersbasicprinciplesfrom
theperspectiveofmulti-coreandparallelsystems.Theneedforsuchahighqualitytextbookwrittenfrom
thisperspectiveisoverdue,andtheauthorsofthistexthavedoneagoodjobinorganizingandrevamping
topicstoprovidethenextgenerationofcomputerarchitectswiththebasicprinciplestheywillneedto
designmulti-coreandmany-coresystems.”
DavidKaeli,DirectoroftheNUComputerArchitectureResearchLaboratory,NEU
“Anexcellentbookinanareathathaslongcriedoutfortutorialmaterial–itwillbeanindispensable
resourcetostudentsandeducatorsinparallelcomputerarchitecture.”
JosepTorrellas,UniversityofIllinois

Parallel Computer
Organization and
Design
MICHEL DUBOIS
UniversityofSouthernCalifornia,USA
MURALI ANNAVARAM
UniversityofSouthernCalifornia,USA
PER STENSTRO¨ M
ChalmersUniversityofTechnology,Sweden
cambridge university press
Cambridge,NewYork,Melbourne,Madrid,CapeTown,
Singapore,Sa˜oPaulo,Delhi,MexicoCity
CambridgeUniversityPress
TheEdinburghBuilding,CambridgeCB28RU,UK
PublishedintheUnitedStatesofAmericabyCambridgeUniversityPress,NewYork
www.cambridge.org
Informationonthistitle:www.cambridge.org/9780521886758
(cid:2)C CambridgeUniversityPress2012
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2012
PrintedintheUnitedKingdomattheUniversityPress,Cambridge
AcatalogrecordforthispublicationisavailablefromtheBritishLibrary
LibraryofCongressCataloginginPublicationdata
Dubois,Michel,1953–
Parallelcomputerorganizationanddesign/MichelDubois,MuraliAnnavaram,
PerStenström.
pages cm
Includesindex.
ISBN978-0-521-88675-8
1.Parallelcomputers. 2.Computerorganization. I.Annavaram,Murali.
II.Stenström,Per. III.Title.
QA76.5.D754 2012
005.2(cid:3)75–dc23 2012010634
ISBN978-0-521-88675-8Hardback
Additionalresourcesforthispublicationatwww.cambridge.org/9780521886758
CambridgeUniversityPresshasnoresponsibilityforthepersistenceor
accuracyofURLsforexternalorthird-partyinternetwebsitesreferredto
inthispublication,anddoesnotguaranteethatanycontentonsuch
websitesis,orwillremain,accurateorappropriate.
CONTENTS
Preface pagexi
1 Introduction 1
1.1 Whatiscomputerarchitecture? 2
1.2 Componentsofaparallelarchitecture 5
1.3 Parallelisminarchitectures 13
1.4 Performance 17
1.5 Technologicalchallenges 26
Exercises 30
2 Impact of technology 36
2.1 Chapteroverview 36
2.2 Basiclawsofelectricity 37
2.3 TheMOSFETtransistorandCMOSinverter 39
2.4 Technologyscaling 43
2.5 Powerandenergy 45
2.6 Reliability 54
Exercises 71
3 Processor microarchitecture 74
3.1 Chapteroverview 74
3.2 Instructionsetarchitecture 75
3.3 Staticallyscheduledpipelines 91
3.4 Dynamicallyscheduledpipelines 111
3.5 VLIWmicroarchitectures 140
3.6 EPICmicroarchitectures 157
3.7 Vectormicroarchitectures 158
Exercises 165
4 Memory hierarchies 193
4.1 Chapteroverview 193
4.2 Thepyramidofmemorylevels 194
4.3 Cachehierarchy 198
4.4 Virtualmemory 212
Exercises 224
viii Contents
5 Multiprocessor systems 232
5.1 Chapteroverview 232
5.2 Parallel-programmingmodelabstractions 233
5.3 Message-passingmultiprocessorsystems 239
5.4 Bus-basedshared-memorysystems 246
5.5 Scalableshared-memorysystems 276
5.6 Cache-onlyshared-memorysystems 293
Exercises 298
6 Interconnection networks 309
6.1 Chapteroverview 309
6.2 Designspaceofinterconnectionnetworks 311
6.3 Switchingstrategies 319
6.4 Topologies 322
6.5 Routingtechniques 330
6.6 Switcharchitecture 337
Exercises 339
7 Coherence, synchronization, and memory consistency 342
7.1 Chapteroverview 342
7.2 Background 344
7.3 Coherenceandstoreatomicity 350
7.4 Sequentialconsistency 375
7.5 Synchronization 388
7.6 Relaxedmemory-consistencymodels 398
7.7 Speculativeviolationsofmemoryorders 411
Exercises 415
8 Chip multiprocessors 425
8.1 Chapteroverview 425
8.2 RationalebehindCMPs 426
8.3 Coremulti-threading 429
8.4 Chipmultiprocessorarchitectures 446
8.5 Programmingmodels 459
Exercises 482
9 Quantitative evaluations 488
9.1 Chapteroverview 488
9.2 Taxonomyofsimulators 490
9.3 Integratingsimulators 498
9.4 Multiprocessorsimulators 500