High Performance Datacenter Networks. Architectures,Algorithms,and Opportunities PDF
Preview High Performance Datacenter Networks. Architectures,Algorithms,and Opportunities
SSSeeerrireiieess sI IISSSSSSNNN: ::1 11999333555---333222333555 AAA BBB MMM TTT SSSYYYNNNTTTHHHEEESSSIIISSS LLLEEECCCTTTUUURRREEESSS OOONNN S • KS • KS • K &&&CCC MMMooorrrgggaaannn& & & CCClllaaayyypppoooooolll PPPuuubbbllliiissshhheeerrrsss III CCCOOOMMMPPPUUUTTTEEERRR AAARRRCCCHHHIIITTTEEECCCTTTUUURRREEE MMM SSSeeerrriiieeesss E EEdddiiitttooorrr::: M MMaaarrrkkk D DD. .. H HHiiillllll, ,, UUUnnniiivvveeerrrsssiiitttyyy o oofff W WWiiissscccooonnnsssiiinnn HHHiiiggghhh PPPeeerrrfffooorrrmmmaaannnccceee HHHiiiggghhh PPPeeerrrfffooorrrmmmaaannnccceee DDDaaatttaaaccceeennnttteeerrr NNNeeetttwwwooorrrkkksss AAArrrccchhhiiittteeeccctttuuurrreeesss,,, AAAlllgggooorrriiittthhhmmmsss,,, aaannnddd OOOppppppooorrrtttuuunnniiitttyyy HHH III DDDaaatttaaaccceeennnttteeerrr NNNeeetttwwwooorrrkkksss GGG HHH DDDeeennnnnniiisss A AAbbbtttsss,, ,G GGoooooogggllelee I IInnnccc.. . a aannnddd J JJooohhhnnn K KKiiimmm,, ,K KKooorrreeeaaa A AAdddvvvaaannnccceeeddd I IInnnssstttiiitttuuuttteee o oofff S SSccceeeiiinnnccceee a aannnddd T TTeeeccchhhnnnooolloloogggyyy PPP EEE RRR DDDaaatttaaaccceeennnttteeerrr n nneeetttwwwooorrrkkksss p pprrrooovvviididdeee t tthhheee c ccooommmmmmuuunnniiciccaaatttiioioonnn s ssuuubbbssstttrrraaattteee f ffooorrr l lalaarrrgggeee p ppaaarrraaallllleleell l c ccooommmpppuuuttteeerrr s ssyyysssttteeemmmsss t tthhhaaattt FFF AAArrrccchhhiiittteeeccctttuuurrreeesss,,, AAAlllgggooorrriiittthhhmmmsss,,, aaannnddd OOOppppppooorrrtttuuunnniiitttyyy OOO fffooorrrmmm t tthhheee e eecccooosssyyysssttteeemmm f ffooorrr h hhiigigghhh p ppeeerrrfffooorrrmmmaaannnccceee c ccooommmpppuuutttiininnggg ( ((HHHPPPCCC))) s ssyyysssttteeemmmsss a aannnddd m mmooodddeeerrrnnn I IInnnttteeerrrnnneeettt a aappppppllili-i-- RRR MMM cccaaatttiioioonnnsss. .. T TThhheee d ddeeesssiigiggnnn o oofff n nneeewww d ddaaatttaaaccceeennnttteeerrr n nneeetttwwwooorrrkkksss i isiss m mmoootttiivivvaaattteeeddd b bbyyy a aannn a aarrrrrraaayyy o oofff a aapppppplliliciccaaatttiioioonnnsss r rraaannngggiininnggg AAA fffrrrooommm c ccooommmmmmuuunnniiciccaaatttiioioonnn i ininnttteeennnsssiivivveee c ccllilimimmaaatttooolloloogggyyy, ,, c ccooommmppplleleexxx m mmaaattteeerrriiaiaall l s ssiimimmuuullalaatttiioioonnnsss a aannnddd m mmooolleleecccuuullalaarrr d ddyyynnnaaammmiiciccsss NNN CCC tttooo s ssuuuccchhh I IInnnttteeerrrnnneeettt a aapppppplliliciccaaatttiioioonnnsss a aasss W WWeeebbb s sseeeaaarrrccchhh, ,, l lalaannnggguuuaaagggeee t ttrrraaannnsssllalaatttiioioonnn, ,, c ccooolllllalaabbbooorrraaatttiivivveee I IInnnttteeerrrnnneeettt a aapppppplliliciccaaatttiioioonnnsss,,, EEE ssstttrrreeeaaammmiininnggg v vviididdeeeooo a aannnddd v vvoooiicicceee---ooovvveeerrr---IIIPPP. .. F FFooorrr b bbooottthhh S SSuuupppeeerrrcccooommmpppuuutttiininnggg a aannnddd C CClloloouuuddd C CCooommmpppuuutttiininnggg t tthhheee n nneeetttwwwooorrrkkk DDD AAA eeennnaaabbblleleesss d ddiisisstttrrriibibbuuuttteeeddd a aapppppplliliciccaaatttiioioonnnsss t ttooo c ccooommmmmmuuunnniiciccaaattteee a aannnddd i ininnttteeerrrooopppeeerrraaattteee i ininn a aannn o oorrrccchhheeessstttrrraaattteeeddd a aannnddd e eeffffffiicicciieieennnttt TTT AAA wwwaaayyy... CCC EEE DDDeeennnnnniiisss AAAbbbtttsss TTThhhiisiss b bbooooookkk d ddeeessscccrrriibibbeeesss t tthhheee d ddeeesssiigiggnnn a aannnddd e eennngggiininneeeeeerrriininnggg t ttrrraaadddeeeoooffffffsss o oofff d ddaaatttaaaccceeennnttteeerrr n nneeetttwwwooorrrkkksss. .. I IIttt d ddeeessscccrrriibibbeeesss NNN TTT iininnttteeerrrcccooonnnnnneeeccctttiioioonnn n nneeetttwwwooorrrkkksss f ffrrrooommm t ttooopppooolloloogggyyy a aannnddd n nneeetttwwwooorrrkkk a aarrrccchhhiititteeeccctttuuurrreee t ttooo r rrooouuutttiininnggg a aallglggooorrriititthhhmmmsss, ,, a aannnddd p pprrreeessseeennntttsss EEE JJJooohhhnnn KKKiiimmm RRR oooppppppooorrrtttuuunnniitittiieieesss f ffooorrr t ttaaakkkiininnggg a aadddvvvaaannntttaaagggeee o oofff t tthhheee e eemmmeeerrrgggiininnggg t tteeeccchhhnnnooolloloogggyyy t ttrrreeennndddsss t tthhhaaattt a aarrreee i ininnffflluluueeennnccciininnggg r rrooouuuttteeerrr NNN mmmiiciccrrroooaaarrrccchhhiititteeeccctttuuurrreee. .. W WWiititthhh t tthhheee e eemmmeeerrrgggeeennnccceee o oofff “ ““mmmaaannnyyy---cccooorrreee”” ” p pprrroooccceeessssssooorrr c cchhhiipippsss, ,, i ititt i isiss e eevvviididdeeennnttt t tthhhaaattt w wweee w wwiilillll l a aallslssooo EEE TTT nnneeeeeeddd “ ““mmmaaannnyyy---pppooorrrttt”” ” rrrooouuutttiininnggg ccchhhiipippsss tttooo ppprrrooovvviididdeee aaa bbbaaannndddwwwiididdttthhh---rrriicicchhh nnneeetttwwwooorrrkkk tttooo aaavvvoooiididd ttthhheee pppeeerrrfffooorrrmmmaaannnccceee WWW OOO llilimimmiitittiininnggg eeeffffffeeeccctttsss ooofff AAAmmmdddaaahhhll’ls’’ss LLLaaawww. .. W WWeee ppprrrooovvviididdeee aaannn ooovvveeerrrvvviieieewww ooofff cccooonnnvvveeennntttiioioonnnaaall l tttooopppooolloloogggiieieesss aaannnddd ttthhheeeiirirr RRR rrrooouuutttiininnggg a aallglggooorrriititthhhmmmsss a aannnddd s sshhhooowww h hhooowww t tteeeccchhhnnnooolloloogggyyy, ,, s ssiigiggnnnaaallilininnggg r rraaattteeesss a aannnddd c ccooosssttt---eeeffffffeeeccctttiivivveee o ooppptttiiciccsss a aarrreee m mmoootttiivivvaaatttiininnggg KKK SSS nnneeewww nnneeetttwwwooorrrkkk tttooopppooolloloogggiieieesss ttthhhaaattt ssscccaaallelee uuuppp tttooo mmmiilillllilioioonnnsss ooofff hhhooossstttsss. .. T TThhheee bbbooooookkk aaallslssooo ppprrrooovvviididdeeesss dddeeetttaaaiilileleeddd cccaaassseee ssstttuuudddiieieesss o oofff t ttwwwooo h hhiigigghhh p ppeeerrrfffooorrrmmmaaannnccceee p ppaaarrraaallllleleell l c ccooommmpppuuuttteeerrr s ssyyysssttteeemmmsss a aannnddd t tthhheeeiirirr n nneeetttwwwooorrrkkksss... AAAbbbooouuuttt S SSYYYNNNTTTHHHEEESSSIIIsss MMM TTThhhiisiss v vvooolluluummmeee i isiss a aa p pprrriininntteteeddd v vveeerrrsssiioioonnn o ooff f a aa w wwooorrrkkk t ththhaaatt t a aappppppeeeaaarrrsss i ininn t ththheee S SSyyynnntththheeesssiisiss OOO DDDiigiggiititataall lL LLiibibbrrraaarrryyy o ooff fE EEnnngggiininneeeeeerrriininnggg a aannnddd C CCooommmpppuuutteteerrr S SSccciieieennnccceee. ..S SSyyynnntththheeesssiisiss L LLeeecccttutuurrreeesss RRR ppprroroovvvidiiddeee c ccooonnncccisiisseee, ,,o oorrirgiigginiinnaaal llp pprrereessseeennnttataattitoiioonnnsss o ooff fi miimmpppooorrrttataannntt tr rereessseeeaaarrcrcchhh a aannnddd d ddeeevvveeelolloopppmmmeeennnttt GGG AAA ttotoopppiiciccsss, ,,p ppuuubbblillisisshhheeeddd q qquuuiicicckkkllylyy, ,,i ininn d ddiigiggiititataal lla aannnddd p pprrriininntt tf fofoorrrmmmaaattstss. ..F FFooorrr m mmooorreree i ininnffofoorrrmmmaaattitioioonnn NNN vvviisissiitit t w wwwwwwww..m.mmooorrrgggaaannncccllalaayyypppooooooll.l.c.ccooommm &&& CCC SSSYYYNNNTTTHHHEEESSSIIISSS LLLEEECCCTTTUUURRREEESSS OOONNN IIISSSBBBNNN::: 999777888---111---666000888444555---444000222---000 LLL MMMooorrrgggaaannn & & & CCClllaaayyypppoooooolll PPPuuubbbllliiissshhheeerrrsss 999000000000000 AAA YYY CCCOOOMMMPPPUUUTTTEEERRR AAARRRCCCHHHIIITTTEEECCCTTTUUURRREEE PPP wwwwwwwww...mmmooorrrgggaaannnccclllaaayyypppoooooolll...cccooommm OOO 999777888111666000888444555444000222000 OOO MMMaaarrrkkk DDD.. . HHHiiillllll,, , SSSeeerrriiieeesss E EEdddiiitttooorrr LLL High Performance Datacenter Networks Architectures,Algorithms,andOpportunities Synthesis Lectures on Computer Architecture Editor MarkD.Hill,UniversityofWisconsin SynthesisLecturesonComputerArchitecturepublishes50-to100-pagepublicationsontopics pertainingtothescienceandartofdesigning,analyzing,selectingandinterconnectinghardware componentstocreatecomputersthatmeetfunctional,performanceandcostgoals.Thescopewill largelyfollowthepurviewofpremiercomputerarchitectureconferences,suchasISCA,HPCA, MICRO,andASPLOS. HighPerformanceDatacenterNetworks:Architectures,Algorithms,andOpportunities DennisAbtsandJohnKim 2011 QuantumComputingforArchitects,SecondEdition TzvetanMetodi,FredChong,andArvinFaruque 2011 ProcessorMicroarchitecture:AnImplementationPerspective AntonioGonzález,FernandoLatorre,andGrigoriosMagklis 2010 TransactionalMemory,2ndedition TimHarris,JamesLarus,andRaviRajwar 2010 ComputerArchitecturePerformanceEvaluationMethods LievenEeckhout 2010 IntroductiontoReconfigurableSupercomputing MarcoLanzagorta,StephenBique,andRobertRosenberg 2009 On-ChipNetworks NatalieEnrightJergerandLi-ShiuanPeh 2009 iii TheMemorySystem:YouCan’tAvoidIt,YouCan’tIgnoreIt,YouCan’tFakeIt BruceJacob 2009 FaultTolerantComputerArchitecture DanielJ.Sorin 2009 TheDatacenterasaComputer:AnIntroductiontotheDesignofWarehouse-ScaleMachines freeaccess LuizAndréBarrosoandUrsHölzle 2009 ComputerArchitectureTechniquesforPower-Efficiency StefanosKaxirasandMargaretMartonosi 2008 ChipMultiprocessorArchitecture:TechniquestoImproveThroughputandLatency KunleOlukotun,LanceHammond,andJamesLaudon 2007 TransactionalMemory JamesR.LarusandRaviRajwar 2006 QuantumComputingforComputerArchitects TzvetanS.MetodiandFredericT.Chong 2006 Copyright© 2011byMorgan&Claypool Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedin anyformorbyanymeans—electronic,mechanical,photocopy,recording,oranyotherexceptforbriefquotationsin printedreviews,withoutthepriorpermissionofthepublisher. HighPerformanceDatacenterNetworks:Architectures,Algorithms,andOpportunities DennisAbtsandJohnKim www.morganclaypool.com ISBN:9781608454020 paperback ISBN:9781608454037 ebook DOI10.2200/S00341ED1V01Y201103CAC014 APublicationintheMorgan&ClaypoolPublishersseries SYNTHESISLECTURESONCOMPUTERARCHITECTURE Lecture#14 SeriesEditor:MarkD.Hill,UniversityofWisconsin SeriesISSN SynthesisLecturesonComputerArchitecture Print1935-3235 Electronic1935-3243 High Performance Datacenter Networks Architectures,Algorithms,andOpportunities Dennis Abts GoogleInc. John Kim KoreaAdvancedInstituteofScienceandTechnology(KAIST) SYNTHESISLECTURESONCOMPUTERARCHITECTURE#14 M &C Morgan &cLaypool publishers ABSTRACT Datacenternetworksprovidethecommunicationsubstrateforlargeparallelcomputersystemsthat form the ecosystem for high performance computing (HPC) systems and modern Internet appli- cations.The design of new datacenter networks is motivated by an array of applications ranging fromcommunicationintensiveclimatology,complexmaterialsimulationsandmoleculardynamics tosuchInternetapplicationsasWebsearch,languagetranslation,collaborativeInternetapplications, streamingvideoandvoice-over-IP.ForbothSupercomputingandCloudComputingthenetwork enables distributed applications to communicate and interoperate in an orchestrated and efficient way. This book describes the design and engineering tradeoffs of datacenter networks. It de- scribes interconnection networks from topology and network architecture to routing algorithms, and presents opportunities for taking advantage of the emerging technology trends that are influ- encing router microarchitecture.With the emergence of“many-core” processor chips,it is evident thatwewillalsoneed“many-port”routingchipstoprovideabandwidth-richnetworktoavoidthe performancelimitingeffectsofAmdahl’sLaw.Weprovideanoverviewofconventionaltopologies andtheirroutingalgorithmsandshowhowtechnology,signalingratesandcost-effectiveopticsare motivatingnewnetworktopologiesthatscaleuptomillionsofhosts.Thebookalsoprovidesdetailed casestudiesoftwohighperformanceparallelcomputersystemsandtheirnetworks. KEYWORDS networkarchitectureanddesign,topology,interconnectionnetworks,fiberoptics,par- allelcomputerarchitecture,systemdesign vii Contents Preface .................................................................. xi Acknowledgments.......................................................xiii NotetotheReader........................................................xv 1 Introduction ..............................................................1 1.1 FromSupercomputingtoCloudComputing................................ 3 1.2 Beowulf:TheClusterisBorn............................................. 3 1.3 OverviewofParallelProgrammingModels................................. 4 1.4 Puttingitalltogether ................................................... 5 1.5 QualityofService(QoS)requirements..................................... 6 1.6 Flowcontrol ........................................................... 7 1.6.1 Lossyflowcontrol ................................................ 7 1.6.2 Losslessflowcontrol .............................................. 8 1.7 Theriseofethernet ..................................................... 9 1.8 Summary.............................................................. 9 2 Background ............................................................ 13 2.1 Interconnectionnetworks ............................................... 13 2.2 Technologytrends ..................................................... 13 2.3 Topology,RoutingandFlowControl ..................................... 16 2.4 CommunicationStack.................................................. 16 3 TopologyBasics ......................................................... 19 3.1 Introduction .......................................................... 19 3.2 TypesofNetworks ..................................................... 20 3.3 Mesh,Torus,andHypercubes ........................................... 20 3.3.1 Nodeidentifiers ................................................. 22 3.3.2 k-aryn-cubetradeoffs............................................ 22 viii 4 High-RadixTopologies .................................................. 25 4.1 TowardsHigh-radixTopologies.......................................... 25 4.2 TechnologyDrivers .................................................... 26 4.2.1 PinBandwidth.................................................. 26 4.2.2 EconomicalOpticalSignaling ..................................... 29 4.3 High-RadixTopology .................................................. 30 4.3.1 High-DimensionHypercube,Mesh,Torus .......................... 30 4.3.2 Butterfly ....................................................... 30 4.3.3 High-RadixFolded-Clos ......................................... 31 4.3.4 FlattenedButterfly .............................................. 34 4.3.5 Dragonfly ...................................................... 34 4.3.6 HyperX ........................................................ 37 5 Routing ................................................................ 39 5.1 RoutingBasics ........................................................ 39 5.1.1 ObjectivesofaRoutingAlgorithm................................. 40 5.2 MinimalRouting ...................................................... 40 5.2.1 DeterministicRouting ........................................... 40 5.2.2 ObliviousRouting ............................................... 41 5.3 Non-minimalRouting.................................................. 41 5.3.1 Valiant’salgorithm(VAL) ........................................ 42 5.3.2 UniversalGlobalAdaptiveLoad-Balancing(UGAL) ................ 42 5.3.3 ProgressiveAdaptiveRouting(PAR) ............................... 43 5.3.4 Dimensionally-Adaptive,Load-balanced(DAL)Routing ............. 43 5.4 IndirectAdaptiveRouting .............................................. 43 5.5 RoutingAlgorithmExamples ........................................... 44 5.5.1 Example1:Folded-Clos .......................................... 45 5.5.2 Example2:FlattenedButterfly .................................... 45 5.5.3 Example3:Dragonfly ............................................ 49 6 ScalableSwitchMicroarchitecture......................................... 51 6.1 RouterMicroarchitectureBasics ......................................... 51 6.2 Scalingbaselinemicroarchitecturetohighradix ............................ 52 6.3 FullyBufferedCrossbar ................................................ 54 6.4 HierarchicalCrossbarArchitecture....................................... 55 6.5 ExamplesofHigh-RadixRouters ........................................ 57 ix 6.5.1 CrayYARCRouter .............................................. 57 6.5.2 MellanoxInfiniScaleIV .......................................... 59 7 SystemPackaging ....................................................... 63 7.1 Packaginghierarchy.................................................... 63 7.2 Powerdeliveryandcooling .............................................. 63 7.3 TopologyandPackagingLocality ........................................ 68 8 CaseStudies ............................................................ 73 8.1 CrayBlackWidowMultiprocessor ....................................... 73 8.1.1 BlackWidowNodeOrganization .................................. 73 8.1.2 High-radixFolded-ClosNetwork.................................. 74 8.1.3 SystemPackaging ............................................... 75 8.1.4 High-radixFat-tree.............................................. 76 8.1.5 PacketFormat .................................................. 77 8.1.6 NetworkLayerFlowControl ..................................... 78 8.1.7 Data-linkLayerProtocol ......................................... 78 8.1.8 Serializer/Deserializer............................................ 80 8.2 CrayXTMultiprocessor................................................ 80 8.2.1 3-Dtorus ...................................................... 81 8.2.2 Routing ....................................................... 82 8.2.3 FlowControl ................................................... 84 8.2.4 SeaStarRouterMicroarchitecture ................................. 84 8.3 Summary............................................................. 88 9 ClosingRemarks ........................................................ 91 9.1 Programmingmodels .................................................. 91 9.2 Wireprotocols ........................................................ 91 9.3 Opportunities ......................................................... 92 Bibliography............................................................ 93 Authors’Biographies .................................................... 99
The list of books you might like
