Table Of Content

SOURCEBOOK OF PARALLEL COMPUTING JACK DONGARRA UniversityofTennessee IAN FOSTER ArgonneNationalLaboratory GEOFFREY FOX IndianaUniversity WILLIAM GROPP ArgonneNationalLaboratory KEN KENNEDY RiceUniversity LINDA TORCZON RiceUniversity ANDY WHITE LosAlamosNationalLaboratory SeniorEditor DenisePenrose PublishingServicesManager EdwardWade ProductionEditor HowardSeverson EditorialCoordinator EmiliaThiuri CoverDesign FrancesBaca TextDesign DettaPenna Illustration DartmouthPublishing,Inc. Composition WindfallSoftware,usingZzTEX Copyeditor BarbaraKohl Proofreader CarolLeyba Indexer SteveRath Printer TheMaple-VailBookManufacturingGroup Covercredit:PaulKlee,Greenchurchsteepleatcenter,1917.©Nimatallah/ArtResource,NY. Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks orregisteredtrademarks.InallinstancesinwhichMorganKaufmannPublishersisawareofa claim,theproductnamesappearininitialcapitalorallcapitalletters.Readers,however,should contacttheappropriatecompaniesformorecompleteinformationregardingtrademarksand registration. MorganKaufmannPublishers AnimprintofElsevierScience 340PineStreet,SixthFloor SanFrancisco,CA94104-3205 www.mkp.com ©2003bybyElsevierScience(USA) Allrightsreserved. PrintedintheUnitedStatesofAmerica 07 06 05 04 03 5 4 3 2 1 Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedin anyformorbyanymeans—electronic,mechanical,photocopying,orotherwise—withoutthe priorwrittenpermissionofthepublisher. LibraryofCongressControlNumber:2002107244 ISBN:1-55860-871-0 Thisbookisprintedonacid-freepaper. PREFACE Duringits11-yearlifetime,theCenterforResearchonParallelComputation(CRPC), aNationalScienceFoundation(NSF)ScienceandTechnologyCenter,wasfocusedon researchandtechnologydevelopmentthatwouldbeneededtomakeparallelcom- puting“trulyusable.”Overthatperiod,aremarkablenumberofapplicationswere convertedtoworkonscalableparallelcomputers,andanequallyremarkablenum- ber of software tools were developed to assist in that process. These developments emerged both from within CRPC and from the community at large. Although the toolsandstrategiesdevelopedandrefinedduringthisperiodarewidelyknowninthe communityofprofessionalparallelapplicationdevelopers,toourknowledgethere existsnocompendiumofmaterialdescribingtheseadvancesinaformsuitablefor use by a newcomer to the field. This volume attempts to fill that gap by bringing togetheracollectionofworkscoveringbothapplicationsandthetechnologiesused toconvertthemtorunonscalableparallelsystems. The principal goal of this book is to make it easy for newcomers to the field of parallel computing to understand the technologies available and how to apply them.Thebookisintendedforstudentsandpractitionersoftechnicalcomputing whoneedtounderstandboththepromiseandpracticeofhigh-performanceparallel computation.Itassumesthatthereaderhasagoodbackgroundineitherapplications orcomputationalscience,butnotinparallelcomputation.Thus,thebookincludes asignificantamountoftutorialmaterial. Wealsointendthatthebookserveasausefulreferenceforthepracticingparallel applicationdeveloper.Thus,itcontainsin-depthtreatmentsofspecifictechnologies andmethodsthatareusefulinparallelcomputing.Thesemethodsareeasilyaccessi- blebystartingfromtheapplicationoverviewsectionsorbyreadingthetechnology overviewchaptersprovidedatthebeginningofeachmajorpart. xiii xiv Preface Wehopethatyoufindthisbookusefulandthatithelpsyouexploittheknowledge gained over the past fifteen years, while avoiding some of the pitfalls that we ourselvesencounteredingainingthatknowledge. Overview of Content Thebookisorganizedintofivemajorparts.PartI,entitledParallelism,isatutorial introductiontothefieldofparallelcomputersandcomputing,withseparatechapters providingabroadoverviewofthefield,anin-depthtreatmentofthearchitectureof modernparallelcomputersystems,andasurveyoftheissuesthatshouldbetaken intoconsiderationwhenprogrammingthem. Part II, Applications, is designed to help new developers understand how high- performancecomputationcanbeappliedinavarietyofspecificapplicationareas.It consistsofanoverviewoftheprocessbywhichoneidentifiesappropriatesoftware andalgorithmsandtheissuesinvolvedinimplementation.Thistreatmentiscom- plemented by chapters containing in-depth studies in the areas of computational fluiddynamics,environmentalengineeringandenergy,andcomputationalchem- istry,andbyaseparatechapterwith11vignettesthatbrieflydescribesuccessfuluses ofparallelsystemsinotherareas.Thesehavebeenchosentocoverabroadrangeof bothscientificareasandnumericalapproaches.Theapplicationsarecross-referenced to the material in later chapters that cover the needed software technologies and algorithmsindepth. PartIII,SoftwareTechnologies,discussestheprogressmadeonavarietyoftechno- logicalstrategiesforapplicationdevelopment, includingmessage-passinglibraries; run-timelibrariesforparallelcomputing,suchasclasslibrariesforHPC++,languages likeHPF,Co-ArrayFortran,andHPC++;performanceanalysisandtuningtoolssuch asPablo;andhigh-levelprogrammingsystems.Thegoalofthispartistoprovidea survey of progress with hints to the user that will help in selecting the right tech- nologyforuseinagivenapplication. PartIV,EnablingTechnologiesandAlgorithms,treatsnumericalalgorithmsandcov- ersparallelnumericalalgorithmsforavarietyofproblemsinscienceandengineer- ing,includinglinearalgebra,continuousanddiscreteoptimization,andsimulation. Eachchaptercoversadifferentalgorithmicarea.Thegoalhereistoserveasaresource fortheapplicationdeveloperseekinggoodalgorithmsfordifficultproblems. Thefinalpartofthebookisdevotedtoasummaryoftheissuesandadiscussion of important future problems for the high-performance science and engineering community,includingdistributedcomputinginagridenvironment. Using This Book Thisbookcanbeusedinseveraldifferentways.Thenewcomertoparallelcompu- tationseekingatutorialintroductionshouldreadallofPartI,alongwithChapters 4,9,16,17,and25.Chapters4,9,and17provideoverviewsofPartsII,III,andIV, Preface xv respectively,sothesemayprovidehintsonotherchaptersthatmaybeofinterestto thereader. Ontheotherhand, thedeveloperwishingtounderstandtechnologiesthatcan helpwithaspecificapplicationareashouldstartwithPartII,whichcoversanumber ofapplicationsalongwiththestrategiesusedtoparallelizethem.Oncethedeveloper hasidentifiedasimilarapplication,heorshecanfollowthecross-referencestofind in-depthtreatmentsoftheusefultechnologiesinPartsIIIandIV. Finally,theparallelcomputingprofessionalcanusethebookasareference.The introductorychaptersatthebeginningofeachmajorpartprovideexcellentguides tothematerialintherestofthepart. Thebookshouldserveasaresourceforusersofsystemsthatspantherangefrom smalldesktopSMPsandPCclusterstohigh-endsupercomputerscosting$100mil- lionormore.Itfocusesonsoftwaretechnologies, alongwiththelarge-scaleappli- cationsenabledbythem.Ineacharea,thetextcontainsageneraldiscussionofthe stateofthefieldfollowedbydetaileddescriptionsofkeytechnologiesormethods. Insomecases,suchasMPIformessagepassing,thisisthedominantapproach.In others,suchasthediscussionofproblem-solvingenvironments,theauthorschoose systemsrepresentingkeyconceptsinanemergingarea. Supplementary Online Material Look for links to the Sourcebook of Parallel Computing website at www.mkp.com, whereyoucanfindnumeroussupplementarymaterials.Inadditiontoupdatesand correctionsdiscoveredafterpublication,thereaderwillfindlinkstoservers(suchas Netlib)fromwhichsoftwaretoolsandlibrariescanbedownloaded. Acknowledgments This book was inspired by the research carried out in the high-performance parallel computing commmunity during the lifetime of the CRPC, an NSF Science and TechnologyCenterthatincludedRiceUniversity,CaliforniaInstituteofTechnology, ArgonneNationalLaboratory,LosAlamosNationalLaboratory,SyracuseUniversity, the University of Tennessee–Knoxville, and the University of Texas–Austin. With- out the original grant from the NSF that established CRPC, the book would not have been possible. In addition, the support for research provided through other grants and contracts from NSF, the Department of Energy, the Department of De- fense(especiallytheDefenseAdvancedResearchProjectsAgency),andtheNational Aeronautics and Space Administration has been critical to the advances described in this volume. Substantive additional support for research and technology devel- opment was provided by the CRPC home institutions, and the home institutions ofmanyofthechapterauthors.Inparticular, matchingfundsforCRPCfromRice Universitywereusedtoprovidetechnicalsupportforthecompletionofthisvolume aftertheNSFfundingforCRPCended. xvi Preface Theeditorswouldalsoliketothankthepeoplewhohelpedusfinishthisvolume. TeresaParksimprovedtheentiremanuscriptwithhereditorialandtechnicalwriting skills.SarahGonzalesprovidedtechnicalsupportandpreparedthegraphicsforpub- lication.GailPiepercarefullyandknowledgeablyeditedsomeofthechapters.Keith CooperconsultedwithusonthedeepermysteriesofLaTEX.EllenButlerandShiliang Chang collected, collated, merged, and corrected the bibliographic entries for this volume.PennyAndersonmanagedanumberofproductiontasksontheRiceside, includingassemblingauthorbiographiesandcollectingcopyrighttransferformsand permissionsforuseofcopyrightedmaterials.EllenButler,KathrynO’Brien,Penny Anderson, andTheresaChatmanarrangedthemeetingsandteleconferences, han- dledthecorrespondence,managedthemanycontactlists,andlenthandswherever they were needed. All these people deserve our thanks; without their efforts, this volumewouldnothavebeenfinished. Denise Penrose, our editor at Morgan Kaufmann, provided valuable advice on the preparation of the manuscript and arranged for independent reviews of the manuscript that were very helpful. Emilia Thiuri provided editorial support and extensive guidance as the final manuscript neared completion. Howard Severson, the production editor, did a superb job of bringing the book to completion on a tight schedule and managed to keep his sense of humor throughout the process. Theproductionstaff,includingthecopyeditorBarbaraKohl,theproofreaderCarol Leyba,andtheindexerSteveRath,weretirelessintheireffortstobringoutahigh- qualitybookontime.Wearealsogratefultothosewhocontributedtotheoverall appearanceofthebook:DettaPennafortextdesign,FrancesBacaforcoverdesign, DartmouthPublishingforillustration,andWindfallSoftwareforcomposition.We especiallyappreciatethecompositor’seffortstoincorporatethesubstantivenumber of changes we made late in the production cycle. Finally, the reviewers provided many intelligent and helpful comments on the book that significantly affected its structure,content,andquality. Toallofthesepeople,andtotheauthorsofindividualchaptersinthistext,we extendourheartfeltthanks. Part I Parallelism1.................................................. Introduction ........................................................ 3 Parallel Computing Hardware .................................. 4 What Have We Learned from Applications? ............ 8 Software and Algorithms.......................................... 11 Toward a Science of Parallel Computation.............. 13 Parallel Computer Architectures ........................ 15 Uniprocessor Architecture........................................ 16 The CPU.............................................................. 17 Memory................................................................ 21 I/O and Networking.............................................. 25 Design Tradeoffs ................................................. 26 Parallel Architectures ............................................... 26 Memory Parallelism............................................. 26 Interconnects....................................................... 33 CPU Parallelism .................................................. 35 I/O and Networking for Parallel Processors......... 38 Support for Programming Models........................ 39 Parallel Architecture Design Tradeoffs................ 39 Future Directions for Parallel Architectures.............. 40 Conclusion ............................................................... 41 Parallel Programming Considerations ............... 43 Architectural Considerations.................................... 45 Shared Memory................................................... 45 Distributed Memory ............................................. 46 Hybrid Systems ................................................... 47 Memory Hierarchy ............................................... 47 Decomposing Programs for Parallelism................... 49 Identi.cation of Parallelism................................... 49 Decomposition Strategy ...................................... 50 Programming Models .......................................... 51 Implementation Styles ......................................... 51 A Simple Example ............................................... 54 Enhancing Parallel Performance.............................. 56 Scalability and Load Balance .............................. 57 Pipeline Parallelism............................................. 59 Regular versus Irregular Problems...................... 60 Memory-Hierarchy Management.............................. 63 Uniprocessor Memory-Hierarchy Management... 63 Multiprocessor Memory Hierarchies.................... 65 Parallel Debugging................................................... 66 Performance Analysis and Tuning........................... 67 Parallel Input/Output ................................................ 69 Conclusion ............................................................... 70 Further Reading....................................................... 70 Part II Applications7..3............................................. General Application Issues ................................ 75 Application Characteristics in a Simple Example..... 75 Communication Structure in Jacobi(cid:144)s Method for Poisson(cid:144)s Equation .................................................. 79 Communication Overhead for More General Update Stencils........................................................ 82 Applications as Basic Complex Systems ................. 84 Time-Stepped and Event-Driven Simulations .......... 87 Temporal Structure of Applications.......................... 88 Summary of Parallelization of Basic Complex Systems ................................................................... 89 Meta-Problems......................................................... 90 Conclusion ............................................................... 91 Parallel Computing in Computational Fluid Dynamics ........................................................... 93 Introduction to Computational Fluid Dynamics......... 94 Basic Equations of Fluid Dynamics ..................... 94 Physical Regimes and Dimensionless Variables.............................................................. 95 The Role of High-Performance Computing ......... 97 Incompressible Flows............................................... 98 Semi-discrete Formulation................................... 99 Spectral Element Methods .................................. 100 Basic Operations ................................................. 108 Global Matrix Operations..................................... 111 Data Structures.................................................... 113 Solution Techniques............................................ 116 Adaptive Mesh Re.nement .................................. 121 Implementation for Parallel Architectures............ 126 An Example(cid:132)The Cylinder Wake .......................... 129 Compressible Flows................................................. 132 Governing Equations of Motion........................... 133 Numerical Methods for Hyperbolic Conservation Laws.............................................. 134 An Application: The Richtmyer (cid:133)Meshkov Instability.............................................................. 137 Adaptive Mesh Re.nement .................................. 138 Conclusion ............................................................... 144 Parallel Computing in Environment and Energy................................................................ 145 Subsurface-Flow Modeling....................................... 146 IPARS Motivation ................................................ 147 IPARS Description............................................... 148 IPARS and Grid Computing by NetSolve................. 152 Integrating IPARS into NetSolve ......................... 153 Client-Side Web-Browser Interface ..................... 154 Tracking and Interactive Simulation in IPARS ......... 155 An Interactive Computational Collaboration: DISCOVER.......................................................... 157 Integrating IPARS with DISCOVER..................... 158 Surface-Water Simulation ........................................ 159 A Water Quality Model: CE-QUAL-ICM............... 159 A Parallel Water-Quality Model: PCE-QUAL-ICM .................................................. 160 Parallel Algorithm ................................................ 161 A Coupled Simulation of Flow and Transport with ADR.......................................................................... 162 The Active Data Repository................................. 163 Implementation.................................................... 164 Conclusion ............................................................... 165 Parallel Computational Chemistry: An Overview of NWChem ....................................... 167 Molecular Quantum Chemistry................................. 168