DataFlow Supercomputing Essentials: Algorithms, Applications and Implementations

157 Pages·2017·3.52 MB·English

by  Veljko Milutinovic

Computer Communications and Networks
Veljko Milutinovic
Milos Kotlar
Marko Stojanovic
Igor Dundic
Nemanja Trifunovic
Zoran Babovic
DataFlow
Supercomputing
Essentials
Algorithms, Applications and
Implementations
Computer Communications and Networks
Serieseditor
A.J.Sammes
CentreforForensicComputing
CranfieldUniversity,ShrivenhamCampus
Swindon,UK
The Computer Communications and Networks series is a range of textbooks,
monographs and handbooks. It sets out to provide students, researchers, and non-
specialists alike with a sure grounding in current knowledge, together with
comprehensibleaccesstothelatestdevelopmentsincomputercommunicationsand
networking.
Emphasis is placed on clear and explanatory styles that support a tutorial
approach, so that even the most complex of topics is presented in a lucid and
intelligiblemanner.
Moreinformationaboutthisseriesathttp://www.springer.com/series/4198
Veljko Milutinovic • Milos Kotlar
Marko Stojanovic • Igor Dundic
Nemanja Trifunovic • Zoran Babovic
DataFlow Supercomputing
Essentials
Algorithms, Applications
and Implementations
123
VeljkoMilutinovic MilosKotlar
SchoolofElectricalEngineering SchoolofElectricalEngineering
UniversityofBelgrade UniversityofBelgrade
Belgrade,Serbia Belgrade,Serbia
MarkoStojanovic IgorDundic
SchoolofElectricalEngineering UniversityofBern
UniversityofBelgrade Bern/Fribourg/Neuchâtel
Belgrade,Serbia Switzerland
NemanjaTrifunovic ZoranBabovic
MaxelerTechnologies SchoolofElectricalEngineering
London,UK UniversityofBelgrade
Belgrade,Serbia
ISSN1617-7975 ISSN2197-8433 (electronic)
ComputerCommunicationsandNetworks
ISBN978-3-319-66124-7 ISBN978-3-319-66125-4 (eBook)
DOI10.1007/978-3-319-66125-4
LibraryofCongressControlNumber:2017951392
©SpringerInternationalPublishingAG2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Ingeneral,thetwomajorapproachestocomputingare:controlflowanddataflow.
In the control-flow approach, the program micro-controls the flow of data through
the hardware. In the dataflow approach, the program configures the hardware; in
theory, it is the voltage difference between the input and output of the hardware
thatmicro-controlstheflowofdatathroughthehardwareinthetheoreticallyideal
dataflowmachines(inreality,fornow,thistheoreticalidealhasnotbeenachieved
yet).
DataflowProgramming
The most effective approach to dataflow is the one in which a dataflow machine
serves as an accelerator. Only such approaches are the subject of the rest of
this text. This means that the basic application continues to run on the classical
controlflow machine; only the execution of time-consuming loops gets moved to
the dataflow accelerator. This goal is implemented as follows: loops that satisfy
specific conditions get extracted, skins get inserted instead (send data and receive
results),andthenthekernelsgetwritten(typically,oneperloopextracted),which
explainthesemanticsofthelooptothedataflowcompiler.
DataflowCompilation
One enters a dataflow compiler with one or more kernel programs (responsible
for algorithm), one manager program (responsible for communications), and a
few more utility programs (responsible for related logistics). In the first step, the
compiler generates an execution graph, which is characterized with edges that are
v
vi Preface
asshortaspossible,sincethatensuresgoodspeedandlowpowerconsumption.In
thesecondstep,theexecutiongraphsgettranslatedintoabinaryfilethatconfigures
thehardware.
ProgrammingMaxeler
One has to prepare the abovementioned programs (kernel(s), manager, etc.) in
a DSL (domain-specific language). If Maxeler dataflow machines are used, the
DSL to be used is Maxeler Java. Maxeler Java is a superset of the classical Java,
with extensions going into two different directions: (a) a large number of built-
in Java classes that enable the programmers to be more effective and (b) a new
set of variables; classical java variables, referred to here as software variables, are
responsibleforprocessesthattakeplaceatcompiletime;thenewsetofvariables,
referred to here as hardware variables, physically flow through the hardware and
producetheresultofinterestfortheapplicationdomain.
ApplicationIssues
Therearefourthingsthatonecando,inordertomaximizeperformance.Onecan
(a) reform the input data choreography, (b) utilize the internal pipelines generated
by the compiler in its effort to ensure the integrity of the code in conditions when
different data items use different data paths, (c) use the changes in the algorithm
(association,distribution,commutation),and(d)tunetheprecision,especiallyifthe
floating-pointcomputationisused.
Conclusion
Theeffectivenessofdataflowcomputingdependsonvariousfactors:(a)thehigher
the contribution of loops to the overall code run time, the higher the acceleration
that the dataflow concept brings, especially if the level of data reusability within
loopishighandifdataarrivesequentiallyinastream,and(b)themoreskillfulthe
programmer,thebettertheutilizationoftheissuesdiscussedearlierinthistext.In
conclusion,theprogrammingofdataflowmachinesisnotmoredifficult;onlymore
up-fronttrainingisneeded.
Belgrade,Serbia VeljkoMilutinovic
July2017
Contents
PartI Algorithms
1 ImplementingNeuralNetworksbyUsingtheDataFlowParadigm.... 3
1.1 Introduction ............................................................. 3
1.2 ExistingSolutions....................................................... 4
1.3 EssenceofthedataflowParadigm...................................... 6
1.4 ArtificialNeuralNetworks ............................................. 11
1.5 SingleLayerPerceptron ................................................ 14
1.5.1 Algorithm ....................................................... 14
1.5.2 ThedataflowImplementation.................................. 18
1.6 MultilayerPerceptronAlgorithm....................................... 27
1.6.1 BackpropagationAlgorithm.................................... 28
1.6.2 ThedataflowImplementation.................................. 31
1.7 PerformanceEvaluation ................................................ 39
1.7.1 SingleLayerPerceptron........................................ 39
1.7.2 MultilayerPerceptron .......................................... 40
1.8 Conclusion .............................................................. 42
References..................................................................... 44
PartII Applications
2 SolvingthePoissonEquationbyUsingDataFlowTechnology.......... 47
2.1 Introduction ............................................................. 47
2.2 MathematicalBackground.............................................. 48
2.2.1 BoundaryConditions........................................... 50
2.3 OverviewofExistingSolutions ........................................ 51
2.3.1 FourierTechniques ............................................. 52
2.3.2 IterativeTechniques ............................................ 56
2.3.3 MultigridTechniques........................................... 63
vii
viii Contents
2.4 OverviewoftheGoalSolution......................................... 65
2.4.1 MaxelerdataflowComputingTechnologyConcept........... 65
2.4.2 ConditionsofApplicability .................................... 68
2.4.3 ImplementationDetails......................................... 68
2.4.4 UserGuide...................................................... 79
2.5 Measurements,Results,andPerformances............................ 81
2.6 PracticalApplicability.................................................. 86
2.6.1 Astrophysics .................................................... 87
2.6.2 Electrostatics.................................................... 87
2.6.3 BiologyandRelatedDisciplines............................... 88
2.6.4 TeamBehaviorAnalysis........................................ 88
2.6.5 OtherApplications.............................................. 89
2.7 Conclusion .............................................................. 90
References..................................................................... 91
3 BinarySearchintheDataFlowParadigm ................................ 93
3.1 Introduction ............................................................. 93
3.2 SystemRealizationRequirements...................................... 94
3.2.1 TheProblem .................................................... 94
3.2.2 AboutMaxelerTechnology .................................... 94
3.2.3 ConditionsandAssumptions................................... 95
3.2.4 ExistingSolutions .............................................. 95
3.3 AlgorithmDescription.................................................. 95
3.3.1 DataPreparation................................................ 95
3.3.2 BinarySearchAlgorithm....................................... 95
3.4 DataFlowAlgorithmRealization....................................... 97
3.4.1 KernelCode..................................................... 97
3.4.2 CCode .......................................................... 99
3.4.3 ManagerCode .................................................. 102
3.5 PerformanceAnalysis................................................... 104
3.5.1 TestResults ..................................................... 104
3.5.2 LongerInputSequenceAnalysis............................... 105
3.5.3 BottleneckAnalysis ............................................ 105
3.6 Conclusion .............................................................. 106
References..................................................................... 107
PartIII Implementations
4 IntroductoryOverviewonImplementationTools........................ 111
4.1 IntroductiontodataflowProgramming ................................ 111
4.2 DataFlowEngines(DFEs)andMaxJProgramming .................. 112
4.3 PortingaProgramtodataflowSystem................................. 113
4.3.1 ProgramAnalysis............................................... 115
4.3.2 EvaluatingDifferentArchitecturalOptions ................... 117
4.3.3 ImplementingtheKernels...................................... 117
Contents ix
4.3.4 IntegrationoftheAcceleratedKernelsBackintothe
OriginalProgram ............................................... 117
4.3.5 DebuggingandOptimizingtheImplementationtoGet
MaximalPerformance.......................................... 118
4.4 Interfaces ................................................................ 118
4.4.1 SAPISingle-DFESingle-ActionInterface.................... 119
4.4.2 MAPIMultipleActionInterface............................... 119
4.4.3 DAPIDFECreationAPI....................................... 119
4.5 ToolsOverview.......................................................... 120
4.5.1 Maxtop .......................................................... 120
4.5.2 MaxIDE......................................................... 120
4.5.3 MaxPower....................................................... 120
4.5.4 MonitoringDashboard.......................................... 121
4.5.5 Maxdebug....................................................... 121
4.5.6 Sliccompile...................................................... 123
4.5.7 MaxSkins(DFEsSpeakYourLanguage)...................... 124
4.5.8 MaxOrchestrator................................................ 125
References..................................................................... 125
5 DataFlowSystems:FromTheirOriginstoFutureApplications
inDataAnalytics,DeepLearning,andtheInternetofThings.......... 127
5.1 Introduction ............................................................. 127
5.2 High-PerformanceComputingTechnology............................ 128
5.2.1 StateoftheControl-FlowTechnology......................... 128
5.2.2 Special-PurposeSystems....................................... 129
5.2.3 DataFlowSystems.............................................. 129
5.2.4 DataFlowLanguages ........................................... 131
5.2.5 SystolicArchitecture ........................................... 131
5.2.6 MultiscaledataflowComputingApproach .................... 133
5.3 IdeasforNewApplicationsofdataflowSystems...................... 137
5.3.1 SoftwareDistributeddataflowSystemsforBigData
Analytics ........................................................ 137
5.3.2 IntegrationofMDCSystemwithApacheSpark.............. 138
5.3.3 DeepLearningApplications ................................... 142
5.3.4 TheInternetofThingsApplications........................... 144
5.4 Conclusion .............................................................. 146
References..................................................................... 147
Index............................................................................... 149