Hardware Architectures for Deep Learning

329 Pages·2020·17.123 MB·English

by  Masoud Daneshtalab (editor), Mehdi Modarressi (editor)

IETMATERIALS,CIRCUITSANDDEVICESSERIES55
Hardware Architectures for
Deep Learning
Othervolumesinthisseries:
Volume2 AnalogueICDesign:Thecurrent-modeapproachC.Toumazou,F.J.Lidgeyand
D.G.Haigh(Editors)
Volume3 Analogue–Digital ASICs: Circuit techniques, design tools and applications
R.S.Soin,F.MalobertiandJ.France(Editors)
Volume4 Algorithmic and Knowledge-Based CAD for VLSI G.E. Taylor and G. Russell
(Editors)
Volume5 SwitchedCurrents:AnanaloguetechniquefordigitaltechnologyC.Toumazou,
J.B.C.HughesandN.C.Battersby(Editors)
Volume6 High-FrequencyCircuitEngineeringF.Nibleretal.
Volume8 Low-PowerHigh-FrequencyMicroelectronics:AunifiedapproachG.Machado
(Editor)
Volume9 VLSITesting:Digitalandmixedanalogue/digitaltechniquesS.L.Hurst
Volume10 Distributed Feedback Semiconductor Lasers J.E. Carroll, J.E.A. Whiteaway and
R.G.S.Plumb
Volume11 SelectedTopicsinAdvancedSolidStateandFibreOpticSensors
S.M.Vaezi-Nejad(Editor)
Volume12 StrainedSiliconHeterostructures:Materialsanddevices
C.K.Maiti,N.B.ChakrabartiandS.K.Ray
Volume13 RFICandMMICDesignandTechnologyI.D.RobertsonandS.Lucyzyn(Editors)
Volume14 DesignofHighFrequencyIntegratedAnalogueFiltersY.Sun(Editor)
Volume15 FoundationsofDigitalSignalProcessing: Theory, algorithmsandhardware
designP.Gaydecki
Volume16 WirelessCommunicationsCircuitsandSystemsY.Sun(Editor)
Volume17 TheSwitchingFunction:AnalysisofpowerelectroniccircuitsC.Marouchos
Volume18 SystemonChip:NextgenerationelectronicsB.Al-Hashimi(Editor)
Volume19 TestandDiagnosisofAnalogue,Mixed-SignalandRFIntegratedCircuits:The
systemonchipapproachY.Sun(Editor)
Volume20 Low Power and Low Voltage Circuit Design with the FGMOS Transistor
E.Rodriguez-Villegas
Volume21 TechnologyComputerAidedDesignforSi,SiGeandGaAsIntegratedCircuits
C.K.MaitiandG.A.Armstrong
Volume22 NanotechnologiesM.Wauteletetal.
Volume23 UnderstandableElectricCircuitsM.Wang
Volume24 Fundamentals of Electromagnetic Levitation: Engineering sustainability
throughefficiencyA.J.Sangster
Volume25 OpticalMEMSforChemicalAnalysisandBiomedicineH.Jiang(Editor)
Volume26 HighSpeedDataConvertersA.M.A.Ali
Volume27 Nano-ScaledSemiconductorDevicesE.A.Gutiérrez-D(Editor)
Volume28 SecurityandPrivacyforBigData,CloudComputingandApplicationsL.Wang,
W.Ren,K.R.ChooandF.Xhafa(Editors)
Volume29 Nano-CMOS and Post-CMOS Electronics: Devices and modelling Saraju P.
MohantyandAshokSrivastava
Volume30 Nano-CMOSandPost-CMOSElectronics:CircuitsanddesignSarajuP.Mohanty
andAshokSrivastava
Volume32 Oscillator Circuits: Frontiers in design, analysis and applications Y. Nishio
(Editor)
Volume33 HighFrequencyMOSFETGateDriversZ.ZhangandY.Liu
Volume34 RFandMicrowaveModuleLevelDesignandIntegrationM.Almalkawi
Volume35 DesignofTerahertzCMOSIntegratedCircuitsforHigh-SpeedWireless
CommunicationM.FujishimaandS.Amakawa
Volume38 SystemDesignwithMemristorTechnologiesL.GuckertandE.E.SwartzlanderJr.
Volume39 Functionality-EnhancedDevices:AnalternativetoMoore’slawP.-E.Gaillardon
(Editor)
Volume40 DigitallyEnhancedMixedSignalSystemsC.Jabbour,P.DesgreysandD.Dallett
(Editors)
Volume43 NegativeGroupDelayDevices: FromconceptstoapplicationsB.Ravelo
(Editor)
Volume45 Characterisation and Control of Defects in Semiconductors F. Tuomisto
(Editor)
Volume47 UnderstandableElectricCircuits:Keyconcepts,2ndEditionM.Wang
Volume51 ModellingMethodologiesinAnalogueIntegratedCircuitDesign
G.DundarandM.B.Yelten(Editors)
Volume53 VLSIArchitecturesforFutureVideoCodingM.Martina(Editor)
Volume54 AdvancesinHigh-PowerFiberandDiodeLaserEngineering
IvanDivliansky(Editor)
Volume58 Magnetorheological Materials and their Applications S. Choi and W. Li
(Editors)
Volume60 IPCoreProtectionandHardware-AssistedSecurityforConsumer
ElectronicsA.SenguptaandS.Mohanty
Volume64 Phase-Locked Frequency generation and Clocking: Architectures and
circuitsformodemwirelessandwirelinesystemsW.Rhee(Editor)
Volume67 FrontiersinSecuringIPCores:forensicdetectivecontrolandobfuscation
techniquesASengupta
Volume68 High-QualityLiquidCrystalDisplaysandSmartDevices:Vol.1andVol.2
S.Ishihara,S.KobayashiandY.Ukai(Editors)
Volume69 FibreBraggGratingsinHarshandSpaceEnvironments:Principlesand
applicationsB.Aïssa,E.I.Haddad,R.V.Kruzelecky,andW.R.Jamroz
Volume70 Self-HealingMaterials:Fromfundamentalconceptstoadvancedspace
andelectronicsapplications,2ndEditionB.Aïssa,E.I.Haddad,
R.V.Kruzelecky,andW.R.Jamroz
Volume71 RadioFrequencyandMicrowavePowerAmplifiers:Vol.1andVol.2
A.Grebennikov(Editor)
Volume73 VLSIandPost-CMOSElectronicsVolume1:VLSIandPost-CMOS
ElectronicsandVolume2:Materials,devicesandinterconnects
R.DhimanandR.Chandel(Editors)

Hardware Architectures for
Deep Learning
Edited by
Masoud Daneshtalab and Mehdi Modarressi
TheInstitutionofEngineeringandTechnology
Published by The Institution of Engineering and Technology, London, United Kingdom
The Institution of Engineering and Technology is registered as a Charity in England & Wales
(no. 211014) and Scotland (no. SC038698).
© The Institution of Engineering and Technology 2020
First published 2020
This publication is copyright under the Berne Convention and the Universal Copyright
Convention. All rights reserved. Apart from any fair dealing for the purposes of research
or private study, or criticism or review, as permitted under the Copyright, Designs and
Patents Act 1988, this publication may be reproduced, stored or transmitted, in any
form or by any means, only with the prior permission in writing of the publishers, or in
the case of reprographic reproduction in accordance with the terms of licenses issued
by the Copyright Licensing Agency. Enquiries concerning reproduction outside those
terms should be sent to the publisher at the undermentioned address:
The Institution of Engineering and Technology
Michael Faraday House
Six Hills Way, Stevenage
Herts, SG1 2AY, United Kingdom
www.theiet.org
While the authors and publisher believe that the information and guidance given in this
work are correct, all parties must rely upon their own skill and judgement when making
use of them. Neither the authors nor publisher assumes any liability to anyone for any
loss or damage caused by any error or omission in the work, whether such an error or
omission is the result of negligence or any other cause. Any and all such liability
is disclaimed.
The moral rights of the authors to be identified as authors of this work have been
asserted by them in accordance with the Copyright, Designs and Patents Act 1988.
British Library Cataloguing in Publication Data
A catalogue record for this product is available from the British Library
ISBN 978-1-78561-768-3 (hardback)
ISBN 978-1-78561-769-0 (PDF)
Typeset in India by MPS Limited
Contents
Abouttheeditors xv
Preface xvii
Acknowledgments xxi
PartI Deeplearningandneuralnetworks:conceptsandmodels 1
1 Anintroductiontoartificialneuralnetworks 3
AhmadKalhor
1.1 Introduction 3
1.1.1 NaturalNNs 3
1.1.2 Artificialneuralnetworks 6
1.1.3 PreliminaryconceptsinANNs 7
1.2 ANNsinclassificationandregressionproblems 11
1.2.1 ANNsinclassificationproblems 11
1.2.2 ANNsinregressionproblems 12
1.2.3 Relationbetweenclassificationandregression 13
1.3 WidelyusedNNmodels 14
1.3.1 Simplestructurenetworks 14
1.3.2 MultilayeranddeepNNs 16
1.4 Convolutionalneuralnetworks 20
1.4.1 Convolutionlayers 21
1.4.2 Poolinglayers 22
1.4.3 LearninginCNNs 23
1.4.4 CNNexamples 24
1.5 Conclusion 25
References 25
2 Hardwareaccelerationforrecurrentneuralnetworks 27
SimaSinaeiandMasoudDaneshtalab
2.1 Recurrentneuralnetworks 28
2.1.1 Longshort-termmemory 30
2.1.2 Gatedrecurrentunits 36
viii Hardwarearchitecturesfordeeplearning
2.2 HardwareaccelerationforRNNinference 37
2.2.1 Softwareimplementation 37
2.2.2 Hardwareimplementation 38
2.3 HardwareimplementationofLSTMs 39
2.3.1 Modelcompression 40
2.3.2 DatatypeandQuantization 44
2.3.3 Memory 46
2.4 Conclusion 48
References 48
3 Feedforwardneuralnetworksonmassivelyparallelarchitectures 53
RezaHojabr,AhmadKhonsari,MehdiModarressi,
andMasoudDaneshtalab
3.1 Relatedwork 55
3.2 Preliminaries 57
3.3 ClosNN:acustomizedClosforneuralnetwork 59
3.4 CollectivecommunicationsonClosNN 60
3.5 ClosNNcustomizationandareareduction 62
3.6 FoldedClosNN 65
3.7 Leafswitchoptimization 67
3.8 ScalingtolargerNoCs 67
3.9 Evaluation 68
3.9.1 Performancecomparisonundersynthetictraffic 69
3.9.2 Performanceevaluationunderrealisticworkloads 70
3.9.3 Powercomparison 71
3.9.4 Sensitivitytoneuralnetworksize 72
3.10 Conclusion 73
References 73
PartII Deeplearningandapproximatedatarepresentation 77
4 Stochastic-binaryconvolutionalneuralnetworkswithdeterministic
bit-streams 79
M.HassanNajafi,S.RasoulFaraji,BingzheLi,DavidJ.Lilja,
andKiaBazargan
4.1 Overview 79
4.2 Introduction 79
4.3 Background 81
4.3.1 Stochasticcomputing 81
4.3.2 Deterministiclow-discrepancybit-streams 82
4.3.3 Convolutionalneuralnetworks 84
4.4 Relatedwork 84
Contents ix
4.5 Proposedhybridbinary-bit-streamdesign 85
4.5.1 Multiplicationsandaccumulation 86
4.5.2 Handlingnegativeweights 86
4.6 Experimentalresults 88
4.6.1 Performancecomparison 88
4.6.2 Costcomparison 90
4.7 Summary 92
Acknowledgment 92
References 92
5 Binaryneuralnetworks 95
NajmehNazariandMostafaE.Salehi
5.1 Introduction 95
5.2 Binaryneuralnetworks 96
5.2.1 Binaryandternaryweightsforneuralnetworks 97
5.2.2 Binarizedandternarizedneuralnetworks 100
5.3 BNNoptimizationtechniques 109
5.4 HardwareimplementationofBNNs 111
5.5 Conclusion 112
References 113
PartIII Deeplearningandmodelsparsity 117
6 Hardwareandsoftwaretechniquesforsparsedeep
neuralnetworks 119
AliShafiee,LiuLiu,LeiWang,andJosephHassoun
6.1 Introduction 119
6.2 Differenttypesofsparsitymethods 120
6.3 Softwareapproachforpruning 122
6.3.1 Hardpruning 122
6.3.2 Softpruning,structuralsparsity,
andhardwareconcern 122
6.3.3 Questioningpruning 122
6.4 Hardwaresupportforsparsity 123
6.4.1 Advantagesofsparsityfordenseaccelerator 124
6.4.2 Supportingactivationsparsity 125
6.4.3 Supportingweightsparsity 125
6.4.4 Supportingbothweightandactivationsparsity 131
6.4.5 Supportingoutputsparsity 137
6.4.6 Supportingvaluesparsity 140
6.5 Conclusion 143
References 143