memristive crossbar arrays for machine learning systems

105 Pages·2015·4.34 MB·English

MEMRISTIVE CROSSBAR ARRAYS
FOR MACHINE LEARNING
SYSTEMS
AthesissubmittedtotheUniversityofManchesterforthedegreeof
MasterofPhilosophy
inthe
FacultyofScienceandEngineering
by
MANU V NAIR
2015
SchoolofElectricalandElectronicsEngineering
Contents
Abstract 7
Declaration 8
Copyright 9
Acknowledgements 10
TheAuthor 11
1 Introduction 12
2 Hardwareneuralnetworks 15
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2 NeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3 DigitalversusAnalogueNeuralNetworks . . . . . . . . . . . . . . . . . . 18
2.4 HardwareNeuralNetworkArchitectures . . . . . . . . . . . . . . . . . . . 20
2.4.1 Pulse-streamarithmeticbasednetworks . . . . . . . . . . . . . . . 20
2.4.2 TrueNorth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.3 SpiNNaker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.4.3.1 Nodearchitecture . . . . . . . . . . . . . . . . . . . . . 28
2.4.3.2 Eventdrivenoperation . . . . . . . . . . . . . . . . . . . 29
2.4.3.3 Networkcommunication . . . . . . . . . . . . . . . . . 29
2.4.3.4 NeuronandSynapsemodel . . . . . . . . . . . . . . . . 30
2.4.4 Neurogrid . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4.4.1 Shareddendritestructure . . . . . . . . . . . . . . . . . 31
2.4.4.2 NeuronandSynapse . . . . . . . . . . . . . . . . . . . . 32
1
CONTENTS 2
2.4.4.3 Communication . . . . . . . . . . . . . . . . . . . . . . 33
2.5 Programmingneuromorphichardware . . . . . . . . . . . . . . . . . . . . 35
2.6 Othernoteworthyneuromorphicsystems . . . . . . . . . . . . . . . . . . . 35
2.7 ConcludingRemarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3 MemristiveLearning 39
3.1 Memristors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.1.1 Boundaryconditionmodel . . . . . . . . . . . . . . . . . . . . . . 41
3.2 CrossbarArrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3 Memristorcircuitsandsystems . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.1 Crossbars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3.2 Memristorprogrammingschemes . . . . . . . . . . . . . . . . . . 46
3.3.2.1 Unregulatedwrite . . . . . . . . . . . . . . . . . . . . . 46
3.3.2.2 Regulatedwrite . . . . . . . . . . . . . . . . . . . . . . 48
3.3.3 STDP-basedalgorithmsinmemristivecrossbararrays . . . . . . . 51
3.3.4 Back-propagationalgorithm . . . . . . . . . . . . . . . . . . . . . 53
3.3.5 Dynamicalsystemsandothercircuitapplications . . . . . . . . . . 54
4 Gradient-descentincrossbararrays 55
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2 Gradientdescentalgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.3 Gradientdescentforlinearclassifiers . . . . . . . . . . . . . . . . . . . . . 58
4.4 Gradientdescentincrossbararrays . . . . . . . . . . . . . . . . . . . . . . 59
4.5 Unregulatedstepdescent . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.6 USDversusothermethods . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.7 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.8 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.8.1 SimulationSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.8.2 Initialconditionanalysis . . . . . . . . . . . . . . . . . . . . . . . 67
4.8.3 Effectofdeviceparametersonperformance . . . . . . . . . . . . . 69
4.8.4 Effectofvariabilityonperformance . . . . . . . . . . . . . . . . . 72
4.8.5 Comparisonagainstfloatingpointimplementation . . . . . . . . . 73
4.8.6 PerformanceontheMNISTdatabase . . . . . . . . . . . . . . . . 74
CONTENTS 3
4.9 USDforotheralgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.9.1 MatrixInversion . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.9.2 Auto-encoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.9.3 RestrictedBoltzmannMachines . . . . . . . . . . . . . . . . . . . 77
4.10 Concludingremarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5 Conclusion 80
Bibliography 83
A NeuralNetworksandrelatedalgorithms 93
A.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
A.2 Rosenblatt’sperceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
A.3 Hopfieldnetwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
A.4 BoltzmannMachines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
A.5 Theself-organizingmap(SOM) . . . . . . . . . . . . . . . . . . . . . . . 98
A.6 SpikingNeuralNetworks(SNN) . . . . . . . . . . . . . . . . . . . . . . . 100
List of Figures
2.1 Pulsestreamneuron[13](cid:13)c IEEE . . . . . . . . . . . . . . . . . . . . . . 22
2.2 Atransconductancemultiplier[8](cid:13)c IEEE . . . . . . . . . . . . . . . . . . 22
2.3 Self-timedasynchronouscommunicationscheme[8](cid:13)c IEEE . . . . . . . 23
2.4 Inputpulseprobabilityversusoutputpulseprobability . . . . . . . . . . . 24
2.5 TrueNortharchitecture[15](cid:13)c IEEE . . . . . . . . . . . . . . . . . . . . . 25
2.6 ASpiNNakernode[10](cid:13)c IEEE . . . . . . . . . . . . . . . . . . . . . . . 28
2.7 TheSpiNNakermachine[10](cid:13)c IEEE . . . . . . . . . . . . . . . . . . . . 30
2.8 Anetworkof4neuronsand16synapseswith1-to-1connectivity[11](cid:13)c IEEE
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.9 Ashareddendritenetworkof4neuronsand4synapses[11](cid:13)c IEEE . . . 32
2.10 ANeurogridneuron[11](cid:13)c IEEE . . . . . . . . . . . . . . . . . . . . . . 33
2.11 BlockdiagramofaNeurogridtree[11](cid:13)c IEEE . . . . . . . . . . . . . . . 34
3.1 Thenewcircuitelement: Memristor[3](cid:13)c IEEE . . . . . . . . . . . . . . 40
3.2 TreeandCrossbararchitectureusedinTeramac. [43](cid:13)c IEEE . . . . . . . 43
3.3 Crossbararrayofmemristors: 3-Dand2-Drepresentation . . . . . . . . . 44
3.4 A simple circuit schematic for (a) Unregulated writing into memristor and
(b)Readingthestateofthememristor . . . . . . . . . . . . . . . . . . . . 47
3.5 Schematicofacontinuousfeedbackwritescheme . . . . . . . . . . . . . 48
3.6 Sneakpathsinacrossbararray . . . . . . . . . . . . . . . . . . . . . . . . 49
3.7 Memristor-basedanaloguememory/computingunit[49](cid:13)c IEEE . . . . . . 50
3.8 A1T1Mcrossbararrayaccesstothetopleftelement[33](cid:13)c IEEE . . . . . 50
3.9 Segmentedcrossbararchitecture[50](cid:13)c IEEE . . . . . . . . . . . . . . . . 51
3.10 Voltage across the synapse ξ(∆T) for various action potential shapes [51]
(cid:13)c FrontiersofNeuroscience . . . . . . . . . . . . . . . . . . . . . . . . . 52
4
LISTOFFIGURES 5
4.1 Gradient descent for a 2-dimensional objective function. F(w ,w ) in the
0 1
figureisthesameasF(w,x,y). . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2 Blockdiagramoftrainingmodule[68] . . . . . . . . . . . . . . . . . . . 59
4.3 Convergence of USD algorithm when finding the minima of a paraboloid
function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.4 4-phasetrainingscheme. Thevoltagelevelsin thepulsingschemecanonly
takethreelevels: +V ,0,and−V [68] . . . . . . . . . . . . . . . . 64
drive drive
4.5 Convergencebehaviourfordifferentweightinitializations . . . . . . . . . 67
4.6 Weightupdatesversusiterationswheninitializedcloseto0. . . . . . . . . 68
4.7 Weightupdatesversusiterationswheninitializedto Gratio. . . . . . . . . . 68
2
4.8 Weightupdatesversusiterationswheninitializedcloseto1. . . . . . . . . 69
4.9 Weight updates versus iterations when initialized to uniformly distributed
randomvaluesbetween0and1. . . . . . . . . . . . . . . . . . . . . . . . 70
4.10 Evolutionofclassificationerrorwithiterations[68] . . . . . . . . . . . . . 71
4.11 Classification error versus G . The bands straddle the maximum and
ratio
minimumclassificationerrorbecauseofsettlingerroratconvergence[68] . 71
4.12 Settlingtime(Niters)versusG fordifferentvaluesofα(Alpha). [68] . 72
ratio
4.13 )ClassificationError(Error)versusσ(sigma). . . . . . . . . . . . . . . . 73
4.14 Number of training iterations (Niters) versus σ (Sigma). Training time in-
creasesasvariabilityincreases. [68](cid:13)c IEEE . . . . . . . . . . . . . . . . . 73
4.15 Effectof inthespreadofG (y-axis)andG (x-axis)valuesshownusing
on off
1000 memristive device samples with mean G = 0.003 S, Mean G =
off ratio
100. Notethatbothxandy-axisareinlog-scale. [68](cid:13)c IEEE . . . . . . . 74
4.16 PerformanceofthestochasticUSDimplementationincomparisontofloating-
pointlogisticregressionforvariousvaluesof p [68](cid:13)c IEEE . . . . . . . . 75
e
A.1 Theperceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
A.2 AHopfieldnetwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
A.3 BoltzmannmachinesandRestrictedBoltzmannmachines . . . . . . . . . 97
A.4 Aself-organizingmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
A.5 Asynapse. Theblocksshownarestate-dependentandcharacterizedbyvar-
iousdynamicaleffectsleadingtodifferenttypesofspikes . . . . . . . . . 100
A.6 IPSP,EPSP,andactionpotential . . . . . . . . . . . . . . . . . . . . . . . 101
LISTOFFIGURES 6
A.7 Data-typesandsymbolsusedintheLIFneuronmodel[9](cid:13)c IEEE . . . . . 102
A.8 STDPwaveformfortheactionpotentialshowninFigureA.6 . . . . . . . 103
Abstract
This thesis is a study of specialized circuits and systems targeted towards machine learning
algorithms. These systems operate on a computing paradigm that is different from tradi-
tional Von-Neumann architectures and can potentially reduce power consumption and im-
prove performance over traditional computers when running specialized tasks. In order to
study them, case studies covering implementations such as TrueNorth, SpiNNaker, Neuro-
grid, Pulse-stream based neural networks, and memristor-based systems, were done. The
use of memristive crossbar arrays for machine learning was found particularly interesting
andchosenastheprimaryfocusofthiswork.
This thesis presents an Unregulated Step Descent (USD) algorithm that can be used for
trainingmemristivecrossbararraystorunalgorithmsbasedongradient-descentlearning. It
describeshowtheUSDalgorithmcanaddresshardwarelimitationssuchasvariability,poor
devicemodels,complexityoftrainingarchitectures,etc. Thelinearclassifieralgorithmwas
primarily used in the experiments designed to study these features. This algorithm was
chosen because its crossbar architecture can easily be extended to larger networks. More
importantly, using a simple algorithm makes it easier to draw inferences from experimen-
tal results. Datasets used for these experiments included randomly generated data and the
MNIST digits dataset. The results indicate that performance of crossbar arrays that have
beentrainedusingtheUSDalgorithmisreasonablyclosetothatofthecorrespondingfloat-
ingpointimplementation. Theseexperimentalobservationsalsoprovideablueprintofhow
training and device parameters affect the performance of a crossbar array and how it might
be improved. The thesis also covers how other machine learning algorithms such as logis-
tic regressions, multi-layer perceptrons, and restricted Boltzmann machines may be imple-
mentedoncrossbararraysusingtheUSDalgorithm.
7
Declaration
No portion of the work referred to in the thesis has been submitted in support of an appli-
cation for another degree or qualification of this or any other university or other institute of
learning.
8
Copyright
1. The author of this thesis (including any appendices and/or schedules to this thesis)
owns any copyright in it (the Copyright) and he has given The University of Manch-
ester the right to use such Copyright for any administrative, promotional, educational
and/orteachingpurposes.
2. Copies of this thesis, either in full or in extracts, may be made only in accordance
with the regulations of the John Rylands University Library of Manchester. Details
oftheseregulationsmaybeobtainedfromtheLibrarian. Thispagemustformpartof
anysuchcopiesmade.
3. The ownership of any patents, designs, trade marks and any and all other intellectual
property rights except for the Copyright (the Intellectual Property Rights) and any
reproductions of copyright works, for example graphs and tables (Reproductions),
which may be described in this thesis, may not be owned by the author and may be
owned by third parties. Such Intellectual Property Rights and Reproductions cannot
and must not be made available for use without the prior written permission of the
owner(s)oftherelevantIntellectualPropertyRightsand/orReproductions.
4. Further information on the conditions under which disclosure, publication and ex-
ploitation of this thesis, the Copyright and any Intellectual Property Rights and/or
Reproductions described in it may take place is available from the Head of School
of Electrical and Electronic Engineering (or the Vice-President) and the Dean of the
FacultyofLifeSciences,forFacultyofLifeSciencescandidates.
9