Neuromorphic Cognitive Systems: A Learning and Memory Centered Approach

180 Pages·2017·9.493 MB·English

by  Qiang Yu, Huajin Tang, Jun Hu, Kay Tan Chen (auth.)

Intelligent Systems Reference Library 126
Qiang Yu
Huajin Tang
Jun Hu
Kay Chen Tan
Neuromorphic
Cognitive
Systems
A Learning and Memory Centered
Approach
Intelligent Systems Reference Library
Volume 126
Series editors
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
e-mail: [email protected]
Lakhmi C. Jain, University of Canberra, Canberra, Australia;
Bournemouth University, UK;
KES International, UK
e-mail: [email protected]; [email protected];
URL: http://www.kesinternational.org/organisation.php
About this Series
The aim of this series is to publish a Reference Library, including novel advances
and developments in all aspects of Intelligent Systems in an easily accessible and
well structured form. The series includes reference works, handbooks, compendia,
textbooks,well-structuredmonographs,dictionaries,andencyclopedias.Itcontains
well integrated knowledge and current information in the field of Intelligent
Systems. The series covers the theory, applications, and design methods of
IntelligentSystems.Virtuallyalldisciplinessuchasengineering,computerscience,
avionics, business, e-commerce, environment, healthcare, physics and life science
are included.
More information about this series at http://www.springer.com/series/8578
Qiang Yu Huajin Tang Jun Hu
(cid:129) (cid:129)
Kay Chen Tan
Neuromorphic Cognitive
Systems
A Learning and Memory Centered Approach
123
QiangYu Jun Hu
Institute for Infocomm Research AGI Technologies
Singapore Singapore
Singapore Singapore
HuajinTang Kay ChenTan
Collegeof Computer Science Department ofComputer Science
SichuanUniversity City University of HongKong
Chengdu KowloonTong
China Hong Kong
ISSN 1868-4394 ISSN 1868-4408 (electronic)
Intelligent Systems Reference Library
ISBN978-3-319-55308-5 ISBN978-3-319-55310-8 (eBook)
DOI 10.1007/978-3-319-55310-8
LibraryofCongressControlNumber:2017933943
©SpringerInternationalPublishingAG2017
Reuse of the materials of “Cheu, E.Y., Yu, J., Tan, C.H. and Tang, H. Synaptic conditions for
auto-associativememorystorageandpatterncompletioninJensenetal.’smodelofhippocampalarea
CA3.JournalofComputationalNeuroscience,2012,33(3):435–447”,withpermissionfromSpringer
Publisher.
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
To our family, for their loves and supports.
Qiang Yu
Huajin Tang
Jun Hu
Kay Chen Tan
Preface
Thepowerfulandyetmysterioushumanbrainsystemattractsnumerousresearchers
devoting themselves to characterizing what nervous systems do, determining how
they function, and understanding why they operate in particular ways.
Encompassing various studies of biology, physics, psychology, mathematics, and
computer science, theoretical neuroscience provides a quantitative basis for
uncoveringthegeneralprinciplesbywhichthenervous systems operate.Basedon
these principles, neuromorphic cognitive systems introduce some basic mathe-
matical and computational methods to describe and utilize schemes at a cognitive
level. Since the mechanisms how human memory cognitively operates and how to
utilize the bioinspired mechanisms to practical applications are rarely known, the
study of neuromorphic cognitive systems is urgently demanded.
This book presents the computational principles underlying spike-based infor-
mationprocessingandcognitivecomputationwithaspecificfocusonlearningand
memory. Specifically, the action potential timing is utilized for sensory neuronal
representations and computation, and spiking neurons are considered as the basic
information processing unit. The topics covered in this book vary from neuronal
level to system level, including neural coding, learning in both single- and multi-
layered networks, cognitive memory, and applied developments of information
processing systems with spiking neurons. From the neuronal level, synaptic
adaptation plays an important role on learning patterns. In order to perform higher
level cognitive functions such as recognition and memory, spiking neurons with
learningabilitiesareconsistentlyintegratedwitheachother,buildingasystemwith
the functionality of encoding, learning, and decoding. All these aspects are
described with details in this book.
Theories, concepts, methods, and applications are provided to motivate
researchers in this exciting and interdisciplinary area. Theoretical modeling and
analysisaretightlyboundedwithpracticalapplications,whichwouldbepotentially
vii
viii Preface
beneficial for readers in the area of neuromorphic computing. This book presents
the computational ability of bioinspired systems and gives a better understanding
of the mechanisms by which the nervous system might operate.
Singapore Qiang Yu
Chengdu, China Huajin Tang
Singapore Jun Hu
Kowloon Tong, Hong Kong Kay Chen Tan
December 2016
Contents
1 Introduction.... .... .... ..... .... .... .... .... .... ..... .... 1
1.1 Background .... .... ..... .... .... .... .... .... ..... .... 1
1.2 Spiking Neurons. .... ..... .... .... .... .... .... ..... .... 2
1.2.1 Biological Background ... .... .... .... .... ..... .... 2
1.2.2 Generations of Neuron Models. .... .... .... ..... .... 3
1.2.3 Spiking Neuron Models .. .... .... .... .... ..... .... 4
1.3 Neural Codes ... .... ..... .... .... .... .... .... ..... .... 5
1.3.1 Rate Code.... ..... .... .... .... .... .... ..... .... 6
1.3.2 Temporal Code..... .... .... .... .... .... ..... .... 7
1.3.3 Temporal Code Versus Rate Code .. .... .... ..... .... 7
1.4 Cognitive Learning and Memory in the Brain ... .... ..... .... 8
1.4.1 Temporal Learning .. .... .... .... .... .... ..... .... 8
1.4.2 Cognitive Memory in the Brain .... .... .... ..... .... 11
1.5 Objectives and Contributions .... .... .... .... .... ..... .... 12
1.6 Outline of the Book .. ..... .... .... .... .... .... ..... .... 14
References.. .... .... .... ..... .... .... .... .... .... ..... .... 15
2 Rapid Feedforward Computation by Temporal Encoding
and Learning with Spiking Neurons . .... .... .... .... ..... .... 19
2.1 Introduction .... .... ..... .... .... .... .... .... ..... .... 19
2.2 The Spiking Neural Network .... .... .... .... .... ..... .... 21
2.3 Single-Spike Temporal Coding... .... .... .... .... ..... .... 22
2.4 Temporal Learning Rule.... .... .... .... .... .... ..... .... 26
2.4.1 The Tempotron Rule. .... .... .... .... .... ..... .... 26
2.4.2 The ReSuMe Rule... .... .... .... .... .... ..... .... 27
2.4.3 The Tempotron-Like ReSuMe Rule . .... .... ..... .... 28
2.5 Simulation Results ... ..... .... .... .... .... .... ..... .... 29
2.5.1 The Data Set and the Classification Problem... ..... .... 29
2.5.2 Encoding Images.... .... .... .... .... .... ..... .... 30
2.5.3 Choosing Among Temporal Learning Rules... ..... .... 30
ix
x Contents
2.5.4 The Properties of Tempotron Rule .. .... .... ..... .... 32
2.5.5 Recognition Performance . .... .... .... .... ..... .... 34
2.6 Discussion . .... .... ..... .... .... .... .... .... ..... .... 37
2.6.1 Encoding Benefits from Biology.... .... .... ..... .... 37
2.6.2 Types of Synapses .. .... .... .... .... .... ..... .... 37
2.6.3 Schemes of Readout . .... .... .... .... .... ..... .... 38
2.6.4 Extension of the Network for Robust Sound
Recognition... ..... .... .... .... .... .... ..... .... 38
2.7 Conclusion. .... .... ..... .... .... .... .... .... ..... .... 39
References.. .... .... .... ..... .... .... .... .... .... ..... .... 39
3 A Spike-Timing Based Integrated Model for Pattern
Recognition .... .... .... ..... .... .... .... .... .... ..... .... 43
3.1 Introduction .... .... ..... .... .... .... .... .... ..... .... 43
3.2 The Integrated Model. ..... .... .... .... .... .... ..... .... 45
3.2.1 Neuron Model and General Structure .... .... ..... .... 45
3.2.2 Latency-Phase Encoding.. .... .... .... .... ..... .... 46
3.2.3 Supervised Spike-Timing Based Learning. .... ..... .... 49
3.3 Numerical Simulations ..... .... .... .... .... .... ..... .... 50
3.3.1 Network Architecture and Encoding
of Grayscale Images . .... .... .... .... .... ..... .... 51
3.3.2 Learning Performance.... .... .... .... .... ..... .... 52
3.3.3 Generalization Capability . .... .... .... .... ..... .... 53
3.3.4 Parameters Evaluation.... .... .... .... .... ..... .... 55
3.3.5 Capacity of the Integrated System... .... .... ..... .... 57
3.4 Related Works .. .... ..... .... .... .... .... .... ..... .... 59
3.5 Conclusions .... .... ..... .... .... .... .... .... ..... .... 60
References.. .... .... .... ..... .... .... .... .... .... ..... .... 61
4 Precise-Spike-Driven Synaptic Plasticity for Hetero
Association of Spatiotemporal Spike Patterns.. .... .... ..... .... 65
4.1 Introduction .... .... ..... .... .... .... .... .... ..... .... 65
4.2 Methods... .... .... ..... .... .... .... .... .... ..... .... 67
4.2.1 Spiking Neuron Model ... .... .... .... .... ..... .... 67
4.2.2 PSD Learning Rule.. .... .... .... .... .... ..... .... 68
4.3 Results .... .... .... ..... .... .... .... .... .... ..... .... 71
4.3.1 Association of Single-Spike and Multi-spike Patterns. .... 71
4.3.2 Generality to Different Neuron Models... .... ..... .... 76
4.3.3 Robustness to Noise . .... .... .... .... .... ..... .... 77
4.3.4 Learning Capacity... .... .... .... .... .... ..... .... 79
4.3.5 Effects of Learning Parameters . .... .... .... ..... .... 81
4.3.6 Classification of Spatiotemporal Patterns.. .... ..... .... 83
4.4 Discussion and Conclusion.. .... .... .... .... .... ..... .... 84
References.. .... .... .... ..... .... .... .... .... .... ..... .... 86