Neuro-inspired Information Processing

236 Pages·2020·11.475 MB·English

by  Alain Cappy

Neuro-inspired Information Processing
To my mentors, Professors Georges Salmer
and Eugène Constant, who passed on to me their
passion for research into semiconductor device physics

To Nadine, Hélène and Pierre
Series Editor
Robert Baptist
Neuro-inspired
Information Processing

Alain Cappy
First published 2020 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers,
or in the case of reprographic reproduction in accordance with the terms and licenses issued by the
CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the
undermentioned address:
ISTE Ltd John Wiley & Sons, Inc.
27-37 St George’s Road 111 River Street
London SW19 4EU Hoboken, NJ 07030
UK USA
www.iste.co.uk www.wiley.com

© ISTE Ltd 2020
The rights of Alain Cappy to be identified as the author of this work have been asserted by him in
accordance with the Copyright, Designs and Patents Act 1988.
Library of Congress Control Number: 2019957598

British Library Cataloguing-in-Publication Data
A CIP record for this book is available from the British Library
ISBN 978-1-78630-472-8
Contents
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Chapter 1. Information Processing . . . . . . . . . . . . . . . . . . . . . . . 1
1.1. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1. Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2. Memorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2. Information processing machines . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1. The Turing machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.2. von Neumann architecture . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.3. CMOS technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.2.4. Evolution in microprocessor performance . . . . . . . . . . . . . . . 14
1.3. Information and energy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.3.1. Power and energy dissipated in
CMOS gates and circuits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.4. Technologies of the future . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.4.1. Evolution of the “binary
coding/von Neumann/CMOS” system . . . . . . . . . . . . . . . . . . . . . 27
1.4.2. Revolutionary approaches . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.5. Microprocessors and the brain . . . . . . . . . . . . . . . . . . . . . . . . . 40
1.5.1. Physical parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
1.5.2. Information processing . . . . . . . . . . . . . . . . . . . . . . . . . . 43
1.5.3. Memorization of information . . . . . . . . . . . . . . . . . . . . . . . 45
1.6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
vi Neuro-inspired Information Processing
Chapter 2. Information Processing in the Living . . . . . . . . . . . . . . 47
2.1. The brain at a glance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.1.1. Brain functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.1.2. Brain anatomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.2. Cortex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.2.1. Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.2.2. Hierarchical organization of the cortex . . . . . . . . . . . . . . . . . 52
2.2.3. Cortical columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.2.4. Intra- and intercolumnar connections . . . . . . . . . . . . . . . . . . 55
2.3. An emblematic example: the visual cortex . . . . . . . . . . . . . . . . . 57
2.3.1. Eye and retina . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.3.2. Optic nerve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.3.3. Cortex V1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.3.4. Higher level visual areas V2, V3,
V4, V5 and IT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.3.5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
2.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Chapter 3. Neurons and Synapses . . . . . . . . . . . . . . . . . . . . . . . 67
3.1. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.1.1. Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.1.2. Synapses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2. Cell membrane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.1. Membrane structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.2. Intra- and extracellular media . . . . . . . . . . . . . . . . . . . . . . . 74
3.2.3. Transmembrane proteins . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3. Membrane at equilibrium. . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.3.1. Resting potential, V . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
r
3.4. The membrane in dynamic state . . . . . . . . . . . . . . . . . . . . . . . . 85
3.4.1. The Hodgkin–Huxley model . . . . . . . . . . . . . . . . . . . . . . . 89
3.4.2. Beyond the Hodgkin–Huxley model . . . . . . . . . . . . . . . . . . 100
3.4.3. Simplified HH models . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
3.4.4. Application of membrane models . . . . . . . . . . . . . . . . . . . . 111
3.5. Synapses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
3.5.1. Biological characteristics . . . . . . . . . . . . . . . . . . . . . . . . . 122
3.5.2. Synaptic plasticity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
3.6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Contents vii
Chapter 4. Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . 129
4.1. Software neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
4.1.1. Neuron and synapse models. . . . . . . . . . . . . . . . . . . . . . . . 130
4.1.2. Artificial Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . 133
4.1.3. Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
4.1.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
4.2. Hardware neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
4.2.1. Comparison of the physics of biological
systems and semiconductors . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
4.2.2. Circuits simulating the neuron . . . . . . . . . . . . . . . . . . . . . . 154
4.2.3. Circuits simulating the synapse . . . . . . . . . . . . . . . . . . . . . . 189
4.2.4. Circuits for learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
4.2.5. Examples of hardware neural networks . . . . . . . . . . . . . . . . . 201
4.3. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
Acknowledgments
I wish to thank my colleagues, Virginie Hoel, Christophe Loyez,
François Danneville, Kevin Carpentier and Ilias Sourikopoulos, who
have accompanied my work on neuro-inspired information processing.
This book would not have been possible without our numerous discussions
on this new research theme.
I would also like to thank Marie-Renée Friscourt for her diligent
and efficient proofreading of the manuscript, and for the many insightful
remarks made for the benefit of its improvement.
Introduction
The invention of the junction transistor in 1947 was undoubtedly the
most significant innovation of the 20th Century, with our day-to-day lives
coming to entirely depend on it. Since this date, which we will come back to
later, the world has “gone digital”, with virtually all information processed
in binary form by microprocessors.
In order to attain the digital world we know today, several steps were
essential, such as the manufacture of the first integrated circuit in 1958. It
soon became apparent that integrated circuits not only enabled the
processing of analog signals, such as those used in radio, but also digital
signals. Such digital circuits were used in the Apollo XI mission that led
humankind onto the moon, on July 21, 1969. Astronauts only had very
limited computing means at their disposal to achieve this spectacular feat.
The flight controller was a machine that we might consider very basic by
today’s standards. Composed of 2,800 integrated circuits, each comprising
two three-input “NOR” gates, 2,048 words RAM1 and 38,000 words ROM2
for programs, it worked at a clock frequency of 80 kHz and weighed no
more than 32 kg for 55 W power consumption.
The exploit was thus essentially based on “human” or “cortical”
processing of information: processing power, too often advanced today, is
not always the sine qua non condition for success!
In order to reduce the weight of processing systems, while improving
their performance, it is necessary to incorporate a large number of logic

1 Memory that can be both written to and read from.
2 Read-only memory.