Hardware Annealing in Analog VLSI Neurocomputing

234 Pages·1991·0.27 MB·English

by  Bang W. Lee

HARDWARE ANNEALING
IN ANALOG VLSI
NEUROCOMPUTING
THE KLUWER INTERNATIONAL SERIES
IN ENGINEERING AND COMPUTER SCIENCE
VLSI, COMPUTER ARCIllTECfURE AND
DIGITAL SIGNAL PROCESSING
Consulting Editor
Jonathan Allen
Wafer Level Integrated Systelll<i: Implementation Issues, S.K. Tewksbury,
ISBN: 0-7923-9003
The Annealing Algorithm, R.Hl.M. Otten, L.P.P. van Ginneken,
ISBN: 0-7923-9022-9
UnifIed Methods for VLSI Simulation and Test Generation, K.T. Cheng,
V.D. Agrawal, ISBN: 0-7923-9025-3
ASIC System Design With VHDL: A Paradigm, S.S. Leung, M.A. Shanblatt,
ISBN: 0-7923-9032-6
BiCMOS Technology and Applications, A. R. Alvarez, Editor
ISBN: 0-7923-9033-4
Analog VLSI ImpleDJentationofNeural Systems, C. Mead, M.Ismail (Editors),
ISBN: 0-7923-9040-7
The MIPS-X RISC Mkroprocessor, P. Chow. ISBN: 0-7923-9045-8
Nonlinear Digital Filters: Principles and Applications, I . Pitas, A.N.
Venetsanopoulos, ISBN: 0-7923-9049-0
Algorithmk and Register-Transfer Level Synthesis: The System Architect's
Workbench, D.E. Thomas, E.D. Lagnese, R.A Walker, J.A. Nestor, J.V. Ragan,
R.L.B1ackbum, ISBN: 0-7923-9053-9
VLSI Design for Manufacturing: Yield Enhancement, S.W. . Director, W. Maly,
A.J. Strojwas, ISBN: 0-7923-9053-7
Testing and Reliable Design of CMOS Circuits, N.K. Jha, S. Kundu,
ISBN: 0-7923-9056-3
Hierarchkal Modeling for VLSI Circuit Testing, D. Bhattacharya, J.P.
Hayes, ISBN: 0-7923-9058-X
Steady-State Methods for Simulating Analog and Mkrowave Circuits,
K. Kundert, A Sangiovanni-Vincentelli, J. White,
ISBN: 0-7923-9069-5
Introduction to Analog VLSI Design Automation, M. Ismail, J. Franca,
ISBN: 0-7923-9102-0
Gallium Arsentide Digital Circuits, O. Wing, ISBN: 0-7923-9081-4
Principles ofVLSI System Planning, AM. Dewey ISBN: 0-7923-9102
Mixed-Mode Simulation, R. Saleh, A.R. Newton, ISBN: 0-7923-9107-1
Automatic Programming Applied to VLSI CAD Software: A Case Study,
D. Setliff, R.A. Rutenbar, ISBN: 0-7923-9112-8
Models for Large Integrated Circuits, P. Dewilde, Z.O. Ning
ISBN: 0-7923-9115-2
Hardware Design and Simulation in VAUVHDL, L.M. Augustin, D.C..Luckham,
B.A.Gennart, Y.Huh, AG.Stanculescu ISBN: 0-7923-9087-3
Subband Image Coding, J. Woods, editor, ISBN: 0-7923-9093-8
Low-Noise Wide-Band Amplif"Iers in Bipolar and CMOTechnologies,Z.Y.Chang,
W.M.C.Sansen, ISBN: 0-7923-9096-2
Iterative Identification and Restoration Images, R. LLagendijk, J. Biemond
ISBN: 0-7923-9097-0
VLSI Design of Neural Networks, U. Ramacher, U. Ruckert
ISBN: 0-7923-9127-6
HARDWARE ANNEALING
IN ANALOG VLSI
NEUROCOMPUTING
by
BangW. Lee
University of Southern California
and
Bing J. Sheu
University of Southern California
~.
"
Springer-Science+Business Media, LLC
Libnry of Coogress Catalogiog-io-PubUcatioo Data
Lee, Bang W.
Hardware annealing in analog VLSI neurocomputing / by Bang W. Lee
and Bing J. Sheu.
p. cm. - (The Kluwer international series in engineering and
computer science. VLSI, computer architecture, and digital signal
processing)
Includes bibliographical references and index.
ISBN 978-1-4613-6780-2 ISBN 978-1-4615-3984-1 (eBook)
DOI 10.1007/978-1-4615-3984-1
1. Neural networks (Computer science) 2. Neural computers
-Circuits. 3. Integrated circuits-Very large scale integration.
4. Simulated annealing (Mathematics) 1. Sheu, Bing Jay.
II. Title. III. Series.
QA76.87.L44 1991
006.3-dc20 90-19256
CIP
Copyright © 1991 by Springer Science+Business Media New York
Originally published by Kluwer Academic Publishers in 1991
Softcover reprint of the hardcover 1s t edition 1991
Ali rights reserved. No part of this publication may be reproduced, stored in a retrieval
system or transmitted in any form or by any means, mechanical, photo-copying, recording,
or otherwise, without the prior written permission of the publisher, Springer-Science+
Business Media, LLC
Printed on acid-free paper.
Table of Contents
L1· St 0 fF·1 9ures..... .................................... ....... ............................. ... ... V.I.I.I
List of Tables .... .... .... .... ........ .... .... ............ .... .... .... .... .... .... ..... ....... ..... XlII
Preface ................................................................................................ xv
Acknowledgement ........ .... .... .... .... .... ........ ........ .... .... ................. ........ XIX
1. Introduction ................................................................................ 1
1.1 Overview of Neural Architectures .... ..... .... ... ........ ..... ....... ..... 4
1.2 VLSI Neural Network Design Methodology.... .... ..... ............ 11
2. VLSI Hopfield Networks ........................................ ............ ..... 21
2.1 Circuit Dynamics of Hopfield Networks ................................ 23
2.2 Existence of Local Minima .... .... .... ........ .... .... .... ........ ....... ..... 25
2.3 Elimination of Local Minima ........ .... ..................................... 28
2.4 Neural-Based AID Converter Without Local Minima ............ 32
2.4.1 The Step Function Approach .... .... .... ..... ....... ......... ... ... 38
2.4.2 The Correction Logic Approach .... ........ ..... ....... .......... 43
2.5 Traveling Salesman Problem .... .... .... .... .... .... .... .... ........ .......... 50
2.5.1 Competitive-Hop field Network Approach .... ............ ... 54
2.5.2 Search for Optimal Solution ........................................ 56
3. Hardware Annealing Theory.... .... ..... .... ... .................... .......... 67
3.1 Simulated Annealing in Software Computation .... ..... ... ......... 67
3.2 Hardware Annealing ..... ....... .... .... .... .... .... .... ..... ....... ........ ....... 70
3.2.1 Starting Voltage Gain of the Cooling Schedule .......... 71
3.2.2 Final Voltage Gain of the Cooling Schedule .............. 75
3.3 Application to the Neural-Based AID Converter .................. 77
3.3.1 Neuron Gain Requirement ........................................ 78
3.3.2 Relaxed Gain Requirement Using Modified Synapse
Weightings ............................................................... 82
4. Programmable Synapses and Gain-Adjustable Neurons. 89
4.1 Compact and Programmable Neural Chips ........................... 90
4.2 Medium-Term and Long-Term Storage of Synapse Weight .. 100
4.2.1 DRAM-Style Weight Storage .................................... 100
4.2.2 EEPROM-Style Weight Storage ................................ 104
5. System Integration for VLSI Neurocomputing ................. 117
5.1 System Module Using Programmable Neural Chip .............. 117
5.2 Application Examples ......................................................... 121
5.2.1 Hopfield Neural-Based AID Converter ...................... 121
5.2.2 Modified Hopfield Network for Image Restoration .... 131
6. Alternative VLSI Neural Chips ........................................... 143
6.1 Neural Sensory Chips ......................................................... 145
6.2 Various Analog Neural Chips .............................................. 151
6.2.1 Analog Neurons ....................................................... 152
6.2.2 Synapses with Fixed Weights ................................... 155
6.2.3 Programmable Synapses ........................................... 161
6.3 Various Digital Neural Chips .............................................. 164
7. Conclusions and Future Work ............................................ 175
vi
Appendixes . ...... .......... ........ .................... .................. .............. ...... 179
A. Program for Neural-Based NO Conversion and Traveling
Salesman Problems .............................................................. 179
B. Non-Saturated Input Stage for Wide-Range Synapse
Circuits ............................... .................... ........ .... .. .... .. ...... .. 203
C. SPICE CMOS LEVEL-2 and LEVEL-4 Model Files ............ 209
Bibliography ................................................................................. 219
Index ............................................................................................. 231
vii
List of Figures
1.1 : Advanced neural computing system ............. .... ... .... ..... ....... .... .... 3
1.2 : Neuron Model....... ..... .... .... ..... ... .... .... .... .... .... ........ .... .... ............. 4
1.3 : Several neural networks .............................................................. 6
1.4 : A VLSI neuron with direct resistor implementation ................... 12
1.5 : Network dynamics of a lO-city TSP ........................................... 14
2.1 : A Hopfield neural network with inverting and non-inverting
amplifiers as neurons ..... ... .... ............. ................ .... ... ............ ...... 22
2.2 : Calculation of the characteristic parameter GAP ............. ... ........ 26
2.3 : Modified Hopfield neural network without local minima .. ... ...... 29
2.4 : A 4-bit neural-based AID converter ............ ........ .... .... ........ .... .... 33
2.5 : Digital output versus analog input characteristics of the AID
converter shown in Fig. 2.4 ..... .... ... .... .... .... ... ......... ......... ... ........ 37
2.6 : Energy function of the original Hopfield AID converter .... ........ 40
2.7 : Energy function of a modified Hopfield AID converter ............. 41
2.8 : A 5-bit modified neural-based AID converter ........ .... .... .... .... ..... 42
2.9 : Measured transfer characteristics of 5-bit neural-based AID
converter ............ .... ........ ..... ....... ........ .... .... ............ ........ ............. 44
2.10 : Circuit schematic of a modified 4-bit Hopfield AID conver-
ter with the correction logic gate ........ .... .... ........ .... .... ........ .... .... 45
2.11 : SPICE simulation results on the transfer characteristics of the
original Hopfield AID converter and the modified AID conver-
ter ............................................................................................... 48
2.12 : Die photo of the modified AID converter with correction logic
: gates ............................................................................................. 49
2.13 : Measured transfer characteristics of neural-based NO conver-
ter ................................................................................................. 51
2.14 : Transient responses .................................................................... 52
2.15 : A competitive-Hopfield neural networlc ..................................... 55
2.16 : One 10-city TSP ......................................................................... 58
2.17 : Another lO-city TSP .... ..... ... ........ .... ..... ... .... ..... ....... ........ .......... 59
2.18 : A 20-city TSP ........... ..... .... ..... ....... .... .... .... .... .... ................ ........ 60
2.19 : One 20-city TSP with averaged correction scheme ................... 61
2.20 : Another 20-city TSP with averaged correction scheme .... .... ..... 62
3.1 : Analogy between the annealing temperature of a Boltzmann
machine and the amplifier gain of an electronic neuron .... ........ 69
3.2 : A Hopfield neural network with non-inverting amplifiers as
neurons ....................................................................................... 71
3.3 : Radius of eigenvalues with different amplifier gains .... .... .......... 74
3.4 : Stiffness of the system function of a neural-based NO conver-
ter ............................................................................................... 77
3.5 : Trajectory of the largest eigenvalue of Hopfield NO converter
against neuron gain ........ ..... ....... ............................. ....... ............. 79
3.6 : Neural-based NO converter outputs versus amplifier gain ........ 79
3.7 : SPICE simulation results of a 4-bit Hopfield neural-based NO
converter in time domain analysis with V in = 1.4 V.... .... .... .... 80
3.8 : Trajectories of the NO converter output versus the amplifier
gain ............................................................................................. 81
3.9 : A 4-bit neural-based NO converter with unity synapse weigh-
ting .............................................................................................. 83
3.10 : Transfer characteristics of a 4-bit Hopfield NO converter ........ 85
3.11 : Different annealing styles ........................................................... 86
ix
4.1 : A programmable synapse cell in [8] ........................................... 91
4.2 : Transconductance amplifier as the core for the programmable
synapse cell .... ........ ........ .... .... ........ .... .... .... ........ .... .... ..... ... .... .... 93
4.3 : Simulated synapse characteristics ............................ .... ............... 94
4.4 : Circuit schematic of the programmable synapse and related
neurons ....................................................................................... 96
4.5 : The Programmable synapse cell consisting of a non-saturated
transconductance amplifier ....... , .... .... .... ........ .... .... .... .... .... .... ..... 99
4.6 : Simulated charge retention characteristics in DRAM-style cell .. 101
4.7 : DRAM-style synapse cell ........................................................... 102
4.8 : Die photos ................................................................................... 103
4.9 : Floating-gate transistor ................................................................ 105
4.10 : Measured results of several programming schemes ................... 107
4.11 : Output characteristics of a double-polysilicon floating-gate
transistor with WI L = 18 Jlm / 2 Jlm ........ ....... .................... ..... 109
4.12 : Charge retention characteristics of a simple floating-gate
transistor ..................................................................................... 111
4.13 : Die photos .................................................................................. 112
5.1 : Neural computing board diagram using DRAM-style program-
mable synapses ............ ............ ........... ..................... ................... 118
5.2 : Neural computing board diagram using EEPROM-style progra-
mmable synapses ........... ............................. ................ ................ 119
5.3 : Real-time neural computing system with a multi-layer network 121
5.4 : Circuit schematic of an 8-bit neural-based AID converter with
the programmable synapses and gain-adjustable neurons ........... 122
5.5 : Die photos of the 8-bit neural-based AID converter in Fig. 5.4 . 123
5.6 : Response time characteristics of 4-bit neural-based converters .. 125
5.7 : Transient response ....................................................................... 126
x