Neural Networks: An Introduction

339 Pages·1995·9.787 MB·English

by  Professor Dr. Berndt Müller, Dr. Joachim Reinhardt, Michael T. Strickland (auth.)

Physics of Neural Networks
Series Editors:
E. Domany J. L. van Hemmen K. Schulten
Advisory Board:
H. Axelrad
R. Eckmiller
J. A. Hertz
J. J. Hopfield
P. I. M. Johannesma
P. Peretto
D. Sherrington
M. A. Virasoro
Springer-Ve rlag Berlin Heidelberg GmbH
B. MUller J. Reinhardt
M. T. Strickland
Neural Networks
An Introduction
Second Updated and Corrected Edition
With a MS-DOS Programm Diskette
and 105 Figures
Springer
Professor Dr. Berndt Mtiller Michael T. Strickland
Departrnent of Physics, Duke University Department of Physics, Duke University
Durham, NC 27706, USA Durham, NC 27706, USA
Dr. Joachim Reinhardt
Institut fUr Theoretische Physik
J.-W.-Goethe-Universităt,
Postfach I I I 9 32
D-60054 Frankfurt , Gennany
Series Editors: Professor J. Leo van Hemmen
Institut flir Theoretische Physik
Technische Universităt MUnchen
D-85747 Garching bei MUnchen
Germany
Professor Eytan Domany Professor Klaus Schulten
Department of Electronics Department of Physics
Weizmann Institute of Science and Beckman Institute
76100 Rehovot, Israel University of Illinois
Urbana,IL6I801, USA
Additional material to this book can be downloaded from http://extras.springer.com.
ISBN 978-3-540-60207-1
L lbrary of Con gresa Caulog lng-tn-Pub lleat ton Olto
Hu ller. Berndt.
Neural networks : an tntroductlon 1 B. Huller, J. Relnhardt, H.T.
Strlckland. -- 2nd updated and eorr. ed.
p. ca. -- <Phystcs of neural networks)
Ineludes blbllographlcal raferanees and Index.
ISBN 978-3-540-60207-1 ISBN 978-3-642-57760-4 (eBook)
DOI 10.1007/978-3-642-57760-4
1. Neural networks <Coeputer sctence) 1. Ratnhardt. 4.
(Joachlol, 1952- II. Strlckland, H. T. !Hichael Thooas>. 1969-
III. Tltla. IV. Serios.
OA76. 87.H85 1995
006. 3--de20 95-24948
This work is subject to copyright. Ali rigbts are reserved. wbetber tbe whole or part of tbe material is concerned,
specifically the rights of translation. reprinting. reuse of illustrations. recitation. broadcasting. reproduction on
microfilm or in any other way. and storage in data banks. Duplication of tbis publication or parts tbereof is
permitted only under the provisions of the German Copyright Law of September 9. 1965. in its current versi on.
and permissioo for use must always be obtained from Springer-Verlag. Violatioos are liable for prosecution under
the German Copyright Law.
O Springer-Verlag Berlin Heidelberg 1990,1995
Originally published by Springer-Ver1ag Berlin Heidelberg New York 1995
Please note: Before using tbe programs in tbis book. please consult the technical manuals provided by tbe manufacturer
of the computer-and of any additiooal plug-in boards-to be used. Tbe autbors and tbe publisher accept no legal
responsibility for any damage by improper use of tbe iostructions aod programs contaioed herein. Althougb these
programs bave been tested with extreme care. we can offer oo formal guarantee that they will function correctly.
Tbe use of general descriptive oames. registered oames. trademarks. etc. in tbis publication does oot imply, even in
the absence of a specific statement, that sucb oames are exempt from the relevant protective laws and regulations and
tberefore free for general use.
The programs oo the eoclosed diskette are under copyright-protection and may not be reproduced witbout written
permission by Springer-Verlag. Ooe copy of tbe programs may be made as a back-up, but ali further copies offeod
copyright law.
Typesetting: Camera ready copy from the authors.
SPIN 10120349 54/3144-S 4 3 2 1 O-Printed on acid-free paper
In memory of
Jorg Briechle
Preface to the Second Edition
Since the publication of the first edition, the number of studies of the proper
ties and applications of neural networks has grown considerably. The field is
now so diverse that a complete survey of its recent developments is impossible
within the scope of an introductory text. Fortunately, a wealth of material
has become available in the public domain via the Internet and the World
Wide Web (WWW). We trust that the reader who is interested in some of
the latest developments will explore this source of information on their own.
For those interested we have compiled a short list of Internet resources that
provide access to up to the date information on neural networks.
Neural-Network related WWW Sites
http://www.neuronet.ph.kcl.ac.uk:80/
NEuroNet - King's College London
http://glimpse.cs.arizona.edu:1994/bib/
Bibliographies on Neural Networks and More
http://http2.sils.umich.edu/Public/nirg/nirg1.html
Neurosciences Internet Resource Guide
http://www1.cern.ch/NeuralNets/nnwlnHep.html
Neural Networks in High Energy Physics
http://www.cs.cmu.edu:8001/afs/cs.cmu.edu/project
/nnspeech/WorldWideWeb/PUBLIC/nnspeech/
The Neural Net Speech Group at Carnegie Mellon
ftp://archive.cis.ohio-state.edu/pub/neuroprose/
The Neuroprose preprint archive
Listservers
Connectionists-RequestCcs.cmu.edu
Connectionists - Neural Computation
neuron-requestCpsych.upenn.edu
Neuron-Digest - Natural and Artificial Neural Nets
LISTSERVCUICVM.UIC.EDU
Neurol-L - Neuroscience Resources
mbCtce.ing.uniroma1.it (151.100.8.30)
Cellular Neural Networks
VIII
U senet N ewsgroups
comp.ai.neural-nets
bionet.neuroscience
We have, therefore, resisted attempting to completely update the text
and references with the latest developments and rather concentrated our ef
forts on eliminating two shortcomings of the first edition. We have added
a separate chapter on applying genetic algorithms to neural-network learn
ing, complete with a new demonstration program neurogen. A new section
on back-propagation for recurrent networks has been added along with a
new demonstration program btt which illustrates back-propagation through
time. In addition, we have expanded the chapter on applications of neural
networks, covering recent developments in pattern recognition and time series
prediction.
One of us, Michael Strickland, would like to thank his wife Melissa Trach
tenberg and all of his friends who helped him with proofreading and pep
talks.
It is our hope that with the introduction of the new chapters and programs
the second edition will be interesting and stimulating to new readers starting
their own exploration of this fascinating new field of human science.
Durham and Frankfurt Berndt Muller
May 1995 Joachim Reinhardt
Michael Strickland
Email addresses and homepages of the authors
http://wvw.phy.duke.edu/-muller/
Berndt Muller - [email protected]
http://www.th.physik.uni-frankfurt.de/-jr/jr.html
Joachim Reinhardt - [email protected]
http://www.phy.duke.edu/-strickla/
Michael Strickland - [email protected]
References to the demonstration programs are indicated
in the main text by this "PC logo". We encourage all
readers to do the exercises and play with these programs.
Preface to the First Edition
The mysteries of the human mind have fascinated scientists and philosophers
for centuries. Descartes identified our ability to think as the foundation of
ontological philosophy. Others have taken the human mind as evidence of the
existence of supernatural powers, or even of God. Serious scientific investiga
tion, which began about half a century ago, has partially answered some of
the simpler questions (such as how the brain processes visual information),
but has barely touched upon the deeper issues concerned with the nature of
consciousness and the existence of mental features transcending the biological
substance of the brain, usually encapsulated in the concept of the "mind".
Besides the physiological and philosophical approaches to these questions,
so impressively presented and contrasted in the book by Popper and Eccles
[P077], studies of formal networks composed of binary-valued information
processing units, highly abstracted versions of biological neurons, either by
mathematical analysis or by computer simulation, have emerged as a third
route towards a better understanding of the brain, and possibly of the human
mind. Long remaining - with the exception of a brief period in the early 1960s
-' a rather obscure research interest of a small group of dedicated scientists
scattered around the world, neural network research has recently sprung into
the limelight as a fashionable research field. Much of this surge of attention
results, not from interest in neural networks as models of the brain, but rather
from their promise to provide solutions to technical problems of "artificial
intelligence" that the traditional, logic-based approach did not yield easily.l
The quick rise to celebrity (and the accompanying struggle for funding
and fame) has also led to the emergence of a considerable amount of exag
geration of the virtues of present neural network models. Even in the most
successful areas of application of neural networks, i.e. content-addressable (as
sociative) memory and pattern recognition, relatively little has been learned
which would look new to experts in the various fields. The difficult problem of
position- and distortion-invariant pattern recognition has not yet been solved
by neural networks in a satisfactory way, although we all know from experi
ence that our brains can often do this. In fact, it is difficult to pinpoint any
1 Neural networks are already being routinely used in some fields. For example, in
high energy physics experiments they are being used as triggering software for
complex detector systems.
x
technical problem where neural networks have been shown to yield solutions
that are clearly superior to those previously known. The standard argument,
that neural networks can do anything a traditional computer can, but do not
need to be programmed, does not weigh as strongly as one might think. A
great deal of thought must go into the design of the network architecture and
training strategy appropriate for a specific problem, but little experience and
few rules are there to help.
Then why, the reader may ask, have we as nonexperts taken the trouble
of writing this book and devising the computer demonstrations? One moti
vation, quite honestly, was our own curiosity. We wanted to see for ourselves
what artificial neural networks can do, what their merits are and what their
failures. Not having done active research in the field, we have no claim to
fame. Whether our lack of prejudice outweighs our lack of experience, and
maybe expertise, the reader must judge for herself (or himself).
The other, deeper reason is our firm belief that neural networks are, and
will continue to be, an indispensable tool in the quest for understanding the
human brain and mind. When the reader feels that this aspect has not re
ceived its due attention in our book, we would not hesitate to agree. However,
we felt that we should focus more on the presentation of physical concepts
and mathematical techniques that have been found to be useful in neural
network studies. Knowledge of proven tools and methods is basic to progress
in a field that still has more questions to discover than it has learned to ask,
let alone answer.
To those who disagree (we know some), and to the experts who know
everything much better, we apologize. The remaining readers, if there are
any, are invited to play with our computer programs, hopefully capturing
some of the joy we had while devising them. We hope that some of them
may find this book interesting and stimulating, and we would feel satisfied
if someone is inspired by our presentation to think more deeply about the
problems concerning the human mind.
This book developed out of a graduate level course on neural network
models with computer demonstrations that we taught to physics students at
the J.W. Goethe University in the winter semester 1988/89. The interest in
the lecture notes accompanying the course, and the kind encouragement of
Dr. H.-U. Daniel of Springer-Verlag, have provided the motivation to put it
into a form suitable for publication. In line with its origin, the present mono
graph addresses an audience mainly of physicists, but we have attempted to
limit the "hard-core" physics sections to those contained in Part II, which
readers without an education in theoretical physics may wish to skip. We
have also attempted to make the explanations of the computer programs
contained on the enclosed disk self-contained, so that readers mainly inter
ested in "playing" with neural networks can proceed directly to Part III.
Durham and Frankfurt Berndt Muller
July 1990 Joachim Reinhardt
Table of Contents
Part I. Models of Neural Networks
1. The Structure of the Central Nervous System . .. .. .. .. .. . 3
1.1 The Neuron .................... ....... ... .. ... ........ 3
1.2 The Cerebral Cortex. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2. Neural Networks Introduced. .. .. . . .. . .. .. . . . ... . . . .. ... .. 13
2.1 A Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . 1. 3 . . . . . . . . . .
2.2 A Brief History of Neural Network Models. . . . . . . . . . . .. .. 1.3 .
2.3 Why Neural Networks? . . . . . . . . . . . . . . . . . . . . . . . . .1 8 . . . . . . . . .
2.3.1 Parallel Distributed Processing. . . . . . . . . . . . . . . .1 8. . . ..
2.3.2 Understanding How the Brain Works ... .... ... ..... 21
2.3.3 General Literature on Neural Network Models .... ... 23
3. Associative Memory . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 2. 4. . . . . . . .
3.1 Associative Information Storage and Recall . . . . . . . . . . ... . 2. 4.
3.2 Learning by Hebb's Rule . . . . . . . . . . . . . . . . . . . . . . . 26. . . . . . . ..
3.3 Neurons and Spins. . . . . . . . . . . . . . . . . . . . . . . . . . .2 .9 . . . . . . . . ..
3.3.1 The "Magnetic" Connection . . . . . . . . . . . . . . . .. . .2 .9 . . .
3.3.2 Parallel versus Sequential Dynamics ..... ........ ... 32
3.4 Neural "Motion Pictures" .. ... ............... ........... 34
4. Stochastic Neurons.. ... . . .. . .. . ... .. .. .. .... .. . ... . ... ... 38
4.1 The Mean-Field Approximation. . . . . . . . . . . . . . . . .. .. . 3.8 . . . .
4.2 Single Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 4. 0 . . . . . . . . .
4.3 Several Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 . . . . . . . . . ..
5. Cybernetic Networks. ... .. .. .. . ... . .. .. . .... .. . .. .. . .. ... 46
5.1 Layered Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . 46. . . . . . . . . . .
5.2 Simple Perceptrons . . . . . . . . . . . . . . . . . . . . . . . .. .. . 4.7 . . . . . . . .
5.2.1 The Perceptron Learning Rule. . . . . . . . . . . . . . . .4 .7 . . . .
5.2.2 "Gradient" Learning. . . . . . . . . . . . . . . . . . . . . .4 8. . . . . . . .
5.2.3 A Counterexample: The Exclusive-OR Gate. . . . . . .. .. 49