Simulating Neural Networks

360 Pages·12.573 MB·English

■

SIMULATING
flEURAL
NETWORKS
with
Mathematica
James A. Freeman
Loral Space Information Systems
and
University of Houston-Clear Lake
I
I
1
§
▲ ft
TT
ADDISON-WESLEY PUBLISHING COMPANY
Reading, Massachusetts • Menlo Park, California
New York • Don Mills, Ontario • Wokingham, England
Amsterdam • Bonn • Sydney • Singapore • Tokyo
Madrid • San Juan • Milan • Paris
Mathematica is not associated with Mathematica Inc., Mathematica Policy Research, Inc.,
or MathTech, Inc.
Many of the designations used by manufacturers and sellers to distinguish their products
are claimed as trademarks. Where those designations appear in this book, and Addison-
Wesley was aware of a trademark claim, the designations have been printed in caps or
initial caps.
The programs and applications presented in this book have been included for their instruc¬
tional value. They have been tested with care, but are not guaranteed for any particular
purpose. The publisher does not offer any warranties or representations, nor does it accept
any liabilities with respect to the programs or applications.
Library of Congress Cataloging-in-Publication Data
Freeman, James A.
Simulating neural networks with Mathematica / James A. Freeman,
p. cm.
Includes bibliographical references and index.
ISBN 0-201-56629-X
1. Neural networks (Computer science) 2. Mathematica (Computer
program) I. Title.
QA76.87.F72 1994
006.3-dc20
92-2345
CIP
Reproduced by Addison-Wesley from camera-ready copy supplied by the author.
Copyright (c) 1994 by Addison-Wesley Publishing Company, Inc.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval
system, or transmitted, in any form or by any means, electronic, mechanical, photocopying,
recording, or otherwise, without the prior written permission of the publisher. Printed in
the United States of America.
123456789 10-DOC-97 96 95 94 93
Preface
I sat across the dinner table at a restaurant recently with a researcher from
Los Alamos National Laboratory. We were discussing a collaboration
on a neural-network project while eight of us had dinner at a quaint
restaurant in White Rock, just outside of Los Alamos. Fortified by a few
glasses of wine, I took the opportunity to mention that I had just reached
an agreement with my publisher to write a new book on neural networks
with Mathematica. My dinner companion looked over at me and asked a
question: "Why?"
I was somewhat surprised by this response, but reflecting for a mo¬
ment on the environment in which this person works, I realized that the
answer to his question was not necessarily so obvious. This person had
access to incredible computational resources: large Crays, and a 64,000-
node Connection Machine, for example. Considering the computational
demands of most neural networks, why would anyone working with
them want to incur the overhead of an interpreted language like Mathe¬
matica? To me the answer was obvious, but to him, and possibly to you,
the answer may require some explanation.
During the course of preparing the manuscript for an earlier text. Neu¬
ral Networks: Algorithms, Applications, and Programming Techniques (hence¬
forth "Neural Networks" ), I used Mathematica extensively for a variety
of purposes. I simulated the operation of most of the neural networks
described in that text using Mathematica. Mathematica gave me the ability
to confirm nuances of network behavior and performance, as well as to
develop examples and exercises pertaining to the networks. In addition
I used Mathematica's graphics capability to illustrate numerous points
throughout the text. The ease and speed with which I was able to im¬
plement a new network spoke highly of using Mathematica as a tool for
exploring neural-network technology.
iv Preface
The idea for Simulating Neural Networks with Mathematica grew along
with my conviction that Mathematica had saved me countless hours of
programming time. Even though Mathematica is an interpreted language
(much like the old BASIC) and neural networks are notorious CPU hogs,
there seemed to me much insight to be gained by using Mathematica in
the early stages of network development.
I explained all of these ideas (somewhat vigorously) to my dinner
companion. No doubt feeling a bit of remorse for asking the question in
the first place, he promised to purchase a copy of the book as soon as it
was available. I hope he finds it useful. I have the same wish for you.
This book will introduce you to the subject of neural networks within
the context of the interactive Mathematica environment. There are two
main thrusts of this text: teaching about neural networks and showing
how Mathematica can be used to implement and experiment with neural-
network architectures.
In Neural Networks my coauthor and I stated that you should do
some programming of your own, in a high-level language such as C,
FORTRAN, or Pascal, in order to gain a complete understanding of the
networks you study. I do not wish to retract that philosophy, and this
book is not an attempt to show you how you can avoid eventual trans¬
lation of your neural networks into executable software. As I stated in
the previous paragraph, most neural networks are computationally quite
intensive. A neural network of any realistic size for an actual application
would likely overwhelm Mathematica. Nevertheless, a researcher can use
Mathematica to experiment with variants of architectures, debug a new
training algorithm, design techniques for the analysis of network per¬
formance, and perform many other analyses that would prove far more
time-consuming if done in traditional software or by hand. This book
illustrates many of those techniques.
The book is suitable for a course in neural networks at the upper-level
undergraduate or beginning graduate level in computer science, electrical
engineering, applied mathematics, and related areas. The book is also
suitable for self-study. The best way to study the material presented
here is interactively, executing statements and trying new ideas as you
progress.
This book does not assume expertise with either neural networks or
Mathematica, although I expect that the readers will most likely know
something of Mathematica. If you have read through the first chapter
of Stephen Wolfram's Mathematica book and have spent an hour or two
Preface v
interacting with Mathematica, you will have more than sufficient back¬
ground for the Mathematica syntax in this book.
I have kept the Mathematica syntax as understandable as possible. It
is too easy to spend an inordinate amount of time trying to decipher
complex Mathematica expressions and programs, thereby missing the for¬
est for the trees. Moreover, many individual functions are quite long
with embedded print statements and code segments that I could have
written as separate functions. I chose not to, however, so the code stands
as it is. For these reasons, some Mathematica experts may find the coding
a bit nonelegant: To them I extend my apologies in advance. I have
also chosen to violate one of the "rules" of Mathematica programming:
using complete English words for function and variable names. As an
example, I use bpnMomentum instead of the more "correct" backprop-
agationNetworkWithMomentum. The former term is easily interpreted,
requires less space leading to more understandable expressions, and is
less prone to typographical errors.
Readers with no prior experience with neural networks should have
no trouble following the text, although the theoretical development is not
as complete as in Neural Networks. I do assume a familiarity with basic
linear algebra and calculus. Chapter 1 is a prerequisite to all other chap¬
ters in the book; the material in this chapter is fairly elementary, however,
both in terms of the neural-network theory and the use of Mathematica.
Readers who possess at least a working knowledge of each subject can
safely skip Chapter 1. If you have never studied the gradient-descent
method of learning, then you should also study Chapter 2 before pro¬
ceeding to later chapters. The discussion of the Elman and Jordan net¬
works in Chapter 6 requires an understanding of the backpropagation
algorithm given in Chapter 3.
The text comprises eight chapters; each one, with the exception of the
last, deals with a major topic related to neural networks or to a specific
type of network architecture. Each chapter also includes a simulation of
the networks using Mathematica and demonstrates the use of Mathemat¬
ica to explore the characteristics of the network and to experiment with
variations in many instances.
The last chapter introduces the subject of genetic algorithms (GAs).
We will study the application of GAs to a scaled-down version of the
traveling salesperson problem. To tie the subject back to neural networks,
we will look at one method of using GAs to find optimum weights for a
neural network. A brief description of the chapters follows.
vi Preface
Chapter 1: Introduction to Neural Networks and Mathematica. This
chapter contains basic information about neural networks and defines
many of the conventions in notation and terminology that are used
throughout the text. I also introduce the concept of learning in neu¬
ral networks, and that of Hinton diagrams, and show how to construct
them with Mathematica.
Chapter 2: Training by Error Minimization. This chapter introduces
the topic of gradient descent on an error surface as a method of learning
in a simple neural network. The chapter begins with the Adaline, show¬
ing both the theoretical calculation of the optimum weight vector, as well
as the least-mean-square algorithm. I also describe techniques for simu¬
lating the Adaline using Mathematica and for analyzing the performance
of the network.
Chapter 3: Backpropagation and Its Variants. In this chapter we de¬
velop the generalized delta rule and apply it to a multilayered network
generally known as a backpropagation network. To demonstrate how
Mathematica can be used to experiment with different network architec¬
tures, we investigate several modifications to the basic network structure
in an attempt to improve the performance of the network. We also study
a related network known as the functional link network.
Chapter 4: Probability and Neural Networks. In this chapter we
explore the use of probability concepts in neural networks using two
different types of networks. After describing the deterministic Hopfield
network, we examine some concepts in the thermodynamics of physi¬
cal systems and see how they relate to the Hopfield network. Then we
use a technique called simulated annealing to develop a stochastic Hop-
field network. We also examine the probabilistic neural network, which
is an example of a network that implements a traditional classification
algorithm based on Bayesian probability theory.
Chapter 5: Optimization and Constraint Satisfaction with Neural
Networks. In this chapter we explore the concept of constraint satisfac¬
tion using the familiar traveling salesperson problem. Then we illustrate
how this problem, and other constraint-satisfaction problems, map onto
the Hopfield network.
Chapter 6: Feedback and Recurrent Networks. In this chapter we
describe and simulate networks that depart from the simple feedforward
structures typical of the backpropagation network. We begin with a two-
layer network called a bidirectional associative memory (BAM). We then
proceed to networks that can recognize and reproduce time-sequences of