Structure Level Adaptation for Artificial Neural Networks

223 Pages·1991·5.04 MB·English

by  Tsu-Chang Lee (auth.)

STRUCTURE LEVEL
ADAPTATION FOR
ARTIFICIAL NEURAL
NETWORKS
THEKLUWER INTERNATIONALSERIES
IN ENGINEERINGAND COMPUTERSCIENCE
KNOWLEDGE REPRESENTATION, LEARNINGAND
EXPERTSYSTEMS
ConsultingEditor
Tom Mitchell
CarnegieMellon University
UNIVERSALSUBGOALINGANDCHUNKINGOFGOALHIERARCHIES,J.
Laird,P. Rosenbloom,A. Newell
ISBN: 0-89838-213-0
MACHINELEARNING:AGuidetoCurrentResearch,T. Mitchell,J. Carbonell,
R. Michalski
ISBN: 0-89838-214-9
MACHINELEARNINGOFINDUCTIVEBIAS,P. Vtgoff
ISBN: 0-89838-223-8
ACONNECTIONISTMACHINEFORGENETICHILLCLIMBING,
D.H.Ackley
ISBN:0-89838-236-X
LEARNINGFROMGOODANDBADDATA, P. D.Laird
ISBN: 0-89838-263-7
MACHINELEARNINGOFROBOTASSEMBLYPLANS, A. M. Segre
ISBN: 0-89838-269-6
AUTOMATINGKNOWLEDGEACQUISITIONFOREXPERTSYSTEMS,
S.Marcus,Editor
ISBN: 0-89838-294-7
MACHINELEARNING,META·REASONINGANDWGICS, P. B.Brazdil,
K. Konolige
ISBN: 0-7923-9047-4
CHANGEOFREPRESENTATIONANDINDUCTIVEBIAS:D. P.Benjamin
ISBN: 0-7923·9055-5
KNOWLEDGEACQUISITION:SELECTEDRESEARCHAND
COMMENTARY,S.Marcus,Editor
ISBN: 0-7923-9062-8
LEARNINGWITHNESTEDGENERALIZEDEXEMPLARS, S. L. Salzberg
ISBN: 0-7923-9110-1
INCREMENTALVERSION·SPACEMERGING:AGeneralFramework
forConceptLearning,H. Hirsh
ISBN: 0-7923-9119-5
COMPETITIVELY INHIBITEDNEURALNETWORKSFORADAPTIVE
PARAMETERESTIMATION, M. Lemmon
ISBN: 0-7923-9086-5
STRUCTURE LEVEL
ADAPTATION FOR
ARTIFICIAL NEURAL
NETWORKS
by
Tsu-Chang Lee
Stanford University/
Cadence Design Systems
foreword by
Joseph W. Goodman
Stanford University
.
.,
~
Springer Science+Business Media, LLC
Library or Congras Cataloging.in-Publlcation Data
Lee, Tsu-Chang, 1961-
Structure level adaptation for artificial neural nefWOrlcs I by Tsu
-Chang Lee; foreword by Joseph W. Goodman.
p. CfD. --(The K1uwer international series in engineering and
computer science. Knowledge representation, learning, and expert
systems)
Includes bibliographical references and index.
ISBN 978-1-4613-6765-9 ISBN 978-1-4615-3954-4 (eBook)
DOI 10.1007/978--1-4615-3954-4
1. Neural networks (Computer science) 1. TItle. II. Seties.
QA7 8.87 .lA43 1991
91-2251
CIP
Copyright © 1991 by Springer Science+Business Media New York
Originally published by K1uwer Academic Publishers in 1991
Softcover reprint ofthe hardcover Ist edition 1991
Ali rights reserved. No pari of Ihis publication may be reproduced, SlOred in a relrieval
system or transmi lIed in any form or by any means, mechanical, photo-<:opying, record ing.
or otherwise. wilhoutlhe prior written permission of Ihe publisher,
Springer Science+Business Media. LLC.
Printed an aeid·free paper.
This book is dedicated to
my parents
Ying-Jian Lee
Chou-Feng Lu
my wife
Yih-ching Salina An
and my daughters
Teresa Tai-yi Lee
Jennifer Jen-yi Lee
Contents
Foreword xv
Preface xvii
1 Introduction 1
1.1 Background......... 1
1.2 Neural Network Paradigms 3
1.3 The Frame Problem in Artificial Neural Networks. 13
1.4 Approach . . . . . . . . 17
1.5 Overview of This Book . 20
2 Basic Framework 23
2.1 Introduction.. 23
2.2 Formal Neurons. 25
2.3 Formal Neural Networks 35
2.4 Multi-Level Adaptation Formalism 43
2.5 Activity-Based Structural Adaptation 52
2.5.1 Neuron generation . . . . . . . 55
2.5.2 Neuron Annihilation . . . . . . 58
2.5.3 Structural Relationship Modification 61
2.6 Summary . . . . . . . . . . . . . . . . 61
Vll
3 Multi-Layer Feed-Forward Networks 63
3.1 Introduction. . . . . . . . . 63
3.2 Function Level Adaptation 64
3.3 Parameter Level Adaptation. 67
3.4 Structure Level Adaptation 70
3.4.1 Neuron Generation . 70
3.4.2 Neuron Annihilation 72
3.5 Implementation . . . . . 74
3.6 An Illustrative Example 77
........
3.7 Summary 79
4 Competitive Signal Clustering Networks 93
4.1 Introduction. . 93
4.2 Basic Structure 94
4.3 Function Level Adaptation 96
4.4 Parameter Level Adaptation . 101
4.5 Structure Level Adaptation 104
4.5.1 Neuron Generation Process 107
4.5.2 Neuron Annihilation and Coalition Process 114
4.5.3 Structural Relation Adjustment. 116
4.6 Implementation . . 119
4.7 Simulation Results 122
4.8 Summary ..... 134
5 Application Example: An Adaptive Neural Network Source
Coder 135
5.1 Introduction.......... 135
5.2 Vector Quantization Problem 136
5.3 VQ Using Neural Network Paradigms 139
Vlll
5.3.1 Basic Properties . 140
5.3.2 Fast Codebook Search Procedure 141
5.3.3 Path Coding Method. . . . . . . 143
5.3.4 Performance Comparison .... 144
5.3.5 Adaptive SPAN Coder/Decoder 147
5.4 Summary . . . . . . . . . . . . . . . . . 152
6 Conclusions 155
6.1 Contributions 155
6.2 Recommendations 157
A Mathematical Background 159
A.1 Kolmogorov's Theorem . 160
A.2 Networks with One Hidden Layer are Sufficient 161
B Fluctuated Distortion Measure 163
B.1 Measure Construction . 163
B.2 The Relation Between Fluctuation and Error 166
C SPAN Convergence Theory 171
C.1 Asymptotic Value of Wi 172
C.2 Energy Function .. 175
D Operational Measures 179
D.1 Averaging Mechanism . . . . . . . 179
D.2 Summary of Operational Measures 181
E Glossary of Symbols and Acronyms 183
Bibliography 189
Index 207
IX
List of Figures
1.1 The schematic diagram of a neuron . 7
1.2 Neural network paradigms ..... 14
1.3 The contribution tree of this book. 22
2.1 A formal neuron (up to the P-Level).. 29
2.2 The function tree ofa formal neuron.. 30
2.3 Treelize a function graph. . . . . . . . 32
2.4 The temporal dependency of variables 33
2.5 A formal neural network (up to the P-Level). 38
.........
2.6 Eliminate state inconsistency 40
2.7 A structure level neural network with centralized control
mechanism. ................ 49
2.8 A network structure evolution element.. 51
2.9 A typical input weight convergence process of a neuron. 54
3.1 A Multi-layer feed forward neural network. 65
3.2 A typical sigmoidal function. ........ 66
3.3 The data structure for the FUNNET simulator 75
3.4 The control flow chart for the FUNNET simulator. 76
3.5 The input pattern distribution for the illustrative example 78
Xl
3.6 The error convergence behavior for the adaptable struc-
ture neural network. . . . . . . . . . . . . . . . . . . . .. 81
3.7 The error convergence behavior: 2 hidden layer neurons
case. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 84
3.8 The error convergence behavior: 4 hidden layer neurons
case . 87
4.1 Homogeneous neighborhood systems 94
4.2 The SPAN function level structure 97
4.3 The spatial impulse response of SPAN 99
4.4 Output activity building up in SPAN. 100
4.5 Lattice expansion . . . . . . . . . . . . 111
4.6 Local axes and the directional distribution ofdistortion. 112
4.7 Lattice shrinkage . . . . . 116
4.8 The axis merging process 117
4.9 Doubly linked list representation of SPAN. . 119
4.10 An example showing the network growing process. 124
4.11 An example showing the network reduction process. 130
5.1 A path code example. . . . . . . . . . . . . . . . . . 144
5.2 Performance comparison between LBG and SPAN coders 148
5.3 The codebook searching efficiency of the SPAN coder. 149
5.4 A proposed adaptive SPAN coding/decoding system 151
6.1 Future research directions. . . . . . . . . . . . . . . . 158
xii