Networks of Learning Automata: Techniques for Online Stochastic Optimization

274 Pages·2004·14.944 MB·English

by  M. A. L. Thathachar, P. S. Sastry (auth.)

NETWORKS OF LEARNING
AUTOMATA
Techniques for Online Stochastic
Optimization
NETWORKS OF LEARNING
AUTOMATA
Techniques for Online Stochastic
Optim izatio n
M. A. L. Thathachar
s.
P. Sastry
Dept. of Electrical Engineering
Indian Institute of Science
Bangalore, India
SPRINGER SCIENCE+BUSINESS MEDIA, LLC
Library of Congress Cataloging-in-Publication Data
Networks of Leaming Automata: Techniques for Online Stochastic Optimization
M.A.L. Thathachar and P.S. Sastry
ISBN 978-1-4613-4775-0 ISBN 978-1-4419-9052-5 (eBook)
DOI 10.1007/978-1-4419-9052-5
Copyright © 2004 by Springer Science+Business Media New York
Originally published by Kluwer Academic Publishers in 2004
Softcover reprint ofthe hardcover Ist edition 2004
AII rights reserved. No part of this work may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, electronic, mechanical,
photocopying, microfilming, recording, or otherwise, without prior written
permission from the Publisher, with the exception of any material supplied
specifically for the purpose of being entered and executed on a computer system,
for exclusive use by the purchaser of the work.
Printed on acid-free paper.
TO OUR PARENTS
Contents
Dedication v
Preface xiii
1. INTRODUCTION 1
1.1 MachineIntelligence and Learning 1
1.2 LearningAutomata 8
1.3 TheFinite Action LearningAutomaton (FALA) 10
1.3.1 The Automaton 10
1.3.2 The Random Environment 11
1.3.3 Operation ofFALA 12
1.4 Some ClassicalLearningAlgorithms 14
1.4.1 LinearReward-Inaction (LR-I) Algorithm 14
1.4.2 OtherLinearAlgorithms 17
1.4.3 EstimatorAlgorithms 18
1.4.4 Simulation Results 20
1.5 TheDiscretized Probability FALA 20
1.5.1 DLR-I Algorithm 22
1.5.2 Discretized PursuitAlgorithm 22
1.6 TheContinuous Action LearningAutomaton (CALA) 24
1.6.1 Analysis of the Algorithm 26
1.6.2 Simulation Results 30
1.6.3 AnotherContinuousAction Automaton 31
1.7 TheGeneralizedLearning Automaton (GLA) 33
1.7.1 Learning Algorithm 36
1.7.2 An Example 38
1.8 TheParameterizedLearning Automaton (PLA) 41
1.8.1 Learning Algorithm 44
viii NETWORKS OFLEARNINGAUTOMATA
1.9 MultiautomataSystems 46
1.10 Supplementary Remarks 47
2. GAMES OF LEARNINGAUTOMATA 51
2.1 Introduction 51
2.2 A MultiplePayoffStochastic Game of Automata 54
2.2.1 The Learning Algorithm 56
2.3 Analysisof the AutomataGame Algorithm 63
2.3.1 Analysis of the Approximating ODE 65
2.4 Game with Common Payoff 74
2.5 Games ofFALA 76
2.5.1 CommonPayoffGames ofFALA 78
2.5.2 PursuitAlgorithmfor aTeamof FALA 79
2.5.3 OtherTypes ofGames 82
2.6 CommonPayoffGames ofCALA 83
2.6.1 Stochastic Approximation Algorithms and CALA 85
2.7 Applications 89
2.7.1 SystemIdentification 89
2.7.2 Learning ConjunctiveConcepts 92
2.8 Discussion 102
2.9 Supplementary Remarks 103
3. FEEDFORWARD NETWORKS 105
3.1 Introduction 105
3.2 Networks ofFALA 106
3.3 The LearningModel 107
3.3.1 G-Environment 107
3.3.2 The Network 109
3.3.3 Network Operation 113
3.4 The LearningAlgorithm 116
3.5 Analysis 116
3.6 Extensions 127
3.6.1 OtherNetwork Structures 127
3.6.2 Other Learning Algorithms 128
3.7 Convergenceto the Global Maximum 129
3.7.1 The Network 130
3.7.2 The Global Learning Algorithm 131
3.7.3 Analysis of theGlobal Algorithm 132
Contents ix
3.8 Networks ofGLA 136
3.9 Discussion 137
3.10 SupplementaryRemarks 138
4. LEARNINGAUTOMATAFOR
PATTERNCLASSIFICATION 139
4.1 Introduction 139
4.2 Pattern Recognition 140
4.3 Common Payoff Game of Automata forPR 146
4.3.1 Pattern Classification withFALA 148
4.3.2 Pattern Classification withCALA 150
4.3.3 Simulations 151
4.4 Automata Network for Pattern Recognition 154
4.4.1 Simulations 158
4.4.2 Network of Automatafor Learning Global Maximum 161
4.5 DecisionTree Classifiers 164
4.5.1 LearningDecision Trees using GLA and CALA 166
4.5.2 Learning Piece-wiseLinearFunctions 170
4.6 Discussion 173
4.7 SupplementaryRemarks 176
5. PARALLELOPERATION OF
LEARNINGAUTOMATA 177
5.1 Introduction 177
5.2 Parallel Operation ofFALA 178
5.2.1 Analysis 180
5.2.2 e-optimality 181
5.2.3 Speed of Convergence andModule Size 184
5.2.4 Simulation Studies 185
5.3 Parallel Operation of CALA 185
5.4 Parallel Pursuit Algorithm 187
5.4.1 Simulation Studies 190
5.5 GeneralProcedure 190
5.6 Parallel Operation ofGames ofFALA 192
5.6.1 Analysis 194
5.6.2 Common PayoffGame 196
5.7 Parallel Operationof Networks ofFALA 197
5.7.1 Analysis 199
x NETWORKSOFLEARNINGAUTOMATA
5.7.2 Modules ofParameterized Learning Automata (PLA) 199
5.7.3 Modules ofGeneralized Learning Automata (GLA) 201
5.7.4 Pattern ClassificationExample 202
5.8 Discussion 203
5.9 SupplementaryRemarks 204
6. SOME RECENT APPLICATIONS 205
6.1 Introduction 205
6.2 SupervisedLearning of Perceptual Organization in Computer
Vision 206
6.3 Distributed Control ofBroadcast Communication Networks 213
6.4 Other Applications 218
6.5 Discussion 221
EPILOGUE 223
Appendices
A The ODE Approach to AnalysisofLearning Algorithms 227
A.1 Introduction 227
A.2 Derivation oftheODE Approximation 229
A.2.1 Assumptions 230
A.2.2 Analysis 231
A.3 Approximating ODEsforSome Automata Algorithms 234
A.3.1 LR-I Algorithm for aSingle Automaton 235
A.3.2 The CALA Algorithm 236
A.3.3 Automata TeamAlgorithms 239
A.4 Relaxing the Assumptions 239
B Proofs ofConvergencefor Pursuit Algorithm 241
B.l ProofofTheorem 1.1 241
B.2 ProofofTheorem 5.7 245
C WeakConvergence and SDE Approximations 247
C.1 Introduction 247
C.2 WeakConvergence 247
C.3 Convergence toSDE 248
C.3.1 Application to GlobalAlgorithms 250
C.4 Convergence to ODE 251
Contents xi
References 253
Index 265
Preface
The idea of machines which provide intelligent service to humans without
complaints offatigue or boredom, has been around for a long time. It moved
from theworld ofdreams totherealmofpossibility onlyinrecent times. While
there isnorigorous definition ofwhatconstitutes themultidimensionalconcept
of intelligence, it is commonly conceded that the intelligence component of
machinesofvarious types hasbeen steadily increasing inthepastfewdecades,
although it is still nowhere near thelevelobserved inhuman beings. There are
many ongoing efforts in different disciplines, all contributing in the direction
of improving the intelligence of machines inseveral dimensions.
Learningisacrucial aspect ofintelligence andmachine learning hasemerged
as a vibrant discipline with the avowedobjective of developing machines with
learning capabilities. This book represents an effort in the same direction and
discusses one approach tothe problem of learning from examples.
The book considers synthesis of complex learning structures from simple
building blocks. The simplest element is a learning automaton which learns
toselect the best action by repeated interactions with an unknown random en
vironment. It uses stochastic algorithms for refining probabilities of selecting
actions for this purpose. The paradigm isthat of learning from a probabilistic
teacherand constitutes areinforcementlearning model. Using thesingle learn
ingautomaton as thebuilding block, systems consisting ofseveral learning au
tomata such as games and feedforward networks are constructed. Mathemati
cal analysis of their behavior withsuitable learning algorithms isprovided and
adetailed discussion of howlearning automata solutions can beconstructed in
a variety of applications ispresented.
This book may beregarded asanatural successorofthebook, K.S.Narendra
and M.A.L.Thathachar, 'Learning Automata: An Introduction', Prentice Hall,
1989. However, the present book can be read independently. We provide a
fairly comprehensive account of learning automata models with emphasis on