Classification and Approximation with Rule-Based Networks

113 Pages·1999·0.91 MB·English

Classification and Approximation
with Rule-Based Networks
Charles M. Higgins, Jr.
Department of Electrical Engineering
California Institute of Technology
Pasadena, California
1993
CLASSIFICATION AND APPROXIMATION
WITH RULE-BASED NETWORKS
Thesis by
Charles M. Higgins, Jr.
In Partial Ful(cid:12)llment of the Requirements
for the Degree of
Doctor of Philosophy
California Institute of Technology
Pasadena, California
1993
(Defended May 12, 1993)
ii
Publication History:
(cid:15) Thesisapproved by defensecommittee (defense May 12, 1993):
Dr. Edward Posner, EE May 12
Dr. Joel Franklin, AMa May 12
Dr. Yaser Abu-Mostafa, EE May 12
Dr. Rodney Goodman, EE (Chair) May 19
Dr. Richard Murray, ME May 25
(cid:15) Final thesis delivered to Graduate O(cid:14)ce May 27, 1993
(cid:15) First printing, June ’93 - 15 copies
(cid:15) Second printing - special hardback edition, June ’93 - 3 copies
(to author’s parents, author, and advisor)
(cid:13)c 1993
Charles M. Higgins, Jr.
All Rights Reserved
iii
I would never have written this thesis without the love and support of my parents
through my ten-year college ordeal. They have consistently supported me both
monetarily and emotionally, and it is to them that I truly owe this achievement.
This thesis is dedicated to my parents, with love.
iv
v
Acknowledgements
First, Iwouldliketo thank my advisor,RodGoodman, forhispracticaladvice ondi(cid:14)cult
problems,hisendlessenthusiasmfor research, andhisuncriticalsupport. Iwouldalso like
to thank Rod for providing me with the best equipment and virtuallyunlimitedresources
for pursuing my many ideas over the last four years.
I wouldliketo thank the membersof the MicroSystemsGroup, bothpast and present,
for enlightening research discussions, mathematical consultations, late-night computer
game playing, snack runs to the stockroom, and a generally pleasant atmosphere in which
to work. I thank you all for your tolerance of the noise and disturbance my experimenta-
tion made.
Every member of the group has at some point helped me to work on the model car
experiment(whichdidn’tmake itinto thisthesis!), butIwouldnever have gotten asfaras
I did on it without the insightfulsuggestions of Bhusan Gupta, Chris Ulmer, Je(cid:11) Dickson
and Andrew Lundsten.
My special thanks to Je(cid:11) Dickson for helping me get started on the ball-and-beam
hardware.
I would like to thank Padhraic Smyth, on whose research all my work is based, for
four years of consultation on a wide range of issues, culminated by a 15-minute discussion
in which he suggested a solution to all the theoretical issues I had been unable to resolve
for three years.
Part two of this thesis, and especially chapter nine, bene(cid:12)tted greatly from the com-
ments of Dr. Richard Murray. I am indebted to him not only for his practical advice, but
also for helping me to see my work in a control systems perspective.
Finally, I would like to thank Michelle Stratton for endless spins on the dance (cid:13)oor
and for helping me to release all of the pressures of graduate life. Your love and devotion
makes this all worthwhile.
vi
vii
Preface
Systems that can Explain
We live in an age in which most of the di(cid:14)cult problems { includingpattern classi(cid:12)cation
andfunctionapproximation,thetwo problemsaddressedinthisthesis{havebeenstudied
in depth. The breadth of research in any given (cid:12)eld is absolutely astounding. There is a
largebodyoftheorywhichtellsushowwellwecandoando(cid:11)ersamultitudeofmethodsfor
achievingnearlythat performance. What, then,islefttoberesearchedintheseproblems?
The availability of computing power is at an all time high, so optimizing the speed of a
solution is no longer of primary concern. Are not existing methods good enough for any
application?
If the only concern is performance, the answer is yes. We can achieve near-optimal
classi(cid:12)cation performance with a number of algorithms; the same can be said for function
approximation. However, these algorithms may be unsatisfactory for many applications
because it is di(cid:14)cult to understand what the system is doing; they lack the ability to
explain to the user what has been learned about the problem.
Can we retain near-optimal performance while making systems easier to understand
and use? We can, and that philosophy pervades this thesis. Systems which express their
problem-speci(cid:12)c knowledge in the form of rules seem quite intuitive to humans and, as
shown in the pages to follow, can achieve performance comparable with other paradigms.
These e(cid:11)orts are a major step towards computing systems which can not only solve a
di(cid:14)cult problem, but also explain how they have done it.
viii
ix
Abstract
Thisthesisdescribesthearchitectureoflearningsystems whichcanexplaintheirdecisions
through a rule-based knowledge representation. Two problems in learning are addressed:
pattern classi(cid:12)cation and function approximation.
InPartI,apatternclassi(cid:12)erfordiscrete-valuedproblemsispresented. Thesystemuti-
lizes an information-theoretic algorithm for constructing informative rules from example
data. These rules are then used to construct a computational network to perform parallel
inference and posterior probability estimation. The network can be extended incremen-
tally; that is, new data can be incorporated without repeating the training on previous
data. It is shown that this technique performs comparably with other techniques includ-
ing the backpropagation network while having unique advantages in incremental learning
capability, training e(cid:14)ciency, and knowledge representation. Examples are shown of rule-
based classi(cid:12)cation and explanation.
In Part II, we present a method for the learning of fuzzy logic membership functions
and rules to predict a numerical function from examples of the function and its indepen-
dent variables. This method uses a three-step approach to building a complete function
approximation system: (cid:12)rst, learning the membership functions and creating a cell-based
rulerepresentation; second, simplifyingthe cell-basedrulesusingan information-theoretic
approach for inductionof rulesfrom discrete-valueddata; and (cid:12)nally, constructing a com-
putational network to compute the function value given its independent variables. Appli-
cations of the system to adaptive control are suggested, includinga method for learning a
complete control system for an unknown plant. Experimental validation of the suggested
methods using a ball-and-beam system is shown.