An Introduction to Pattern Recognition

561 Pages·2001·4.375 MB·English

by  Alder M.

An Introduction to
Pattern Recognition
Michael Alder
HeavenForBooks.com
An Introduction
to Pattern
Recognition
by
Michael Alder
HeavenForBooks.com
An Introduction to Pattern Recognition
This Edition ©Mike Alder, 2001
Warning: This edition is not to be
copied, transmitted excerpted or printed except
on terms authorised by the publisher
HeavenForBooks.com
An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear.

Next: Contents
An Introduction to Pattern Recognition:
Statistical, Neural Net and Syntactic
methods of getting robots to see and
hear.
Michael D. Alder
September 19, 1997
Preface
Automation, the use of robots in industry, has not progressed with the speed that many had hoped it
would. The forecasts of twenty years ago are looking fairly silly today: the fact that they were produced
largely by journalists for the benefit of boardrooms of accountants and MBA's may have something to do
with this, but the question of why so little has been accomplished remains.
The problems were, of course, harder than they looked to naive optimists. Robots have been built that
can move around on wheels or legs, robots of a sort are used on production lines for routine tasks such as
welding. But a robot that can clear the table, throw the eggshells in with the garbage and wash up the
dishes, instead of washing up the eggshells and throwing the dishes in the garbage, is still some distance
off.
Pattern Classification, more often called Pattern Recognition, is the primary bottleneck in the task of
automation. Robots without sensors have their uses, but they are limited and dangerous. In fact one might
plausibly argue that a robot without sensors isn't a real robot at all, whatever the hardware manufacturers
may say. But equipping a robot with vision is easy only at the hardware level. It is neither expensive nor
technically difficult to connect a camera and frame grabber board to a computer, the robot's `brain'. The
problem is with the software, or more exactly with the algorithms which have to decide what the robot is
looking at; the input is an array of pixels, coloured dots, the software has to decide whether this is an
image of an eggshell or a teacup. A task which human beings can master by age eight, when they decode
the firing of the different light receptors in the retina of the eye, this is computationally very difficult, and
we have only the crudest ideas of how it is done. At the hardware level there are marked similarities
between the eye and a camera (although there are differences too). At the algorithmic level, we have only
a shallow understanding of the issues.
http://ciips.ee.uwa.edu.au/~mike/PatRec/ (1 of 11) [12/12/2000 4:01:56 AM]
An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear.
Human beings are very good at learning a large amount of information about the universe and how it can
be treated; transferring this information to a program tends to be slow if not impossible.
This has been apparent for some time, and a great deal of effort has been put into research into practical
methods of getting robots to recognise things in images and sounds. The Centre for Intelligent
Information Processing Systems (CIIPS), of the University of Western Australia, has been working in the
area for some years now. We have been particularly concerned with neural nets and applications to
pattern recognition in speech and vision, because adaptive or learning methods are clearly of great
potential value. The present book has been used as a postgraduate textbook at CIIPS for a Master's level
course in Pattern Recognition. The contents of the book are therefore oriented largely to image and to
some extent speech pattern recognition, with some concentration on neural net methods.
Students who did the course for which this book was originally written, also completed units in
Automatic Speech Recognition Algorithms, Engineering Mathematics (covering elements of Information
Theory, Coding Theory and Linear and Multilinear algebra), Artificial Neural Nets, Image Processing,
Sensors and Instrumentation and Adaptive Filtering. There is some overlap in the material of this book
and several of the other courses, but it has been kept to a minimum. Examination for the Pattern
Recognition course consisted of a sequence of four micro-projects which together made up one
mini-project.
Since the students for whom this book was written had a variety of backgrounds, it is intended to be
accessible. Since the major obstructions to further progress seem to be fundamental, it seems pointless to
try to produce a handbook of methods without analysis. Engineering works well when it is founded on
some well understood scientific basis, and it turns into alchemy and witchcraft when this is not the case.
The situation at present in respect of our scientific basis is that it is, like the curate's egg, good in parts.
We are solidly grounded at the hardware level. On the other hand, the software tools for encoding
algorithms (C, C++, MatLab) are fairly primitive, and our grasp of what algorithms to use is negligible. I
have tried therefore to focus on the ideas and the (limited) extent to which they work, since progress is
likely to require new ideas, which in turn requires us to have a fair grasp of what the old ideas are. The
belief that engineers as a class are not intelligent enough to grasp any ideas at all, and must be trained to
jump through hoops, although common among mathematicians, is not one which attracts my sympathy.
Instead of exposing the fundamental ideas in algebra (which in these degenerate days is less intelligible
than Latin) I therefore try to make them plain in English.
There is a risk in this; the ideas of science or engineering are quite diferent from those of philosophy (as
practised in these degenerate days) or literary criticism (ditto). I don't mean they are about different
things, they are different in kind. Newton wrote `Hypotheses non fingo', which literally translates as `I do
not make hypotheses', which is of course quite untrue, he made up some spectacularly successful
hypotheses, such as universal gravitation. The difference between the two statements is partly in the
hypotheses and partly in the fingo. Newton's `hypotheses' could be tested by observation or calculation,
whereas the explanations of, say, optics, given in Lucretius De Rerum Naturae were recognisably
`philosophical' in the sense that they resembled the writings of many contemporary philosophers and
literary critics. They may persuade, they may give the sensation of profound insight, but they do not
reduce to some essentially prosaic routine for determining if they are actually true, or at least useful.
Newton's did. This was one of the great philosophical advances made by Newton, and it has been
underestimated by philosophers since.
http://ciips.ee.uwa.edu.au/~mike/PatRec/ (2 of 11) [12/12/2000 4:01:56 AM]
The reader should therefore approach the discussion about the underlying ideas with the attitude
of irreverence and disrespect that most engineers, quite properly, bring to non-technical prose.
He should ask: what procedures does this lead to, and how may they be tested? We deal with
high level abstractions, but they are aimed always at reducing our understanding of something
prodigiously complicated to something simple.
It is necessary to make some assumptions about the reader and only fair to say what they are.
I assume, first, that the reader has a tolerably good grasp of Linear Algebra concepts. The
concepts are more important than the techniques of matrix manipulation, because there are
excellent packages which can do the calculations if you know what to compute. There is a
splendid book on Linear Algebra available from the publisher HeavenForBooks.com
I assume, second, a moderate familiarity with elementary ideas of Statistics, and also of
contemporary Mathematical notation such as any Engineer or Scientist will have encountered in
a modern undergraduate course. I found it necessary in this book to deal with underlying ideas
of Statistics which are seldom mentioned in undergraduate courses.
I assume, finally, the kind of general exposure to computing terminology familiar to anyone
who can read, say, Byte magazine, and also that the reader can program in C or some similar
language.
I do not assume the reader is of the male sex. I usually use the pronoun `he' in referring to the
reader because it saves a letter and is the convention for the generic case. The proposition that
this will depress some women readers to the point where they will give up reading and go off
and become subservient housewives does not strike me as sufficiently plausible to be worth
considering further.
This is intended to be a happy, friendly book. It is written in an informal, one might almost say
breezy, manner, which might irritate the humourless and those possessed of a conviction that
intellectual respectability entails stuffiness. I used to believe that all academic books on difficult
subjects were obliged for some mysterious reason to be oppressive, but a survey of the better
writers of the past has shown me that this is in fact a contemporary habit and in my view a bad
one. I have therefore chosen to abandon a convention which must drive intelligent people away
from Science and Engineering in large numbers.
The book has jokes, opinionated remarks and pungent value judgments in it, which might serve
to entertain readers and keep them on their toes, so to speak. They may also irritate a few who
believe that the pretence that the writer has no opinions should be maintained even at the cost of
making the book boring. What this convention usually accomplishes is a sort of bland porridge
which discourages critical thought about fundamental assumptions, and thought about
fundamental assumptions is precisely what this area badly needs.
An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear.
So I make no apology for the occasional provocative judgement; argue with me if you disagree. It is
quite easy to do that via the net, and since I enjoy arguing (it is a pleasant game), most of my
provocations are deliberate. Disagreeing with people in an amiable, friendly way, and learning something
about why people feel the way they do, is an important part of an education; merely learning the correct
things to say doesn't get you very far in Mathematics, Science or Engineering. Cultured men or women
should be able to dissent with poise, to refute the argument without losing the friend.
The judgements are, of course, my own; CIIPS and the Mathematics Department and I are not
responsible for each other. Nor is it to be expected that the University of Western Australia should ensure
that my views are politically correct. If it did that, it wouldn't be a university. In a good university, It is a
case of Tot homines, quot sententiae, there are as many opinions as people. Sometimes more!
I am most grateful to my colleagues and students at the Centre for assistance in many forms; I have
shamelessly borrowed their work as examples of the principles discussed herein. I must mention Dr.
Chris deSilva with whom I have worked over many years, Dr. Gek Lim whose energy and enthusiasm
for Quadratic Neural Nets has enabled them to become demonstrably useful, and Professor Yianni
Attikiouzel, director of CIIPS, without whom neither this book nor the course would have come into
existence.

l C ontents
l B asic Concepts
m M easurement and Representation
n F rom objects to points in space
n T elling the guys from the gals
n P aradigms
m D ecisions, decisions..
n M etric Methods
n N eural Net Methods (Old Style)
n S tatistical Methods
n P arametric
n N on-parametric
n C ART et al
m C lustering: supervised v unsupervised learning
m D ynamic Patterns
m S tructured Patterns
m A lternative Representations
http://ciips.ee.uwa.edu.au/~mike/PatRec/ (4 of 11) [12/12/2000 4:01:57 AM]
An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear.
n S trings, propositions, predicates and logic
n F uzzy Thinking
n R obots
m S ummary of this chapter
m E xercises
m B ibliography
l Im age Measurements
m P reliminaries
n Im age File Formats
m G eneralities
m I mage segmentation: finding the objects
n M athematical Morphology
n L ittle Boxes
n B order Tracing
n C onclusions on Segmentation
m M easurement Principles
n Is sues and methods
n In variance in practice
m M easurement practice
n Q uick and Dumb
n S canline intersections and weights
n M oments
n Z ernike moments and the FFT
n H istorical Note
n M asks and templates
n In variants
n S implifications and Complications
m S yntactic Methods
m S ummary of OCR Measurement Methods
m O ther Kinds of Binary Image
m G reyscale images of characters
n S egmentation: Edge Detection
m G reyscale Images in general
http://ciips.ee.uwa.edu.au/~mike/PatRec/ (5 of 11) [12/12/2000 4:01:57 AM]
An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear.
n S egmentation
n M easuring Greyscale Images
n Q uantisation
n T extures
m C olour Images
n G eneralities
n Q uantisation
n E dge detection
n M arkov Random Fields
n M easurements
m S pot counting
m I R and acoustic Images
m Q uasi-Images
m D ynamic Images
m S ummary of Chapter Two
m E xercises
m B ibliography
l S tatistical Ideas
m H istory, and Deep Philosophical Stuff
n T he Origins of Probability: random variables
n H istograms and Probability Density Functions
n M odels and Probabilistic Models
m P robabilistic Models as Data Compression Schemes
n M odels and Data: Some models are better than others
m M aximum Likelihood Models
n W here do Models come from?
m B ayesian Methods
n B ayes' Theorem
n B ayesian Statistics
n S ubjective Bayesians
m M inimum Description Length Models
n C odes: Information theoretic preliminaries
n C ompression for coin models
http://ciips.ee.uwa.edu.au/~mike/PatRec/ (6 of 11) [12/12/2000 4:01:57 AM]
An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear.
n C ompression for pdfs
n S ummary of Rissanen Complexity
m S ummary of the chapter
m E xercises
m B ibliography
l D ecisions: Statistical methods
m T he view into
m C omputing PDFs: Gaussians
n O ne Gaussian per cluster
n D imension 2
n L ots of Gaussians: The EM algorithm
n T he EM algorithm for Gaussian Mixture Modelling
n O ther Possibilities
m B ayesian Decision
n C ost Functions
n N on-parametric Bayes Decisions
n O ther Metrics
m H ow many things in the mix?
n O verhead
n E xample
n T he Akaike Information Criterion
n P roblems with EM
m S ummary of Chapter
m E xercises
m B ibliography
l D ecisions: Neural Nets(Old Style)
m H istory: the good old days
n T he Dawn of Neural Nets
n T he death of Neural Nets
n T he Rebirth of Neural Nets
n T he End of History
m T raining the Perceptron
n T he Perceptron Training Rule
http://ciips.ee.uwa.edu.au/~mike/PatRec/ (7 of 11) [12/12/2000 4:01:57 AM]