Table Of Content

An Introduction to Pattern Recognition Michael Alder HeavenForBooks.com An Introduction to Pattern Recognition by Michael Alder HeavenForBooks.com An Introduction to Pattern Recognition This Edition ©Mike Alder, 2001 Warning: This edition is not to be copied, transmitted excerpted or printed except on terms authorised by the publisher HeavenForBooks.com An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear. Next: Contents An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear. Michael D. Alder September 19, 1997 Preface Automation, the use of robots in industry, has not progressed with the speed that many had hoped it would. The forecasts of twenty years ago are looking fairly silly today: the fact that they were produced largely by journalists for the benefit of boardrooms of accountants and MBA's may have something to do with this, but the question of why so little has been accomplished remains. The problems were, of course, harder than they looked to naive optimists. Robots have been built that can move around on wheels or legs, robots of a sort are used on production lines for routine tasks such as welding. But a robot that can clear the table, throw the eggshells in with the garbage and wash up the dishes, instead of washing up the eggshells and throwing the dishes in the garbage, is still some distance off. Pattern Classification, more often called Pattern Recognition, is the primary bottleneck in the task of automation. Robots without sensors have their uses, but they are limited and dangerous. In fact one might plausibly argue that a robot without sensors isn't a real robot at all, whatever the hardware manufacturers may say. But equipping a robot with vision is easy only at the hardware level. It is neither expensive nor technically difficult to connect a camera and frame grabber board to a computer, the robot's `brain'. The problem is with the software, or more exactly with the algorithms which have to decide what the robot is looking at; the input is an array of pixels, coloured dots, the software has to decide whether this is an image of an eggshell or a teacup. A task which human beings can master by age eight, when they decode the firing of the different light receptors in the retina of the eye, this is computationally very difficult, and we have only the crudest ideas of how it is done. At the hardware level there are marked similarities between the eye and a camera (although there are differences too). At the algorithmic level, we have only a shallow understanding of the issues. http://ciips.ee.uwa.edu.au/~mike/PatRec/ (1 of 11) [12/12/2000 4:01:56 AM] An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear. Human beings are very good at learning a large amount of information about the universe and how it can be treated; transferring this information to a program tends to be slow if not impossible. This has been apparent for some time, and a great deal of effort has been put into research into practical methods of getting robots to recognise things in images and sounds. The Centre for Intelligent Information Processing Systems (CIIPS), of the University of Western Australia, has been working in the area for some years now. We have been particularly concerned with neural nets and applications to pattern recognition in speech and vision, because adaptive or learning methods are clearly of great potential value. The present book has been used as a postgraduate textbook at CIIPS for a Master's level course in Pattern Recognition. The contents of the book are therefore oriented largely to image and to some extent speech pattern recognition, with some concentration on neural net methods. Students who did the course for which this book was originally written, also completed units in Automatic Speech Recognition Algorithms, Engineering Mathematics (covering elements of Information Theory, Coding Theory and Linear and Multilinear algebra), Artificial Neural Nets, Image Processing, Sensors and Instrumentation and Adaptive Filtering. There is some overlap in the material of this book and several of the other courses, but it has been kept to a minimum. Examination for the Pattern Recognition course consisted of a sequence of four micro-projects which together made up one mini-project. Since the students for whom this book was written had a variety of backgrounds, it is intended to be accessible. Since the major obstructions to further progress seem to be fundamental, it seems pointless to try to produce a handbook of methods without analysis. Engineering works well when it is founded on some well understood scientific basis, and it turns into alchemy and witchcraft when this is not the case. The situation at present in respect of our scientific basis is that it is, like the curate's egg, good in parts. We are solidly grounded at the hardware level. On the other hand, the software tools for encoding algorithms (C, C++, MatLab) are fairly primitive, and our grasp of what algorithms to use is negligible. I have tried therefore to focus on the ideas and the (limited) extent to which they work, since progress is likely to require new ideas, which in turn requires us to have a fair grasp of what the old ideas are. The belief that engineers as a class are not intelligent enough to grasp any ideas at all, and must be trained to jump through hoops, although common among mathematicians, is not one which attracts my sympathy. Instead of exposing the fundamental ideas in algebra (which in these degenerate days is less intelligible than Latin) I therefore try to make them plain in English. There is a risk in this; the ideas of science or engineering are quite diferent from those of philosophy (as practised in these degenerate days) or literary criticism (ditto). I don't mean they are about different things, they are different in kind. Newton wrote `Hypotheses non fingo', which literally translates as `I do not make hypotheses', which is of course quite untrue, he made up some spectacularly successful hypotheses, such as universal gravitation. The difference between the two statements is partly in the hypotheses and partly in the fingo. Newton's `hypotheses' could be tested by observation or calculation, whereas the explanations of, say, optics, given in Lucretius De Rerum Naturae were recognisably `philosophical' in the sense that they resembled the writings of many contemporary philosophers and literary critics. They may persuade, they may give the sensation of profound insight, but they do not reduce to some essentially prosaic routine for determining if they are actually true, or at least useful. Newton's did. This was one of the great philosophical advances made by Newton, and it has been underestimated by philosophers since. http://ciips.ee.uwa.edu.au/~mike/PatRec/ (2 of 11) [12/12/2000 4:01:56 AM] The reader should therefore approach the discussion about the underlying ideas with the attitude of irreverence and disrespect that most engineers, quite properly, bring to non-technical prose. He should ask: what procedures does this lead to, and how may they be tested? We deal with high level abstractions, but they are aimed always at reducing our understanding of something prodigiously complicated to something simple. It is necessary to make some assumptions about the reader and only fair to say what they are. I assume, first, that the reader has a tolerably good grasp of Linear Algebra concepts. The concepts are more important than the techniques of matrix manipulation, because there are excellent packages which can do the calculations if you know what to compute. There is a splendid book on Linear Algebra available from the publisher HeavenForBooks.com I assume, second, a moderate familiarity with elementary ideas of Statistics, and also of contemporary Mathematical notation such as any Engineer or Scientist will have encountered in a modern undergraduate course. I found it necessary in this book to deal with underlying ideas of Statistics which are seldom mentioned in undergraduate courses. I assume, finally, the kind of general exposure to computing terminology familiar to anyone who can read, say, Byte magazine, and also that the reader can program in C or some similar language. I do not assume the reader is of the male sex. I usually use the pronoun `he' in referring to the reader because it saves a letter and is the convention for the generic case. The proposition that this will depress some women readers to the point where they will give up reading and go off and become subservient housewives does not strike me as sufficiently plausible to be worth considering further. This is intended to be a happy, friendly book. It is written in an informal, one might almost say breezy, manner, which might irritate the humourless and those possessed of a conviction that intellectual respectability entails stuffiness. I used to believe that all academic books on difficult subjects were obliged for some mysterious reason to be oppressive, but a survey of the better writers of the past has shown me that this is in fact a contemporary habit and in my view a bad one. I have therefore chosen to abandon a convention which must drive intelligent people away from Science and Engineering in large numbers. The book has jokes, opinionated remarks and pungent value judgments in it, which might serve to entertain readers and keep them on their toes, so to speak. They may also irritate a few who believe that the pretence that the writer has no opinions should be maintained even at the cost of making the book boring. What this convention usually accomplishes is a sort of bland porridge which discourages critical thought about fundamental assumptions, and thought about fundamental assumptions is precisely what this area badly needs. An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear. So I make no apology for the occasional provocative judgement; argue with me if you disagree. It is quite easy to do that via the net, and since I enjoy arguing (it is a pleasant game), most of my provocations are deliberate. Disagreeing with people in an amiable, friendly way, and learning something about why people feel the way they do, is an important part of an education; merely learning the correct things to say doesn't get you very far in Mathematics, Science or Engineering. Cultured men or women should be able to dissent with poise, to refute the argument without losing the friend. The judgements are, of course, my own; CIIPS and the Mathematics Department and I are not responsible for each other. Nor is it to be expected that the University of Western Australia should ensure that my views are politically correct. If it did that, it wouldn't be a university. In a good university, It is a case of Tot homines, quot sententiae, there are as many opinions as people. Sometimes more! I am most grateful to my colleagues and students at the Centre for assistance in many forms; I have shamelessly borrowed their work as examples of the principles discussed herein. I must mention Dr. Chris deSilva with whom I have worked over many years, Dr. Gek Lim whose energy and enthusiasm for Quadratic Neural Nets has enabled them to become demonstrably useful, and Professor Yianni Attikiouzel, director of CIIPS, without whom neither this book nor the course would have come into existence. l C ontents l B asic Concepts m M easurement and Representation n F rom objects to points in space n T elling the guys from the gals n P aradigms m D ecisions, decisions.. n M etric Methods n N eural Net Methods (Old Style) n S tatistical Methods n P arametric n N on-parametric n C ART et al m C lustering: supervised v unsupervised learning m D ynamic Patterns m S tructured Patterns m A lternative Representations http://ciips.ee.uwa.edu.au/~mike/PatRec/ (4 of 11) [12/12/2000 4:01:57 AM] An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear. n S trings, propositions, predicates and logic n F uzzy Thinking n R obots m S ummary of this chapter m E xercises m B ibliography l Im age Measurements m P reliminaries n Im age File Formats m G eneralities m I mage segmentation: finding the objects n M athematical Morphology n L ittle Boxes n B order Tracing n C onclusions on Segmentation m M easurement Principles n Is sues and methods n In variance in practice m M easurement practice n Q uick and Dumb n S canline intersections and weights n M oments n Z ernike moments and the FFT n H istorical Note n M asks and templates n In variants n S implifications and Complications m S yntactic Methods m S ummary of OCR Measurement Methods m O ther Kinds of Binary Image m G reyscale images of characters n S egmentation: Edge Detection m G reyscale Images in general http://ciips.ee.uwa.edu.au/~mike/PatRec/ (5 of 11) [12/12/2000 4:01:57 AM] An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear. n S egmentation n M easuring Greyscale Images n Q uantisation n T extures m C olour Images n G eneralities n Q uantisation n E dge detection n M arkov Random Fields n M easurements m S pot counting m I R and acoustic Images m Q uasi-Images m D ynamic Images m S ummary of Chapter Two m E xercises m B ibliography l S tatistical Ideas m H istory, and Deep Philosophical Stuff n T he Origins of Probability: random variables n H istograms and Probability Density Functions n M odels and Probabilistic Models m P robabilistic Models as Data Compression Schemes n M odels and Data: Some models are better than others m M aximum Likelihood Models n W here do Models come from? m B ayesian Methods n B ayes' Theorem n B ayesian Statistics n S ubjective Bayesians m M inimum Description Length Models n C odes: Information theoretic preliminaries n C ompression for coin models http://ciips.ee.uwa.edu.au/~mike/PatRec/ (6 of 11) [12/12/2000 4:01:57 AM] An Introduction to Pattern Recognition: Statistical, Neural Net and Syntactic methods of getting robots to see and hear. n C ompression for pdfs n S ummary of Rissanen Complexity m S ummary of the chapter m E xercises m B ibliography l D ecisions: Statistical methods m T he view into m C omputing PDFs: Gaussians n O ne Gaussian per cluster n D imension 2 n L ots of Gaussians: The EM algorithm n T he EM algorithm for Gaussian Mixture Modelling n O ther Possibilities m B ayesian Decision n C ost Functions n N on-parametric Bayes Decisions n O ther Metrics m H ow many things in the mix? n O verhead n E xample n T he Akaike Information Criterion n P roblems with EM m S ummary of Chapter m E xercises m B ibliography l D ecisions: Neural Nets(Old Style) m H istory: the good old days n T he Dawn of Neural Nets n T he death of Neural Nets n T he Rebirth of Neural Nets n T he End of History m T raining the Perceptron n T he Perceptron Training Rule http://ciips.ee.uwa.edu.au/~mike/PatRec/ (7 of 11) [12/12/2000 4:01:57 AM]