Table Of Content

38mm RED BOX RULES ARE FOR PROOF STAGE ONLY. DELETE BEFORE FINAL PRINTING. Webb STATISTICAL Copsey PATTERN RECOGNITION S T Third Edition A T Andrew R. Webb and Keith D. Copsey I S Mathematics and Data Analysis Consultancy, Malvern, UK T I C Statistical pattern recognition relates to the use of statistical techniques for analysing data A measurements in order to extract information and make justified decisions. It is a very active area of study and research, which has seen many advances in recent years. Applications such as data L mining, web searching, multimedia data retrieval, face recognition, and cursive handwriting recognition, all require robust and efficient pattern recognition techniques. This third edition provides an introduction to statistical pattern theory and techniques, with P material drawn from a wide range of fields, including the areas of engineering, statistics, computer A STATISTICAL science and the social sciences. The book has been updated to cover new methods and applications, T and includes a wide range of techniques such as Bayesian methods, neural networks, support T vector machines, feature selection and feature reduction techniques. Technical descriptions and E motivations are provided, and the techniques are illustrated using real examples. R PATTERN N Statistical Pattern Recognition, Third Edition: • Provides a self-contained introduction to statistical pattern recognition. • Includes new material presenting the analysis of complex networks. R RECOGNITION • Introduces readers to methods for Bayesian density estimation. E • Presents descriptions of new applications in biometrics, security, finance and C condition monitoring. O • Provides descriptions and guidance for implementing techniques, which will be G T h i r d E d i t i o n invaluable to software engineers and developers seeking to develop real applications. N • Describes mathematically the range of statistical pattern recognition techniques. • Presents a variety of exercises including more extensive computer projects. IT I O The in-depth technical descriptions make this book suitable for senior undergraduate and graduate Andrew R. Webb students in statistics, computer science and engineering. Statistical Pattern Recognition is also an N excellent reference source for technical professionals. Chapters have been arranged to facilitate Keith D. Copsey implementation of the techniques by software engineers and developers in non-statistical engineering fields. T h i r d E d i t i o n www.wiley.com/go/statistical_pattern_recognition Cover design: Gary Thompson P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome Statistical Pattern Recognition P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome Statistical Pattern Recognition Third Edition Andrew R. Webb • Keith D. Copsey MathematicsandDataAnalysisConsultancy, Malvern,UK A John Wiley & Sons, Ltd., Publication P1: OTA/XYZ P2: ABC JWST102-fm JWST102-Webb September 8, 2011 8:52 Printer Name: Yet to Come This edition first published 2011 © 2011 John Wiley & Sons, Ltd Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com. TherightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththeCopyright, DesignsandPatentsAct1988. Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inanyform orbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbytheUKCopyright, DesignsandPatentsAct1988,withoutthepriorpermissionofthepublisher. Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynotbeavailablein electronicbooks. Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrandnamesandproduct namesusedinthisbookaretradenames,servicemarks,trademarksorregisteredtrademarksoftheirrespectiveowners. Thepublisherisnotassociatedwithanyproductorvendormentionedinthisbook.Thispublicationisdesignedtoprovide accurateandauthoritativeinformationinregardtothesubjectmattercovered.Itissoldontheunderstandingthatthe publisherisnotengagedinrenderingprofessionalservices.Ifprofessionaladviceorotherexpertassistanceisrequired,the servicesofacompetentprofessionalshouldbesought. LibraryofCongressCataloging-in-PublicationData Webb,A.R.(AndrewR.) Statisticalpatternrecognition/AndrewR.Webb,KeithD.Copsey.–3rded. p. cm. Includesbibliographicalreferencesandindex. ISBN978-0-470-68227-2(hardback)–ISBN978-0-470-68228-9(paper) 1.Patternperception–Statisticalmethods. I. Copsey,KeithD. II. Title. Q327.W432011 006.4–dc23 2011024957 AcataloguerecordforthisbookisavailablefromtheBritishLibrary. HBISBN:978-0-470-68227-2 PBISBN:978-0-470-68228-9 ePDFISBN:978-1-119-95296-1 oBookISBN:978-1-119-95295-4 ePubISBN:978-1-119-96140-6 MobiISBN:978-1-119-96141-3 Typesetin10/12ptTimesbyAptaraInc.,NewDelhi,India P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome ToRosemary, Samuel, Miriam,JacobandEthan P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome Contents Preface xix Notation xxiii 1 IntroductiontoStatisticalPatternRecognition 1 1.1 StatisticalPatternRecognition 1 1.1.1 Introduction 1 1.1.2 TheBasicModel 2 1.2 StagesinaPatternRecognitionProblem 4 1.3 Issues 6 1.4 ApproachestoStatisticalPatternRecognition 7 1.5 ElementaryDecisionTheory 8 1.5.1 Bayes’DecisionRuleforMinimumError 8 1.5.2 Bayes’DecisionRuleforMinimumError–RejectOption 12 1.5.3 Bayes’DecisionRuleforMinimumRisk 13 1.5.4 Bayes’DecisionRuleforMinimumRisk–RejectOption 15 1.5.5 Neyman–PearsonDecisionRule 15 1.5.6 MinimaxCriterion 18 1.5.7 Discussion 19 1.6 DiscriminantFunctions 20 1.6.1 Introduction 20 1.6.2 LinearDiscriminantFunctions 21 1.6.3 PiecewiseLinearDiscriminantFunctions 23 1.6.4 GeneralisedLinearDiscriminantFunction 24 1.6.5 Summary 26 1.7 MultipleRegression 27 1.8 OutlineofBook 29 1.9 NotesandReferences 29 Exercises 31 2 DensityEstimation–Parametric 33 2.1 Introduction 33 P1:OTA/XYZ P2:ABC JWST102-fm JWST102-Webb September8,2011 8:52 PrinterName:YettoCome viii CONTENTS 2.2 EstimatingtheParametersoftheDistributions 34 2.2.1 EstimativeApproach 34 2.2.2 PredictiveApproach 35 2.3 TheGaussianClassifier 35 2.3.1 Specification 35 2.3.2 DerivationoftheGaussianClassifierPlug-InEstimates 37 2.3.3 ExampleApplicationStudy 39 2.4 DealingwithSingularitiesintheGaussianClassifier 40 2.4.1 Introduction 40 2.4.2 Na¨ıveBayes 40 2.4.3 ProjectionontoaSubspace 41 2.4.4 LinearDiscriminantFunction 41 2.4.5 RegularisedDiscriminantAnalysis 42 2.4.6 ExampleApplicationStudy 44 2.4.7 FurtherDevelopments 45 2.4.8 Summary 46 2.5 FiniteMixtureModels 46 2.5.1 Introduction 46 2.5.2 MixtureModelsforDiscrimination 48 2.5.3 ParameterEstimationforNormalMixtureModels 49 2.5.4 NormalMixtureModelCovarianceMatrixConstraints 51 2.5.5 HowManyComponents? 52 2.5.6 MaximumLikelihoodEstimationviaEM 55 2.5.7 ExampleApplicationStudy 60 2.5.8 FurtherDevelopments 62 2.5.9 Summary 63 2.6 ApplicationStudies 63 2.7 SummaryandDiscussion 66 2.8 Recommendations 66 2.9 NotesandReferences 67 Exercises 67 3 DensityEstimation–Bayesian 70 3.1 Introduction 70 3.1.1 Basics 72 3.1.2 RecursiveCalculation 72 3.1.3 Proportionality 73 3.2 AnalyticSolutions 73 3.2.1 ConjugatePriors 73 3.2.2 EstimatingtheMeanofaNormalDistributionwith KnownVariance 75 3.2.3 EstimatingtheMeanandtheCovarianceMatrixofaMultivariate NormalDistribution 79 3.2.4 UnknownPriorClassProbabilities 85 3.2.5 Summary 87 3.3 BayesianSamplingSchemes 87 3.3.1 Introduction 87