Table Of Content

Paul Fieguth An Introduction to Pattern Recognition and Machine Learning An Introduction to Pattern Recognition and Machine Learning Paul Fieguth An Introduction to Pattern Recognition and Machine Learning PaulFieguth DepartmentofSystemsDesignEngineering, FacultyofEngineering UniversityofWaterloo Waterloo,ON,Canada ISBN978-3-030-95993-7 ISBN978-3-030-95995-1 (eBook) https://doi.org/10.1007/978-3-030-95995-1 ©SpringerNatureSwitzerlandAG2022 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar ordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Preface In 1996, I had the good fortune of being hired by the Department of Sys- tems Design Engineering at the University of Waterloo. As a department whichemphasizessystemstheory,designmethodology,andout-of-the-box thinking,itwasanaturalhomeforinterdisciplinarytopicssuchaspattern recognition,andIhadtheopportunitytoteachpatternrecognitionatboth theundergraduateandgraduatelevels. Atthetime,patternrecognitionandmachinelearningweresomethingofa nichearea,notofferedinmanyprogramsofstudy,andifatall,thenatthe graduate/research level, making systems design engineering all the more uniqueinhavingofferedapatternrecognitioncourseattheundergraduate levelsincethe1970s.Today,ofcourse,thesefieldshaveexplodedtobecome oneofthehottestareasofstudyandresearch,withstudentsfromnearlyevery fieldclamouringtoenrolinsuchcoursesandwithresearchersinnearlyevery lineofworkseekingwaysinwhichmachinelearning,morebroadly,ordeep learning,morespecifically,mightbeappliedtotheirdomain. Thisexplosioninpopularityisnotwithoutreason.Therearetwokeychanges which,amongmanyothers,havechangedthefield: 1.Patternrecognitionhasbecomemuchmorecapable:deep/convolutional neuralnetworkshaveallowedhigh-level,highlyabstract,andnonlinear problemstobesolved,openingupvastdomainsofdataprocessingwhich, onlyafewyearsearlier,werenotevenremotelyunderconsideration. 2.SensingandMeasurementhavebecomefarmoreubiquitous:thebillionsof cell-phones,cars,andInternet-of-Thingsdeviceshavecreatedanetworked worldofsensors,ofdata,andofopportunitiestomakeinferencesfrom suchdata. v vi Preface Althoughthereareagreatmanyonlinetutorialsandhow-toguidesfordeep learning,toreallyunderstandtheprocessofclassificationandinferencedata isquitenuanced.Thegoalofthisbookistoexplorethatnuanceandassociated insights,andhopefullytoprovidesomebalanceandcontexttothebreathless excitementandhype/exaggerationwhichiscommonlyencounteredtodayin anyconceptassociatedwithartificialintelligence. Anumberofpeopleneedthankinginmakingitpossibleformetoundertake aprojectofthisscope.1Firstandforemost,agreatdealofloveandthanksto mywife,Betty,whoisanardentsupporterofallofmyprojects,bothacademic andnon-academic,andwhethertheymakeanysenseornot. ThetextwouldneverhavecomeintobeingwithoutProfessorEdJernigan,the departmentchairwhohiredmeintosystemsdesign,whohadtheforesight tocreateanundergraduatepatternrecognitioncourselongbeforethatwas a normal thing to do, and who warmly included me as a colleague in his researchlab. Thetextwouldprobablyalsonothavecomeintobeingwithouttheenthu- siasticandtirelesslyenergeticProfessorAlexWong,whowasaverystrong supporterofanundergraduatetext.Hehadextensivesuggestionsandpro- videdmanyideasandinnovationsfromhisownyearsinteachingpattern recognition. Myappreciationforthepatienceofmychair(ProfessorLisaAultmann-Hall) and of my dean (Professor Mary Wells), in having a colleague who was somewhatdistracted,oratleastwhosetimewassomewhatcompromised, particularlythroughoutthe2020–2021Covid-19period.ThanksalsotoAnya Fieguthforallowingherartwork(page355,page378)toappear. Itis,asanauthor,veryeasytoreachapointofcognitivedissonance,whereby oneisconvincedoftheclarityoreleganceofanexposition,makingitcritical to have other eyes provide honest feedback. My thanks to Alex Wong and EddieParkfortheirfeedback,butmostparticularlytoNicholasPellegrino whoprovidedanamazinglyextensive,careful,anddetailedreading. 1Producingallofthefiguresinthistextrequiredthedevelopmentof15,933linesof code,withhindsightaperhapsill-adviseduseoftheauthor’ssparetime. Preface vii Finally, I would like to dedicate this text to two close colleagues who will sorelybemissed: InmemoryofIgor(Nov27,2020),whowasapassionate,inspiredteacher, andtrulydemonstratedaloveandcommitmenttoallofhisstudents. InmemoryofPearl(Nov28,2020),whowasaclosecolleaguefor10years, andhadagenerousheartandboundlessenergy. PaulFieguth Waterloo,ON,Canada Contents Preface v TableofContents ix ListofExamples xv ListofAlgorithms xvii Notation xix 1 Overview 1 2 IntroductiontoPatternRecognition 5 2.1 WhatIsPatternRecognition? . . . . . . . . . . . . . . . . . . . 5 2.2 MeasuredPatterns . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.3 Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.4 Classification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.5 TypesofClassificationProblems . . . . . . . . . . . . . . . . . 16 CaseStudy2:Biometrics . . . . . . . . . . . . . . . . . . . . . . . . . 19 NumericalLab2: TheIrisDataset . . . . . . . . . . . . . . . . . . 23 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 References 28 3 Learning 29 CaseStudy3:TheNetflixPrize . . . . . . . . . . . . . . . . . . . . . 44 NumericalLab3: OverfittingandUnderfitting . . . . . . . . . . . 46 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 References 53 ix x Contents 4 RepresentingPatterns 55 4.1 Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 4.2 ClassShape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 4.3 ClusterSynthesis . . . . . . . . . . . . . . . . . . . . . . . . . . 73 CaseStudy4:DefectDetection . . . . . . . . . . . . . . . . . . . . . 74 NumericalLab4: WorkingwithRandomNumbers . . . . . . . . 76 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 References 82 5 FeatureExtractionandSelection 83 5.1 FundamentalsofFeatureExtraction . . . . . . . . . . . . . . . 83 5.2 FeatureExtractionandSelection . . . . . . . . . . . . . . . . . 93 CaseStudy5:ImageSearching . . . . . . . . . . . . . . . . . . . . . 103 NumericalLab5: ExtractingFeaturesandPlottingClasses . . . . 104 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 References 114 6 Distance-BasedClassification 117 6.1 DefinitionsofDistance . . . . . . . . . . . . . . . . . . . . . . . 118 6.2 ClassPrototype . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 6.3 Distance-BasedClassification . . . . . . . . . . . . . . . . . . . 132 6.4 ClassifierVariations . . . . . . . . . . . . . . . . . . . . . . . . . 134 CaseStudy6:Hand-writingRecognition . . . . . . . . . . . . . . . . 138 NumericalLab6: Distance-BasedClassifiers . . . . . . . . . . . . 141 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 References 150 7 InferringClassModels 151 7.1 ParametricEstimation . . . . . . . . . . . . . . . . . . . . . . . 152 7.2 ParametricModelLearning . . . . . . . . . . . . . . . . . . . . 154 7.3 NonparametricModelLearning . . . . . . . . . . . . . . . . . . 164 7.3.1 HistogramEstimation . . . . . . . . . . . . . . . . . . . 165 7.3.2 Kernel-BasedEstimation . . . . . . . . . . . . . . . . . . 168 7.3.3 Neighbourhood-basedEstimation . . . . . . . . . . . . 172 7.4 DistributionAssessment . . . . . . . . . . . . . . . . . . . . . . 174 CaseStudy7:ObjectRecognition . . . . . . . . . . . . . . . . . . . . 179 NumericalLab7: ParametricandNonparametricEstimation . . . 180 Contents xi FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 References 191 8 Statistics-BasedClassification 193 8.1 Non-BayesianClassification:MaximumLikelihood . . . . . . 194 8.2 BayesianClassification:MaximumaPosteriori . . . . . . . . . 198 8.3 StatisticalClassificationforNormalDistributions . . . . . . . 201 8.4 ClassificationError . . . . . . . . . . . . . . . . . . . . . . . . . 204 8.5 OtherStatisticalClassifiers . . . . . . . . . . . . . . . . . . . . . 211 CaseStudy8:MedicalAssessments . . . . . . . . . . . . . . . . . . . 213 NumericalLab8: StatisticalandDistance-BasedClassifiers . . . . 218 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 References 230 9 ClassifierTestingandValidation 231 9.1 WorkingwithData . . . . . . . . . . . . . . . . . . . . . . . . . 231 9.2 ClassifierEvaluation . . . . . . . . . . . . . . . . . . . . . . . . 239 9.3 ClassifierValidation. . . . . . . . . . . . . . . . . . . . . . . . . 249 CaseStudy9:AutonomousVehicles . . . . . . . . . . . . . . . . . . 255 NumericalLab9: Leave-One-OutValidation . . . . . . . . . . . . 257 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 References 265 10 Discriminant-BasedClassification 267 10.1 LinearDiscriminants . . . . . . . . . . . . . . . . . . . . . . . . 269 10.2 DiscriminantModelLearning . . . . . . . . . . . . . . . . . . . 271 10.3 NonlinearDiscriminants . . . . . . . . . . . . . . . . . . . . . . 280 10.4 Multi-ClassProblems . . . . . . . . . . . . . . . . . . . . . . . . 285 CaseStudy10:DigitalCommunications . . . . . . . . . . . . . . . . 288 NumericalLab10: Discriminants . . . . . . . . . . . . . . . . . . . 291 FurtherReading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 References 298 11 EnsembleClassification 299 11.1 CombiningClassifiers . . . . . . . . . . . . . . . . . . . . . . . 301 11.2 ResamplingStrategies . . . . . . . . . . . . . . . . . . . . . . . 305 11.3 SequentialStrategies . . . . . . . . . . . . . . . . . . . . . . . . 312