Feature Extraction: Foundations and Applications PDF
Preview Feature Extraction: Foundations and Applications
IsabelleGuyon,SteveGunn,MasoudNikravesh,LotfiA.Zadeh(Eds.) FeatureExtraction StudiesinFuzzinessandSoftComputing,Volume 207 Editor-in-chief Prof.JanuszKacprzyk SystemsResearchInstitute PolishAcademyofSciences ul.Newelska6 01-447Warsaw Poland E-mail:[email protected] Furthervolumesofthisseries Vol.199.ZhongLi canbefoundonourhomepage: FuzzyChaoticSystems,2006 ISBN3-540-33220-0 springer.com Vol.200.KaiMichels,FrankKlawonn, RudolfKruse,AndreasNürnberger Vol.191.MartinV.Butz FuzzyControl,2006 Rule-BasedEvolutionaryOnlineLearning ISBN3-540-31765-1 Systems,2006 ISBN3-540-25379-3 Vol.201.CengizKahraman(Ed.) FuzzyApplicationsinIndustrial Vol.192.JoseA.Lozano,PedroLarrañaga, Engineering,2006 IñakiInza,EndikaBengoetxea(Eds.) ISBN3-540-33516-1 TowardsaNewEvolutionaryComputation, 2006 Vol.202.PatrickDoherty,Witold ISBN3-540-29006-0 Łukaszewicz,AndrzejSkowron,Andrzej Szałas Vol.193.IngoGlöckner KnowledgeRepresentationTechniques:A FuzzyQuantifiers:AComputationalTheory, RoughSetApproach,2006 2006 ISBN3-540-33518-8 ISBN3-540-29634-4 Vol.203.GloriaBordogna,GiuseppePsaila Vol.194.DawnE.Holmes,LakhmiC.Jain (Eds.) (Eds.) FlexibleDatabasesSupportingImprecision InnovationsinMachineLearning,2006 andUncertainty,2006 ISBN3-540-30609-9 ISBN3-540-33288-X Vol.195.ZongminMa Vol.204.ZongminMa(Ed.) FuzzyDatabaseModelingofImpreciseand SoftComputinginOntologiesandSemantic UncertainEngineeringInformation,2006 Web,2006 ISBN3-540-30675-7 ISBN3-540-33472-6 Vol.196.JamesJ.Buckley Vol.205.MikaSato-Ilic,LakhmiC.Jain FuzzyProbabilityandStatistics,2006 InnovationsinFuzzyClustering,2006 ISBN3-540-30841-5 ISBN3-540-34356-3 Vol.197.EnriqueHerrera-Viedma,Gabriella Vol.206.AshokSengupta(Ed.) Pasi,FabioCrestani(Eds.) Chaos,Nonlinearity,Complexity,2006 SoftComputinginWebInformation ISBN3-540-31756-2 Retrieval,2006 ISBN3-540-31588-8 Vol.207.IsabelleGuyon,SteveGunn, MasoudNikravesh,LotfiA.Zadeh(Eds.) Vol.198.HungT.Nguyen,BerlinWu FeatureExtraction,2006 FundamentalsofStatisticswithFuzzyData, ISBN3-540-35487-5 2006 ISBN3-540-31695-7 Isabelle Guyon Steve Gunn Masoud Nikravesh Lotfi A. Zadeh (Eds.) Feature Extraction Foundations and Applications ABC IsabelleGuyon MasoudNikravesh Clopinet DepartmentofElectrical 955CrestonRoad Engineering&Computer 94708Berkeley,USA Science–EECS E-mail:[email protected] UniversityofCalifornia 94720Berkeley,USA E-mail:[email protected] SteveGunn SchoolofElectronics LotfiA.Zadeh andComputerSciences UniversityofSouthampton DivisionofComputerScience SO171BJSouthampton Lab.ElectronicsResearch Highfield,UnitedKingdom UniversityofCalifornia E-mail:[email protected] SodaHall387 94720-1776Berkeley,CA,USA E-mail:[email protected] LibraryofCongressControlNumber:2006928001 ISSNprintedition:1434-9922 ISSNelectronicedition:1860-0808 ISBN-10 3-540-35487-5SpringerBerlinHeidelbergNewYork ISBN-13 978-3-540-35487-1SpringerBerlinHeidelbergNewYork Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis concerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broadcasting, reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationofthispublication orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9, 1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.Violationsare liableforprosecutionundertheGermanCopyrightLaw. SpringerisapartofSpringerScience+BusinessMedia springer.com (cid:2)c Springer-VerlagBerlinHeidelberg2006 PrintedinTheNetherlands Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnotimply, evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevantprotectivelaws andregulationsandthereforefreeforgeneraluse. Typesetting:bytheauthorsandtechbooksusingaSpringerLATEXmacropackage Coverdesign:ErichKirchner,Heidelberg Printedonacid-freepaper SPIN:10966471 89/techbooks 543210 To our friends and foes Foreword Everyonelovesagoodcompetition.AsIwritethis,twobillionfansareeagerly anticipating the 2006 World Cup. Meanwhile, a fan base that is somewhat smaller (but presumably includes you, dear reader) is equally eager to read allabouttheresultsof theNIPS2003 FeatureSelection Challenge, contained herein. Fans of Radford Neal and Jianguo Zhang (or of Bayesian neural net- works and Dirichlet diffusion trees) are gloating “I told you so” and looking forproofthattheirwinwasnotafluke.Butthematterisbynomeanssettled, and fans of SVMs are shouting “wait ’til next year!” You know this book is a bit more edgy than your standard academic treatise as soon as you see the dedication: “To our friends and foes.” Competition breeds improvement. Fifty years ago, the champion in 100m butterflyswimmingwas22percentslowerthantoday’schampion;thewomen’s marathonchampionfromjust30yearsagowas26percentslower.Whoknows how much better our machine learning algorithms would be today if Turing in 1950 had proposed an effective competition rather than his elusive Test? Butwhatmakesaneffectivecompetition?ThefieldofSpeechRecognition hashadNIST-runcompetitionssince1988;errorrateshavebeenreducedbya factorofthreeormore,butthefieldhasnotyethadtheimpactexpectedofit. InformationRetrievalhashaditsTRECcompetitionsince1992;progresshas been steady and refugees from the competition have played important roles in the hundred-billion-dollar search industry. Robotics has had the DARPA GrandChallengeforonlytwoyears,butinthattimewehaveseentheresults go from complete failure to resounding success (although it may have helped that the second year’s course was somewhat easier than the first’s). I think there are four criteria that define effective technical competitions: 1. The task must be approachable. Non-experts should be able to enter, to see some results, and learn from their better-performing peers. 2. The scoring must be incremental. A pass-fail competition where everyone always fails (such as the Turing Test) makes for a boring game and dis- couragesfurthercompetition.OnthisscoretheLoebnerPrize,despiteits VIII Foreword faults,isabettercompetitionthantheoriginalTuringTest.Inonesense, everyone failed the DARPA Grand Challenge in the first year (because noentrant finished the race),butin another sensetherewereincremental scores: the distance each robot travelled, and the average speed achieved. 3. The results should be open. Participants and spectators alike should be able to learn the best practices of all participants. This means that each participant should describe their approaches in a written document, and thatthedata,auxiliaryprograms,andresultsshouldbepubliclyavailable. 4. The task should be relevant to real-world tasks. One of the problems with early competitions in speech recognition was that the emphasis on reducingworderrorratesdidnotnecessarilyleadtoastrongspeechdialog system—you could get almost all the words right and still have a bad dialog,andconverselyyoucouldmissmanyofthewordsandstillrecover. Morerecentcompetitionshavedoneabetterjobofconcentratingontasks that really matter. The Feature Selection Challenge meets the first three criteria easily. Sev- entyfiveteamsentered,sotheymusthavefounditapproachable.Thescoring didagoodjobofseparatingthetopperformerswhilekeepingeveryoneonthe scale. And the results are all available online, in this book, and in the accom- panying CD. All the data and Matlab code is provided, so the Challenge is easily reproducible. The level of explication provided by the entrants in the chapters of this book is higher than in other similar competitions. The fourth criterion, real-world relevance, is perhaps the hardest to achieve. Only time will tell whether the Feature Selection Challenge meets this one. In the mean time,thisbooksetsahighstandardasthepublicrecordofaninterestingand effective competition. Palo Alto, California Peter Norvig January 2006 Preface Feature extraction addresses the problem of finding the most compact and informative set of features, to improve the efficiency or data storage and processing. Defining feature vectors remains the most common and conve- nient means of data representation for classification and regression problems. Data can then be stored in simple tables (lines representing “entries”, “data points, “samples”, or “patterns”, and columns representing “features”). Each feature results from a quantitative or qualitative measurement, it is an “at- tribute” or a “variable”. Modern feature extraction methodology is driven by the size of the data tables, which is ever increasing as data storage becomes more and more efficient. After many years of parallel efforts, researchers in Soft-Computing, Sta- tistics, Machine Learning, and Knowledge Discovery, who are interested in predictive modeling are uniting their effort to advance the problem of fea- ture extraction. The recent advances made in both sensor technologies and machine learning techniques make it possible to design recognition systems, whicharecapableofperformingtasksthatcouldnotbeperformedinthepast. Featureextractionliesatthecenteroftheseadvanceswithapplicationsinthe pharmaco-medical industry, oil industry, industrial inspection and diagnosis systems, speech recognition, biotechnology, Internet, targeted marketing and many of other emerging applications. The present book is organized around the results of a benchmark that took place in 2003. Dozens of research groups competed on five large feature selection problems from various application domains: medical diagnosis, text processing, drug discovery, and handwriting recognition. The results of this effortpavetheway toanewgeneration ofmethodscapableofanalyzingdata tables with million of lines and/or columns. Part II of the book summarizes the results of the competition and gath- ers the papers describing the methods used by the top ranking participants. Following the competition, a NIPS workshop took place in December 2003 to discuss the outcomes of the competition and new avenues in feature ex- traction. The contributions providing new perspectives are found in Part III X Preface of the book. Part I provides all the necessary foundations to understand the recent advances made in Parts II and III. The book is complemented by ap- pendices and by a web site. The appendices include fact sheets summarizing themethodsusedinthecompetition,tablesofresultsofthecompetition,and a summary of basic concepts of statistics. This book is directed to students, researchers, and engineers. It presents recent advances in the field and complements an earlier book (Liu and Mo- toda, 1998), which provides a thorough bibliography and presents methods of historical interest, but explores only small datasets and ends before the new era of kernel methods. Readers interested in the historical aspects of the problemarealsodirectedto(DevijverandKittler,1982).Acompletelynovice reader will find all the necessary elements to understand the material of the book presented in the tutorial chapters of Part I. The book can be used as teachingmaterialforagraduateclassinstatisticsandmachinelearning,Part I supporting the lectures, Part II and III providing readings, and the CD providing data for computer projects. Zu¨rich, Switzerland Isabelle Guyon Southampton, UK Steve Gunn Berkeley, California Masoud Nikravesh and Lofti A. Zadeh November 2005 References P.A.DevijverandJ.Kittler. PatternRecognition:AStatisticalApproach. Prentice- Hall, 1982. H. Liu and H. Motoda. Feature Extraction, Construction and Selection: A Data Mining Perspective. Kluwer Academic, 1998. Contents An Introduction to Feature Extraction Isabelle Guyon, Andr´e Elisseeff .................................... 1 1 Feature Extraction Basics.................................... 1 2 What is New in Feature Extraction?........................... 7 3 Getting Started............................................. 9 4 Advanced Topics and Open Problems.......................... 16 5 Conclusion ................................................. 22 References ...................................................... 23 A Forward Selection with Gram-Schmidt Orthogonalization ........ 24 B Justification of the Computational Complexity Estimates ........ 25 Part I Feature Extraction Fundamentals 1 Learning Machines Norbert Jankowski, Krzysztof Grabczewski........................... 29 1.1 Introduction................................................ 29 1.2 The Learning Problem....................................... 29 1.3 Learning Algorithms ........................................ 35 1.4 Some Remarks on Learning Algorithms ........................ 57 References ...................................................... 58 2 Assessment Methods G´erard Dreyfus, Isabelle Guyon .................................... 65 2.1 Introduction................................................ 65 2.2 A Statistical View of Feature Selection: Hypothesis Tests and Random Probes......................................... 66 2.3 A Machine Learning View of Feature Selection.................. 78 2.4 Conclusion ................................................. 86 References ...................................................... 86
The list of books you might like
