Pattern Classification Using Ensemble Methods

237 Pages·2010·1.972 MB·English

by  Lior Rokach

PATTERN CLASSIFICATION
USING
ENSEMBLE METHODS
SERIES IN MACHINE PERCEPTION AND ARTIFICIAL INTELLIGENCE*
Editors: H. Bunke (Univ. Bern, Switzerland)
P. S. P. Wang (Northeastern Univ., USA)
Vol. 60: Robust Range Image Registration Using Genetic Algorithms
and the Surface Interpenetration Measure
(L. Silva, O. R. P. Bellon and K. L. Boyer)
Vol. 61: Decomposition Methodology for Knowledge Discovery and Data Mining:
Theory and Applications
(O. Maimon and L. Rokach)
Vol. 62: Graph-Theoretic Techniques for Web Content Mining
(A. Schenker, H. Bunke, M. Last and A. Kandel)
Vol. 63: Computational Intelligence in Software Quality Assurance
(S. Dick and A. Kandel)
Vol. 64: The Dissimilarity Representation for Pattern Recognition: Foundations
and Applications
(Elóbieta P“kalska and Robert P. W. Duin)
Vol. 65: Fighting Terror in Cyberspace
(Eds. M. Last and A. Kandel)
Vol. 66: Formal Models, Languages and Applications
(Eds. K. G. Subramanian, K. Rangarajan and M. Mukund)
Vol. 67: Image Pattern Recognition: Synthesis and Analysis in Biometrics
(Eds. S. N. Yanushkevich, P. S. P. Wang, M. L. Gavrilova and
S. N. Srihari)
Vol. 68: Bridging the Gap Between Graph Edit Distance and Kernel Machines
(M. Neuhaus and H. Bunke)
Vol. 69: Data Mining with Decision Trees: Theory and Applications
(L. Rokach and O. Maimon)
Vol. 70: Personalization Techniques and Recommender Systems
(Eds. G. Uchyigit and M. Ma)
Vol. 71: Recognition of Whiteboard Notes: Online, Offline and Combination
(Eds. H. Bunke and M. Liwicki)
Vol. 72: Kernels for Structured Data
(T Gärtner)
Vol. 73: Progress in Computer Vision and Image Analysis
(Eds. H. Bunke, J. J. Villanueva, G. Sánchez and X. Otazu)
Vol. 74: Wavelet Theory Approach to Pattern Recognition (2nd Edition)
(Y Y Tang)
Vol. 75: Pattern Classification Using Ensemble Methods
(L Rokach)
*For the complete list of titles in this series, please write to the Publisher.
Series in Machine Perception and Artificial Intelligence – Vol. 75
PATTERN CLASSIFICATION
USING
ENSEMBLE METHODS
Lior Rokach
Ben-Gurion University of the Negev, Israel
World Scientific
NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • HONG KONG • TAIPEI • CHENNAI
Published by
World Scientific Publishing Co. Pte. Ltd.
5 Toh Tuck Link, Singapore 596224
USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601
UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE
British Library Cataloguing-in-Publication Data
A catalogue record for this book is available from the British Library.
Series in Machine Perception and Artificial Intelligence — Vol. 75
PATTERN CLASSIFICATION USING ENSEMBLE METHODS
Copyright © 2010 by World Scientific Publishing Co. Pte. Ltd.
All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means,
electronic or mechanical, including photocopying, recording or any information storage and retrieval
system now known or to be invented, without written permission from the Publisher.
For photocopying of material in this volume, please pay a copying fee through the Copyright
Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to
photocopy is not required from the publisher.
ISBN-13 978-981-4271-06-6
ISBN-10 981-4271-06-3
Printed in Singapore.
To my wife, Ronit and
my three boys, Yarden, Roy and
Amit who was born the same day this book was completed
–L.R.
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
Preface
Ensemble methodology imitates our secondnature to seek severalopinions
before making a crucial decision. The core principle is to weigh several
individual pattern classifiers, and combine them in order to reach a classi-
fication that is better than the one obtained by each of them separately.
Researchersfromvarious disciplines suchas pattern recognition,statis-
tics,andmachinelearninghaveexploredtheuseofensemblemethodssince
thelateseventies. Giventhegrowinginterestinthefield,itisnotsurprising
that researchers and practitioners have a wide variety of methods at their
disposal. Pattern Classification Using Ensemble Methods aims to provide
a methodic and well structured introduction into this world by presenting
a coherent and unified repository of ensemble methods, theories, trends,
challenges and applications.
Its informative, factual pages will provide researchers, students and
practitioners in industry with a comprehensive,yet concise and convenient
referencesourcetoensemblemethods. Thebookdescribesindetailtheclas-
sicalmethods,aswellasextensionsandnovelapproachesthatwererecently
introduced. Alongwithalgorithmicdescriptionsofeachmethod,thereader
isprovidedwithadescriptionofthe settingsinwhichthis methodisappli-
cable and with the consequences and the trade-offs incurred by using the
method. This book is dedicated entirely to the field of ensemble methods
and covers all aspects of this important and fascinating methodology.
The book consists of seven chapters. Chapter 1 presents the pattern
recognition foundations that are required for reading the book. Chapter
2 introduces the basic algorithmic framework for building an ensemble of
classifiers. Chapters 3-6 present specific building blocks for designing and
implementing ensemble methods. Finally, Chapter 7 discusses how ensem-
bles should be evaluated. Several selection criteria are proposed - all are
vii
viii PatternClassification Using Ensemble Methods
presentedfroma practitioner’s standpoint- for choosingthe mosteffective
ensemble method.
Throughout the book, special emphasis was put on the extensive use
of illustrative examples. Accordingly, in addition to ensemble theory, the
readeris also providedwith anabundance of artificialas wellas real-world
applications from a wide range of fields. The data referred to in this book,
as well as most of the Java implementations of the presented algorithms,
can be obtained via the Web.
Oneofthekeygoalsofthisbookistoprovideresearchersinthefieldsof
pattern recognition, information systems, computer science, statistics and
management with a vital source of ensemble techniques. In addition, the
book will prove to be highly beneficial to those engaged in research in so-
cial sciences, psychology, medicine, genetics, and other fields that confront
complex data-processing problems.
The material in this book constitutes the core of undergraduate and
graduates courses at Ben-Gurion University. The book can also serve as
anexcellentreferencebookforgraduateaswellasadvancedundergraduate
coursesinpatternrecognition,machinelearninganddatamining. Descrip-
tionsofreal-worlddata-miningprojectsthatutilizeensemblemethodsmay
be of particular interest to the practitioners among the readers. The book
is rigorous and requires comprehension of problems and solutions via their
mathematicaldescriptions. Nevertheless,onlybasicbackgroundknowledge
of basic probability theory and computer science (algorithms) in assumed
in most of the book.
Duetothebroadspectrumofensemblemethods,itisimpossibletocover
all techniques and algorithms in a single book. The interested reader can
refertotheexcellentbook“patternclassifiers: methodsandalgorithms”by
Ludmila Kuncheva (John Wiley & Sons, 2004). Other key sources include
journalsandconferences’proceedings. TheInformationFusionJournaland
the JournalofAdvances inInformationFusionare largelydedicated to the
field of ensemble methodology. Nevertheless, many pattern recognition,
machine learning and data mining journals include research papers on en-
semble techniques. Moreover, major conferences such as the International
WorkshoponMultipleClassifierSystems(MCS)andtheInternationalCon-
ference on Information Fusion (FUSION) are especially recommended as
sources for additional information.
Many colleagues generously gave comments on drafts or counsel other-
wise. Dr. AlonSchclardeservesspecialmentionforhisparticularlydetailed
and insightful comments. I am indebted to Prof. Oded Maimon for lend-
Preface ix
ing his insight to this book. Thanks also to Prof. Horst Bunke and Prof.
Patrick Shen-Pei Wang for including my book in their important series in
machine perception and artificial intelligence. The author would also like
to thank Mr. Steven Patt, Editor, and staff members of World Scientific
Publishingfortheirkindcooperationthroughoutthewritingprocessofthis
book.
Last, but certainly not least, I owe my special gratitude to my family
and friends for their patience, time, support, and encouragement.
Lior Rokach
Ben-Gurion University of the Negev
Beer-Sheva, Israel
September 2009