Data Clustering: Algorithms and Applications

648 Pages·2014·10.72 MB·English

by  Charu C. Aggarwal

Chapman & Hall/CRC
Data Mining and Knowledge Discovery Series
DATA
CLUSTERING
Algorithms and
Applications
© 2014 by Taylor & Francis Group, LLC
Chapman & Hall/CRC
Data Mining and Knowledge Discovery Series
SERIES EDITOR
Vipin Kumar
University of Minnesota
Department of Computer Science and Engineering
Minneapolis, Minnesota, U.S.A.
AIMS AND SCOPE
This series aims to capture new developments and applications in data mining and knowledge
discovery, while summarizing the computational tools and techniques useful in data analysis. This
series encourages the integration of mathematical, statistical, and computational methods and
techniques through the publication of a broad range of textbooks, reference works, and hand-
books. The inclusion of concrete examples and applications is highly encouraged. The scope of the
series includes, but is not limited to, titles in the areas of data mining and knowledge discovery
methods and applications, modeling, algorithms, theory and foundations, data and knowledge
visualization, data mining systems and tools, and privacy and security issues.
PUBLISHED TITLES
ADVANCES IN MACHINE LEARNING AND DATA MINING FOR ASTRONOMY
Michael J. Way, Jeffrey D. Scargle, Kamal M. Ali, and Ashok N. Srivastava
BIOLOGICAL DATA MINING
Jake Y. Chen and Stefano Lonardi
COMPUTATIONAL INTELLIGENT DATA ANALYSIS FOR SUSTAINABLE DEVELOPMENT
Ting Yu, Nitesh V. Chawla, and Simeon Simoff
COMPUTATIONAL METHODS OF FEATURE SELECTION
Huan Liu and Hiroshi Motoda
CONSTRAINED CLUSTERING: ADVANCES IN ALGORITHMS, THEORY, AND APPLICATIONS
Sugato Basu, Ian Davidson, and Kiri L. Wagstaff
CONTRAST DATA MINING: CONCEPTS, ALGORITHMS, AND APPLICATIONS
Guozhu Dong and James Bailey
DATA CLUSTERING: ALGORITHMS AND APPLICATIONS
Charu C. Aggarawal and Chandan K. Reddy
DATA CLUSTERING IN C++: AN OBJECT-ORIENTED APPROACH
Guojun Gan
DATA MINING FOR DESIGN AND MARKETING
Yukio Ohsawa and Katsutoshi Yada
DATA MINING WITH R: LEARNING WITH CASE STUDIES
Luís Torgo
FOUNDATIONS OF PREDICTIVE ANALYTICS
James Wu and Stephen Coggeshall
GEOGRAPHIC DATA MINING AND KNOWLEDGE DISCOVERY, SECOND EDITION
Harvey J. Miller and Jiawei Han
HANDBOOK OF EDUCATIONAL DATA MINING
Cristóbal Romero, Sebastian Ventura, Mykola Pechenizkiy, and Ryan S.J.d. Baker
© 2014 by Taylor & Francis Group, LLC
INFORMATION DISCOVERY ON ELECTRONIC HEALTH RECORDS
Vagelis Hristidis
INTELLIGENT TECHNOLOGIES FOR WEB APPLICATIONS
Priti Srinivas Sajja and Rajendra Akerkar
INTRODUCTION TO PRIVACY-PRESERVING DATA PUBLISHING:
CONCEPTS AND TECHNIQUES
Benjamin C. M. Fung, Ke Wang, Ada Wai-Chee Fu, and Philip S. Yu
KNOWLEDGE DISCOVERY FOR COUNTERTERRORISM AND LAW ENFORCEMENT
David Skillicorn
KNOWLEDGE DISCOVERY FROM DATA STREAMS
João Gama
MACHINE LEARNING AND KNOWLEDGE DISCOVERY FOR
ENGINEERING SYSTEMS HEALTH MANAGEMENT
Ashok N. Srivastava and Jiawei Han
MINING SOFTWARE SPECIFICATIONS: METHODOLOGIES AND APPLICATIONS
David Lo, Siau-Cheng Khoo, Jiawei Han, and Chao Liu
MULTIMEDIA DATA MINING: A SYSTEMATIC INTRODUCTION TO CONCEPTS AND THEORY
Zhongfei Zhang and Ruofei Zhang
MUSIC DATA MINING
Tao Li, Mitsunori Ogihara, and George Tzanetakis
NEXT GENERATION OF DATA MINING
Hillol Kargupta, Jiawei Han, Philip S. Yu, Rajeev Motwani, and Vipin Kumar
PRACTICAL GRAPH MINING WITH R
Nagiza F. Samatova, William Hendrix, John Jenkins, Kanchana Padmanabhan,
and Arpan Chakraborty
RELATIONAL DATA CLUSTERING: MODELS, ALGORITHMS, AND APPLICATIONS
Bo Long, Zhongfei Zhang, and Philip S. Yu
SERVICE-ORIENTED DISTRIBUTED KNOWLEDGE DISCOVERY
Domenico Talia and Paolo Trunfio
SPECTRAL FEATURE SELECTION FOR DATA MINING
Zheng Alan Zhao and Huan Liu
STATISTICAL DATA MINING USING SAS APPLICATIONS, SECOND EDITION
George Fernandez
SUPPORT VECTOR MACHINES: OPTIMIZATION BASED THEORY, ALGORITHMS,
AND EXTENSIONS
Naiyang Deng, Yingjie Tian, and Chunhua Zhang
TEMPORAL DATA MINING
Theophano Mitsa
TEXT MINING: CLASSIFICATION, CLUSTERING, AND APPLICATIONS
Ashok N. Srivastava and Mehran Sahami
THE TOP TEN ALGORITHMS IN DATA MINING
Xindong Wu and Vipin Kumar
UNDERSTANDING COMPLEX DATASETS:
DATA MINING WITH MATRIX DECOMPOSITIONS
David Skillicorn
© 2014 by Taylor & Francis Group, LLC
© 2014 by Taylor & Francis Group, LLC
DATA
CLUSTERING
Algorithms and
Applications
Edited by
Charu C. Aggarwal
Chandan K. Reddy
© 2014 by Taylor & Francis Group, LLC
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2014 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20130508
International Standard Book Number-13: 978-1-4665-5822-9 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid-
ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti-
lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy-
ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the
publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
© 2014 by Taylor & Francis Group, LLC
Contents
Preface xxi
EditorBiographies xxiii
Contributors xxv
1 AnIntroductiontoClusterAnalysis 1
CharuC.Aggarwal
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 CommonTechniquesUsedinClusterAnalysis . . . . . . . . . . . . . . . . . . 3
1.2.1 FeatureSelectionMethods. . . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.2 ProbabilisticandGenerativeModels . . . . . . . . . . . . . . . . . . . 4
1.2.3 Distance-BasedAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.4 Density-andGrid-BasedMethods. . . . . . . . . . . . . . . . . . . . . 7
1.2.5 LeveragingDimensionalityReductionMethods . . . . . . . . . . . . . 8
1.2.5.1 GenerativeModelsforDimensionalityReduction . . . . . . . 8
1.2.5.2 MatrixFactorizationandCo-Clustering . . . . . . . . . . . . 8
1.2.5.3 SpectralMethods . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.6 TheHighDimensionalScenario . . . . . . . . . . . . . . . . . . . . . . 11
1.2.7 ScalableTechniquesforClusterAnalysis . . . . . . . . . . . . . . . . . 13
1.2.7.1 I/OIssuesinDatabaseManagement . . . . . . . . . . . . . . 13
1.2.7.2 StreamingAlgorithms . . . . . . . . . . . . . . . . . . . . . 14
1.2.7.3 TheBigDataFramework . . . . . . . . . . . . . . . . . . . . 14
1.3 DataTypesStudiedinClusterAnalysis . . . . . . . . . . . . . . . . . . . . . . 15
1.3.1 ClusteringCategoricalData . . . . . . . . . . . . . . . . . . . . . . . . 15
1.3.2 ClusteringTextData . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.3.3 ClusteringMultimediaData . . . . . . . . . . . . . . . . . . . . . . . . 16
1.3.4 ClusteringTime-SeriesData . . . . . . . . . . . . . . . . . . . . . . . . 17
1.3.5 ClusteringDiscreteSequences. . . . . . . . . . . . . . . . . . . . . . . 17
1.3.6 ClusteringNetworkData . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.3.7 ClusteringUncertainData . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.4 InsightsGainedfromDifferentVariationsofClusterAnalysis . . . . . . . . . . . 19
1.4.1 VisualInsights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4.2 SupervisedInsights . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4.3 MultiviewandEnsemble-BasedInsights . . . . . . . . . . . . . . . . . 21
1.4.4 Validation-BasedInsights . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.5 DiscussionandConclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
vii
© 2014 by Taylor & Francis Group, LLC
viii Contents
2 FeatureSelectionforClustering:AReview 29
SalemAlelyani,JiliangTang,andHuanLiu
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.1.1 DataClustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.1.2 FeatureSelection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.1.3 FeatureSelectionforClustering . . . . . . . . . . . . . . . . . . . . . . 33
2.1.3.1 FilterModel . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.1.3.2 WrapperModel . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.1.3.3 HybridModel . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2 FeatureSelectionforClustering . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2.1 AlgorithmsforGenericData . . . . . . . . . . . . . . . . . . . . . . . 36
2.2.1.1 SpectralFeatureSelection(SPEC) . . . . . . . . . . . . . . . 36
2.2.1.2 LaplacianScore(LS) . . . . . . . . . . . . . . . . . . . . . . 36
2.2.1.3 FeatureSelectionforSparseClustering . . . . . . . . . . . . 37
2.2.1.4 Localized Feature Selection Based on Scatter Separability
(LFSBSS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.2.1.5 MulticlusterFeatureSelection(MCFS) . . . . . . . . . . . . 39
2.2.1.6 FeatureWeightingk-Means. . . . . . . . . . . . . . . . . . . 40
2.2.2 AlgorithmsforTextData . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.2.2.1 TermFrequency(TF) . . . . . . . . . . . . . . . . . . . . . . 41
2.2.2.2 InverseDocumentFrequency(IDF) . . . . . . . . . . . . . . 42
2.2.2.3 TermFrequency-InverseDocumentFrequency(TF-IDF) . . . 42
2.2.2.4 ChiSquareStatistic . . . . . . . . . . . . . . . . . . . . . . . 42
2.2.2.5 FrequentTerm-BasedTextClustering . . . . . . . . . . . . . 44
2.2.2.6 FrequentTermSequence . . . . . . . . . . . . . . . . . . . . 45
2.2.3 AlgorithmsforStreamingData . . . . . . . . . . . . . . . . . . . . . . 47
2.2.3.1 Text Stream Clustering Based on Adaptive Feature Selection
(TSC-AFS) . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.2.3.2 High-DimensionalProjectedStreamClustering(HPStream) . 48
2.2.4 AlgorithmsforLinkedData . . . . . . . . . . . . . . . . . . . . . . . . 50
2.2.4.1 ChallengesandOpportunities. . . . . . . . . . . . . . . . . . 50
2.2.4.2 LUFS: An Unsupervised Feature Selection Framework for
LinkedData . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.2.4.3 ConclusionandFutureWorkforLinkedData . . . . . . . . . 52
2.3 DiscussionsandChallenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
2.3.1 TheChickenortheEggDilemma . . . . . . . . . . . . . . . . . . . . . 53
2.3.2 ModelSelection:Kandl . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3.3 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3.4 Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3 ProbabilisticModelsforClustering 61
HongboDengandJiaweiHan
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
3.2 MixtureModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.2.2 GaussianMixtureModel. . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.3 BernoulliMixtureModel . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.2.4 ModelSelectionCriteria . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.3 EMAlgorithmandItsVariations . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3.1 TheGeneralEMAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3.2 MixtureModelsRevisited . . . . . . . . . . . . . . . . . . . . . . . . . 73
© 2014 by Taylor & Francis Group, LLC
Contents ix
3.3.3 LimitationsoftheEMAlgorithm . . . . . . . . . . . . . . . . . . . . . 75
3.3.4 ApplicationsoftheEMAlgorithm . . . . . . . . . . . . . . . . . . . . 76
3.4 ProbabilisticTopicModels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.4.1 ProbabilisticLatentSemanticAnalysis . . . . . . . . . . . . . . . . . . 77
3.4.2 LatentDirichletAllocation . . . . . . . . . . . . . . . . . . . . . . . . 79
3.4.3 VariationsandExtensions . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.5 ConclusionsandSummary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4 ASurveyofPartitionalandHierarchicalClusteringAlgorithms 87
ChandanK.ReddyandBhanukiranVinzamuri
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4.2 PartitionalClusteringAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2.1 K-MeansClustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2.2 MinimizationofSumofSquaredErrors. . . . . . . . . . . . . . . . . . 90
4.2.3 FactorsAffectingK-Means . . . . . . . . . . . . . . . . . . . . . . . . 91
4.2.3.1 PopularInitializationMethods . . . . . . . . . . . . . . . . . 91
4.2.3.2 EstimatingtheNumberofClusters . . . . . . . . . . . . . . . 92
4.2.4 VariationsofK-Means . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.2.4.1 K-MedoidsClustering . . . . . . . . . . . . . . . . . . . . . 93
4.2.4.2 K-MediansClustering . . . . . . . . . . . . . . . . . . . . . 94
4.2.4.3 K-ModesClustering . . . . . . . . . . . . . . . . . . . . . . 94
4.2.4.4 FuzzyK-MeansClustering . . . . . . . . . . . . . . . . . . . 95
4.2.4.5 X-MeansClustering. . . . . . . . . . . . . . . . . . . . . . . 95
4.2.4.6 IntelligentK-MeansClustering . . . . . . . . . . . . . . . . . 96
4.2.4.7 BisectingK-MeansClustering . . . . . . . . . . . . . . . . . 97
4.2.4.8 KernelK-MeansClustering . . . . . . . . . . . . . . . . . . . 97
4.2.4.9 MeanShiftClustering. . . . . . . . . . . . . . . . . . . . . . 98
4.2.4.10 WeightedK-MeansClustering . . . . . . . . . . . . . . . . . 98
4.2.4.11 GeneticK-MeansClustering . . . . . . . . . . . . . . . . . . 99
4.2.5 MakingK-MeansFaster . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.3 HierarchicalClusteringAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.3.1 AgglomerativeClustering . . . . . . . . . . . . . . . . . . . . . . . . . 101
4.3.1.1 SingleandCompleteLink . . . . . . . . . . . . . . . . . . . 101
4.3.1.2 GroupAveragedandCentroidAgglomerativeClustering . . . 102
4.3.1.3 Ward’sCriterion . . . . . . . . . . . . . . . . . . . . . . . . 103
4.3.1.4 AgglomerativeHierarchicalClusteringAlgorithm . . . . . . . 103
4.3.1.5 Lance–WilliamsDissimilarityUpdateFormula . . . . . . . . 103
4.3.2 DivisiveClustering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.3.2.1 IssuesinDivisiveClustering . . . . . . . . . . . . . . . . . . 104
4.3.2.2 DivisiveHierarchicalClusteringAlgorithm . . . . . . . . . . 105
4.3.2.3 MinimumSpanningTree-BasedClustering . . . . . . . . . . 105
4.3.3 OtherHierarchicalClusteringAlgorithms. . . . . . . . . . . . . . . . . 106
4.4 DiscussionandSummary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
5 Density-BasedClustering 111
MartinEster
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.2 DBSCAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.3 DENCLUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.4 OPTICS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5.5 OtherAlgorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
© 2014 by Taylor & Francis Group, LLC