Type-2 Fuzzy Graphical Models for Pattern Recognition

210 Pages·2015·9.203 MB·English

by  Jia Zeng, Zhi-Qiang Liu (auth.)

Studies in Computational Intelligence 666
Jia Zeng
Zhi-Qiang Liu
Type-2 Fuzzy
Graphical Models
for Pattern
Recognition
Studies in Computational Intelligence
Volume 666
Series editor
Janusz Kacprzyk, Polish Academy of Sciences, Warsaw, Poland
e-mail: [email protected]
About this Series
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
mentsandadvancesinthevariousareasofcomputationalintelligence—quicklyand
with a high quality. The intent is to cover the theory, applications, and design
methods of computational intelligence, as embedded in the fields of engineering,
computer science, physics and life sciences, as well as the methodologies behind
them. The series contains monographs, lecture notes and edited volumes in com-
putational intelligence spanning the areas of neural networks, connectionist sys-
tems, genetic algorithms, evolutionary computation, artificial intelligence, cellular
automata, self-organizing systems, soft computing, fuzzy systems, and hybrid
intelligent systems. Of particular value to both the contributors and the readership
are the short publication timeframe and the world-wide distribution, which enable
both wide and rapid dissemination of research output.
More information about this series at http://www.springer.com/series/7092
Jia Zeng Zhi-Qiang Liu
(cid:129)
Type-2 Fuzzy Graphical
Models for Pattern
Recognition
123
Jia Zeng Zhi-QiangLiu
School ofComputer Science School ofCreative Media
andTechnology CityUniversity ofHong Kong
Soochow University Hong Kong
Suzhou China
China
ISSN 1860-949X ISSN 1860-9503 (electronic)
ISBN 978-3-662-44689-8 ISBN 978-3-662-44690-4 (eBook)
DOI 10.1007/978-3-662-44690-4
LibraryofCongressControlNumber:2014948756
SpringerHeidelbergNewYorkDordrechtLondon
JointlypublishedwithTsinghuaUniversityPress,Beijing
ISBN:978-7-302-36890-8,TsinghuaUniversityPress,Beijing
©TsinghuaUniversityPress,BeijingandSpringer-VerlagBerlinHeidelberg2015
Thisworkissubjecttocopyright.AllrightsarereservedbythePublishers,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodology now known or hereafter developed. Exempted from this legal reservation are brief
excerpts in connection with reviews or scholarly analysis or material supplied specifically for the
purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe
work. Duplication of this publication or parts thereof is permitted only under the provisions of
the Copyright Law of the Publishers’ locations, in its current version, and permission for use must
always be obtained from Springer. Permissions for use may be obtained through RightsLink at the
CopyrightClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublisherscanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishersmakenowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
This book is dedicated to my family: Tian-Yi
Zeng and Xiao-Qin Cao
Preface
Thisstate-of-the-artbookdescribesimportantadvancesintype-2fuzzysystemsthat
have been made in the past decade for real-world pattern recognition problems,
such as speech recognition, handwriting recognition, and topic modeling. The
success of type-2 fuzzy sets has been largely attributed to their three-dimensional
membership functions to handle both randomness and fuzziness uncertainties in
real-world problems. In pattern recognition, both features and models have
uncertainties,suchasnonstationarybabblenoiseinspeechsignals,largevariations
of handwritten Chinese character shapes, uncertain meaning of words in topic
modeling, and uncertain parameters of models because of insufficient and noisy
trainingdata.Alltheseuncertaintiesmotivateustointegratetype-2fuzzysetswith
probabilistic graphical models to achieve better overall performance in terms of
robustness, generalization ability, or recognition accuracy. For example, we inte-
grate type-2 fuzzy sets with graphical models such as Gaussian mixture models,
hidden Markov models, Markov random fields, and latent Dirichlet allocation-
based topic models for pattern recognition. The type-2 fuzzy Gaussian mixture
models can describe uncertain densities of observations. The type-2 fuzzy hidden
Markov models incorporate the first-order Markov chain into the type-2 fuzzy
Gaussian mixture models, which is suitable for modeling uncertain speech signals
under babblenoise.Thetype-2fuzzyMarkovrandom fieldscombinetype-2fuzzy
sets with Markov random fields, which is able to handle large variations in struc-
tural patterns such as handwritten Chinese characters. The type-2 fuzzy topic
modelsfocusonuncertainmixedmembershipofwordstodifferenttopicalclusters,
which is effective to partition the observed (visual) words into semantically
meaningful topical themes. In conclusion, these real-world pattern recognition
applications demonstrate the effectiveness of type-2 fuzzy graphical models for
handling uncertainties.
Suzhou, May 2013 Jia Zeng
Hong Kong Zhi-Qiang Liu
vii
Contents
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Pattern Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Uncertainties. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Book Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Probabilistic Graphical Models. . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1 The Labeling Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Markov Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 The Bayesian Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . 11
2.3.1 Descriptive and Generative Models . . . . . . . . . . . . . . . . 14
2.3.2 Statistical–Structural Pattern Recognition . . . . . . . . . . . . 15
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3 Type-2 Fuzzy Sets for Pattern Recognition . . . . . . . . . . . . . . . . . . 17
3.1 Type-2 Fuzzy Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.2 Operations on Type-2 Fuzzy Sets. . . . . . . . . . . . . . . . . . . . . . . 22
3.3 Type-2 Fuzzy Logic Systems . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3.1 Fuzzifier. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3.2 Rule Base and Inference. . . . . . . . . . . . . . . . . . . . . . . . 27
3.3.3 Type Reducer and Defuzzifier. . . . . . . . . . . . . . . . . . . . 28
3.4 Pattern Recognition Using Type-2 Fuzzy Sets. . . . . . . . . . . . . . 32
3.5 The Type-2 Fuzzy Bayesian Decision Theory . . . . . . . . . . . . . . 35
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
ix
x Contents
4 Type-2 Fuzzy Gaussian Mixture Models . . . . . . . . . . . . . . . . . . . . 45
4.1 Gaussian Mixture Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Type-2 Fuzzy Gaussian Mixture Models. . . . . . . . . . . . . . . . . . 48
4.3 Multi-category Pattern Classification . . . . . . . . . . . . . . . . . . . . 53
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5 Type-2 Fuzzy Hidden Moarkov Models. . . . . . . . . . . . . . . . . . . . . 57
5.1 Hidden Markov Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.1.1 The Forward-Backward Algorithm. . . . . . . . . . . . . . . . . 59
5.1.2 The Viterbi Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . 60
5.1.3 The Baum–Welch Algorithm . . . . . . . . . . . . . . . . . . . . 61
5.2 Type-2 Fuzzy Hidden Markov Models. . . . . . . . . . . . . . . . . . . 62
5.2.1 Elements of a Type-2 FHMM. . . . . . . . . . . . . . . . . . . . 64
5.2.2 The Type-2 Fuzzy Forward-Backward Algorithm . . . . . . 64
5.2.3 The Type-2 Fuzzy Viterbi Algorithm. . . . . . . . . . . . . . . 66
5.2.4 The Learning Algorithm. . . . . . . . . . . . . . . . . . . . . . . . 67
5.2.5 Type-Reduction and Defuzzification . . . . . . . . . . . . . . . 71
5.2.6 Computational Complexity. . . . . . . . . . . . . . . . . . . . . . 71
5.3 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.3.1 Automatic Speech Recognition System . . . . . . . . . . . . . 72
5.3.2 Phoneme Classification . . . . . . . . . . . . . . . . . . . . . . . . 74
5.3.3 Phoneme Recognition . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
6 Type-2 Fuzzy Markov Random Fields. . . . . . . . . . . . . . . . . . . . . . 85
6.1 Markov Random Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6.1.1 The Neighborhood System. . . . . . . . . . . . . . . . . . . . . . 87
6.1.2 Clique Potentials. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.1.3 Relaxation Labeling. . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.2 Type-2 Fuzzy Markov Random Fields . . . . . . . . . . . . . . . . . . . 91
6.2.1 The Type-2 Fuzzy Relaxation Labeling . . . . . . . . . . . . . 94
6.2.2 Computational Complexity. . . . . . . . . . . . . . . . . . . . . . 96
6.3 Stroke Segmentation of Chinese Character . . . . . . . . . . . . . . . . 96
6.3.1 Gabor Filters-Based Cyclic Observations . . . . . . . . . . . . 97
6.3.2 Stroke Segmentation Using MRFs. . . . . . . . . . . . . . . . . 100
6.3.3 Stroke Extraction of Handprinted Chinese Characters. . . . 102
6.3.4 Stroke Extraction of Cursive Chinese Characters. . . . . . . 102
6.4 Handwritten Chinese Character Recognition . . . . . . . . . . . . . . . 105
6.4.1 MRFs for Character Structure Modeling. . . . . . . . . . . . . 109
6.4.2 Handwritten Chinese Character Recognition (HCCR). . . . 115
6.4.3 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
Contents xi
7 Type-2 Fuzzy Topic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.1 Latent Dirichlet Allocation . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
7.1.1 Factor Graph for the Collapsed LDA. . . . . . . . . . . . . . . 132
7.1.2 Loopy Belief Propagation (BP). . . . . . . . . . . . . . . . . . . 134
7.1.3 An Alternative View of BP . . . . . . . . . . . . . . . . . . . . . 137
7.1.4 Simplified BP (siBP). . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.1.5 Relationship to Previous Algorithms . . . . . . . . . . . . . . . 139
7.1.6 Belief Propagation for ATM. . . . . . . . . . . . . . . . . . . . . 141
7.1.7 Belief Propagation for RTM. . . . . . . . . . . . . . . . . . . . . 143
7.2 Speedup Topic Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
7.2.1 Fast Topic Modeling Techniques. . . . . . . . . . . . . . . . . . 147
7.2.2 Residual Belief Propagation . . . . . . . . . . . . . . . . . . . . . 148
7.2.3 Active Belief Propagation. . . . . . . . . . . . . . . . . . . . . . . 149
7.3 Type-2 Fuzzy Latent Dirichlet Allocation. . . . . . . . . . . . . . . . . 155
7.3.1 Topic Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7.3.2 Type-2 Fuzzy Topic Models (T2 FTMs) . . . . . . . . . . . . 161
7.4 Topic Modeling Performance . . . . . . . . . . . . . . . . . . . . . . . . . 168
7.4.1 Belief Propagation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
7.4.2 Residual Belief Propagation . . . . . . . . . . . . . . . . . . . . . 175
7.4.3 Active Belief Propagation. . . . . . . . . . . . . . . . . . . . . . . 179
7.5 Human Action Recognition. . . . . . . . . . . . . . . . . . . . . . . . . . . 189
7.5.1 Feature Extraction and Vocabulary Formation. . . . . . . . . 190
7.5.2 Results on KTH Data Set. . . . . . . . . . . . . . . . . . . . . . . 191
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
8 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
8.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
8.2 Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201