ebook img

Multiview machine learning PDF

155 Pages·2019·1.398 MB·English
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Multiview machine learning

Shiliang Sun · Liang Mao · Ziang Dong · Lidan Wu Multiview Machine Learning Multiview Machine Learning Shiliang Sun Liang Mao (cid:129) (cid:129) Ziang Dong Lidan Wu (cid:129) Multiview Machine Learning 123 Shiliang Sun LiangMao Department ofComputer Science Department ofComputer Science andTechnology andTechnology EastChinaNormal University EastChinaNormal University Shanghai, China Shanghai, China ZiangDong LidanWu Department ofComputer Science Department ofComputer Science andTechnology andTechnology EastChinaNormal University EastChinaNormal University Shanghai, China Shanghai, China ISBN978-981-13-3028-5 ISBN978-981-13-3029-2 (eBook) https://doi.org/10.1007/978-981-13-3029-2 LibraryofCongressControlNumber:2018963292 ©SpringerNatureSingaporePteLtd.2019 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodologynowknownorhereafterdeveloped. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd. Theregisteredcompanyaddressis:152BeachRoad,#21-01/04GatewayEast,Singapore189721, Singapore Preface During the past two decades, multiview learning as an emerging direction in machine learning became a prevailing research topic in artificial intelligence (AI). Its success and popularity were largely motivated by the fact that real-world applicationsgeneratevariousdataasdifferentviewswhilepeopletrytomanipulate and integrate those data for performance improvements. In the data era, this situ- ation will continue. We think the multiview learning research will be active for a longtime,andfurtherdevelopmentandin-depthstudiesareneededtomakeitmore effective and practical. In 2013, a review paper of mine, entitled “A Survey of Multi-view Machine Learning”(NeuralComputingandApplications,2013),waspublished.Itgenerates agooddisseminationandpromotionofmultiviewlearningandhasbeenwellcited. Sincethen,muchmoreresearchhasbeendeveloped.Thisbookaimstoprovidean in-depth and comprehensive introduction to multiview learning and hope to be helpful for AI researchers and practitioners. Ihave beenworking inthe machine learning area for more than 15years. Most ofmyworkintroducedinthisbookwascompletedafterIgraduatedfromTsinghua UniversityandjoinedEastChinaNormalUniversityin2007.Andwealsoinclude many important andrepresentative worksfrom other researchers tomake thebook contentcompleteandcomprehensive.Duetospaceandtimelimits,wemaynotbe able to include all relevant works. I owe many thanks to the past and current members of my Pattern Recognition and Machine Learning Research Group, East China Normal University, for their hardworktomakeresearchdoneintime.Therelationshipbetweenmeandthemis not just professors and students, but also comrades-in-arms. Shanghai, China Shiliang Sun September 2018 v Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Definition of Multiview Machine Learning and Related Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.3 Typical Application Fields in Artificial Intelligence . . . . . . . . . . . 2 1.4 Why Can Multiview Learning Be Useful. . . . . . . . . . . . . . . . . . . 4 1.5 Book Structure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2 Multiview Semi-supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.2 Co-training Style Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 Co-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.2 Co-EM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2.3 Robust Co-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Co-regularization Style Methods . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3.1 Co-regularization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3.2 Bayesian Co-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.3.3 Multiview Laplacian SVM. . . . . . . . . . . . . . . . . . . . . . . . 16 2.3.4 Multiview Laplacian Twin SVM . . . . . . . . . . . . . . . . . . . 18 2.4 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3 Multiview Subspace Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2 Canonical Correlation Analysis and Related Methods . . . . . . . . . . 24 3.2.1 Canonical Correlation Analysis. . . . . . . . . . . . . . . . . . . . . 24 3.2.2 Kernel Canonical Correlation Analysis . . . . . . . . . . . . . . . 26 vii viii Contents 3.2.3 Probabilistic Canonical Correlation Analysis . . . . . . . . . . . 28 3.2.4 Bayesian Canonical Correlation Analysis . . . . . . . . . . . . . 29 3.3 Multiview Subspace Learning with Supervision . . . . . . . . . . . . . . 31 3.3.1 Multiview Linear Discriminant Analysis . . . . . . . . . . . . . . 31 3.3.2 Multiview Uncorrelated Linear Discriminant Analysis . . . . 33 3.3.3 Hierarchical Multiview Fisher Discriminant Analysis. . . . . 35 3.4 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4 Multiview Supervised Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.2 Multiview Large Margin Classifiers. . . . . . . . . . . . . . . . . . . . . . . 40 4.2.1 SVM-2K. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 4.2.2 Multiview Maximum Entropy Discriminant. . . . . . . . . . . . 42 4.2.3 Soft Margin-Consistency-Based Multiview Maximum Entropy Discrimination . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.3 Multiple Kernel Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.3.1 Kernel Combination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.3.2 Linear Combination of Kernels and Support Kernel Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.3.3 SimpleMKL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 4.4 Multiview Probabilistic Models. . . . . . . . . . . . . . . . . . . . . . . . . . 52 4.4.1 Multiview Regularized Gaussian Processes . . . . . . . . . . . . 52 4.4.2 Sparse Multiview Gaussian Processes . . . . . . . . . . . . . . . . 53 4.5 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 5 Multiview Clustering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.2 Multiview Spectral Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 5.2.1 Co-trained Spectral Clustering . . . . . . . . . . . . . . . . . . . . . 60 5.2.2 Co-regularized Spectral Clustering . . . . . . . . . . . . . . . . . . 61 5.3 Multiview Subspace Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.3.1 Multiview Clustering via Canonical Correlation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.3.2 Multiview Subspace Clustering. . . . . . . . . . . . . . . . . . . . . 64 5.3.3 Joint Nonnegative Matrix Factorization. . . . . . . . . . . . . . . 66 5.4 Distributed Multiview Clustering. . . . . . . . . . . . . . . . . . . . . . . . . 67 5.5 Multiview Clustering Ensemble. . . . . . . . . . . . . . . . . . . . . . . . . . 69 5.6 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Contents ix 6 Multiview Active Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.2 Co-testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.3 Bayesian Co-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.4 Multiple-View Multiple-Learner . . . . . . . . . . . . . . . . . . . . . . . . . 78 6.5 Active Learning with Extremely Spare Labeled Examples. . . . . . . 80 6.6 Combining Active Learning with Semi-supervising Learning . . . . 82 6.7 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 7 Multiview Transfer Learning and Multitask Learning . . . . . . . . . . . 85 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 7.2 Multiview Transfer Learning with a Large Margin . . . . . . . . . . . . 86 7.3 Multiview Discriminant Transfer Learning. . . . . . . . . . . . . . . . . . 88 7.4 Multiview Transfer Learning with Adaboost . . . . . . . . . . . . . . . . 90 7.4.1 Adaboost. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 7.4.2 Multiview Transfer Learning with Adaboost . . . . . . . . . . . 92 7.4.3 Multisource Transfer Learning with Multiview Adaboost. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.5 Multiview Multitask Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 7.5.1 Graph-Based Interative Multiview Multitask Learning . . . . 95 7.5.2 Co-regularized Multiview Multitask Learning Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 7.5.3 Convex Shared Structure Learning Algorithm for Multiview Multitask Learning. . . . . . . . . . . . . . . . . . . 100 7.6 Other Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 8 Multiview Deep Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 8.2 Joint Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 8.2.1 Probabilistic Graphical Models. . . . . . . . . . . . . . . . . . . . . 106 8.2.2 Fusion of Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 8.2.3 Sequential Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 8.3 Complementary Structured Space . . . . . . . . . . . . . . . . . . . . . . . . 116 8.3.1 Deep Canonical Correlation Analysis . . . . . . . . . . . . . . . . 117 8.3.2 Methods Based on Autoencoders . . . . . . . . . . . . . . . . . . . 119 8.3.3 Similarity Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 8.4 View Mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 8.4.1 Generative Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 8.4.2 Retrieval-Based Methods . . . . . . . . . . . . . . . . . . . . . . . . . 132 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 x Contents 9 View Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 9.2 Feature Set Partition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 9.2.1 Random Split . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 9.2.2 Genetic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 9.2.3 Pseudo Multiview Co-training . . . . . . . . . . . . . . . . . . . . . 142 9.3 Purifying . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 9.4 Noising . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 9.5 Sequence Reversing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 9.6 Multi-module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 9.7 Conditional Generative Model. . . . . . . . . . . . . . . . . . . . . . . . . . . 146 9.7.1 Conditional Generative Adversarial Nets. . . . . . . . . . . . . . 146 9.7.2 Conditional Variational Autoencoders. . . . . . . . . . . . . . . . 148 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Chapter 1 Introduction 1.1 Background Inmanyscientificdataanalysistasks,dataareoftencollectedthroughdifferentmea- suringmethods,suchasvariousfeatureextractorsorsensors,asusuallythesingle particular measuring method cannot comprehensively describe all the information ofthedata.Inthiscase,thefeaturesofeachdataexamplecanbenaturallydivided intogroups,eachofwhichcanbeconsideredasaview.Forinstance,forimagesand videos,colorfeaturesandtexturefeaturescanberegardedastwoviews.Itisimpor- tanttomakegooduseoftheinformationfromdifferentviews.Multiviewmachine learning(referredtoasmultiviewlearningforshort)isabranchofmachinelearn- ingwhichstudiesandutilizestheinformationandrelationshipbetweenviews.Two representative works of multiview learning in the early days are canonical corre- lationanalysis(CCA)(Hotelling1936)andco-training(BlumandMitchell1998). Especiallyafterco-trainingwasproposed,thestudyofmultiviewlearningisonthe rise. A multiview learning workshop was held at the International Conference on MachineLearningin2005,whichfurtherboostedtheresearchofmultiviewlearn- ing.Sofar,multiviewlearninghasmadegreatprogressintheoryandpractice.The relatedideashavealsobeenincorporatedintoseveralareasofmachinelearningand havebeendevelopedcontinually.Awell-designedmultiviewlearningstrategymay bringbenefitstothedevelopmentsoftheseareas. 1.2 DefinitionofMultiviewMachineLearningandRelated Concepts Multiviewlearningisanimportantbranchofmachinelearning(Sun2013).During learning,multiviewlearningexplicitlyusesmultipledistinctrepresentationsofdata, andmodelstherelationshipbetweenthemand/ortheresultsfromdownstreamcom- putations.Here,these“representation”caneitherbetheoriginalfeaturesofthedata orthefeaturesobtainedthroughsomecomputations.Theabruptapproachtoutilize these representations is to simply concatenate them into a single representation to performlearning.However,twodrawbacksofthisstrategyarethatover-fittingmay ©SpringerNatureSingaporePteLtd.2019 1 S.Sunetal.,MultiviewMachineLearning, https://doi.org/10.1007/978-981-13-3029-2_1

See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.