Latent Factor Analysis for High-dimensional and Sparse Matrices: A particle swarm optimization-based approach

99 Pages·2022·3.976 MB·English

by  Ye Yuan, Xin Luo

SpringerBriefs in Computer Science
Ye Yuan · Xin Luo
Latent Factor Analysis
for High-dimensional
and Sparse Matrices
A particle swarm
optimization-based
approach
SpringerBriefs in Computer Science
SeriesEditors
StanZdonik,BrownUniversity,Providence,RI,USA
ShashiShekhar,UniversityofMinnesota,Minneapolis,MN,USA
XindongWu,UniversityofVermont,Burlington,VT,USA
LakhmiC.Jain,UniversityofSouthAustralia,Adelaide,SA,Australia
DavidPadua,UniversityofIllinoisUrbana-Champaign,Urbana,IL,USA
XueminShermanShen,UniversityofWaterloo,Waterloo,ON,Canada
BorkoFurht,FloridaAtlanticUniversity,BocaRaton,FL,USA
V.S.Subrahmanian,UniversityofMaryland,CollegePark,MD,USA
MartialHebert,CarnegieMellonUniversity,Pittsburgh,PA,USA
KatsushiIkeuchi,UniversityofTokyo,Tokyo,Japan
BrunoSiciliano,UniversitàdiNapoliFedericoII,Napoli,Italy
SushilJajodia,GeorgeMasonUniversity,Fairfax,VA,USA
NewtonLee,InstituteforEducation,ResearchandScholarships,LosAngeles,CA,
USA
SpringerBriefs present concise summaries of cutting-edge research and practical
applications across a wide spectrum of fields. Featuring compact volumes of 50 to
125pages,theseriescoversarangeofcontentfromprofessionaltoacademic.
Typicaltopicsmightinclude:
(cid:129) Atimelyreportofstate-of-theartanalyticaltechniques
(cid:129) A bridge between new research results, as published in journal articles, and a
contextualliteraturereview
(cid:129) Asnapshotofahotoremergingtopic
(cid:129) Anin-depthcasestudyorclinicalexample
(cid:129) A presentation of core concepts that students must understand in order to make
independentcontributions
Briefs allow authors to present their ideas and readers to absorb them with
minimal time investment. Briefs will be published as part of Springer’s eBook
collection, with millions of users worldwide. In addition, Briefs will be available
forindividualprintandelectronicpurchase.Briefsarecharacterizedbyfast,global
electronic dissemination, standard publishing contracts, easy-to-use manuscript
preparation and formatting guidelines, and expedited production schedules. We
aim for publication 8–12 weeks after acceptance. Both solicited and unsolicited
manuscriptsareconsideredforpublicationinthisseries.
**Indexing:ThisseriesisindexedinScopus,Ei-Compendex,andzbMATH**
(cid:129)
Ye Yuan Xin Luo
Latent Factor Analysis
for High-dimensional
and Sparse Matrices
A particle swarm optimization-based approach
YeYuan XinLuo
ComputerandInformationScience ComputerandInformationScience
SouthwestUniversity SouthwestUniversity
Chongqing,China Chongqing,China
ISSN2191-5768 ISSN2191-5776 (electronic)
SpringerBriefsinComputerScience
ISBN978-981-19-6702-3 ISBN978-981-19-6703-0 (eBook)
https://doi.org/10.1007/978-981-19-6703-0
©TheAuthor(s),underexclusivelicensetoSpringerNatureSingaporePteLtd.2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseof
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by
similarordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors, and the editorsare safeto assume that the adviceand informationin this
bookarebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
High-dimensionalandsparse(HiDS)matricesarecommonlyseeninmanybigdata-
related industrial applications like electronic commerce, cloud services, social net-
works, and wireless sensor networks. Despite its extreme sparse, an HiDS matrix
contains rich knowledge regarding desired patterns like users’ potential favorites,
item clusters, and topological neighbors. Hence, how to efficiently and accurately
extractdesiredknowledge fromit forvariousdata analysis tasks becomesahighly
interestingissue.
Latent factor analysis (LFA) has proven to be highly efficient for addressing an
HiDSmatrixowingtoitshighscalabilityandefficiency.However,anLFAmodel’s
performance relies heavily on its hyper-parameters, which should be chosen with
care.Thecommonstrategyistoemploygrid-searchtotunethesehyper-parameters.
However, it requires expensive computational cost to search the whole candidate
space,especiallywhenbuildinganLFAmodelonanHiDSmatrix.Hence,howto
implementahyper-parameter-freeLFAmodelbecomesasignificantissue.
Inthis book, we incorporatethe principle of particle swarm optimization(PSO)
into latent factor analysis, thereby achieving four effective hyper-parameter-free
latent factor analysis models. The first model is a Learning rate-free LFA (L2FA)
model,whichutilizesastandardPSOalgorithmintothelearningprocessbybuilding
aswarmoflearningratesappliedtothesamegroup.ThesecondmodelisaLearning
rateandRegularizationcoefficient-freeLFA(LRLFA)model,whichbuildaswarm
bytakingthelearningrateandregularizationcoefficientofeverysingleLFA-based
model as particles, and then apply particle swarm optimization to make them
adaptationaccordingtoapredefinedfitnessfunction.ThethirdmodelisaGeneral-
ized and Adaptive LFA (GALFA) model, which implement self-adaptation of the
regularizationcoefficientandmomentumcoefficientforexcellent practicabilityvia
PSO.ThelastmodelisanAdvancedLearningrate-freeLFA(AL2FA)model.Before
building this model, we first propose a novel position-transitional particle swarm
optimization (P2SO) algorithm by incorporate more dynamic information into the
particle’sevolutionforpreventingprematureconvergence.Andthen,aP2SOalgo-
rithmisutilizedintothetrainingprocesstomakethelearningrateadaptationwithout
v
vi Preface
accuracyloss.Sincehyper-parameteradaptationisdonewithinonlyonefulltraining
process,therebygreatlyreducingcomputationalcost.Hence,itfitstheneedofreal
applicationswithhighscalabilityandefficiency.
Thisisthefirstbookabout how toincorporateparticle swarm optimization into
latent factor analysis for implementing effective hyper-parameter adaptation. It is
intendedforprofessionalsandgraduatestudentsinvolvedinLFAonHiDSdata.Itis
assumedthatthereaderhasabasicknowledgeofmathematics,aswellasacertain
background in data mining. The reader can get an overview on how to implement
efficienthyper-parameteradaptationinanLFAmodel.Wehopethismonographwill
be a useful reference for students, researchers, and professionals to understand the
basicmethodologiesofhyper-parameteradaptationviaPSO.Thereaderscanimme-
diatelyconductextensiveresearchesandexperimentsontherealapplicationsHiDS
datainvolvedinthisbook.
Chongqing,China YeYuan
June2022 XinLuo
Contents
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Background. . . . . . . . . .. . . . . . . . . .. . . . . . . . . .. . . . . . . . . . 1
1.2 Preliminaries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 SymbolandAbbreviationAppointment. . . . . . . . . . . . . . 2
1.2.2 AnLFAModel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.3 ParticleSwarmOptimization. . . . . . . . . . . . . . . . . . . . . . 6
1.3 BookOrganization. . . . . . . . . .. . . . . . . . . . .. . . . . . . . . . . .. . 6
References. . . . .. . . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. 7
2 LearningRate-FreeLatentFactorAnalysisviaPSO. . . . . . . . . . . . 11
2.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 AnLFAModelwithSGDAlgorithm. . . . . . . . . . . . . . . . . . . . . 12
2.3 TheProposedL2FAModel. . .. . . . . . . .. . . . . . . . .. . . . . . . .. 14
2.3.1 LearningRateAdaptationviaPSO. . . . . . . . . . . . . . . . . . 14
2.3.2 AlgorithmDesignandAnalysis. . . . . . . . . . . . . . . . . . . . 15
2.4 ExperimentalResultsandAnalysis. . . . . . . . . . . . . . . . . . . . . . . 17
2.4.1 GeneralSettings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.2 PerformanceComparison. . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.3 EffectofSwarmSize. . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.4 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
References. . . . .. . . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. 25
3 LearningRateandRegularizationCoefficient-FreeLatentFactor
AnalysisviaPSO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.2 AnSGD-BasedLFAModel. . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.3 TheProposedLRLFAModel. . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3.1 LearningRateandRegularizationCoefficientAdaptation
viaPSO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3.2 LinearlyDecreasingInertiaWeightIncorporation. . . . . . . 35
vii
viii Contents
3.3.3 AlgorithmDesignandAnalysis. . . . . . . . . . . . . . . . . . . . 36
3.4 ExperimentalResultsandAnalysis. . . . . . . . . . . . . . . . . . . . . . . 37
3.4.1 GeneralSettings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.4.2 EffectofSwarmSize. . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.3 PerformanceComparison. . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.4 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.5 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
References. . . . .. . . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. 45
4 RegularizationandMomentumCoefficient-FreeNon-negative
LatentFactorAnalysisviaPSO. . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 AnSLF-NMU-BasedNLFAModel. . . . . . . . . . . . . . . . . . . . . . 53
4.3 TheProposedGALFAModel. . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3.1 AGeneralizedLearningObjective. . . . . . . . . . . . . . . . . . 54
4.3.2 AGeneralizedSLF-NMU-BasedLearningRule. . . . . . . . 55
4.3.3 Generalized-MomentumIncorporation. . . . . . . . . . . . . . . 56
4.3.4 RegularizationandMomentumCoefficientAdaptation
viaPSO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.3.5 AlgorithmDesignandAnalysis. . . . . . . . . . . . . . . . . . . . 60
4.4 ExperimentalResultsandAnalysis. . . . . . . . . . . . . . . . . . . . . . . 63
4.4.1 GeneralSettings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.4.2 ParameterSensitivity. . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.4.3 ComparisonwithState-of-the-ArtModels. . . . . . . . . . . . . 68
4.4.4 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.5 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
References. . . . .. . . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. 72
5 AdvancedLearningRate-FreeLatentFactorAnalysisviaP2SO. . . 75
5.1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.2 AnSGD-BasedLFAModel. . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
5.3 TheProposedAL2FAModel. . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.3.1 AP2SOAlgorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.3.2 LearningRateAdaptationviaP2SO. . . . . . . . . . . . . . . . . 80
5.3.3 AlgorithmDesignandAnalysis. . . . . . . . . . . . . . . . . . . . 81
5.4 ExperimentalResultsandAnalysis. . . . . . . . . . . . . . . . . . . . . . . 83
5.4.1 GeneralSettings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.4.2 Effectofρ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
5.4.3 ComparisonResults. . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.4.4 Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
5.5 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
References. . . . .. . . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. 88
6 ConclusionandFutureDirections. . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.1 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.2 Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
References. . . . .. . . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. 92
Chapter 1
Introduction
1.1 Background
Withtherapiddevelopmentofbasiccomputingandstoragefacilities,thebig-data-
relatedindustrialapplications,i.e.,recommendationsystem[1–5],cloudservices[6–
10], social networks [11–16], and wireless sensor networks [17–22], continue to
expand. Consequently, the data involved in these applications grow exponentially.
Such data commonly contains a mass of entities. With the increasing number of
involvedentities,itbecomesimpossibletoobservetheirwholeinteractionmapping.
Hence, a High-dimensional and sparse (HiDS) matrix [23–31, 32] is commonly
adopted to describe such specific relationships among entities. For instance, the
Doubanmatrix[33]collectedbytheChineselargestonlinebook,movieandmusic
database includes 129,490 users and 58,541 items. However, it only contains
16,830,839knownratingsandthedensityis0.22%.
Althoughan HiDS matrix canbe extremely sparse, it possesses rich knowledge
regardingdesiredpatternslikeusers’potentialfavorites[34],itemclusters[35],and
topological neighbors [36]. How to precisely discover such knowledge from an
HiDS matrix for various analysis tasks, i.e., accurate recommendation [1–3], com-
munitydetection[37,38],andwebserviceselection[30,39],isaresearchfieldwith
highscientific research value, economicbenefitsandsocialsignificance.However,
conventional data analysis models [40–46] mostly fails to consider the high-
dimensional, sparse and fragmented of an HiDS matrix comprehensively, thereby
achievingunsatisfactoryresultsandunavailable.
Considering existing HiDS matrix analysis models [1–3, 25–27], a latent factor
analysis (LFA) model has proven to be highly efficient for addressing an HiDS
matrix owing to its high scalability and efficiency. It models each row or column
entity with an equally-sized latent factor (LF) vector, and then optimizes these LF
vectors on an objective defined on an HiDS matrix’s known data only. The well-
optimizedLFvectorsactuallyembedcorrespondingrowandcolumnentitiesintothe
low-dimensional LF space based on an HiDS matrix’s known data. Hence, it
©TheAuthor(s),underexclusivelicensetoSpringerNatureSingaporePteLtd.2022 1
Y.Yuan,X.Luo,LatentFactorAnalysisforHigh-dimensionalandSparseMatrices,
SpringerBriefsinComputerScience,https://doi.org/10.1007/978-981-19-6703-0_1