Table Of Content

Machine Learning: Foundations, Methodologies, and Applications Yaochu Jin · Hangyu Zhu · Jinjin Xu · Yang Chen Federated Learning Fundamentals and Advances Machine Learning: Foundations, Methodologies, and Applications SeriesEditors KayChenTan,DepartmentofComputing,HongKongPolytechnicUniversity, HongKong,China DachengTao,UniversityofTechnology,Sydney,Australia Bookspublishedinthisseriesfocusonthetheoryandcomputationalfoundations, advanced methodologies and practical applications of machine learning, ideally combiningmathematicallyrigoroustreatmentsofacontemporarytopicsinmachine learningwithspecificillustrationsinrelevantalgorithmdesignsanddemonstrations in real-world applications. The intended readership includes research students and researchersincomputerscience,computerengineering,electricalengineering,data science,andrelatedareasseekingaconvenientmediumtotracktheprogressesmade inthefoundations,methodologies,andapplicationsofmachinelearning. Topicsconsideredincludeallareasofmachinelearning,includingbutnotlimited to: (cid:129) Decisiontree (cid:129) Artificialneuralnetworks (cid:129) Kernellearning (cid:129) Bayesianlearning (cid:129) Ensemblemethods (cid:129) Dimensionreductionandmetriclearning (cid:129) Reinforcementlearning (cid:129) Metalearningandlearningtolearn (cid:129) Imitationlearning (cid:129) Computationallearningtheory (cid:129) Probabilisticgraphicalmodels (cid:129) Transferlearning (cid:129) Multi-viewandmulti-tasklearning (cid:129) Graphneuralnetworks (cid:129) Generativeadversarialnetworks (cid:129) Federatedlearning Thisseriesincludesmonographs,introductory,andadvancedtextbooks,andstate- of-the-artcollections.Furthermore,itsupportsOpenAccesspublicationmode. · · · Yaochu Jin Hangyu Zhu Jinjin Xu Yang Chen Federated Learning Fundamentals and Advances YaochuJin HangyuZhu FacultyofTechnology DepartmentofArtificialIntelligence BielefeldUniversity andComputerScience Bielefeld,Germany JiangnanUniversity Wuxi,China JinjinXu IntelligentPerceptionandInteraction YangChen ResearchDepartment SchoolofElectricalEngineering OPPOResearchInstitute ChinaUniversityofMiningandTechnology Shanghai,China Xuzhou,China ISSN 2730-9908 ISSN 2730-9916 (electronic) MachineLearning:Foundations,Methodologies,andApplications ISBN 978-981-19-7082-5 ISBN 978-981-19-7083-2 (eBook) https://doi.org/10.1007/978-981-19-7083-2 ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature SingaporePteLtd.2023 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar ordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore Preface Iheardtheterminology“federatedlearning”forthefirsttimewhenIwaslistening toatalkgivenbyDr.CatherineHuangfromIntelataworkshopoftheIEEESympo- siumSeriesonComputationalIntelligenceinDecember2016inAthens,Greece.I was fascinated by the idea of federated learning and immediately recognized the paramount importance of preserving data privacy, since deep learning had been increasinglyrelyingonthecollectionofalargeamountofdata,eitherfromindustrial processesorfromhumaneverydaylife. Federatedlearninghasnowbecomeapopularresearchareainmachinelearning andreceivedincreasedinterestfrombothindustryandgovernment.Actuallyalready inApril2016,theEuropeanUnionadoptedtheGeneralDataProtectionRegulation, whichisthemoststrictlawondataprotectionandprivacyintheEuropeanUnionand the European Economic Area and took effective in May 2018. Consequently, data privacyandsecurityhasbecomeindispensableforpracticalapplicationsofmachine learning and artificial intelligence, together with other importance requirements, Includingexplainability,fairness,accountability,robustnessandreliability. Thisbookpresentsacompactyetcomprehensiveandupdatedcoverageoffunda- mentals and recent advances in federated learning. The book is self-contained andsuitedforbothpostgraduatestudents,researchers,andindustrialpractitioners. Chapter 1 starts with an introduction to the most popular neural network models andgradient-basedlearningalgorithms,evolutionaryalgorithms,evolutionaryopti- mization, and multi-objective evolutionary learning. Then, three main privacy- preservingcomputingtechniques,includingsecuremulti-partycomputation,differential privacy, and homomorphic encryption are described. Finally, the basics of federatedlearning,includingacategoryoffederatedlearningalgorithms,thevanilla federated averaging algorithm, federated transfer learning, and main challenges in federated learning over non independent and identically distributed data are presented.Chapter2focusesoncommunicationefficientfederatedlearning,which aimstoreducethecommunicationcostsinfederatedlearningwithoutdeteriorating the learning performance. Two algorithms for reducing communication overhead in federated learning are detailed. The first algorithm takes advantage of the fact that shallow layers in a deep neural network are responsible for learning general v vi Preface featuresacrossdifferentclasseswhiledeeplayerstakecareofclass-specificfeatures. Consequently,thedeeplayerscanbeupdatedlessfrequentlythanthedeeplayers, therebyreducingthenumberofparametersthatneedtobeuploadedanddownloaded. The second algorithm adopts a completely different approach, which dramatically decreases the communication load by converting the weights in real numbers into ternaryvalues.Here,thekeyquestionishowtomaintainthelearningperformance afterternarycompressionoftheweights,whichisachievedbytrainingareal-valued co-efficientforeachlayer.Interestingly,weareabletoprovetheoreticallythatmodel divergencecanevenbemitigatedbyintroducingternaryquantizationoftheweights, inparticularwhenthedataisnotindependentandidenticallydistributed.Whilelayer- wisesynchronousweightupdateandternarycompressiondescribedinChap.2can effectively lower the communication cost, Chap. 3 addresses communication cost bysimultaneouslymaximizingthelearningperformanceandminimizingthemodel complexity (e.g., the number of parameters in the model) using a multi-objective evolutionaryalgorithm.Togoastepfurther,areal-timemulti-objectiveevolutionary federatedlearningalgorithmisgiven,whichsearchesforoptimalneuralarchitectures bymaximizingtheperformance,minimizingthemodelcomplexity,andminimizing thecomputationalperformance(indicatedbyfloatingpointoperationspersecond)at thesametime.Tomakeitpossibleforreal-timeneuralarchitecturesearch,astrategy thatsearchesforsubnetworksbysamplingasupernet,andreducingcomputational costs by sampling clients is introduced. To enhance the security level of federated learning,Chap.4elaboratestwosecurefederatedlearningalgorithmsbyintegrating homomorphicencryptionanddifferentialprivacytechniqueswithfederatedlearning. Thefirstalgorithmisbasedonahorizontalfederatedlearning,inwhichadistributed encryptionalgorithmisappliedtotheweightstobeuploadedontopofternaryquan- tization. By contrast,the second algorithm is meant for vertical federated learning basedondecisiontrees,inwhichsecurenodesplitandconstructionaredeveloped basedonhomomorphicencryptionandpredictedlabelsareaggregatedonthebasis of partial differential privacy. This algorithm does not assume that all labels are storedononeclient,makingitmorepracticalforreal-worldapplications.Thebook isconcludedbyChap.5,providingasummaryofthepresentedalgorithms,andan outlookoffutureresearch. ThetwoalgorithmspresentedinChap.2weredevelopedbytwovisitingPh.D. studentsIhostedatUniversityofSurrey,JinjinXuandYangCheng.Thetwomulti- objectiveevolutionaryfederatedlearningalgorithmswereproposedbymyprevious Ph.D. student, Hangyu Zhu. Finally, the two secure federated learning algorithms weredesignedbyHangyuZhu,incollaborationwithDr.KaitaiLiang,andhisPh.D. student,RuiWang,whowerewiththeDepartmentofComputerScience,University ofSurrey,andarenowwiththeDepartmentofIntelligentSystems,DelftUniversity ofTechnology,TheNetherlands.IwouldliketotakethisopportunitythanktoKaitai andRuifortheircontributionstoChap.4. Preface vii Finally,IwouldliketoacknowledgethatmyresearchisfundedbyanAlexander von Humboldt Professorship for Artificial Intelligence endowed by the German FederalMinistryofEducationandResearch. Bielefeld,Germany YaochuJin August2022 Contents 1 Introduction ................................................... 1 1.1 ArtificialNeuralNetworksandDeepLearning .................. 1 1.1.1 ABriefHistoryofArtificialIntelligence ................ 1 1.1.2 Multi-layerPerceptrons ............................... 4 1.1.3 ConvolutionalNeuralNetworks ........................ 13 1.1.4 LongShort-TermMemory ............................ 20 1.1.5 DecisionTrees ....................................... 26 1.1.6 Gradient-BasedMethods .............................. 31 1.2 EvolutionaryOptimizationandLearning ....................... 36 1.2.1 OptimizationandLearning ............................ 36 1.2.2 GeneticAlgorithms .................................. 37 1.2.3 GeneticProgramming ................................ 42 1.2.4 EvolutionaryMulti-objectiveOptimization .............. 43 1.2.5 EvolutionaryMulti-objectiveLearning .................. 52 1.2.6 EvolutionaryNeuralArchitectureSearch ................ 54 1.3 Privacy-PreservingComputation .............................. 60 1.3.1 SecureMulti-partyComputation ....................... 61 1.3.2 DifferentialPrivacy .................................. 64 1.3.3 HomomorphicEncryption ............................. 66 1.4 FederatedLearning ......................................... 68 1.4.1 HorizontalandVerticalFederatedLearning .............. 68 1.4.2 FederatedAveraging .................................. 71 1.4.3 FederatedTransferLearning ........................... 73 1.4.4 FederatedLearningOverNon-IIDData ................. 76 1.5 Summary .................................................. 83 References ..................................................... 83 2 CommunicationEfficientFederatedLearning ..................... 93 2.1 CommunicationCostinFederatedLearning .................... 93 2.2 MainMethodologies ........................................ 94 ix x Contents 2.2.1 Non-IID/IIDDataandDatasetShift .................... 94 2.2.2 Non-identicalClientDistributions ...................... 95 2.2.3 ViolationsofIndependence ............................ 95 2.3 TemporallyWeightedAveragingandLayer-WiseWeight Update .................................................... 96 2.3.1 TemporallyWeightedAveraging ....................... 98 2.3.2 Layer-WiseAsynchronousWeightUpdate ............... 99 2.3.3 EmpiricalStudies .................................... 103 2.4 TrainedTernaryCompressionforFederatedLearning ............ 109 2.4.1 BinaryandTernaryCompression ....................... 110 2.4.2 TrainedTernaryCompression .......................... 113 2.4.3 TrainedTernaryCompressionforFederatedLearning ..... 114 2.4.4 TheoreticalAnalysis .................................. 119 2.4.5 EmpiricalStudies .................................... 126 2.5 Summary .................................................. 135 References ..................................................... 135 3 EvolutionaryMulti-objectiveFederatedLearning ................. 139 3.1 MotivationsandChallenges .................................. 139 3.2 OfflineEvolutionaryMulti-objectiveFederatedLearning ......... 140 3.2.1 SparseNetworkEncodingwithaRandomGraph ......... 141 3.2.2 EvolutionaryMulti-objectiveNeuralArchitecture Search .............................................. 141 3.2.3 OverallFramework ................................... 144 3.2.4 EmpiricalResults .................................... 146 3.3 Real-Time Evolutionary Federated Neural Architecture Search .................................................... 150 3.3.1 NetworkArchitectureEncodingBasedOnSupernet ....... 151 3.3.2 NetworkSamplingandClientSampling ................. 153 3.3.3 OverallFramework ................................... 154 3.3.4 EmpiricalStudies .................................... 157 3.4 Summary .................................................. 163 References ..................................................... 163 4 SecureFederatedLearning ...................................... 165 4.1 ThreatstoFederatedLearning ................................ 165 4.2 DistributedEncryptionforHorizontalFederatedLearning ........ 166 4.2.1 DistributedDataEncryption ........................... 167 4.2.2 FederatedEncryptionandDecryption ................... 168 4.2.3 TernaryQuantizationandApproximateAggregation ...... 173 4.2.4 OverallFramework ................................... 176 4.2.5 EmpiricalStudies .................................... 178 4.3 SecureVerticalFederatedLearning ........................... 185 4.3.1 VerticalFederatedLearningwithXGBoost .............. 187 4.3.2 SecureNodeSplitandConstruction .................... 192 4.3.3 PartialDifferentialPrivacy ............................ 196