Federated Learning: A Comprehensive Overview of Methods and Applications PDF
Preview Federated Learning: A Comprehensive Overview of Methods and Applications
Heiko Ludwig Nathalie Baracaldo Editors Federated Learning A Comprehensive Overview of Methods and Applications Federated Learning Heiko Ludwig • Nathalie Baracaldo Editors Federated Learning A Comprehensive Overview of Methods and Applications Editors HeikoLudwig NathalieBaracaldo IBMResearch–Almaden IBMResearch–Almaden SanJose,CA,USA SanJose,CA,USA ISBN978-3-030-96895-3 ISBN978-3-030-96896-0 (eBook) https://doi.org/10.1007/978-3-030-96896-0 ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland AG2022 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar ordissimilarmethodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Preface Machine learning has made great strides over the past two decades and has been adoptedinmanyapplicationdomains.Successfulmachinelearningdependslargely onaccesstoqualitydata,bothlabeledandunlabeled. Concerns related to data privacy, security, and sovereignty have caused public and technical discussion on how to use data for machine learning purposes consistentwithregulatoryandstakeholderinterests.Theseconcernsandlegislation have led to the realization that collecting training data in large central repositories maybeatoddswithmaintainingprivacyfordataowners. While distributed learning or model fusion has been discussed since at least a decade, federated machine learning (FL) as a concept has been popularized by MacMahanandotherssince2017.Inthesubsequentyears,muchresearchhasbeen conducted – both, in academia and the industry – and, at the time of writing this book,thefirstviablecommercialframeworksforfederatedlearningarecomingto themarket. Thisbookaimstocapturetheresearchprogressandstateoftheartthathasbeen made in the past years, from the initial conception of the field to first applications and commercial use. To get this broad and deep overview, we invited leading researchers to address the different perspectives of federated learning: the core machinelearningperspective,privacyandsecurity,distributedsystems,andspecific applicationdomains. The book’s title, Federated Learning: A Comprehensive Overview of Methods and Applications, outlines its scope. It presents in depth the most important issuesandapproachestofederatedlearningforresearchersandpractitioners.Some chapters contain a variety of technical content that is relevant to understand the intricaciesofthealgorithmsandparadigmsthatmakeitpossibletodeployfederated learninginmultipleenterprisesettings.Otherchaptersfocusonprovidingclarityon howtoselectprivacyandsecuritysolutionsinawaythatcanbetailoredtospecific usecases,whileotherstakeintoconsiderationthepragmaticsofthesystemswhere thefederatedlearningprocesswillrun. Giventheinherentcross-disciplinarynatureofthetopic,weencounterdifferent notational conventions in different chapters of the book. What might be parties in v vi Preface federatedmachinelearningmaybecalledclientsinthedistributedsystemsperspec- tives.Intheintroductorychapterofthisbook,welayouttheprimaryterminology weuse,andeachchapterexplainshowthediscipline-specificterminologymapsto the common one when it is introduced, if this is the case. With this approach, we makethisbookunderstandabletoreadersfromdiversebackgroundswhilestaying truetotheconventionsofthespecificdisciplinesinvolved. Taken as a whole, this book enables the reader to get a broad state-of-the-art summaryofthemostrecentresearchdevelopments. Editingthisbook,andwritingsomeofthechapters,requiredthehelpofmany, who we want to acknowledge. IBM Research gave us the opportunity to work in thisexcitingfield,notjustacademicallybutalsotoputthistechnologyintopractice andmakeitpartofaproduct.Welearnedinvaluablelessonsonthejourney,andwe havemuchtothanktoourcolleaguesatIBM.Inparticular,wewanttoacknowledge ourdirector,SandeepGopisetty,forgivingusthespacetoworkonthisbook:Gegi Thomas,whomadesureourresearchcontributionsmaketheirwayintotheproduct: andourteammembers. Thechapterauthorsprovidethesubstanceofthisbookandwerepatientwithus withrequestsforchangestotheirchapters. We owe greatest thanks to our families, who patiently put up with us devoting time to the book rather than them over the year of writing and editing this book. Heiko is deeply thankful to his wife, Beatriz Raggio, for making these sacrifices andsupportinghimthroughout.Nathalieisprofoundlythankfultoherhusbandand sons, Santiago and Matthias Bock, for their love and support and for cheering for allherprojects,includingthisone.Shealsothanksherparents,AdrianaandJesus; thisandmanymoreachievementswouldnotbepossiblewithouttheiramazingand continuoussupport. SanJose,CA,USA HeikoLudwig September2021 NathalieBaracaldo Contents 1 IntroductiontoFederatedLearning...................................... 1 HeikoLudwigandNathalieBaracaldo PartI FederatedLearningasaMachineLearningProblem 2 Tree-BasedModelsforFederatedLearningSystems................... 27 YuyaJeremyOng,NathalieBaracaldo,andYiZhou 3 SemanticVectorization:Text-andGraph-BasedModels.............. 53 ShalishaWitherspoon,DeanSteuer,andNirmitDesai 4 PersonalizationinFederatedLearning................................... 71 MayankAgarwal,MikhailYurochkin,andYuekaiSun 5 Personalized,RobustFederatedLearningwithFed+ .................. 99 PengqianYu,AchintyaKundu,LauraWynter,andShiauHongLim 6 Communication-EfficientDistributedOptimizationAlgorithms ..... 125 GauriJoshiandShiqiangWang 7 Communication-EfficientModelFusion................................. 145 MikhailYurochkinandYuekaiSun 8 FederatedLearningandFairness......................................... 177 AnnieAbay,YiZhou,NathalieBaracaldo,andHeikoLudwig PartII SystemsandFrameworks 9 IntroductiontoFederatedLearningSystems............................ 195 SyedZawad,FengYan,andAliAnwar 10 LocalTrainingandScalabilityofFederatedLearningSystems....... 213 SyedZawad,FengYan,andAliAnwar 11 StragglerManagement..................................................... 235 SyedZawad,FengYan,andAliAnwar vii viii Contents 12 SystemsBiasinFederatedLearning ..................................... 259 SyedZawad,FengYan,andAliAnwar PartIII PrivacyandSecurity 13 ProtectingAgainstDataLeakageinFederatedLearning: WhatApproachShouldYouChoose?.................................... 281 NathalieBaracaldoandRunhuaXu 14 PrivateParameterAggregationforFederatedLearning............... 313 K.R.JayaramandAshishVerma 15 DataLeakageinFederatedLearning .................................... 337 XiaoJin,Pin-YuChen,andTianyiChen 16 SecurityandRobustnessinFederatedLearning........................ 363 Ambrish Rawat, Giulio Zizzo, Muhammad Zaid Hameed, andLuisMuñoz-González 17 DealingwithByzantineThreatstoNeuralNetworks................... 391 YiZhou,NathalieBaracaldo,AliAnwar,andKamalaVarma PartIV Beyond Horizontal Federated Learning: Partitioning ModelsandDatainDiverseWays 18 Privacy-PreservingVerticalFederatedLearning ....................... 417 Runhua Xu, Nathalie Baracaldo, Yi Zhou, Annie Abay, andAliAnwar 19 Split Learning: A Resource Efficient Model and Data ParallelApproachforDistributedDeepLearning...................... 439 PraneethVepakommaandRameshRaskar PartV Applications 20 FederatedLearningforCollaborativeFinancialCrimes Detection..................................................................... 455 ToyotaroSuzumura,YiZhou,RyoKawahara,NathalieBaracaldo, andHeikoLudwig 21 FederatedReinforcementLearningforPortfolioManagement....... 467 PengqianYu,LauraWynter,andShiauHongLim 22 ApplicationofFederatedLearninginMedicalImaging ............... 483 Ehsan Degan, Shafiq Abedin, David Beymer, Angshuman Deb, NathanielBraman,BenediktGraf,andVandanaMukherjee 23 AdvancingHealthcareSolutionswithFederatedLearning............ 499 AmoghKamatTarcar Contents ix 24 APrivacy-preservingProductRecommenderSystem ................. 509 TuanM.HoangTrong,MudhakarSrivatsa,andDineshVerma 25 ApplicationofFederatedLearninginTelecommunications andEdgeComputing....................................................... 523 UtpalMangla Chapter 1 Introduction to Federated Learning HeikoLudwigandNathalieBaracaldo Abstract Federated learning (FL) is an approach to machine learning in which the training data is not managed centrally. Data is retained by data parties that participate in the FL process and is not shared with any other entity. This makes FLanincreasinglypopularsolutionformachinelearningtasksforwhichbringing datatogetherinacentralizedrepositoryisproblematic,eitherforprivacy,regulatory, orpracticalreasons.Inthischapter,weintroducethebasicconceptsofFL,provide an overview of its application use cases, and discuss it from a machine learning, distributedcomputing,andprivacyperspective.Wealsoprovideanintroductionto divedeepintothemattercoveredinthesubsequentchapters. 1.1 Overview Machine learning (ML) has become a crucial technique to develop cognitive and analytic functionality difficult and ineffective to develop algorithmically. Applica- tions in computer vision, speech recognition, and natural language understanding progressedbyleapsandboundswiththeadventofDeepNeuralNetworks(DNNs) and the computational hardware to train complex networks effectively. Also, classicalMLtechniquessuchasdecisiontrees,linearregression,andsupportvector models(SVMs)foundincreaseduse,inparticularrelatedtostructureddata. The application of ML critically depends on the availability of high-quality training data. Sometimes, though, privacy considerations prevent training data to be brought to a central data repository to be curated and managed for the ML process.Federatedlearning(FL)isanapproachfirstproposedbythisnamein[28] totrainMLmodelsontrainingdataindisparatelocations,notrequiringthecentral collectionofdata. H.Ludwig((cid:2))·N.Baracaldo IBMResearch–Almaden,SanJose,CA,USA e-mail:[email protected];[email protected] ©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2022 1 H.Ludwig,N.Baracaldo(eds.),FederatedLearning, https://doi.org/10.1007/978-3-030-96896-0_1
The list of books you might like
