Probability and Computing: Randomization and Probabilistic Techniques in Algorithms and Data Analysis

490 Pages·2017·11.686 MB·English

by  Michael Mitzenmacher, Eli Upfal

Probability and Computing
Randomization and probabilistic techniques play an important role in modern computer
science,withapplicationsrangingfromcombinatorialoptimizationandmachinelearning
tocommunicationnetworksandsecureprotocols.
This textbook provides an indispensable teaching tool to accompany a one- or two-
semester course for advanced undergraduate or beginning graduate students in computer
scienceandappliedmathematics.Itoffersacomprehensiveintroductiontotheroleofran-
domizationandprobabilistictechniquesinmoderncomputerscience,inparticulartotech-
niques and paradigms used in the development and probabilistic analysis of algorithms
andfordataanalyses.Itassumesonlyanelementarybackgroundindiscretemathematics
andgivesarigorousyetaccessibletreatmentofthematerial,withnumerousexamplesand
applications.
The first half of the book covers core material, including random sampling, expecta-
tions,Markov’sinequality,Chebyshev’sinequality,Chernoffbounds,balls-and-binsmod-
els, the probabilistic method, and Markov chains. In the second half, the authors delve
intomoreadvancedtopicssuchascontinuousprobability,applicationsoflimitedindepen-
dence,entropy,MarkovchainMonteCarlomethods,coupling,martingales,andbalanced
allocations.
This greatly expanded new edition includes several newly added chapters and sec-
tions, covering topics including normal distributions, sample complexity, VC dimension,
Rademachercomplexity,powerlawsandrelateddistributions,cuckoohashing,andappli-
cations of the Lovász Local Lemma. New material relevant to machine learning and big
dataanalysisenablesstudentstolearnup-to-datetechniquesandapplications.Amongthe
manynewexercisesandexamplesareprogramming-relatedexercisesthatprovidestudents
withpracticalexperienceandtrainingrelatedtothetheoreticalconceptscoveredinthetext.
Michael Mitzenmacher is a Professor of Computer Science in the School of Engineering
and Applied Sciences at Harvard University, where he was also the Area Dean for Com-
puter Science from 2010 to 2013. Michael has authored or co-authored over 200 confer-
enceandjournalpublicationsonavarietyoftopics,includingalgorithmsfortheInternet,
efficient hash-based data structures, erasure and error-correcting codes, power laws, and
compression.Hisworkonlow-densityparity-checkcodessharedthe2002IEEEInforma-
tionTheorySocietyBestPaperAwardandwonthe2009ACMSIGCOMMTestofTime
Award. He is an ACM Fellow, and was elected as the Chair of the ACM Special Interest
GrouponAlgorithmsandComputationTheoryin2015.
EliUpfalisaProfessorofComputerScienceatBrownUniversity,wherehewasalsothe
departmentchairfrom2002to2007.PriortojoiningBrownin1998,hewasaresearcherand
projectmanagerattheIBMAlmadenResearchCenter,andaProfessorofAppliedMath-
ematics and Computer Science at the Weizmann Institute of Science. His main research
interests are randomized algorithms, probabilistic analysis of algorithms, and computa-
tionalstatistics,withapplicationsrangingfromcombinatorialandstochasticoptimization,
computationalbiology,andcomputationalfinance.HeisaFellowofboththeIEEEandthe
ACM.

Probability and Computing
Randomization and Probabilistic
Techniques in Algorithms and
Data Analysis
Second Edition
Michael Mitzenmacher Eli Upfal
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
4843/24,2ndFloor,AnsariRoad,Daryaganj,Delhi-110002,India
79AnsonRoad,#06-04/06,Singapore079906
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781107154889
10.1017/9781316651124
©MichaelMitzenmacherandEliUpfal2017
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2017
PrintedintheUnitedStatesofAmericabySheridanBooks,Inc.
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
LibraryofCongressCataloginginPublicationData
Names:Mitzenmacher,Michael,1969–author.|Upfal,Eli,1954–author.
Title:Probabilityandcomputing/MichaelMitzenmacherEliUpfal.
Description:Secondedition.|Cambridge,UnitedKingdom;
NewYork,NY,USA:CambridgeUniversityPress,[2017]|
Includesbibliographicalreferencesandindex.
Identifiers:LCCN2016041654|ISBN9781107154889
Subjects:LCSH:Algorithms.|Probabilities.|Stochasticanalysis.
Classification:LCCQA274.M574 2017|DDC518/.1–dc23
LCrecordavailableathttps://lccn.loc.gov/2016041654
ISBN978-1-107-15488-9Hardback
Additionalresourcesforthispublicationatwww.cambridge.org/Mitzenmacher.
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracy
ofURLsforexternalorthird-partyInternetWebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchWebsitesis,orwillremain,
accurateorappropriate.
To
Stephanie,Michaela,Jacqueline,andChloe
M.M.
Liane,Tamara,andIlan
E.U.

Contents
PrefacetotheSecondEdition pagexv
PrefacetotheFirstEdition xvii
1 EventsandProbability 1
1.1 Application:VerifyingPolynomialIdentities 1
1.2 AxiomsofProbability 3
1.3 Application:VerifyingMatrixMultiplication 8
1.4 Application:NaïveBayesianClassifier 12
1.5 Application:ARandomizedMin-CutAlgorithm 15
1.6 Exercises 17
2 DiscreteRandomVariablesandExpectation 23
2.1 RandomVariablesandExpectation 23
2.1.1 LinearityofExpectations 25
2.1.2 Jensen’sInequality 26
2.2 TheBernoulliandBinomialRandomVariables 27
2.3 ConditionalExpectation 29
2.4 TheGeometricDistribution 33
2.4.1 Example:CouponCollector’sProblem 35
2.5 Application:TheExpectedRun-TimeofQuicksort 37
2.6 Exercises 40
3 MomentsandDeviations 47
3.1 Markov’sInequality 47
3.2 VarianceandMomentsofaRandomVariable 48
3.2.1 Example:VarianceofaBinomialRandomVariable 51
vii
contents
3.3 Chebyshev’sInequality 51
3.3.1 Example:CouponCollector’sProblem 53
3.4 MedianandMean 55
3.5 Application:ARandomizedAlgorithmforComputingtheMedian 57
3.5.1 TheAlgorithm 58
3.5.2 AnalysisoftheAlgorithm 59
3.6 Exercises 62
4 ChernoffandHoeffdingBounds 66
4.1 MomentGeneratingFunctions 66
4.2 DerivingandApplyingChernoffBounds 68
4.2.1 ChernoffBoundsfortheSumofPoissonTrials 68
4.2.2 Example:CoinFlips 72
4.2.3 Application:EstimatingaParameter 72
4.3 BetterBoundsforSomeSpecialCases 73
4.4 Application:SetBalancing 76
4.5 TheHoeffdingBound 77
4.6∗ Application:PacketRoutinginSparseNetworks 79
4.6.1 PermutationRoutingontheHypercube 80
4.6.2 PermutationRoutingontheButterfly 85
4.7 Exercises 90
5 Balls,Bins,andRandomGraphs 97
5.1 Example:TheBirthdayParadox 97
5.2 BallsintoBins 99
5.2.1 TheBalls-and-BinsModel 99
5.2.2 Application:BucketSort 101
5.3 ThePoissonDistribution 101
5.3.1 LimitoftheBinomialDistribution 105
5.4 ThePoissonApproximation 107
5.4.1∗ Example:CouponCollector’sProblem,Revisited 111
5.5 Application:Hashing 113
5.5.1 ChainHashing 113
5.5.2 Hashing:BitStrings 114
5.5.3 BloomFilters 116
5.5.4 BreakingSymmetry 118
5.6 RandomGraphs 119
5.6.1 RandomGraphModels 119
5.6.2 Application:HamiltonianCyclesinRandomGraphs 121
5.7 Exercises 127
5.8 AnExploratoryAssignment 133
6 TheProbabilisticMethod 135
6.1 TheBasicCountingArgument 135
viii