The Fundamentals of Heavy Tails: Properties, Emergence, and Estimation

265 Pages·2022·9.316 MB·English

by  Jayakrishnan Nair, Adam Wierman, Bert Zwart

The Fundamentals of Heavy Tails
Heavytails–extremeeventsorvaluesmorecommonthanexpected–emergeevery-
where:theeconomy,naturalevents,andsocialandinformationnetworksarejusta
fewexamples.Yetafterdecadesofprogress,theyarestilltreatedasmysterious,sur-
prising,andevencontroversial,primarilybecausethenecessarymathematicalmodels
andstatisticalmethodsarenotwidelyknown.
Thisbook,forthefirsttime,providesarigorousintroductiontoheavy-taileddistri-
butionsaccessibletoanyonewhoknowselementaryprobability.Ittacklesandtames
the zoo of terminology for models and properties, demystifying topics such as the
generalized central limit theorem and regular variation. It tracks the natural emer-
genceofheavy-taileddistributionsfromawidevarietyofgeneralprocesses,building
intuition. And it reveals the controversy surrounding heavy tails to be the result of
flawedstatistics,thenequipsreaderstoidentifyandestimatewithconfidence.Over
100exercisescompletethisengagingpackage.
(cid:3459)(cid:3450)(cid:3474)(cid:3450)(cid:3460)(cid:3467)(cid:3458)(cid:3468)(cid:3457)(cid:3463)(cid:3450)(cid:3463) (cid:3463)(cid:3450)(cid:3458)(cid:3467)isAssociateProfessorinElectricalEngineeringatIITBom-
bay. His research focuses on modeling, performance evaluation, and design issues
in online learning environments, communication networks, queueing systems, and
smartpowergrids.HeistherecipientofbestpaperawardsatIFIPPerformance(2010
and2020)andACMe-Energy(2020).
(cid:3450)(cid:3453)(cid:3450)(cid:3462) (cid:3472)(cid:3458)(cid:3454)(cid:3467)(cid:3462)(cid:3450)(cid:3463)isProfessorofComputingandMathematicalSciencesattheCal-
ifornia Institute of Technology (Caltech). His research develops tools in machine
learning,optimization,control,andeconomicswiththegoalofmakingthenetworked
systemsthatgovernourworldsustainableandresilient.Heisbestknownforhiswork
spearheadingthedesignofalgorithmsforsustainabledatacentersandheistherecip-
ientofnumerousawardsincludingtheACMSigmetricsRisingStaraward,theACM
Sigmetrics Test of Time award, the IEEE Communication Society William Bennet
Prize,andmultipleteachingandbestpaperawards.
(cid:3451)(cid:3454)(cid:3467)(cid:3469) (cid:3475)(cid:3472)(cid:3450)(cid:3467)(cid:3469) is group leader at CWI Amsterdam and Professor of Mathematics
at Eindhoven University of Technology. He has expertise in stochastic operations
research,queueingtheory,andlargedeviations,andinthecontextofheavytails,he
has focused on sample path properties, designing Monte Carlo methods and appli-
cations to computer-communication and energy networks. He was area editor of
OperationsResearch,theflagshipjournalofhisprofession,from2009to2017,and
wastherecipientoftheINFORMSAppliedProbabilitySocietyErlangprize,awarded
everytwoyearstoanoutstandingyoungappliedprobabilist.
CAMBRIDGE SERIES IN STATISTICAL AND
PROBABILISTIC MATHEMATICS
EditorialBoard
Z.Ghahramani(DepartmentofEngineering,UniversityofCambridge)
R.Gill(MathematicalInstitute,LeidenUniversity)
F.P.Kelly(DepartmentofPureMathematicsandMathematicalStatistics,
UniversityofCambridge)
B.D.Ripley(DepartmentofStatistics,UniversityofOxford)
S.Ross(DepartmentofIndustrialandSystemsEngineering,
UniversityofSouthernCalifornia)
M.Stein(DepartmentofStatistics,UniversityofChicago)
Thisseriesofhigh-qualityupper-divisiontextbooksandexpositorymonographscoversallaspects
ofstochasticapplicablemathematics.Thetopicsrangefrompureandappliedstatisticstoprobability
theory,operationsresearch,optimization,andmathematicalprogramming.Thebookscontainclear
presentationsofnewdevelopmentsinthefieldandalsoofthestateoftheartinclassicalmethods.
Whileemphasizingrigoroustreatmentoftheoreticalmethods,thebooksalsocontainapplications
anddiscussionsofnewtechniquesmadepossiblebyadvancesincomputationalpractice.
Acompletelistofbooksintheseriescanbefoundatwww.cambridge.org/statistics..
Recenttitlesincludethefollowing:
27. ModelSelectionandModelAveraging,byGerdaClaeskensandNilsLidHjort
28. BayesianNonparametrics,editedbyNilsLidHjortetal.
29. FromFiniteSampletoAsymptoticMethodsinStatistics,byPranabK.Sen,JulioM.Singer
andAntonioC.PedrosadeLima
30. BrownianMotion,byPeterMörtersandYuvalPeres
31. Probability:TheoryandExamples(FourthEdition),byRickDurrett
32. AnalysisofMultivariateandHigh-DimensionalData,byIngeKoch
33. StochasticProcesses,byRichardF.Bass
34. RegressionforCategoricalData,byGerhardTutz
35. ExercisesinProbability(SecondEdition),byLoïcChaumontandMarcYor
36. StatisticalPrinciplesfortheDesignofExperiments,byR.Mead,S.G.GilmourandA.Mead
37. QuantumStochastics,byMou-HsiungChang
38. Nonparametric Estimation under Shape Constraints, by Piet Groeneboom and
GeurtJongbloed
39. LargeSampleCovarianceMatricesandHigh-DimensionalDataAnalysis,byJianfengYao,
ShurongZhengandZhidongBai
40. Mathematical Foundations of Infinite-Dimensional Statistical Models, by Evarist Giné and
RichardNickl
41. Confidence,Likelihood,Probability,byToreSchwederandNilsLidHjort
42. ProbabilityonTreesandNetworks,byRussellLyonsandYuvalPeres
43. RandomGraphsandComplexNetworks(Volume1),byRemcovanderHofstad
44. Fundamentals of Nonparametric Bayesian Inference, by Subhashis Ghosal and
AadvanderVaart
45. Long-RangeDependenceandSelf-Similarity,byVladasPipirasandMuradS.Taqqu
46. PredictiveStatistics,byBertrandS.ClarkeandJenniferL.Clarke
47. High-DimensionalProbability,byRomanVershynin
48. High-DimensionalStatistics,byMartinJ.Wainwright
49. Probability:TheoryandExamples(FifthEdition),byRickDurrett
50. StatisticalModel-basedClusteringandClassification,byCharlesBouveyronetal.
51. SpectralAnalysisforUnivariateTimeSeries,byDonaldB.PercivalandAndrewT.Walden
52. StatisticalHypothesisTestinginContext,byMichaelP.FayandEricaBrittain
The Fundamentals of Heavy Tails
Properties, Emergence, and Estimation
Jayakrishnan Nair
IndianInstituteofTechnology,Bombay
Adam Wierman
CaliforniaInstituteofTechnology
Bert Zwart
CWIAmsterdamandEindhovenUniversityofTechnology
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,NewDelhi–110025,India
103PenangRoad,#05-06/07,VisioncrestCommercial,Singapore238467
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781316511732
DOI:10.1017/9781009053730
©JayakrishnanNair,AdamWierman,andBertZwart2022
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2022
PrintedintheUnitedKingdombyTJBooksLimited,PadstowCornwall
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
ISBN978-1-316-51173-2Hardback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracyof
URLsforexternalorthird-partyinternetwebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain,
accurateorappropriate.
Contents
Preface pageix
Acknowledgments xiii
1 Introduction 1
1.1 DefiningHeavy-TailedDistributions 5
1.2 ExamplesofHeavy-TailedDistributions 9
1.3 What’sNext 24
1.4 Exercises 24
PartI Properties 27
2 ScaleInvariance,PowerLaws,andRegularVariation 29
2.1 ScaleInvarianceandPowerLaws 30
2.2 ApproximateScaleInvarianceandRegularVariation 32
2.3 AnalyticPropertiesofRegularlyVaryingFunctions 36
2.4 AnExample:ClosurePropertiesofRegularlyVaryingDistributions 48
2.5 AnExample:BranchingProcesses 50
2.6 AdditionalNotes 53
2.7 Exercises 54
3 Catastrophes,Conspiracies,andSubexponentialDistributions 56
3.1 ConspiraciesandCatastrophes 58
3.2 SubexponentialDistributions 62
3.3 AnExample:Randomsums 67
3.4 AnExample:ConspiraciesandCatastrophesinRandomWalks 72
3.5 AdditionalNotes 80
3.6 Exercises 81
4 ResidualLives,HazardRates,andLongTails 85
4.1 ResidualLivesandHazardRates 86
4.2 HeavyTailsandResidualLives 90
4.3 Long-TailedDistributions 93
4.4 AnExample:RandomExtrema 97
4.5 AdditionalNotes 100
4.6 Exercises 101
v
vi Contents
PartII Emergence 105
5 AdditiveProcesses 107
5.1 TheCentralLimitTheorem 108
5.2 GeneralizingtheCentralLimitTheorem 112
5.3 UnderstandingStableDistributions 114
5.4 TheGeneralizedCentralLimitTheorem 118
5.5 AVariation:TheEmergenceofHeavyTailsinRandomWalks 120
5.6 AdditionalNotes 124
5.7 Exercises 125
6 MultiplicativeProcesses 127
6.1 TheMultiplicativeCentralLimitTheorem 128
6.2 VariationsonMultiplicativeProcesses 131
6.3 AnExample:PreferentialAttachmentandYuleProcesses 138
6.4 AdditionalNotes 144
6.5 Exercises 145
7 ExtremalProcesses 148
7.1 ALimitTheoremforMaxima 150
7.2 UnderstandingMax-StableDistributions 154
7.3 TheExtremalCentralLimitTheorem 156
7.4 AnExample:ExtremesofRandomWalks 161
7.5 AVariation:TheTimebetweenRecordBreakingEvents 168
7.6 AdditionalNotes 170
7.7 Exercises 171
PartIII Estimation 175
8 EstimatingPower-LawDistributions:ListentotheBody 177
8.1 ParametricEstimationofPower-LawsUsingLinearRegression 179
8.2 MaximumLikelihoodEstimationforPower-LawDistributions 185
8.3 PropertiesoftheMaximumLikelihoodEstimator 187
8.4 VisualizingtheMLEviaRegression 189
8.5 ARecipeforParametricEstimationofPower-LawDistributions 192
8.6 AdditionalNotes 194
8.7 Exercises 195
9 EstimatingPower-LawTails:LettheTailDotheTalking 197
9.1 TheFailureofParametricEstimation 199
9.2 TheHillEstimator 203
9.3 PropertiesoftheHillEstimator 205
9.4 TheHillPlot 209
9.5 BeyondHillandRegularVariation 215
9.6 WhereDoestheTailBegin? 226
9.7 GuidelinesforEstimatingHeavy-TailedPhenomena 233
Contents vii
9.8 AdditionalNotes 235
9.9 Exercises 236
CommonlyUsedNotation 238
References 240
Index 249

Preface
Heavytailsareacontinualsourceofexcitementandconfusionastheyarerepeatedly“dis-
covered”innewcontexts.Thisistrueacrossthesocialsciences,thephysicalsciences,and
especiallytheinformationsciences,whereheavy-tailspopupseeminglyeverywhere–from
degree distributions in the web and social networks to file sizes and interarrival times of
computejobs.However,despitenearlyacenturyofworkonheavytails,theyarestilltreated
asmysterious,duetotheircounterintuitiveproperties;surprising,sincethecentrallimitthe-
orem trains us to expect the “Normal” distribution; and even controversial, since intuitive
statisticaltoolscanleadtofalsediscoveries.
The goal of this book is to show that heavy-tailed distributions need not be mysterious
and should not be surprising or controversial either. The book strives to demystify heavy-
taileddistributionsbyshowinghowtoreasonformallyabouttheircounterintuitiveproperties
by demonstrating that the emergence of heavy-tailed phenomena should be expected (not
surprising)andbyrevealingthatmostofthecontroversysurroundingheavy-tailsistheresult
of bad statistics and can be avoided by using the proper tools. It proceeds by offering a
coherentcollectionofbite-sizedintroductionstothebigquestions:
Howcanweunderstandtheseeminglycounterintuitivepropertiesofheavy-tailed
distributions?
Whyareheavy-taileddistributionssocommon?
Howcanheavy-taileddistributionsbeestimatedfromdata?
Thesequestionsareimportantnotjusttomathematiciansbutalsotocomputerscientists,
economists,socialscientists,astronomers,andbeyond.Heavy-taileddistributionstrulycome
up everywhere in the world around us, and understanding their consequences is crucial to
an understanding of the world. Despite this, many students learn probability and statistics
without being taught how to think carefully about the heavy tails. Our mission is to make
thispracticeathingofthepast.
Thisbookaimstoprovideanaccessibleintroductiontothefundamentalsofheavytails.It
coversmathematicallydeepconceptsthataretypicallytaughtingraduate-levelcourses,such
asthegeneralizedcentrallimittheorem,extremevaluetheory,andregularvariation;butit
does so using only elementary mathematical tools in order to make these topics accessible
to anyone who has had an introductory probability course. To make this possible we have
explicitlychosen,attimes,nottoproveresultsinfullgenerality.Instead,wehaveworkedto
rigorouslypresentthecoreideainassimpleamanneraspossible,pushingasidetechnicalities
ix