Superconcentration and related topics

156 Pages·2014·3.088 MB·English

by  Sourav Chatterjee (auth.)

Springer Monographs in Mathematics
Sourav Chatterjee
Superconcentration
and Related Topics
Springer Monographs in Mathematics
Forfurthervolumes:
www.springer.com/series/3733
Sourav Chatterjee
Superconcentration
and Related Topics
SouravChatterjee
DepartmentofStatistics
StanfordUniversity
Stanford,CA,USA
ISSN1439-7382 ISSN2196-9922(electronic)
SpringerMonographsinMathematics
ISBN978-3-319-03885-8 ISBN978-3-319-03886-5(eBook)
DOI10.1007/978-3-319-03886-5
SpringerChamHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2014930163
MathematicsSubjectClassification(2010): 60E15,60K35,60G15,82B44,60G60,60G70
©SpringerInternationalPublishingSwitzerland2014
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub-
lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany
errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect
tothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Understandingthefluctuationsofrandomobjectsisoneofthemajorgoalsofproba-
bilitytheory.Thereisawholesubfieldofprobabilityandanalysis,calledconcentra-
tionofmeasure,devotedtounderstandingfluctuationsofrandomobjects.Measure
concentrationhasseentremendousprogressinthelastfortyyears.Andyet,thereis
alargeclassofproblemsinwhichclassicalconcentrationofmeasuregivessubop-
timalboundsontheorderoffluctuations.
In2008and2009,IpostedtwopreprintsonarXivwhereitwasshownthatthe
suboptimality of classical concentration, when it occurs, is not simply a question
ofmathematicalinadequacy.Thesuboptimalityisinfactequivalenttoanumberof
veryinterestingthingsgoingoninthestructureoftherandomobjectunderinves-
tigation.Indeed,theconsequencesarepossiblyinterestingenoughforthesubopti-
mality of classical concentration to deserve a name of its own; I call it ‘supercon-
centration’.
Thismonographisacombinationofthesetwopreprints(whichwillnotbepub-
lishedindividually),togetherwithsomenewmaterialandnewinsights.Themajor-
ity of the results are the same as in the preprints, but the presentation is radically
different.Inparticular,IthinkIachievedasubstantialdegreeofsimplificationand
claritythroughtheuseofthespectralapproach.Thisisquitestandardinthenoise-
sensitivity literature (which is intimately connected with the topic of this mono-
graph),butitisnotthewayIderivedtheresultsinthepreprints.
Inadditiontothetheoremsandproofs,Ihaveinterspersedthedocumentwitha
sizablenumberofopenproblemsforprofessionalmathematiciansandexercisesfor
graduatestudents.
I spent many hours deliberating over whether to keep the book in its present
form,orexpandittoaround250pagesbyincludingadditionalmaterialfromclas-
sicalconcentrationofmeasureandotherrelatedtopics.IntheendIdecidednotto
expand.Therationalebehindthisdecisionistwo-fold:Thefirstreasonisthatthere
areseveralcomprehensivetextsonconcentrationofmeasurealreadyavailableinthe
market,andIdidnotwishtoencroachonthatterritory.Ihadoriginallyintendedthis
booktobeashortandsuccinctexpositionofthesuperconcentrationphenomenon,
andintheendIdecidedtokeepitthatway.ThesecondreasonisthatIamdeeply
v
vi Preface
familiarwithmyprocrastinatingtendencies,whichmademeconfidentthatIwould
never have finished this project if I had planned a major overhaul. However I did
expanda little bit; the original version that I submitted to Springer was even thin-
ner. On the advice of one of the reviewers, I decided to include several additional
examplesthatIhadomittedinthefirstdraft.
ThemainbodyofthismonographgrewoutofasetofsixlecturesIgaveatthe
CornellProbabilitySummerSchoolinJuly2012.Thetaskofexplainingtheresults
and proofs to graduate students forced me to organize the material in a manner
suitable for exposition in a monograph. I thank the organizers of CPSS 2012 for
giving me this opportunity, and one of the students attending the summer school,
MihaiNica,fordoingaterrificjobintakingnotesandtypingthemup.
I am grateful to Persi Diaconis for suggesting that I write this monograph, and
for his constant encouragement and advice. I thank the reviewers for many useful
comments,andChristopheGarban,SusanHolmes,DmitryPanchenkoandMichel
Talagrandforlookingattheearlydrafts,givingsuggestionsforimprovements,and
pointingoutmisattributionsanderrors.Dr.CatrionaByrneofSpringerandheredi-
torialteamhasmygratitudeforbeingexceptionallyhelpfulandresponsiveinevery
stepofthepreparationofthemanuscript.
IwouldliketoacknowledgetheroleplayedbytheNationalScienceFoundation,
the Courant Institute, and UC Berkeley in funding, at various stages, the research
relatedtothisbook.
Andfinally,Imustthankonepersonwhoisoutsidetherealmofmathematicsand
yetplayedanindispensableroleinthecompletionofthisproject:Iwouldhavenever
finished writing the monographwithout the sustained urging, care and patienceof
mywife,Esha.Ithankherforallthatandmuchmore.
Stanford,CA,USA SouravChatterjee
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1 Superconcentration . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Chaos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 MultipleValleys . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 MarkovSemigroups . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1 SemigroupBasics . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 TheOrnstein-UhlenbeckSemigroup . . . . . . . . . . . . . . . 17
3 ConnectionwithMalliavinCalculus . . . . . . . . . . . . . . . 18
4 PoincaréInequalities . . . . . . . . . . . . . . . . . . . . . . . . 18
5 SomeApplicationsoftheGaussianPoincaréInequality . . . . . 19
6 FourierExpansion . . . . . . . . . . . . . . . . . . . . . . . . . 20
3 SuperconcentrationandChaos . . . . . . . . . . . . . . . . . . . . 23
1 DefinitionofSuperconcentration . . . . . . . . . . . . . . . . . 23
2 SuperconcentrationandNoise-Sensitivity . . . . . . . . . . . . . 25
3 DefinitionofChaos . . . . . . . . . . . . . . . . . . . . . . . . 25
4 EquivalenceofSuperconcentrationandChaos . . . . . . . . . . 27
5 SomeApplicationsoftheEquivalenceTheorem . . . . . . . . . 28
6 ChaoticNatureoftheFirstEigenvector . . . . . . . . . . . . . . 29
4 MultipleValleys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1 ChaosImpliesMultipleValleys:TheGeneralIdea . . . . . . . . 33
2 MultipleValleysinGaussianPolymers . . . . . . . . . . . . . . 34
3 MultipleValleysintheSKModel . . . . . . . . . . . . . . . . . 36
4 MultiplePeaksinGaussianFields . . . . . . . . . . . . . . . . . 38
5 MultiplePeaksintheNK FitnessLandscape . . . . . . . . . . . 40
5 Talagrand’sMethodforProvingSuperconcentration . . . . . . . . 45
1 Hypercontractivity . . . . . . . . . . . . . . . . . . . . . . . . . 45
2 Talagrand’sL1–L2 Bound . . . . . . . . . . . . . . . . . . . . . 47
3 Talagrand’sMethodAlwaysWorksforMonotoneFunctions . . . 49
vii
viii Contents
4 TheBenjamini-Kalai-SchrammTrick . . . . . . . . . . . . . . . 51
5 SuperconcentrationinGaussianPolymers. . . . . . . . . . . . . 54
6 SharpnessoftheLogarithmicImprovement . . . . . . . . . . . . 56
6 TheSpectralMethodforProvingSuperconcentration . . . . . . . 57
1 SpectralDecompositionoftheOUSemigroup . . . . . . . . . . 58
2 AnImprovedPoincaréInequality . . . . . . . . . . . . . . . . . 60
3 SuperconcentrationintheSKModel . . . . . . . . . . . . . . . 60
7 IndependentFlips . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
1 TheIndependentFlipsSemigroup . . . . . . . . . . . . . . . . . 63
2 HypercontractivityforIndependentFlips . . . . . . . . . . . . . 65
3 ChaosUnderIndependentFlips . . . . . . . . . . . . . . . . . . 66
8 ExtremalFields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
1 SuperconcentrationinExtremalFields . . . . . . . . . . . . . . 73
2 ASufficientConditionforExtremality . . . . . . . . . . . . . . 78
3 ApplicationtoSpinGlasses . . . . . . . . . . . . . . . . . . . . 79
4 ApplicationtotheDiscreteGaussianFreeField . . . . . . . . . 83
9 FurtherApplicationsofHypercontractivity . . . . . . . . . . . . . 87
1 SuperconcentrationoftheLargestEigenvalue. . . . . . . . . . . 87
2 ADifferentHypercontractiveTool . . . . . . . . . . . . . . . . 90
3 SuperconcentrationinLowCorrelationFields . . . . . . . . . . 92
4 SuperconcentrationinSubfields . . . . . . . . . . . . . . . . . . 93
5 DiscreteGaussianFreeFieldonaTorus . . . . . . . . . . . . . 94
6 GaussianFieldsonEuclideanSpaces . . . . . . . . . . . . . . . 99
10 TheInterpolationMethodforProvingChaos . . . . . . . . . . . . 105
1 AGeneralTheorem . . . . . . . . . . . . . . . . . . . . . . . . 105
2 ApplicationtotheSherrington-KirkpatrickModel . . . . . . . . 111
3 SharpnessoftheInterpolationMethod . . . . . . . . . . . . . . 112
11 VarianceLowerBounds . . . . . . . . . . . . . . . . . . . . . . . . 115
1 SomeGeneralTools . . . . . . . . . . . . . . . . . . . . . . . . 115
2 ApplicationtotheEdwards-AndersonModel . . . . . . . . . . . 118
3 ChaosintheEdwards-AndersonModel . . . . . . . . . . . . . . 119
12 DimensionsofLevelSets . . . . . . . . . . . . . . . . . . . . . . . . 125
1 LevelSetsofExtremalFields . . . . . . . . . . . . . . . . . . . 125
2 InducedDimension . . . . . . . . . . . . . . . . . . . . . . . . 128
3 DimensionofNear-MaximalSets . . . . . . . . . . . . . . . . . 131
4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
AppendixA GaussianRandomVariables . . . . . . . . . . . . . . . . 137
1 TailBounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
2 SizeoftheMaximum . . . . . . . . . . . . . . . . . . . . . . . 137
3 IntegrationbyParts . . . . . . . . . . . . . . . . . . . . . . . . 139
4 TheGaussianConcentrationInequality . . . . . . . . . . . . . . 139
5 ConcentrationoftheMaximum . . . . . . . . . . . . . . . . . . 140
Contents ix
AppendixB Hypercontractivity. . . . . . . . . . . . . . . . . . . . . . 143
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
AuthorIndex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
SubjectIndex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
Chapter 1
Introduction
ThismonographisaboutacertaincuriousfeatureofrandomobjectsthatIcall‘su-
perconcentration’,andtworelatedtopics,‘chaos’and‘multiplevalleys’.Although
superconcentration has been a recognized feature in a number of areas of proba-
bility theory in the last twenty years (under a variety of names), its connections
with chaos and multiple valleys were discovered and explored for the first time in
Chatterjee(2008b).Thisintroductorychaptersketchesthebasicideasbehindthese
threeconceptsthroughsomeexamples.Precisedefinitionsaregiveninlaterchap-
ters.
1 Superconcentration
The theory of concentration of measure gives probability theory a range of tools
to compute upper bounds on the orders of fluctuations of complicated random
variables. Usually, the general techniques of measure concentration are required
when more direct problem-specific approaches do not work. The essential tech-
niques of this theory are adequately summarized in the classic monograph of
Ledoux (2001) and the recent book by Boucheron et al. (2013). Roughly speak-
ing, superconcentration happens when the classical measure concentration tech-
niques give suboptimal bounds on the order of fluctuations. The techniques avail-
able for proving superconcentration are rather inadequate at this point of time.
They give only small improvements on the upper bounds from classical theory.
Onemaywonderwhatissointerestingaboutsuchminorimprovements.Themain
point of this monograph is to demonstrate how any improvement over a classi-
cal upper bound is equivalent to a number of strange and interesting phenom-
ena, such as chaos under small perturbations and the emergence of multiple val-
leys.
Aformaldefinitionofsuperconcentrationwillbegivenlater.Rightnow,thesit-
uationisbestexplainedthroughexamples.
S.Chatterjee,SuperconcentrationandRelatedTopics, 1
SpringerMonographsinMathematics,DOI10.1007/978-3-319-03886-5_1,
©SpringerInternationalPublishingSwitzerland2014