High-Dimensional Statistics A Non-Asymptotic Viewpoint

571 Pages·2019·5.33 MB·english

by  Martin J. Wainwright

“Non-asymptotic, high-dimensional theory is critical for modern statistics and machine
learning.Thisbookisuniqueinprovidingacrystalclear,complete,andunifiedtreatmentof
thearea.Withtopicsrangingfromconcentrationofmeasuretographicalmodels,theauthor
weavestogetherprobabilitytheoryanditsapplicationstostatistics.Idealforgraduatestudents
andresearchers.Thiswillsurelybethestandardreferenceonthetopicformanyyears.”
—LarryWasserman,CarnegieMellonUniversity
“MartinWainwrightbringshislargeboxofanalyticalpowertoolstobearontheproblemsof
theday—theanalysisofmodelsforwidedata.Abroadknowledgeofthisnewareacombines
withhispowerfulanalyticalskillstodeliverthisimpressiveandintimidatingwork—boundto
beanessentialreferenceforallthebravesoulsthattrytheirhand.”
—TrevorHastie,StanfordUniversity
“Thisbookprovidesanexcellenttreatmentofperhapsthefastestgrowingareawithinhigh-
dimensionaltheoreticalstatistics—non-asymptotictheorythatseekstoprovideprobabilistic
boundsonestimatorsasafunctionofsamplesizeanddimension.Itoffersthemostthorough,
clear,andengagingcoverageofthisareatodate,andisthuspoisedtobecomethedefinitive
referenceandtextbookonthistopic.”
—GeneveraAllen,RiceUniversity
“Statistical theory and practice have undergone a renaissance in the past two decades,
with intensive study of high-dimensional data analysis. No researcher has deepened our
understandingofhigh-dimensionalstatisticsmorethanMartinWainwright.Thisbookbrings
the signature clarity and incisiveness of his published research into book form. It will be a
fantasticresourceforbothbeginningstudentsandseasonedresearchers,asthefieldcontinues
tomakeexcitingbreakthroughs.”
—JohnLafferty,YaleUniversity
“This is an outstanding book on high-dimensional statistics, written by a creative and
celebratedresearcherinthefield.Itgivescomprehensivetreatmentsofmanyimportanttopics
instatisticalmachinelearningand,furthermore,isself-contained,fromintroductorymaterial
tothemostup-to-dateresultsonvariousresearchfrontiers.Thisbookisamust-readforthose
whowishtolearnandtodevelopmodernstatisticalmachinetheory,methodsandalgorithms.”
—JianqingFan,PrincetonUniversity
“This book provides an in-depth mathematical treatment and methodological intuition for
high-dimensional statistics. The main technical tools from probability theory are carefully
developedandtheconstructionandanalysisofstatisticalmethodsandalgorithmsforhigh-
dimensional problems are presented in an outstandingly clear way. Martin Wainwright has
writtenatrulyexceptional,inspiring,andbeautifulmasterpiece!”
—PeterBu¨hlmann,ETHZurich
“ThisnewbookbyMartinWainwrightcoversmoderntopicsinhigh-dimensionalstatistical
inference,andfocusesprimarilyonexplicitnon-asymptoticresultsrelatedtosparsityandnon-
parametricestimation.Thisisamust-readforallgraduatestudentsinmathematicalstatistics
and theoretical machine learning, both for the breadth of recent advances it covers and the
depthofresultswhicharepresented.Theexpositionisoutstandinglyclear,startingfromthe
firstintroductorychaptersonthenecessaryprobabilistictools.Then,thebookcoversstate-of-
the-artadvancesinhigh-dimensionalstatistics,withalwaysacleverchoiceofresultswhich
havetheperfectmixofsignificanceandmathematicaldepth.”
—FrancisBach,INRIAParis
“Wainwright’sbookonthosepartsofprobabilitytheoryandmathematicalstatisticscritical
to understanding of the new phenomena encountered in high dimensions is marked by the
clarity of its presentation and the depth to which it travels. In every chapter he starts with
intuitiveexamplesandsimulationswhicharesystematicallydevelopedeitherintopowerful
mathematicaltoolsorcompleteanswerstofundamentalquestionsofinference.Itisnoteasy,
butelegantandrewardingwhetherreadsystematicallyordippedintoasareference.”
—PeterBickel,UCBerkeley
High-Dimensional Statistics
Recentyearshavewitnessedanexplosioninthevolumeandvarietyofdatacollected
in all scientific disciplines and industrial settings. Such massive data sets present a
number of challenges to researchers in statistics and machine learning. This book
provides a self-contained introduction to the area of high-dimensional statistics,
aimed at the first-year graduate level. It includes chapters that are focused on core
methodologyandtheory—includingtailbounds,concentrationinequalities,uniform
laws and empirical process, and random matrices—as well as chapters devoted to
in-depth exploration of particular model classes—including sparse linear models,
matrix models with rank constraints, graphical models, and various types of non-
parametricmodels.
With hundreds of worked examples and exercises, this text is intended both for
coursesandforself-studybygraduatestudentsandresearchersinstatistics,machine
learning,andrelatedfieldswhomustunderstand,apply,andadaptmodernstatistical
methodssuitedtolarge-scaledata.
MARTIN J. WAINWRIGHT is a Chancellor’s Professor at the University of
California,Berkeley,withajointappointmentbetweentheDepartmentofStatistics
andtheDepartmentofElectricalEngineeringandComputerSciences.Hisresearch
liesatthenexusofstatistics,machinelearning,optimization,andinformationtheory,
and he has published widely in all of these disciplines. He has written two other
books,oneongraphicalmodelstogetherwithMichaelI.Jordan,andoneonsparse
learning together with Trevor Hastie and Robert Tibshirani. Among other awards,
he has received the COPSS Presidents’ Award, has been a Medallion Lecturer and
BlackwellLecturerfortheInstituteofMathematicalStatistics,andhasreceivedBest
PaperAwardsfromtheNIPS,ICML,andUAIconferences,aswellasfromtheIEEE
InformationTheorySociety.
CAMBRIDGE SERIES IN STATISTICAL AND
PROBABILISTIC MATHEMATICS
EditorialBoard
Z.Ghahramani(DepartmentofEngineering,UniversityofCambridge)
R.Gill(MathematicalInstitute,LeidenUniversity)
F.P.Kelly(DepartmentofPureMathematicsandMathematicalStatistics,
UniversityofCambridge)
B.D.Ripley(DepartmentofStatistics,UniversityofOxford)
S.Ross(DepartmentofIndustrialandSystemsEngineering,
UniversityofSouthernCalifornia)
M.Stein(DepartmentofStatistics,UniversityofChicago)
This series of high-quality upper-division textbooks and expository monographs covers all
aspectsofstochasticapplicablemathematics.Thetopicsrangefrompureandappliedstatistics
toprobabilitytheory,operationsresearch,optimization,andmathematicalprogramming.The
books contain clear presentations of new developments in the field and also of the state of
theartinclassicalmethods.Whileemphasizingrigoroustreatmentoftheoreticalmethods,the
booksalsocontainapplicationsanddiscussionsofnewtechniquesmadepossiblebyadvances
incomputationalpractice.
Acompletelistofbooksintheseriescanbefoundatwww.cambridge.org/statistics.
Recenttitlesincludethefollowing:
23. AppliedAsymptotics,byA.R.Brazzale,A.C.DavisonandN.Reid
24. RandomNetworksforCommunication,byMassimoFranceschettiandRonaldMeester
25. DesignofComparativeExperiments,byR.A.Bailey
26. SymmetryStudies,byMarlosA.G.Viana
27. ModelSelectionandModelAveraging,byGerdaClaeskensandNilsLidHjort
28. BayesianNonparametrics,editedbyNilsLidHjortetal.
29. FromFiniteSampletoAsymptoticMethodsinStatistics,byPranabK.Sen,JulioM.Singerand
AntonioC.PedrosadeLima
30. BrownianMotion,byPeterMo¨rtersandYuvalPeres
31. Probability:TheoryandExamples(FourthEdition),byRickDurrett
33. StochasticProcesses,byRichardF.Bass
34. RegressionforCategoricalData,byGerhardTutz
35. ExercisesinProbability(SecondEdition),byLo¨ıcChaumontandMarcYor
36. StatisticalPrinciplesfortheDesignofExperiments,byR.Mead,S.G.GilmourandA.Mead
37. QuantumStochastics,byMou-HsiungChang
38. NonparametricEstimationunderShapeConstraints,byPietGroeneboomandGeurtJongbloed
39. LargeSampleCovarianceMatricesandHigh-DimensionalDataAnalysis,byJianfengYao,Shurong
ZhengandZhidongBai
40. MathematicalFoundationsofInfinite-DimensionalStatisticalModels,byEvaristGine´andRichard
Nickl
41. Confidence,Likelihood,Probability,byToreSchwederandNilsLidHjort
42. ProbabilityonTreesandNetworks,byRussellLyonsandYuvalPeres
43. RandomGraphsandComplexNetworks(Volume1),byRemcovanderHofstad
44. FundamentalsofNonparametricBayesianInference,bySubhashisGhosalandAadvanderVaart
45. Long-RangeDependenceandSelf-Similarity,byVladasPipirasandMuradS.Taqqu
46. PredictiveStatistics,byBertrandS.ClarkeandJenniferL.Clarke
47. High-DimensionalProbability,byRomanVershynin
48. High-DimensionalStatistics,byMartinJ.Wainwright
49. Probability:TheoryandExamples(FifthEdition),byRickDurrett

High-Dimensional Statistics
A Non-Asymptotic Viewpoint
Martin J. Wainwright
UniversityofCalifornia,Berkeley
UniversityPrintingHouse,CambridgeCB28BS,UnitedKingdom
OneLibertyPlaza,20thFloor,NewYork,NY10006,USA
477WilliamstownRoad,PortMelbourne,VIC3207,Australia
314–321,3rdFloor,Plot3,SplendorForum,JasolaDistrictCentre,NewDelhi–110025,India
79AnsonRoad,#06–04/06,Singapore079906
CambridgeUniversityPressispartoftheUniversityofCambridge.
ItfurtherstheUniversity’smissionbydisseminatingknowledgeinthepursuitof
education,learning,andresearchatthehighestinternationallevelsofexcellence.
www.cambridge.org
Informationonthistitle:www.cambridge.org/9781108498029
DOI:10.1017/9781108627771
©MartinJ.Wainwright2019
Thispublicationisincopyright.Subjecttostatutoryexception
andtotheprovisionsofrelevantcollectivelicensingagreements,
noreproductionofanypartmaytakeplacewithoutthewritten
permissionofCambridgeUniversityPress.
Firstpublished2019
PrintedintheUnitedKingdombyTJInternationalLtd.PadstowCornwall
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary.
LibraryofCongressCataloging-in-PublicationData
Names:Wainwright,Martin(MartinJ.),author.
Title:High-dimensionalstatistics:anon-asymptoticviewpoint/MartinJ.
Wainwright(UniversityofCalifornia,Berkeley).
Description:Cambridge;NewYork,NY:CambridgeUniversityPress,2019.|
Series:Cambridgeseriesinstatisticalandprobabilisticmathematics;48|
Includesbibliographicalreferencesandindexes.
Identifiers:LCCN2018043475|ISBN9781108498029(hardback)
Subjects:LCSH:Mathematicalstatistics–Textbooks.|Bigdata.
Classification:LCCQA276.18.W352019|DDC519.5–dc23
LCrecordavailableathttps://lccn.loc.gov/2018043475
ISBN978-1-108-49802-9Hardback
CambridgeUniversityPresshasnoresponsibilityforthepersistenceoraccuracy
ofURLsforexternalorthird-partyinternetwebsitesreferredtointhispublication
anddoesnotguaranteethatanycontentonsuchwebsitesis,orwillremain,
accurateorappropriate.
List of chapters
Listofillustrations xv
1 Introduction 1
2 Basictailandconcentrationbounds 21
3 Concentrationofmeasure 58
4 Uniformlawsoflargenumbers 98
5 Metricentropyanditsuses 121
6 Randommatricesandcovarianceestimation 159
7 Sparselinearmodelsinhighdimensions 194
8 Principalcomponentanalysisinhighdimensions 236
9 Decomposabilityandrestrictedstrongconvexity 259
10 Matrixestimationwithrankconstraints 312
11 Graphicalmodelsforhigh-dimensionaldata 347
12 ReproducingkernelHilbertspaces 383
13 Nonparametricleastsquares 416
14 Localizationanduniformlaws 453
15 Minimaxlowerbounds 485
References 524
Subjectindex 540
Authorindex 548
vii