Probability for Statistics and Machine Learning: Fundamentals and Advanced Topics

803 Pages·2011·3.93 MB·English

Springer Texts in Statistics
SeriesEditors:
G.Casella
S.Fienberg
I.Olkin
Forfurthervolumes:
http://www.springer.com/series/417

Anirban DasGupta
Probability for Statistics
and Machine Learning
Fundamentals and Advanced Topics
ABC
AnirbanDasGupta
DepartmentofStatistics
PurdueUniversity
150N.UniversityStreet
WestLafayette,IN47907,USA
[email protected]
Mathematica(cid:2)R isaregisteredtrademarkofWolframResearch,Inc.
ISBN978-1-4419-9633-6 e-ISBN978-1-4419-9634-3
DOI10.1007/978-1-4419-9634-3
SpringerNewYorkDordrechtHeidelbergLondon
LibraryofCongressControlNumber:2011924777
(cid:2)c SpringerScience+BusinessMedia,LLC2011
Allrightsreserved.Thisworkmaynotbetranslatedorcopiedinwholeorinpartwithoutthewritten
permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York,
NY10013, USA),except forbrief excerpts inconnection with reviews orscholarly analysis. Usein
connectionwithanyformofinformationstorageandretrieval,electronicadaptation,computersoftware,
orbysimilarordissimilarmethodologynowknownorhereafterdevelopedisforbidden.
Theuseinthispublicationoftradenames,trademarks,servicemarks,andsimilarterms,eveniftheyare
notidentifiedassuch,isnottobetakenasanexpressionofopinionastowhetherornottheyaresubject
toproprietaryrights.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
To PersiDiaconis,PeterHall,AshokMaitra,
andmymother,with affection

Preface
This is the companionsecond volume to my undergraduatetext Fundamentalsof
Probability:AFirstCourse.Thepurposeofmywritingthisbookistogivegradu-
ate students, instructors, and researchers in statistics, mathematics, and computer
science a lucidly written unique text at the confluence of probability, advanced
stochasticprocesses,statistics, andkeytoolsformachinelearning.Numeroustop-
ics in probability and stochastic processes of current importance in statistics and
machine learning that are widely scattered in the literature in many differentspe-
cialized books are all brought together under one fold in this book. This is done
withanextensivebibliographyforeachtopic,andnumerousworked-outexamples
andexercises.Probability,withallitsmodels,techniques,anditspoignantbeauty,
isanincrediblypowerfultoolforanyonewhodealswithdataorrandomness.The
content and the style of this book reflect that philosophy;I emphasize lucidity, a
widebackground,andthefar-reachingapplicabilityofprobabilityinscience.
Thebookstartswithaself-containedandfairlycompletereviewofbasicprob-
ability,andthentraversesitsway throughtheclassics, to advancedmoderntopics
and tools, including a substantial amountof statistics itself. Because of its nearly
encyclopaedic coverage, it can serve as a graduate text for a year-long probabil-
itysequence,orforfocusedshortcoursesonselectedtopics,forself-study,andas
a nearly unique reference for research in statistics, probability, and computer sci-
ence.Itprovidesanextensivetreatmentofmostofthestandardtopicsinagraduate
probabilitysequence,andintegratesthemwiththebasictheoryandmanyexamples
of several core statistical topics, as well as with some tools of major importance
in machine learning. This is done with unusually detailed bibliographies for the
readerwhowantstodigdeeperintoaparticulartopic,andwithahugerepertoireof
worked-outexamplesand exercises. The total numberof worked-outexamplesin
thisbookis423,andthetotalnumberofexercisesis808.Aninstructorcanrotate
theexercisesbetweensemesters,andusethemforsettingexams,andastudentcan
use themforadditionalexampreparationandself-study.I believethatthe bookis
uniqueinitsrange,unification,bibliographicdetail,anditscollectionofproblems
andexamples.
Topics in core probability, such as distribution theory, asymptotics, Markov
chains, martingales, Poisson processes, random walks, and Brownian motion are
covered in the first 14 chapters. In these chapters, a reader will also find basic
vii
viii Preface
coverageofsuchcorestatisticaltopicsasconfidenceintervals,likelihoodfunctions,
maximumlikelihoodestimates, posteriordensities, sufficiency,hypothesistesting,
variance stabilizing transformations,and extreme value theory,all illustrated with
manyexamples.InChapters15,16,and17,Itreatthreemajortopicsofgreatappli-
cationpotential,empiricalprocessesandVCtheory,probabilitymetrics,andlarge
deviations. Chapters 18, 19, and 20 are specifically directed to the statistics and
machine-learning community, and cover simulation, Markov chain Monte Carlo,
theexponentialfamily,bootstrap,theEMalgorithm,andkernels.
Thebookdoesnotmakeformaluseofmeasuretheory.Idonotintendtomini-
mizetheroleofmeasuretheoryinarigorousstudyofprobability.However,Ibelieve
thata largeamountofprobabilitycanbetaught,understood,enjoyed,andapplied
without needing formal use of measure theory. We do it around the world every
day. At the same time, some theorems cannot be proved without at least a men-
tionofsomemeasuretheoryterminology.Evensomedefinitionsrequireamention
ofsomemeasuretheorynotions.Iincludesomeunavoidablementionofmeasure-
theoretictermsandresults,suchasthestronglawoflargenumbersanditsproof,the
dominatedconvergencetheorem,monotoneconvergence,Lebesguemeasure,anda
fewothers,butonlyintheadvancedchaptersinthebook.
Following the table of contents, I have suggested some possible courses with
differentthemesusingthisbook.Ihavealsomarkedthenonroutineandharderex-
ercises in each chapter with an asterisk. Likewise, some specialized sections with
reference value have also been marked with an asterisk. Generally, the exercises
andtheexamplescomewithacaption,sothatthereaderwillimmediatelyknowthe
contentof an exerciseoran example.Theend of the proofof a theoremhasbeen
markedbya(cid:2)sign.
MydeepestgratitudeandappreciationareduetoPeterHall.Iamluckythatthe
styleandsubstanceofthisbookaresignificantlymoldedbyPeter’sinfluence.Outof
habit,IsenthimthedraftsofnearlyeverychapterasIwasfinishingthem.Itdidn’t
matter where exactly he was, I always received his input and gentle suggestions
for improvement.I have found Peter to be a concerned and warm friend, teacher,
mentor,andguardian,andforthis,Iamextremelygrateful.
Mouli Banerjee, Rabi Bhattacharya, Burgess Davis, Stewart Ethier, Arthur
Frazho, Evarist Gine´, T. Krishnan, S. N. Lahiri, Wei-Liem Loh, Hyun-Sook Oh,
B. V. Rao, Yosi Rinott, Wen-Chi Tsai, Frederi Viens, and Larry Wasserman
graciously went over various parts of this book. I am deeply indebted to each
of them.LarryWasserman, in particular,suggestedthe chaptersonempiricalpro-
cesses, VC theory,concentrationinequalities,the exponentialfamily, and Markov
chainMonteCarlo.TheSpringerserieseditors,PeterBickel,GeorgeCasella,Steve
Fienberg,andIngramOlkinhaveconsistentlysupportedmyefforts,andIamsovery
thankfulto them. Springer’s incoming executiveeditor Marc Strauss saw through
the final production of this book extremely efficiently, and I have much enjoyed
working with him. I appreciated Marc’s gentility and his thoroughly professional
handling of the transition of the production of this book to his oversight. Valerie
Greco did an astonishing job of copyediting the book. The presentation, display,
andthegrammarofthebookaresubstantiallybetterbecauseoftheincrediblecare
Preface ix
and thoughtfulness that she put into correcting my numerous errors. The staff at
SPiTechnologies,Chennai,Indiadidanastoundingandmarvelousjobofproduc-
ingthisbook.Six anonymousreviewersgaveextremelygraciousandconstructive
comments, and their input has helped me in various dimensions to make this a
betterbook.DougCrabillisthegreatestcomputersystemsadministrator,andwith
an infectious pleasantness has bailed me out of my stupidity far too many times.
I also wantto mentionmy fondmemoriesand deep-rootedfeelingsforthe Indian
StatisticalInstitute,whereIhadallofmycollegeeducation.Itwasjustawonderful
placeforresearch,education,andfriendships.NearlyeverythingthatIknowisdue
tomyyearsattheIndianStatisticalInstitute,andforthisIamthankful.
Thisisthe third time thatI havewritten a bookin contractwith JohnKimmel.
John is much more than a nearly unique person in the publishing world. To me,
Johnepitomizessensitivityandprofessionalism,asingularcombination.Ihavenow
knownJohnforalmostsix years,anditisveryverydifficultnottoappreciateand
admirehimawholelotforhiswarmth,style,andpassionforthesubjectsofstatis-
ticsandprobability.Ironically,thedaythatthisbookenteredproduction,thenews
came that John was leaving Springer. I will remember John’s contribution to my
professionalgrowthwithenormousrespectandappreciation.
WestLafayette,Indiana AnirbanDasGupta