Fundamentals of Probability: A First Course

465 Pages·2010·3.89 MB·English

Springer Texts in Statistics
SeriesEditors:
G.Casella
S.Fienberg
I.Olkin
Forothertitlespublishedinthisseries,goto
http://www.springer.com/series/417
Anirban DasGupta
Fundamentals of Probability:
A First Course
123
AnirbanDasGupta
PurdueUniversity
Dept.Statistics&Mathematics
150N.UniversityStreet
WestLafayetteIN47907
USA
[email protected]
EditorialBoard
GeorgeCasella StephenFienberg IngramOkin
DepartmentofStatistics DepartmentofStatistics DepartmentofStatistics
UniversityofFlorida CarnegieMellonUniversity StanfordUniversity
Gainesville,FL32611-8545 Pittsburgh,PA15213-3890 Stanford,CA94305
USA USA USA
Mathematica(cid:2)R isaregisteredtrademarkofWolframResearch,Inc.
ISSN1431-875X
ISBN978-1-4419-5779-5 e-ISBN978-1-4419-5780-1
DOI10.1007/978-1-4419-5780-1
SpringerNewYorkDordrechtHeidelbergLondon
LibraryofCongressControlNumber:2010924739
(cid:2)c SpringerScience+BusinessMedia,LLC2010
Allrightsreserved.Thisworkmaynotbetranslatedorcopiedinwholeorinpartwithoutthewritten
permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York,
NY10013, USA),except forbrief excerpts inconnection with reviews orscholarly analysis. Usein
connectionwithanyformofinformationstorageandretrieval,electronicadaptation,computersoftware,
orbysimilarordissimilarmethodologynowknownorhereafterdevelopedisforbidden.
Theuseinthispublicationoftradenames,trademarks,servicemarks,andsimilarterms,eveniftheyare
notidentifiedassuch,isnottobetakenasanexpressionofopinionastowhetherornottheyaresubject
toproprietaryrights.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
To theMemoryof WilliamFeller,whose
booksinspiredmyloveofprobability,andto
DevBasu,thegreatest teacherIhaveknown
Preface
Probability theory is one branch of mathematics that is simultaneously deep and
immediatelyapplicableindiverseareasofhumanendeavor.Itisasfundamentalas
calculus.Calculusexplainstheexternalworld,andprobabilitytheoryhelpspredict
a lot of it. In addition, problems in probability theory have an innate appeal, and
the answers are often structured and strikingly beautiful. A solid background in
probabilitytheoryandprobabilitymodelswillbecomeincreasinglymoreusefulin
the twenty-first century, as difficult new problems emerge, that will require more
sophisticatedmodelsandanalysis.
Thisisatextonthefundamentalsofthetheoryofprobabilityatanundergraduate
orfirst-yeargraduatelevelforstudentsinscience,engineering,andeconomics.The
only mathematicalbackgroundrequiredis knowledgeof univariate and multivari-
ate calculus and basic linear algebra. The book covers all of the standard topics
in basic probability, such as combinatorial probability, discrete and continuous
distributions, moment generating functions, fundamental probability inequalities,
the central limit theorem, and joint and conditional distributions of discrete and
continuousrandomvariables.But italso hassome uniquefeaturesand a forward-
looking feel. Some unique features of this book are its emphasis on conceptual
discussions,alivelywritingstyle,andonpresentingalargevarietyofunusualand
interestingexamples;carefulandmoredetailedtreatmentofnormalandPoissonap-
proximations(Chapters6and10);betterexposuretodistributiontheory,including
developing superior skills in working with joint and conditional distributions and
thebivariatenormaldistribution(Chapters11,12,and13);acompleteandreadable
accountoffiniteMarkovchains(Chapter14);treatmentofmodernurnmodelsand
statisticalgenetics(Chapter15);specialeffortstomakethebookuser-friendly,with
unusuallydetailedchaptersummaries,andaunifiedcollectionofformulasfromthe
text,andfromalgebra,trigonometry,geometry,andcalculusintheappendixofthe
book,forimmediateandeasyreference;anduseofinterestingUseYourComputer
simulationprojectsaspartofthechapterexercisestohelpstudentsseeatheoretical
resultevolveintheirowncomputerwork.
Theexercisesetsformaprincipalassetofthistext.Theycontainawidemixof
problemsat differentdegreesof difficulty. While many are straightforward,many
othersarechallengingandrequireastudenttothinkhard.Theseharderproblemsare
alwaysmarkedwith an asterisk. Thechapterendingexercisesthatare notmarked
vii
viii Preface
with an asterisk generally require only straightforward skills, and these are also
essentialforgivingastudentconfidenceinproblemsolving.Thebookalsogivesa
setofsupplementaryexercisesforadditionalhomeworkandexampreparation.The
supplementaryproblemsethas185wordproblemsandaverycarefullydesignedset
of120true/falseproblems.Instructorscanusethetrue/falseproblemstoencourage
students to learn to think and also quite possibly for weekly homework.The total
numberofproblemsinthebookis810.
Studentswhotakeacoursefromthisbookshouldbeextremelywell-preparedto
takemoreadvancedprobabilitycoursesandalsocoursesinstatisticaltheoryatthe
levelofBickelandDoksum(2001),andCasellaandBerger(2001).Thisbookalone
shouldgivemanystudentsa solid workingknowledgeof basic probabilitytheory,
together with some experience with applications. The sections in the text that are
markedwithanasteriskareoptional,andtheyarenotessentialforlearningthemost
basictheoryofprobability.However,thesesectionshavesignificantreferencevalue,
andinstructorsmaychoosetocoversomeofthemattheirdiscretion.
The book can be used for a few different types of one-semester courses; for
example,acoursethatprimarilyteachesunivariateprobability,acoursethatcaters
tostudentswhohavealreadyhadsomeunivariateprobability,oracoursethatdoes
abitofboth.Thebookcanalsobeusedtoteachacoursethatdoessometheoryand
thensomeapplications.Afewsuchsampleone-semestercourseoutlinesusingthis
bookare:
Sample course one: Univariate and some urn models Sections 1.1–1.5; 2.1;
3.1–3.4;4.1–4.9,4.10.1;6.1–6.7;7.1–7.5;7.7.1;8.1–8.4;9.1–9.4;10.1–10.5,10.7;
15.4,15.5
Sample course two: Mostly multivariate, with Markov chains and some
urn models A four week review of univariate probability, followed by Sections
11.1–11.5;12.1–12.6;13.1–13.5;8.6;14.1–14.6;15.1,15.2,15.4–15.6
Sample course three: Univariate, discrete multivariate, some Markov chains,
and genetics Sections 1.1–1.5; 3.1–3.4; 4.1–4.6, 4.8, 4.9, 4.12; 6.3, 6.4, 6.6,
6.7, 6.9; 7.1, 7.3–7.5, 7.6.1; 8.1–8.6; 9.1–9.4; 10.1–10.4; 11.1–11.4; 14.1–14.3;
15.7–15.9.
Acompanionsecondvolumeofthisbookisplannedforlate 2010.Thesecond
volumewillcaterprimarilytograduatestudentsinmathematics,statistics,andma-
chine learning and will coveradvanceddistribution theory,asymptotictheory and
characteristicfunctions,randomwalks,Brownianmotionandtheempiricalprocess,
Poissonprocesses,extremevaluetheoryandconcentrationinequalities,asurveyof
models,includingmartingales,copulas,andexponentialfamilies,andanintroduc-
tiontoMCMC.
Peter Hall, Stewart Ethier, Burgess Davis, B.V. Rao, Wei-Liem Loh, Dimitris
Politis,YosiRinott,SaravandeGeer,JayaramSethuraman,andRabiBhattacharya
made scholarly comments on various drafts of this book. I am thankful to all of
them. I am specifically deeply indebted to Peter Hall for the extraordinarynature
ofhiscounselandsupportandforhisenduringandselflessfriendshipandwarmth.
Preface ix
I simplycouldnothavewritten thisbookwithoutPeter’shelpandmentoring.For
this,andforbeingauniquecounselorandfriendtome,IamgratefultoPeter.
I also want to express my deep appreciation for all the help that I received
fromStewartEthierasIwaswritingthisbook.Stewartwasmostgracious,patient,
thoughtful,andkind.BurgessDavisaffectionatelyreadthroughseveralpartsofthe
book,correctedsomeerrors,andwasatrustedcounselor.Eightanonymousreview-
ersmadesuperbcommentsandhelpedmemakethisabetterbook.Springer’sseries
editors,PeterBickel,GeorgeCasella,SteveFeinberg,andIngramOlkin,helpedme
ineverypossiblewayatalltimes.Iamthankfultothem.
John Kimmel, as always, was a pleasure to work with. John’s professionalism
and his personal qualities make him a really dear person. My production editor
SusanWestendorfgraciouslyhandledeveryproductionrelatedissueanditwasmy
pleasuretoworkwithher.MycopyeditorHalHenglindidanunbelievablycareful
and thoughtful job. Indeed, if it was not for Hal, I could not have put this book
outinareadableform.ThetechnicalstaffatSPiTechnologies,Pondicherry,India
did a terrific and timely job of resetting the bookin Springer’stextbooktemplate.
Doug and Cheryl Crabill helped me with my computer questions and solved my
problems with mysterious and magical powers. Shanti Gupta brought me to the
UnitedStatesandcaredformeandwasaguardianandamentorformorethan15
years. I miss Shanti very much. Larry Brown, Persi Diaconis, Jon Wellner, Steve
Lalley, Jim Pitman, C.R. Rao, and Jim Berger have given me supportand sincere
encouragementformanyofmyefforts.Iappreciateallofthem.
Human life is unreasonably fragile. It is important that our fondness for our
friendsnotremainunspoken.I amthankfulto numerouspersonalfriendsfortheir
affection, warmth, and company over the years. It is not possible to name all of
them. But I am especially grateful and fortunate for the magnificent and endear-
ingsupport,camaraderie,andconcernof someof mybestfriends,JeniferBrown,
Len Haff, Peter Hall, Rajeeva Karandikar,T. Krishnan, Wei-Liem Loh, B.V. Rao,
HermanRubin,BillStrawderman,LarryWasserman,andDr.JulieMarshburn,MD.
TheyhavegivenmemuchmorethanIhavecaredtogiveinreturn.Iappreciatethem
andtheirfriendshipmorethanIcanexpress.
I had my core training in probability at the fundamental level in Dev Basu’s
classesattheISI.InevermetanotherteacherlikeBasu. Iwassimplyfortunateto
havehimasmyteacherandtohaveknownhimfortherarehumanbeingthathewas.
BasutoldusthatwemustreadFeller.Icontinuetobelievethatthetwovolumesof
Felleraretwo all-timeclassics, andit’shardnottogetinspiredaboutthestudyof
randomnessonceonehasreadFeller.IdedicatethisbooktoWilliamFellerandDev
Basuforbringingmethejoyofprobabilitytheory.
But most of all, I am in love with my family for their own endless love for as
longasIhavelived.Ihopetheylikethisbook.
WestLafayette,Indiana AnirbanDasGupta
Contents
Preface............................................................................... vii
1 IntroducingProbability...................................................... 1
1.1 ExperimentsandSampleSpaces..................................... 2
1.2 SetTheoryNotationandAxiomsofProbability.................... 3
1.3 HowtoInterpretaProbability........................................ 5
1.4 CalculatingProbabilities ............................................. 7
1.4.1 ManualCounting............................................ 8
1.4.2 GeneralCountingMethods................................. 10
1.5 Inclusion-ExclusionFormula......................................... 12
1.6 (cid:2)BoundsontheProbabilityofaUnion............................. 15
1.7 Synopsis............................................................... 16
1.8 Exercises .............................................................. 16
References...................................................................... 21
2 TheBirthdayandMatchingProblems..................................... 23
2.1 TheBirthdayProblem................................................ 23
2.1.1 *Stirling’sApproximation ................................. 24
2.2 TheMatchingProblem ............................................... 25
2.3 Synopsis............................................................... 26
2.4 Exercises .............................................................. 27
References...................................................................... 27
3 ConditionalProbabilityandIndependence................................ 29
3.1 BasicFormulasandFirstExamples.................................. 29
3.2 MoreAdvancedExamples ........................................... 31
3.3 IndependentEvents................................................... 33
3.4 Bayes’Theorem....................................................... 36
3.5 Synopsis............................................................... 39
3.6 Exercises .............................................................. 39
xi
xii Contents
4 Integer-ValuedandDiscreteRandomVariables.......................... 45
4.1 MassFunction......................................................... 45
4.2 CDFandMedianofaRandomVariable............................. 47
4.2.1 FunctionsofaRandomVariable ........................... 53
4.2.2 IndependenceofRandomVariables........................ 55
4.3 ExpectedValueofaDiscreteRandomVariable..................... 56
4.4 BasicPropertiesofExpectations..................................... 57
4.5 IllustrativeExamples ................................................. 59
4.6 UsingIndicatorVariablestoCalculateExpectations................ 60
4.7 TheTailSumMethodforCalculatingExpectations ................ 62
4.8 Variance,Moments,andBasicInequalities.......................... 63
4.9 IllustrativeExamples ................................................. 65
4.9.1 VarianceofaSumofIndependentRandomVariables .... 67
4.10 Utilityof(cid:2)and(cid:3) asSummaries..................................... 67
4.10.1 Chebyshev’sInequalityandtheWeakLaw
ofLargeNumbers........................................... 68
4.10.2 *BetterInequalities......................................... 70
4.11 (cid:2)OtherFundamentalMomentInequalities.......................... 71
4.11.1 *ApplyingMomentInequalities........................... 73
4.12 TruncatedDistributions............................................... 74
4.13 Synopsis............................................................... 75
4.14 Exercises .............................................................. 76
References...................................................................... 80
5 GeneratingFunctions........................................................ 81
5.1 GeneratingFunctions................................................. 81
5.2 MomentGeneratingFunctionsandCumulants...................... 85
5.2.1 (cid:2)Cumulants................................................. 87
5.3 Synopsis............................................................... 89
5.4 Exercises .............................................................. 89
References...................................................................... 90
6 StandardDiscreteDistributions ............................................ 91
6.1 IntroductiontoSpecialDistributions ................................ 91
6.2 DiscreteUniformDistribution ....................................... 94
6.3 BinomialDistribution................................................. 95
6.4 GeometricandNegativeBinomialDistributions.................... 99
6.5 HypergeometricDistribution.........................................102
6.6 PoissonDistribution ..................................................104
6.6.1 MeanAbsoluteDeviationandtheMode...................108
6.7 PoissonApproximationtoBinomial.................................109
6.8 (cid:2)MiscellaneousPoissonApproximations...........................112
6.9 Benford’sLaw ........................................................114
6.10 DistributionofSumsandDifferences................................115
6.10.1 (cid:2)DistributionofDifferences...............................117