Probability Theory. A first Course in Probability Theory and Statistics

404 Pages·2016·3.461 MB·english

by  Werner Linde

Werner Linde
Probability Theory
A First Course in Probability Theory and Statistics
MathematicsSubjectClassification2010
Primary:60-01,62-01;Secondary:60A05
Author
Prof.Dr.WernerLinde
Friedrich-Schiller-UniversitätJena
FakultätfürMathematik&Informatik
InstitutfürStochastik
Prof.fürStochastischeAnalysis
D-07737Jena
[e-mail]
and
UniversityofDelaware
DepartmentofMathematicalSciences
501EwingHall
NewarkDE,19716
[e-mail]
ISBN978-3-11-046617-1
e-ISBN(PDF)978-3-11-046619-5
e-ISBN(EPUB)978-3-11-046625-6
©2016WalterdeGruyterGmbH,Berlin/Boston
Typesetting:IntegraSoftwareServicesPvt.Ltd.
Printingandbinding:CPIbooksGmbH,Leck
Coverimage:WernerLinde
(cid:2)8Printedonacid-freepaper
PrintedinGermany
www.degruyter.com
Preface
Thisbookisintendedasanintroductorycourseforstudentsinmathematics,phys-
ical sciences, engineering, or in other related fields. It is based on the experience
ofprobabilitylecturestaughtduringthepast25years,wherethespectrumreached
fromtwo-hourintroductorycourses,overMeasureTheoryandadvancedprobability
classes,tosuchtopicsasStochasticProcessesandMathematicalStatistics.Until2012
theselecturesweredeliveredtostudentsattheUniversityofJena(Germany),andsince
2013tothoseattheUniversityofDelawareinNewark(USA).
ThebookisthecompletelyrevisedversionoftheGermanedition“Stochastikfür
dasLehramt,”whichappearedin2014atDeGruyter.AtmostuniversitiesinGermany,
there exist special classes in Probability Theory for students who want to become
teachersofmathematicsinhighschools.BesidesbasicfactsaboutProbabilityTheory,
thesecoursesarealsosupposedtogiveanintroductionintoMathematicalStatistics.
Thus,theoriginalmainintentionfortheGermanversionwastowriteabookthathelps
thosestudentsunderstandProbabilityTheorybetter.Butsoonthebookturnedoutto
alsobeusefulasintroductionforstudentsinotherfields,e.g.inmathematics,phys-
ics,andsoon.Thuswedecided,inordertomakethebookapplicableforabroader
audience,toprovideatranslationintheEnglishlanguage.
DuringnumerousyearsofteachingIlearnedthefollowing:
– Probabilisticquestionsareusuallyeasytoformulate,generallyhaveatightrela-
tiontoeverydayproblems,andthereforeattracttheinterestoftheaudience.Every
studentknowsthephenomenathatoccurwhenonerollsadie,playscards,tosses
acoin,orplaysalottery.Thus,aninitialinterestinProbabilityTheoryexists.
– In contrast, after a short time many students have very serious difficulties with
understanding the presented topics. Consequently, a common opinion among
students is that Probability Theory is a very complicated topic, causing a lot of
problemsandtroubles.
SurelythereexistseveralreasonsforthebadimageofProbabilityTheoryamongstu-
dents.But,aswebelieve,themostimportantoneisasfollows.InProbabilityTheory,
thetypeofproblemsandquestionsconsidered,aswellasthewayofthinking,differs
considerablyfromtheproblems,questions,andthinkinginotherfieldsofmathem-
atics,i.e.,fromfieldswithwhichthestudentsbecameacquaintedbeforeattendinga
probabilitycourse.Forexample,inCalculusafunctionhasawell-describeddomain
ofdefinition;mostlyitisdefinedbyaconcreteformula,hascertainpropertiesascon-
tinuity,differentiability,andsoon.Afunctionissomethingveryconcretewhichcan
bemadevividbydrawingitsgraph.Incontrast,inProbabilityTheoryfunctionsare
mostlyinvestigatedasrandomvariables.Theyaredefinedonacompletelyunimport-
ant,nonspecifiedsamplespace,andtheygenerallydonotpossessaconcreteformula
fortheirdefinition.Itmayevenhappenthatonlytheexistenceofafunction(random
variable) is known. The only property of a random variable which really matters is
VIII Preface
thedistributionofitsvalues.Thisandmanyothersimilartechniquesmakethewhole
theorysomethingmysteriousandnotcompletelycomprehensible.
Consideringthisobservation,weorganizedthebookinawaythattriestomake
probabilistic problems more understandable and that puts the focus more onto ex-
planationsofthedefinitions,notations,andresults.Thetoolsweusetodothisare
examples;wepresentatleastonebeforeanewdefinition,inordertomotivateit,fol-
lowedbymoreexamplesafterthedefinitiontomakeitcomprehensible.Hereweact
uponthemaximexpressedbyEinstein’squote1:
Exampleisn’tanotherwaytoteach,itistheonlywaytoteach.
PresentingthebasicresultsandmethodsinProbabilityTheorywithoutusingresults,
facts,andnotationsfromMeasureTheoryis,inouropinion,asdifficultastosquare
the circle. Either one restricts oneself to discrete probability measures and random
variables or one has to be unprecise. There is no other choice! In some places, it is
possibletoavoidtheuseofmeasuretheoreticfacts,suchastheLebesgueintegral,or
theexistenceofinfiniteproductmeasures,andsoon,butthepriceishigh.2Ofcourse,
IalsostruggledwiththeproblemofmissingfactsfromMeasureTheorywhilewriting
this book. Therefore, I tried to include some ideas and some results about 3-fields,
measures, and integrals, hoping that a few readers become interested and want to
learnmoreaboutMeasureTheory.Forthose,werefertothebooks[Coh13],[Dud02],
or[Bil12]asgoodsources.
Inthiscontext,letusmakesomeremarkabouttheverificationofthepresented
results. Whenever it was possible, we tried to prove the stated results. Times have
changed;whenIwasastudent,everytheorempresentedinamathematicallecture
was proved – really every one. Facts and results without proof were doubtful and
soon forgotten. And a tricky and elegant proof is sometimes more impressive than
theprovenresult(atleasttous).Hopefully,somereaderswilllikesomeoftheproofs
inthisbookasmuchaswedid.
One of most used applications of Probability Theory is Mathematical Statistics.
WhenImetformerstudentsofmine,Ioftenaskedthemwhichkindofmathematics
they are mainly using now in their daily work. The overwhelming majority of them
answeredthatoneoftheirmainfieldsofmathematicalworkisstatisticalproblems.
Therefore,wedecidedtoincludeanintroductorychapteraboutMathematicalStatist-
ics. Nowadays, due to the existence of good and fast statistical programs, it is very
easy to analyze data, to evaluate confidence regions, or to test a given hypotheses.
Butdothosewhousetheseprogramsalsoalwaysknowwhattheyaredoing?Since
1 Seehttp://www.alberteinsteinsite.com/quotes/einsteinquotes.html
2 Forexample,severalyearsago,toavoidtheuseoftheLebesgueintegral,Iintroducedtheexpected
valueofarandomvariableasaRiemannintegralviaitsdistributionfunction.Thisismathematically
correct,butattheendalmostnostudentsunderstoodwhattheexpectedvaluereallyis.Trytoprove
thattheexpectedvalueislinearusingthisapproach!
Preface IX
wedoubtthatthisisso,westressedthefocusinthischaptertothequestionofwhy
themainstatisticalmethodsworkandonwhatmathematicalbackgroundtheyrest.
Wealsoinvestigatehowprecisestatisticaldecisionsareandwhatkindsoferrorsmay
occur.
The organization of this book differs a little bit from those in many other first-
course books about Probability Theory. Having Measure Theory in the back of our
mindscausesustothinkthatprobabilitymeasuresarethemostimportantingredi-
entofProbabilityTheory;randomvariablescomeinsecond.Onthecontrary,many
otherauthorsgoexactlytheotherway.Theystartwithrandomvariables,andprob-
ability measures then occur as their distribution on their range spaces (mostly R).
Inthiscase,astandardnormalprobabilitymeasuredoesnotexist,onlyastandard
normaldistributedrandomvariable.Bothapproacheshavetheiradvantagesanddis-
advantages,butaswesaid,forustheprobabilitymeasuresareinterestingintheirown
right,andthereforewestartwiththeminChapter1,followedbyrandomvariablesin
Section3.
Thebookalsocontainssomefactsandresultsthataremoreadvancedandusually
notpartofanintroductorycourseinProbabilityTheory.Suchtopicsare,forexample,
theinvestigationofproductmeasures,orderstatistics,andsoon.Wehaveassigned
those more involved sections with a star. They may be skipped at a first reading
withoutlossinthefollowingchapters.
Attheendofeachchapter,onefindsacollectionofsomeproblemsrelatedtothe
contentsofthesection.Herewerestrictedourselvestoafewproblemsintheactual
task;thesolutionsoftheseproblemsarehelpfultotheunderstandingofthepresented
topics.Theproblemsaremainlytakenfromourcollectionofhomeworksandexams
during the past years. For those who want to work with more problems we refer to
manybooks,ase.g.[GS01a],[Gha05],[Pao06],or[Ros14],whichcontainahugecollec-
tionofprobabilisticproblems,rangingfromeasytodifficult,fromnaturaltoartificial,
frominterestingtoboring.
FinallyIwanttoexpressmythankstothosewhosupportedmyworkatthetrans-
lationandrevisionofthepresentbook.ManystudentsattheUniversityofDelaware
helped me to improve my English and to correct wrong phrases and wrong expres-
sions. To mention all of them is impossible. But among them were a few students
who read whole chapters and, without them, the book would have never been fin-
ished(orreadable).InparticularIwanttomentionEmilyWagnerandSpencerWalker.
They both did really a great job. Many thanks! Let me also express my gratitude to
ColleenMcInerney,RachelAustin,DanielAtadan,andQuentinDubroff,allstudents
inDelawareandattendingmyclassesforsometime.Theyalsoreadwholesectionsof
thebookandcorrectedmybrokenEnglish.Finally,mythanksgotoProfessorAnne
LeuchtfromtheTechnicalUniversityinBraunschweig(Germany);herfieldofworkis
MathematicalStatistics,andherhintsandremarksaboutChapter8inthisbookwere
importanttome.
X Preface
AndlastbutnotleastIwanttothanktheDepartmentofMathematicalSciences
attheUniversityofDelawarefortheexcellentworkingconditionsaftermyretirement
inGermany.
Newark,Delaware,June6,2016 WernerLinde
Contents
1 Probabilities 1
1.1 ProbabilitySpaces 1
1.1.1 SampleSpaces 1
1.1.2 3-FieldsofEvents 2
1.1.3 ProbabilityMeasures 5
1.2 BasicPropertiesofProbabilityMeasures 9
1.3 DiscreteProbabilityMeasures 13
1.4 SpecialDiscreteProbabilityMeasures 18
1.4.1 DiracMeasure 18
1.4.2 UniformDistributiononaFiniteSet 18
1.4.3 BinomialDistribution 22
1.4.4 MultinomialDistribution 24
1.4.5 PoissonDistribution 27
1.4.6 HypergeometricDistribution 29
1.4.7 GeometricDistribution 33
1.4.8 NegativeBinomialDistribution 35
1.5 ContinuousProbabilityMeasures 39
1.6 SpecialContinuousDistributions 43
1.6.1 UniformDistributiononanInterval 43
1.6.2 NormalDistribution 46
1.6.3 GammaDistribution 48
1.6.4 ExponentialDistribution 51
1.6.5 ErlangDistribution 52
1.6.6 Chi-SquaredDistribution 53
1.6.7 BetaDistribution 54
1.6.8 CauchyDistribution 56
1.7 DistributionFunction 57
1.8 MultivariateContinuousDistributions 64
1.8.1 MultivariateDensityFunctions 64
1.8.2 MultivariateUniformDistribution 66
1.9 ⋆ProductsofProbabilitySpaces 71
1.9.1 Product3-FieldsandMeasures 71
1.9.2 ProductMeasures:DiscreteCase 74
1.9.3 ProductMeasures:ContinuousCase 76
1.10 Problems 79
2 ConditionalProbabilitiesandIndependence 86
2.1 ConditionalProbabilities 86
2.2 IndependenceofEvents 94
2.3 Problems 101
XII Contents
3 RandomVariablesandTheirDistribution 105
3.1 TransformationofRandomValues 105
3.2 ProbabilityDistributionofaRandomVariable 107
3.3 SpecialDistributedRandomVariables 117
3.4 RandomVectors 119
3.5 JointandMarginalDistributions 120
3.5.1 MarginalDistributions:DiscreteCase 123
3.5.2 MarginalDistributions:ContinuousCase 128
3.6 IndependenceofRandomVariables 131
3.6.1 IndependenceofDiscreteRandomVariables 134
3.6.2 IndependenceofContinuousRandomVariables 138
3.7 ⋆OrderStatistics 141
3.8 Problems 146
4 OperationsonRandomVariables 149
4.1 MappingsofRandomVariables 149
4.2 LinearTransformations 154
4.3 CoinTossingversusUniformDistribution 157
4.3.1 BinaryFractions 157
4.3.2 BinaryFractionsofRandomNumbers 160
4.3.3 RandomNumbersGeneratedbyCoinTossing 162
4.4 SimulationofRandomVariables 164
4.5 AdditionofRandomVariables 169
4.5.1 SumsofDiscreteRandomVariables 171
4.5.2 SumsofContinuousRandomVariables 175
4.6 SumsofCertainRandomVariables 177
4.7 ProductsandQuotientsofRandomVariables 189
4.7.1 Student’st-Distribution 192
4.7.2 F-Distribution 194
4.8 Problems 196
5 ExpectedValue,Variance,andCovariance 200
5.1 ExpectedValue 200
5.1.1 ExpectedValueofDiscreteRandomVariables 200
5.1.2 Expected Value of Certain Discrete Random
Variables 203
5.1.3 ExpectedValueofContinuousRandomVariables 208
5.1.4 Expected Value of Certain Continuous Random
Variables 211
5.1.5 PropertiesoftheExpectedValue 215
5.2 Variance 222
5.2.1 HigherMomentsofRandomVariables 222
5.2.2 VarianceofRandomVariables 226
Contents XIII
5.2.3 VarianceofCertainRandomVariables 229
5.3 CovarianceandCorrelation 233
5.3.1 Covariance 233
5.3.2 CorrelationCoefficient 240
5.4 Problems 243
6 NormallyDistributedRandomVectors 248
6.1 RepresentationandDensity 248
6.2 ExpectedValueandCovarianceMatrix 256
6.3 Problems 262
7 LimitTheorems 264
7.1 LawsofLargeNumbers 264
7.1.1 Chebyshev’sInequality 264
7.1.2 ⋆Infinite Sequences of Independent Random
Variables 267
7.1.3 ⋆Borel–CantelliLemma 270
7.1.4 WeakLawofLargeNumbers 276
7.1.5 StrongLawofLargeNumbers 278
7.2 CentralLimitTheorem 283
7.3 Problems 298
8 MathematicalStatistics 301
8.1 StatisticalModels 301
8.1.1 NonparametricStatisticalModels 301
8.1.2 ParametricStatisticalModels 305
8.2 StatisticalHypothesisTesting 307
8.2.1 HypothesesandTests 307
8.2.2 PowerFunctionandSignificanceTests 310
8.3 TestsforBinomialDistributedPopulations 315
8.4 TestsforNormallyDistributedPopulations 319
8.4.1 Fisher’sTheorem 320
8.4.2 Quantiles 323
8.4.3 Z-TestsorGaussTests 326
8.4.4 t-Tests 328
8.4.5 72-TestsfortheVariance 330
8.4.6 Two-SampleZ-Tests 332
8.4.7 Two-Samplet-Tests 334
8.4.8 F-Tests 336
8.5 PointEstimators 337
8.5.1 MaximumLikelihoodEstimation 338
8.5.2 UnbiasedEstimators 346
8.5.3 RiskFunction 351
XIV Contents
8.6 ConfidenceRegionsandIntervals 355
8.7 Problems 362
AAppendix 365
A.1 Notations 365
A.2 ElementsofSetTheory 365
A.3 Combinatorics 367
A.3.1 BinomialCoefficients 367
A.3.2 DrawingBallsoutofanUrn 373
A.3.3 MultinomialCoefficients 376
A.4 VectorsandMatrices 379
A.5 SomeAnalyticTools 382
Bibliography 389
Index 391