Probability, Statistics, and Stochastic Processes

583 Pages·2013·4.02 MB·English

Probability, Statistics, and Stochastic
Processes

Probability, Statistics, and
Stochastic Processes
Peter Olofsson
Mikael Andersson
AWiley-IntersciencePublication
JOHN WILEY & SONS, INC.
NewYork / Chichester / Weinheim / Brisbane / Singapore / Toronto

Preface
TheBook
InNovember2003,Iwascompletingareviewofanundergraduatetextbookinprob-
ability and statistics. In the enclosedevaluationsheet was the question “Have you
ever considered writing a textbook?” and I suddenly realized that the answer was
“Yes,” and had been for quite some time. For several years I had been teaching a
courseoncalculus-basedprobabilityandstatisticsmainlyformathematics,science,
andengineeringstudents. Otherthanthebasicprobabilitytheory,mygoalwastoin-
cludetopicsfromtwoareas: statisticalinferenceandstochasticprocesses. Formany
students this was the only probability/statistics course they would ever take, and I
founditdesirablethattheywerefamiliarwithconfidenceintervalsandthemaximum
likelihoodmethod,aswellasMarkovchainsandqueueingtheory. Whiletherewere
plentyofbookscoveringoneareaortheother,itwassurprisinglydifficulttofindone
thatcoveredbothinasatisfyingwayandontheappropriatelevelofdifficulty. My
solutionwastochooseonetextbookandsupplementitwithlecturenotesinthearea
thatwasmissing. AsIchangedtextsoften,plentyoflecturenotesaccumulatedand
itseemedlikea goodideatoorganizethemintoatextbook. Iwaspleasedtolearn
thatthegoodpeopleatWileyagreed.
It is now more than a year later, and the bookhas been written. The first three
chaptersdevelopprobabilitytheoryandintroducetheaxiomsofprobability,random
variables,andjointdistributions. Thefollowingtwo chaptersareshorterandofan
“introductionto”nature: Chapter4onlimittheoremsandChapter5onsimulation.
Statistical inference is treated in Chapter 6, which includes a section on Bayesian
v
vi PREFACE
statistics, too oftena neglectedtopicin undergraduatetexts. Finally, in Chapter7,
Markov chains in discrete and continuous time are introduced. The reference list
atthe end of thebookis byno meansintendedto be comprehensive;rather, it is a
subjectiveselectionoftheusefulandtheentertaining.
ThroughoutthetextIhavetriedtoconveyanintuitiveunderstandingofconcepts
andresults, whichis whya definitionora propositionisoftenprecededbya short
discussion or a motivating example. I have also attempted to make the exposition
entertainingbychoosingexamplesfromtherichsourceoffunandthought-provoking
probabilityproblems. Thedatasetsusedinthestatisticschapterareofthreedifferent
kinds: real,fakebutrealistic,andunrealisticbutillustrative.
Thepeople
Most textbookauthors start by thankingtheir spouses. I know now that this is far
more than a formality, and I would like to thank Aλκµη´νη not only for patiently
puttingupwithirregularworkhoursandanabsentmindednessgreaterthanusualbut
alsoforvaluablecommentsontheaestheticsofthemanuscript.
A numberof people have commentedon variousparts and aspects of the book.
First, Iwouldlike tothankOlle Ha¨ggstro¨matChalmersUniversityofTechnology,
Go¨teborg,Swedenforvaluablecommentsonall chapters. His remarksare always
accurate and insightful, and never obscured by unnecessary politeness. Second, I
wouldliketothankKjellDoksumattheUniversityofWisconsinforaveryhelpful
reviewofthestatisticschapter. IhavealsoenjoyedtheBayesianenthusiasmofPeter
Mu¨llerattheUniversityofTexasMDAndersonCancerCenter.
Otherpeoplewhohavecommentedonpartsofthebookorbeenotherwisehelpful
are my colleagues Dennis Cox, Kathy Ensor, Rudy Guerra, Marek Kimmel, Rolf
Riedi,JavierRojo,DavidW.Scott,andJimThompsonatRiceUniversity;Prof. Dr.
R.W.J.MeesteratVrijeUniversiteit,Amsterdam,TheNetherlands;TimoSeppa¨la¨inen
at the University of Wisconsin; Tom English at Behrend College; Robert Lund at
ClemsonUniversity;andJaredMartinatShellExplorationandProduction. Forhelp
with solutions to problems, I am grateful to several bright Rice graduate students:
Blair Christian, Julie Cong, Talithia Daniel, GingerDavis, Li Deng, GretchenFix,
HectorFlores, GarrettFox,DarrinGershman,JasonGershman,ShuHan,Shannon
Neeley,RickOtt,GalenPapkov,BoPeng,ZhaoxiaYu,andJennyZhang. Thanksto
MikaelAnderssonatStockholmUniversity,Swedenforcontributionstotheproblem
sections,andtoPatrickKingatODS–Petrodata,Inc. forprovidingdatawithadis-
tinctTexasflavor: oilrigcharterrates. AtWiley,IwouldliketothankSteveQuigley,
SusanneSteitz, andKellsee Chuforalwayspromptlyansweringmyquestions. Fi-
nally,thankstoJohnHaigh,JohnAllenPaulos,JeffreyE.Steif,andananonymous
Dutchmanforagreeingtoappearandbemildlymockedinfootnotes.
PETEROLOFSSON
Houston,Texas,2005
PREFACE vii
PrefacetotheSecondEdition
Thesecondeditionwasmotivatedbycommentsfromseveralusersandreadersthat
thechaptersonstatisticalinferenceandstochasticprocesseswouldbenefitfromsub-
stantial extensions. To accomplish such extensions, I decided to bring in Mikael
Andersson,anoldfriendandcolleaguefromgraduateschool. Beingfivedaysmyju-
nior,hebroughtavigorousandyouthfulperspectivetothetaskandIamverypleased
withtheoutcome. Below,Mikaelwilloutlinethemajorchangesandadditionsintro-
ducedinthesecondedition.
PeterOlofsson
SanAntonio,Texas,2011
The chapter on statistical inference has been extended, reorganized and split into
twonewchapters. Chapter6introducestheprinciplesandconceptsbehindstandard
methodsofstatisticalinferenceingeneralwhiletheimportantspecialcaseofnormally
distributedsamplesistreatedseparatelyinChapter7. Thisisasomewhatdifferent
structurecomparedtomostothertextbooksinstatisticssincecommonmethodsliket
testsandlinearregressioncomeratherlateinthetext. Accordingtomyexperience,
ifmethodsbasedonnormalsamplesarepresentedtooearlyinacourse,theytendto
overshadowotherapproacheslikenonparametricandbayesianmethodsandstudents
becomelessawarethatthesealternativesexist.
New additions in Chapter 6 include consistency of point estimators, large sam-
ple theory, bootstrap simulation, multiple hypothesis testing, Fisher’s exact test,
Kolmogorov-Smirnov’stestandnonparametricconfidenceintervalsaswellasadis-
cussionofinformativeversusnon-informativepriorsandcredibilityintervalsinSec-
tion6.8.
Chapter 7 opens with a detailed treatment of sampling distributions, like the t,
chi-squareandF distributions,derivedfromthenormaldistribution. Therearealso
newsectionsintroducingone-wayanalysisofvarianceandthegenerallinearmodel.
Chapter8havebeenexpandedtoincludethreenewsectionson martingales,re-
newalprocessesandBrownianmotion,respectively. Theseareasareofgreatimpor-
tanceinprobabilitytheoryandstatistics,butsincetheyarebasedonquiteextensive
andadvancedmathematicaltheory,weonlyofferabriefintroductionhere.
Ithasbeenagreatprivilege,responsibilityandpleasuretohavehadtheopportunity
toworkwithsuchanesteemedcolleagueandgoodfriend. Finally,thejointproject
thatwedreamedaboutduringgraduateschoolhascometofruition!
I also have a victim of preoccupation and absentmindedness; my beloved Eva
whomIwanttothankforhersupportandalltheloveandfriendshipwehaveshared
andwillcontinuetoshareformanydaystocome.
MikaelAndersson
Stockholm,Sweden,2011

Contents
Preface v
1 BasicProbabilityTheory 1
1.1 Introduction 1
1.2 SampleSpacesand Events 3
1.3 TheAxiomsofProbability 7
1.4 FiniteSampleSpaces andCombinatorics 16
1.4.1 Combinatorics 18
1.5 ConditionalProbabilityandIndependence 29
1.5.1 IndependentEvents 35
1.6 TheLawof TotalProbabilityand Bayes’Formula 43
1.6.1 Bayes’Formula 49
1.6.2 Geneticsand Probability 56
1.6.3 RecursiveMethods 58
2 RandomVariables 79
2.1 Introduction 79
2.2 DiscreteRandomVariables 81
2.3 ContinuousRandomVariables 86
2.3.1 TheUniformDistribution 94
ix
x CONTENTS
2.3.2 Functionsof RandomVariables 96
2.4 Expected ValueandVariance 99
2.4.1 The Expected Value of a Function of a Random
Variable 104
2.4.2 Varianceofa RandomVariable 108
2.5 SpecialDiscreteDistributions 115
2.5.1 Indicators 116
2.5.2 TheBinomialDistribution 116
2.5.3 TheGeometricDistribution 120
2.5.4 ThePoissonDistribution 122
2.5.5 TheHypergeometricDistribution 125
2.5.6 DescribingData Sets 127
2.6 TheExponentialDistribution 128
2.7 TheNormalDistribution 132
2.8 OtherDistributions 137
2.8.1 TheLognormalDistribution 137
2.8.2 TheGamma Distribution 139
2.8.3 TheCauchyDistribution 140
2.8.4 Mixed Distributions 141
2.9 LocationParameters 142
2.10 TheFailureRateFunction 145
2.10.1UniquenessoftheFailureRateFunction 147
3 JointDistributions 161
3.1 Introduction 161
3.2 TheJointDistributionFunction 161
3.3 DiscreteRandomVectors 163
3.4 JointlyContinuousRandomVectors 166
3.5 ConditionalDistributionsand Independence 169
3.5.1 IndependentRandomVariables 174
3.6 FunctionsofRandomVectors 178
3.6.1 Real-ValuedFunctionsof RandomVectors 178
3.6.2 TheExpected ValueandVarianceofa Sum 182
3.6.3 Vector-ValuedFunctionsofRandomVectors 188
3.7 ConditionalExpectation 191
3.7.1 ConditionalExpectationasa RandomVariable 195
3.7.2 ConditionalExpectationandPrediction 197
3.7.3 ConditionalVariance 198
3.7.4 RecursiveMethods 199