An Introduction to Statistical Computing: A Simulation-based Approach

388 Pages·2013·3.369 MB·English

by  Jochen Voss

An Introduction to
Statistical Computing
WILEYSERIESINCOMPUTATIONALSTATISTICS
ConsultingEditors:
PaoloGiudici
UniversityofPavia,Italy
GeofH.Givens
ColoradoStateUniversity,USA
BaniK.Mallick
TexasA&MUniversity,USA
WileySeriesinComputationalStatisticsiscomprisedofpracticalguidesandcutting
edge research books on new developments in computational statistics. It features
quality authors with a strong applications focus. The texts in the series provide
detailed coverage of statistical concepts, methods and case studies in areas at the
interfaceofstatistics,computing,andnumerics.
With sound motivation and a wealth of practical examples, the books show in
concrete terms how to select and to use appropriate ranges of statistical comput-
ing techniques in particular fields of study. Readers are assumed to have a basic
understandingofintroductoryterminology.
Theseriesconcentratesonapplicationsofcomputationalmethodsinstatisticsto
fieldsofbioinformatics,genomics,epidemiology,business,engineering,financeand
appliedstatistics.
TitlesintheSeries
Biegler,Biros,Ghattas,Heinkenschloss,Keyes,Mallick,Marzouk,Tenorio,
Waanders,Willcox–Large-ScaleInverseProblemsandQuantificationofUncertainty
BillardandDiday–SymbolicDataAnalysis:ConceptualStatisticsandDataMining
Bolstad–UnderstandingComputationalBayesianStatistics
Borgelt,SteinbrecherandKruse–GraphicalModels,2e
Dunne–AStatisticalApproachtoNeutralNetworksforPatternRecognition
Liang,LiuandCarroll–AdvancedMarkovChainMonteCarloMethods
Ntzoufras–BayesianModelingUsingWinBUGS
Tuffe´ry–DataMiningandStatisticsforDecisionMaking
An Introduction to
Statistical Computing
A Simulation-based Approach
Jochen Voss
SchoolofMathematics,UniversityofLeeds,UK
Thiseditionfirstpublished2014
(cid:2)C 2014JohnWiley&Sons,Ltd
Registeredoffice
JohnWiley&Sons,Ltd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,
UnitedKingdom
Fordetailsofourglobaleditorialoffices,forcustomerservicesandforinformationabouthowtoapply
forpermissiontoreusethecopyrightmaterialinthisbookpleaseseeourwebsiteatwww.wiley.com.
Therightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththe
Copyright,DesignsandPatentsAct1988.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,or
transmitted,inanyformorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,
exceptaspermittedbytheUKCopyright,DesignsandPatentsAct1988,withoutthepriorpermissionof
thepublisher.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmay
notbeavailableinelectronicbooks.
Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrand
namesandproductnamesusedinthisbookaretradenames,servicemarks,trademarksorregistered
trademarksoftheirrespectiveowners.Thepublisherisnotassociatedwithanyproductorvendor
mentionedinthisbook.
LimitofLiability/DisclaimerofWarranty:Whilethepublisherandauthorhaveusedtheirbesteffortsin
preparingthisbook,theymakenorepresentationsorwarrantieswithrespecttotheaccuracyor
completenessofthecontentsofthisbookandspecificallydisclaimanyimpliedwarrantiesof
merchantabilityorfitnessforaparticularpurpose.Itissoldontheunderstandingthatthepublisherisnot
engagedinrenderingprofessionalservicesandneitherthepublishernortheauthorshallbeliablefor
damagesarisingherefrom.Ifprofessionaladviceorotherexpertassistanceisrequired,theservicesofa
competentprofessionalshouldbesought.
LibraryofCongressCataloging-in-PublicationData
Voss,Jochen.
Anintroductiontostatisticalcomputing:asimulation-basedapproach/JochenVoss.–Firstedition.
pagescm.–(Wileyseriesincomputationalstatistics)
Includesbibliographicalreferencesandindex.
ISBN978-1-118-35772-9(hardback)
1.Mathematicalstatistics–Dataprocessing. I.Title.
QA276.4.V662013
519.501(cid:3)13–dc23
2013019321
AcataloguerecordforthisbookisavailablefromtheBritishLibrary.
ISBN:978-1-118-35772-9
Typesetin10/12ptTimesbyAptaraInc.,NewDelhi,India
1 2014
Contents
Listofalgorithms ix
Preface xi
Nomenclature xiii
1 Randomnumbergeneration 1
1.1 Pseudorandomnumbergenerators 2
1.1.1 Thelinearcongruentialgenerator 2
1.1.2 Qualityofpseudorandomnumbergenerators 4
1.1.3 Pseudorandomnumbergeneratorsinpractice 8
1.2 Discretedistributions 8
1.3 Theinversetransformmethod 11
1.4 Rejectionsampling 15
1.4.1 Basicrejectionsampling 15
1.4.2 Enveloperejectionsampling 18
1.4.3 Conditionaldistributions 22
1.4.4 Geometricinterpretation 26
1.5 Transformationofrandomvariables 30
1.6 Special-purposemethods 36
1.7 Summaryandfurtherreading 36
Exercises 37
2 Simulatingstatisticalmodels 41
2.1 Multivariatenormaldistributions 41
2.2 Hierarchicalmodels 45
2.3 Markovchains 50
2.3.1 Discretestatespace 51
2.3.2 Continuousstatespace 56
2.4 Poissonprocesses 58
2.5 Summaryandfurtherreading 67
Exercises 67
vi CONTENTS
3 MonteCarlomethods 69
3.1 Studyingmodelsviasimulation 69
3.2 MonteCarloestimates 74
3.2.1 ComputingMonteCarloestimates 75
3.2.2 MonteCarloerror 76
3.2.3 Choiceofsamplesize 80
3.2.4 Refinederrorbounds 82
3.3 Variancereductionmethods 84
3.3.1 Importancesampling 84
3.3.2 Antitheticvariables 88
3.3.3 Controlvariates 93
3.4 Applicationstostatisticalinference 96
3.4.1 Pointestimators 97
3.4.2 Confidenceintervals 100
3.4.3 Hypothesistests 103
3.5 Summaryandfurtherreading 106
Exercises 106
4 MarkovChainMonteCarlomethods 109
4.1 TheMetropolis–Hastingsmethod 110
4.1.1 Continuousstatespace 110
4.1.2 Discretestatespace 113
4.1.3 RandomwalkMetropolissampling 116
4.1.4 Theindependencesampler 119
4.1.5 Metropolis–Hastingswithdifferentmovetypes 120
4.2 ConvergenceofMarkovChainMonteCarlomethods 125
4.2.1 Theoreticalresults 125
4.2.2 Practicalconsiderations 129
4.3 ApplicationstoBayesianinference 137
4.4 TheGibbssampler 141
4.4.1 Descriptionofthemethod 141
4.4.2 Applicationtoparameterestimation 146
4.4.3 Applicationstoimageprocessing 151
4.5 ReversibleJumpMarkovChainMonteCarlo 158
4.5.1 Descriptionofthemethod 160
4.5.2 Bayesianinferenceformixturedistributions 171
4.6 Summaryandfurtherreading 178
4.6 Exercises 178
5 BeyondMonteCarlo 181
5.1 ApproximateBayesianComputation 181
5.1.1 BasicApproximateBayesianComputation 182
5.1.2 ApproximateBayesianComputationwithregression 188
5.2 Resamplingmethods 192
CONTENTS vii
5.2.1 Bootstrapestimates 192
5.2.2 Applicationstostatisticalinference 197
5.3 Summaryandfurtherreading 209
Exercises 209
6 Continuous-timemodels 213
6.1 Timediscretisation 213
6.2 Brownianmotion 214
6.2.1 Properties 216
6.2.2 Directsimulation 217
6.2.3 InterpolationandBrownianbridges 218
6.3 GeometricBrownianmotion 221
6.4 Stochasticdifferentialequations 224
6.4.1 Introduction 224
6.4.2 Stochasticanalysis 226
6.4.3 Discretisationschemes 231
6.4.4 Discretisationerror 236
6.5 MonteCarloestimates 243
6.5.1 BasicMonteCarlo 243
6.5.2 Variancereductionmethods 247
6.5.3 MultilevelMonteCarloestimates 250
6.6 Applicationtooptionpricing 255
6.7 Summaryandfurtherreading 259
Exercises 260
AppendixA Probabilityreminders 263
A.1 Eventsandprobability 263
A.2 Conditionalprobability 266
A.3 Expectation 268
A.4 Limittheorems 269
A.5 Furtherreading 270
AppendixB ProgramminginR 271
B.1 Generaladvice 271
B.2 RasaCalculator 272
B.2.1 Mathematicaloperations 273
B.2.2 Variables 273
B.2.3 Datatypes 275
B.3 Programmingprinciples 282
B.3.1 Don’trepeatyourself! 283
B.3.2 Divideandconquer! 286
B.3.3 Testyourcode! 290
B.4 Randomnumbergeneration 292
B.5 Summaryandfurtherreading 294
Exercises 294
viii CONTENTS
AppendixC Answerstotheexercises 299
C.1 AnswersforChapter1 299
C.2 AnswersforChapter2 315
C.3 AnswersforChapter3 319
C.4 AnswersforChapter4 328
C.5 AnswersforChapter5 342
C.6 AnswersforChapter6 350
C.7 AnswersforAppendixB 366
References 375
Index 379
List of algorithms
Randomnumbergeneration
alg.1.2 linearcongruentialgenerator 2
alg.1.13 inversetransformmethod 12
alg.1.19 basicrejectionsampling 15
alg.1.22 enveloperejectionsampling 19
alg.1.25 rejectionsamplingforconditionaldistributions 22
Simulatingstatisticalmodels
alg.2.9 mixturedistributions 47
alg.2.11 componentwisesimulation 49
alg.2.22 Markovchainswithdiscretestatespace 53
alg.2.31 Markovchainswithcontinuousstatespace 58
alg.2.36 Poissonprocess 61
alg.2.41 thinningmethodforPoissonprocesses 65
MonteCarlomethods
alg.3.8 MonteCarloestimate 75
alg.3.22 importancesampling 85
alg.3.26 antitheticvariables 89
alg.3.31 controlvariates 93
MarkovChainMonteCarlomethods
alg.4.2 Metropolis–Hastingsmethodforcontinuousstatespace 110
alg.4.4 Metropolis–Hastingsmethodfordiscretestatespace 113
alg.4.9 randomwalkMetropolis 117
alg.4.11 independencesampler 119
alg.4.12 Metropolis–Hastingsmethodwithdifferentmovetypes 121
alg.4.27 Gibbssampler 142
alg.4.31 GibbssamplerfortheIsingmodel 155
alg.4.32 Gibbssamplerinimageprocessing 158
alg.4.36 reversiblejumpMarkovChainMonteCarlo 165
x LISTOFALGORITHMS
BeyondMonteCarlo
alg.5.1 basicApproximateBayesianComputation 182
alg.5.6 ApproximateBayesianComputationwithregression 191
alg.5.11 generalbootstrapestimate 196
alg.5.15 bootstrapestimateofthebias 200
alg.5.18 bootstrapestimateofthestandarderror 202
alg.5.20 simplebootstrapconfidenceinterval 205
alg.5.21 BC bootstrapconfidenceinterval 207
a
Continuous-timemodels
alg.6.6 Brownianmotion 217
alg.6.12 Euler–Maruyamascheme 232
alg.6.15 Milsteinscheme 235
alg.6.26 multilevelMonteCarloestimates 251
alg.6.29 Euler–MaruyamaschemefortheHestonmodel 256