Understanding Markov chains. Examples and applications

379 Pages·2018·2.842 MB·English

by  Privault, Nicolas

Springer Undergraduate Mathematics Series
Nicolas Privault
Understanding
Markov Chains
Examples and Applications
Second Edition
Springer Undergraduate Mathematics Series
Advisory Board
M. A. J. Chaplain, University of St. Andrews
A. MacIntyre, Queen Mary University of London
S. Scott, King’s College London
N. Snashall, University of Leicester
E. Süli, University of Oxford
M. R. Tehranchi, University of Cambridge
J. F. Toland, University of Bath
More information about this series at http://www.springer.com/series/3423
Nicolas Privault
Understanding Markov
Chains
Examples and Applications
Second Edition
123
NicolasPrivault
Schoolof Physical andMathematical
Sciences
NanyangTechnological University
Singapore
Singapore
Additional material tothis bookcanbedownloaded from http://extras.springer.com.
ISSN 1615-2085 ISSN 2197-4144 (electronic)
SpringerUndergraduate MathematicsSeries
ISBN978-981-13-0658-7 ISBN978-981-13-0659-4 (eBook)
https://doi.org/10.1007/978-981-13-0659-4
LibraryofCongressControlNumber:2018942179
MathematicsSubjectClassification(2010): 60J10,60J27,60J28,60J20
1stedition:©SpringerScience+BusinessMediaSingapore2013
2ndedition:©SpringerNatureSingaporePteLtd.2018
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
Theregisteredcompanyaddressis:152BeachRoad,#21-01/04GatewayEast,Singapore189721,
Singapore
Preface
Stochastic and Markovian modeling are of importance to many areas of science
including physics, biology, engineering, as well as economics, finance, and social
sciences. This text is an undergraduate-level introduction to the Markovian mod-
eling of time-dependent randomness in discrete and continuous time, mostly on
discrete state spaces, with an emphasis on the understanding of concepts by
examplesandelementaryderivations.Thissecondeditionincludesarevisionofthe
main course content of the first edition, with additional illustrations and applica-
tions.Inparticular,theexercisesectionshavebeenconsiderableexpandedandnow
contain 138 exercises and 11 longer problems.
Thebookismostlyself-containedexceptforitsmainprerequisites,whichconsist
in a knowledge of basic probabilistic concepts. This includes random variables,
discrete distributions (essentially binomial, geometric, and Poisson), continuous
distributions (Gaussian and gamma), and their probability density functions,
expectation, independence, and conditional probabilities, some of which are
recalled in the first chapter. Such basic topics can be regarded as belonging to the
fieldof“static”probability,i.e.,probabilitywithouttimedependence,asopposedto
the contents of this text which is dealing with random evolution over time.
Our treatment of time-dependent randomness revolves around the important
technique of first-step analysis for random walks, branching processes, and more
generallyforMarkovchainsindiscreteandcontinuoustime,withapplicationtothe
computation of ruin probabilities and mean hitting times. In addition to the treat-
mentofMarkovchains,abriefintroductiontomartingalesisgivenindiscretetime.
Thisprovidesadifferent way torecover thecomputations of ruinprobabilities and
meanhittingtimeswhichhavebeenpresentedintheMarkovianframework.Spatial
Poissonprocessesonabstractspacesarealsoconsideredwithoutanytimeordering.
TherealreadyexistmanytextbooksonstochasticprocessesandMarkovchains,
including [BN96, Çin75, Dur99, GS01, JS01, KT81, Med10, Nor98, Ros96,
Ste01].Incomparisonwiththeexistingliterature,whichissometimesdealingwith
structural properties of stochastic processes via a more compact and abstract
treatment,thepresentbooktendstoemphasizeelementaryandexplicitcalculations
v
vi Preface
insteadofquickerargumentsthatmayshortenthepathtothesolution,whilebeing
sometimes difficult to reproduce by undergraduate students.
Some of the exercises have been influenced by [Çin75, JS01, KT81, Med10,
Ros96] and other references, while a number of them are original, and their solu-
tions have been derived independently. The problems, which are longer than the
exercises, are based on various topics of application. This second edition only
contains the answers to selected exercises, and the remaining solutions can be
downloadedinasolutionmanualavailablefromthepublisher’sWebsite,together
with Python and R codes. This text is also illustrated by 41 figures.
Some theorems whose proofs are technical, as in Chaps. 7 and 9, have been
quoted from [BN96, KT81]. The contents of this book have benefited from
numerous questions, comments, and suggestions from undergraduate students in
StochasticProcessesattheNanyangTechnologicalUniversity(NTU)inSingapore.
Singapore, Singapore Nicolas Privault
March 2018
Contents
1 Probability Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Probability Spaces and Events. . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Probability Measures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Conditional Probabilities and Independence . . . . . . . . . . . . . . . 6
1.4 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Expectation of Random Variables . . . . . . . . . . . . . . . . . . . . . . 17
1.7 Moment and Probability Generating Functions . . . . . . . . . . . . . 30
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2 Gambling Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.1 Constrained Random Walk . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.2 Ruin Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.3 Mean Game Duration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3 Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.1 Unrestricted Random Walk . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2 Mean and Variance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.3 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.4 First Return to Zero . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
4 Discrete-Time Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.1 Markov Property. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2 Transition Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.3 Examples of Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.4 Higher-Order Transition Probabilities. . . . . . . . . . . . . . . . . . . . 98
4.5 The Two-State Discrete-Time Markov Chain . . . . . . . . . . . . . . 101
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
vii
viii Contents
5 First Step Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.1 Hitting Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
5.2 Mean Hitting and Absorption Times . . . . . . . . . . . . . . . . . . . . 121
5.3 First Return Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.4 Mean Number of Returns . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6 Classification of States. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.1 Communicating States. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.2 Recurrent States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
6.3 Transient States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
6.4 Positive Versus Null Recurrence . . . . . . . . . . . . . . . . . . . . . . . 156
6.5 Periodicity and Aperiodicity . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7 Long-Run Behavior of Markov Chains. . . . . . . . . . . . . . . . . . . . . . 163
7.1 Limiting Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7.2 Stationary Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
7.3 Markov Chain Monte Carlo. . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
8 Branching Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
8.1 Construction and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
8.2 Probability Generating Functions . . . . . . . . . . . . . . . . . . . . . . . 192
8.3 Extinction Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
9 Continuous-Time Markov Chains. . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.1 The Poisson Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
9.2 Continuous-Time Markov Chains. . . . . . . . . . . . . . . . . . . . . . . 217
9.3 Transition Semigroup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
9.4 Infinitesimal Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
9.5 The Two-State Continuous-Time Markov Chain. . . . . . . . . . . . 235
9.6 Limiting and Stationary Distributions. . . . . . . . . . . . . . . . . . . . 240
9.7 The Discrete-Time Embedded Chain . . . . . . . . . . . . . . . . . . . . 247
9.8 Mean Absorption Time and Probabilities . . . . . . . . . . . . . . . . . 252
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
10 Discrete-Time Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
10.1 Filtrations and Conditional Expectations. . . . . . . . . . . . . . . . . . 263
10.2 Martingales - Definition and Properties . . . . . . . . . . . . . . . . . . 265
10.3 Stopping Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
10.4 Ruin Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
10.5 Mean Game Duration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
Contents ix
11 Spatial Poisson Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
11.1 Spatial Poisson (1781–1840) Processes . . . . . . . . . . . . . . . . . . 281
11.2 Poisson Stochastic Integrals. . . . . . . . . . . . . . . . . . . . . . . . . . . 283
11.3 Transformations of Poisson Measures . . . . . . . . . . . . . . . . . . . 285
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
12 Reliability Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
12.1 Survival Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
12.2 Poisson Process with Time-Dependent Intensity . . . . . . . . . . . . 291
12.3 Mean Time to Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
Exercise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293
Appendix A: Some Useful Identities .... .... .... .... .... ..... .... 295
Appendix B: Solutions to Selected Exercises and Problems.. ..... .... 297
References.... .... .... .... ..... .... .... .... .... .... ..... .... 363
Subject Index.. .... .... .... ..... .... .... .... .... .... ..... .... 365
Author Index.. .... .... .... ..... .... .... .... .... .... ..... .... 371