Understanding Markov Chains: Examples and Applications

357 Pages·2013·2.95 MB·English

by  Nicolas Privault

Springer Undergraduate Mathematics Series
AdvisoryBoard
M.A.J.ChaplainUniversityofDundee,Dundee,Scotland,UK
K.ErdmannUniversityofOxford,Oxford,England,UK
A.MacIntyreQueenMary,UniversityofLondon,London,England,UK
E.SüliUniversityofOxford,Oxford,England,UK
M.R.TehranchiUniversityofCambridge,Cambridge,England,UK
J.F.TolandUniversityofBath,Bath,England,UK
Forfurthervolumes:
www.springer.com/series/3423
Nicolas Privault
Understanding
Markov
Chains
Examples and Applications
Nicolas Privault
School of Physical and Mathematical
Sciences
Nanyang Technological University
Singapore, Singapore
ISSN 1615-2085 Springer Undergraduate Mathematics Series
ISBN 978-981-4451-50-5 ISBN 978-981-4451-51-2 (eBook)
DOI 10.1007/978-981-4451-51-2
Springer Singapore Heidelberg New York Dordrecht London
LibraryofCongressControlNumber:2013942497
©SpringerScience+BusinessMediaSingapore2013
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewhole
or part of the material is concerned, specifically the rights of translation, reprinting, reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysical
way,andtransmissionorinformationstorageandretrieval,electronicadaptation,computer
software, or by similar or dissimilar methodology now known or hereafter developed. Ex-
emptedfromthislegalreservationarebriefexcerptsinconnectionwithreviewsorscholarly
analysisormaterialsuppliedspecificallyforthepurposeofbeingenteredandexecutedona
computer system, for exclusive use by the purchaser of the work. Duplication of this publi-
cation or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrent version,andpermissionforusemustalwaysbeobtained
from Springer. Permissions for use may be obtained through RightsLink at the Copyright
ClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw.
The use of general descriptive names, registered names, trademarks, service marks, etc. in
thispublicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnames
are exempt from the relevant protective laws and regulations and therefore free for general
use.
While the advice and information in this book are believed to be true and accurate at the
date of publication, neither the authors nor the editors nor the publisher can accept any
legal responsibility for any errors or omissions that may be made. The publisher makes no
warranty,expressorimplied,withrespecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
StochasticandMarkovianmodelingareofimportancetomanyareasofscience
including physics, biology, engineering, as well as in economics, finance, and
social sciences. This text is an undergraduate-level introduction to the Marko-
vian modeling of time-dependent randomness in discrete and continuous time,
mostly on discrete state spaces. The emphasis is put on the understanding of
conceptsbyexamplesandelementaryderivations,accompaniedby72exercises
andlongerproblemswhosesolutionsarecompletelyworkedoutandillustrated.
Some more advanced topics on Poisson stochastic integrals are also covered.
The book is mostly self-contained except for its main prerequisites which
consistinaknowledgeofbasicprobabilisticconceptssuchasrandomvariables,
discretedistributions(essentiallybinomial,geometric,andPoisson),continuous
distributions(Gaussianandgamma)anddensities,expectation,independence,
andconditionalprobabilities,whicharerecalledinthefirstchapter.Suchbasic
topics can be regarded as belonging to the field of “static” probability, i.e.
probability without time dependence, as opposed to the contents of this text
which is dealing with random evolution over time.
Ourtreatmentoftime-dependentrandomnessrevolvesaroundtheimportant
technique of first step analysis which is applied to random walks, branching
processes, and more generally to Markov chains in discrete and continuous
time,withapplicationtothecomputationofruinprobabilitiesandmeanhitting
times. In addition to the treatment of Markov chains, a brief introduction to
martingalesisgivenindiscretetime.Thisprovidesadifferentwaytorecoverthe
computation of ruin probabilities and mean hitting times which was originally
v
vi Preface
presented in the Markovian framework. Spatial Poisson processes on abstract
spaces are also considered without any time ordering, with the inclusion of
somerecentresultsondeviationinequalitiesandmomentidentitiesforPoisson
stochastic integrals.
There already exist many textbooks on stochastic processes and Markov
chains, including e.g. [1, 2, 5, 6, 9, 11, 15, 16, 21, 22]. In comparison with the
existing literature, which is sometimes dealing with structural properties of
stochastic processes via a compact and abstract treatment, the present book
tends to emphasize elementary and explicit calculations instead of quicker ar-
guments that may shorten the path to the solution, while being sometimes
difficult to reproduce by undergraduate students.
Thecontentsoftheexerciseshasbeeninfluencedby[2,9,11,15,21],whilea
numberofthemareoriginalandallsolutionshavebeenderivedindependently.
Some theorems have been quoted from [1, 11] when their proof becomes too
technical, as in Chapters 8and 10. The originalmanuscript has benefited from
numerous questions, comments and suggestions from the undergraduate stu-
dents in stochastic processes during academic years 2010/2013at the Nanyang
Technological University (NTU) in Singapore.
Singapore Nicolas Privault
2013
Contents
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Probability Background . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Probability Spaces and Events . . . . . . . . . . . . . . . . . . 7
2.2 Probability Measures . . . . . . . . . . . . . . . . . . . . . . . 11
2.3 Conditional Probabilities and Independence . . . . . . . . . . 11
2.4 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.5 Probability Distributions . . . . . . . . . . . . . . . . . . . . . 15
2.6 Expectation of a Random Variable . . . . . . . . . . . . . . . 21
2.7 Conditional Expectation . . . . . . . . . . . . . . . . . . . . . 27
2.8 Moment and Probability Generating Functions . . . . . . . . . 29
3 Gambling Problems . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.1 Constrained Random Walk . . . . . . . . . . . . . . . . . . . . 37
3.2 Ruin Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3 Mean Game Duration . . . . . . . . . . . . . . . . . . . . . . . 49
4 Random Walks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.1 Unrestricted Random Walk . . . . . . . . . . . . . . . . . . . . 61
4.2 Mean and Variance . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3 Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.4 First Return to Zero . . . . . . . . . . . . . . . . . . . . . . . 64
5 Discrete-Time Markov Chains . . . . . . . . . . . . . . . . . . 77
5.1 Markov Property . . . . . . . . . . . . . . . . . . . . . . . . . 77
vii
viii Contents
5.2 Transition Matrix . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.3 Examples of Markov Chains . . . . . . . . . . . . . . . . . . . 81
5.4 Higher Order Transition Probabilities . . . . . . . . . . . . . . 84
5.5 The Two-State Discrete-Time Markov Chain . . . . . . . . . . 87
6 First Step Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.1 Hitting Probabilities . . . . . . . . . . . . . . . . . . . . . . . 95
6.2 Mean Hitting and Absorption Times . . . . . . . . . . . . . . 98
6.3 First Return Times . . . . . . . . . . . . . . . . . . . . . . . . 103
6.4 Number of Returns . . . . . . . . . . . . . . . . . . . . . . . . 108
7 Classification of States . . . . . . . . . . . . . . . . . . . . . . . 117
7.1 Communicating States . . . . . . . . . . . . . . . . . . . . . . 117
7.2 Recurrent States . . . . . . . . . . . . . . . . . . . . . . . . . . 119
7.3 Transient States . . . . . . . . . . . . . . . . . . . . . . . . . . 121
7.4 Positive and Null Recurrence . . . . . . . . . . . . . . . . . . . 124
7.5 Periodicity and Aperiodicity . . . . . . . . . . . . . . . . . . . 125
8 Long-Run Behavior of Markov Chains . . . . . . . . . . . . . 129
8.1 Limiting Distributions . . . . . . . . . . . . . . . . . . . . . . 129
8.2 Stationary Distributions . . . . . . . . . . . . . . . . . . . . . 130
8.3 Markov Chain Monte Carlo . . . . . . . . . . . . . . . . . . . 139
9 Branching Processes . . . . . . . . . . . . . . . . . . . . . . . . . 149
9.1 Definition and Examples . . . . . . . . . . . . . . . . . . . . . 149
9.2 Probability Generating Functions . . . . . . . . . . . . . . . . 152
9.3 Extinction Probabilities . . . . . . . . . . . . . . . . . . . . . . 154
10 Continuous-Time Markov Chains . . . . . . . . . . . . . . . . 167
10.1 The Poisson Process. . . . . . . . . . . . . . . . . . . . . . . . 167
10.2 Continuous-Time Chains . . . . . . . . . . . . . . . . . . . . . 172
10.3 Transition Semigroup . . . . . . . . . . . . . . . . . . . . . . . 176
10.4 Infinitesimal Generator . . . . . . . . . . . . . . . . . . . . . . 180
10.5 The Two-State Continuous-Time Markov Chain . . . . . . . . 187
10.6 Limiting and Stationary Distributions . . . . . . . . . . . . . . 191
10.7 The Discrete-Time Embedded Chain . . . . . . . . . . . . . . 196
10.8 Mean Absorption Time and Probabilities . . . . . . . . . . . . 200
11 Discrete-Time Martingales . . . . . . . . . . . . . . . . . . . . . 211
11.1 Filtrations and Conditional Expectations . . . . . . . . . . . . 211
11.2 Martingales—Definition and Properties . . . . . . . . . . . . . 212
11.3 Ruin Probabilities . . . . . . . . . . . . . . . . . . . . . . . . . 216
11.4 Mean Game Duration . . . . . . . . . . . . . . . . . . . . . . . 220
Contents ix
12 Spatial Poisson Processes . . . . . . . . . . . . . . . . . . . . . 225
12.1 Spatial Poisson (1781–1840) Processes. . . . . . . . . . . . . . 225
12.2 Poisson Stochastic Integrals . . . . . . . . . . . . . . . . . . . 227
12.3 Transformations of Poisson Measures . . . . . . . . . . . . . . 229
12.4 Moments of Poisson Stochastic Integrals . . . . . . . . . . . . 231
12.5 Deviation Inequalities . . . . . . . . . . . . . . . . . . . . . . . 235
13 Reliability Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 241
13.1 Survival Probabilities . . . . . . . . . . . . . . . . . . . . . . . 241
13.2 Poisson Process with Time-Dependent Intensity . . . . . . . . 243
13.3 Mean Time to Failure . . . . . . . . . . . . . . . . . . . . . . . 244
Some Useful Identities . . . . . . . . . . . . . . . . . . . . . . . . . . 247
Solutions to the Exercises . . . . . . . . . . . . . . . . . . . . . . . . 251
Chapter 2—Probability Background. . . . . . . . . . . . . . . . . . 251
Chapter 3—Gambling Problems . . . . . . . . . . . . . . . . . . . . 260
Chapter 4—Random Walks . . . . . . . . . . . . . . . . . . . . . . 266
Chapter 5—Discrete-Time Markov Chains . . . . . . . . . . . . . . 271
Chapter 6—First Step Analysis . . . . . . . . . . . . . . . . . . . . 272
Chapter 7—Classification of States . . . . . . . . . . . . . . . . . . 290
Chapter 8—Limiting and Stationary Distributions . . . . . . . . . . 292
Chapter 9—Branching Processes. . . . . . . . . . . . . . . . . . . . 312
Chapter 10—Continuous-Time Markov Chains . . . . . . . . . . . . 321
Chapter 11—Discrete-Time Martingales . . . . . . . . . . . . . . . 345
Chapter 12—Spatial Poisson Processes . . . . . . . . . . . . . . . . 346
Chapter 13—Reliability and Renewal Processes . . . . . . . . . . . 348
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
1
Introduction
A stochastic1 process is a mathematical tool used for the modeling of time-
dependent random phenomena. Here, the term “stochastic” means random
and “process” refers to the time evolving status of a given system. Stochastic
processes have applications to multiple fields, and can be useful anytime one
recognizes the role of randomness and unpredictability of events that occur at
random times in a physical, biological, or financial system.
In applications to physics for example one can mention phase transitions,
atomic emission phenomena, etc. In biology the time behavior of live beings
is often subject to randomness, at least when the observer only handles par-
tial information. This point is of importance, as it shows that the notion of
randomness is linked to the concept of information: what appears random to
an observer may not be random to another observer equipped with more in-
formation. Think for example of an observer watching the apparent random
behavior of cars turning at a crossroad versus the point of view of car drivers,
each of whom are acting according to their own decisions. In finance the im-
portance of modeling time-dependent random phenomena is quite clear as no
one can make fully accurate predictions for the future moves of risky assets.
Theconcrete outcomeofsuch modeling liesinthe computation of expectations
or expected values, which often turn out to be more useful than the probability
valuesthemselves.Thelongtermstatisticalbehaviorofsuchsystemsisanother
issue of interest.
1 From the Greek “στo´χoς” (stokhos), meaning “guess”, “conjecture”.
N. Privault, Understanding Markov Chains, Springer Undergraduate 1
Mathematics Series, DOI 10.1007/978-981-4451-51-2 1,
© Springer Science+Business Media Singapore 2013