APM 504: Continuous-time Markov Chains

55 Pages·2015·0.46 MB·English

by  Jay Taylor

APM 504: Continuous-time Markov Chains
Jay Taylor
Spring 2015
JayTaylor (ASU) APM504 Spring2015 1/55
;
Outline
1 Definitions
2 Example: Jukes-Cantor Model
3 The Gillespie Algorithm
4 Kolmogorov Equations
5 Stationary Distributions
6 Poisson Processes
JayTaylor (ASU) APM504 Spring2015 2/55
Definitions
We begin with a general definition that imposes no restrictions on E. As in the
discrete-time setting, the gist of the definition is that the past and the future are
conditionally independent given the present.
Continuous-time Markov Processes
A stochastic process X =(X ;t ≥0) with values in a set E is said to be a
t
continuous-time Markov process if for every sequence of times
0≤t <···<t <t and every set of values x ,··· ,x ∈E, we have
1 n n+1 1 n+1
P(X ∈A|X =x ,··· ,X =x )=P(X ∈A|X =x ),
tn+1 tn n t1 1 tn+1 tn n
whenever A is a subset of E such that {X ∈A} is an event. In this case, the
tn+1
function defined by
p(s,t;x,A)=P(X ∈A|X =x), t ≥s ≥0
t s
is called the transition function of X. If this function depends on s and t only through
the difference t−s, then we say that X is time-homogeneous.
JayTaylor (ASU) APM504 Spring2015 3/55
Definitions
Examples of continuous-time Markov processes encountered in biology include:
continuous-time Markov chains, e.g., nucleotide substitution models (JC69, HKY,
GTR); the Moran model; Kingman’s coalescent; the Moran model
continuous-time branching processes
continuous-time random walks
Poisson processes
diffusion processes and the solutions to stochastic differential equations, e.g.,
Brownian motion and the Ornstein-Uhlenbeck process
L´evy processes, e.g., L´evy flight
JayTaylor (ASU) APM504 Spring2015 4/55
Definitions
Sample Paths
A continuous-time stochastic process X =(X :t ≥0) can be thought of in two
t
different ways.
Ontheonehand,X issimplyacollectionofrandomvariablesdefinedonthesame
probability space, Ω.
On the other hand, we can also think of X as a path- or function-valued random
variable. Inotherwords,givenanoutcomeω∈Ω,wewillviewX(ω)asafunction
from [0,∞) into E defined by
X(ω)(t)≡X (ω)∈E.
t
ThepathtracedoutbyX(ω)ast variesfrom0to∞issaidtobeasample pathofX.
JayTaylor (ASU) APM504 Spring2015 5/55
Definitions
In these notes, we will mainly be concerned with time-homogeneous Markov processes
that take values in countable state spaces: E ={1,2,··· ,n} or E ={1,2,···}. As we
will see, these processes are jump processes.
Continuous-time Markov Chains
A stochastic process X =(X ;t ≥0) with values in a countable set E is said to be a
t
continuous-time Markov chain (CTMC) if there are functions
p :[0,∞)→[0,1], i,j ∈E
ij
such that for every sequence of times 0≤t <···<t <t and every set of values
1 n n+1
x ,··· ,x ∈E we have
1 n+1
P(X =x |X =x ,··· ,X =x ) = P(X =x |X =x )
tn+1 n+1 tn n t1 1 tn+1 n+1 tn n
= p (t −t ).
xn,xn+1 n+1 n
The functions p (t) are said to be the transition functions of X.
ij
JayTaylor (ASU) APM504 Spring2015 6/55
Definitions
Transition Semigroups
If X =(X :t ≥0) is a continuous-time Markov chain, we can define an entire family of
t
transition matrices indexed by time. Specifically, for each t ≥0 and each pair of
elements i,j ∈E, let
p (t)=P(X =j|X =i)
ij t 0
be the transition function corresponding to transitions from state i to state j, and let
P(t) be the matrix with entries p (t). In particular, if E ={1,2,··· ,n} is finite, then
ij
P(t) is the n×n matrix
 p (t) p (t) ··· p (t) 
11 12 1n
 p21(t) p22(t) ··· p2n(t) 
P(t)= . . . .
 . . . 
 . . . 
p (t) p (t) ··· p (t)
n1 n2 nn
JayTaylor (ASU) APM504 Spring2015 7/55
Definitions
It can be shown that the family of transition matrices (P(t):t ≥0) satisfies the
following properties:
For each t ≥0, P(t) is a stochastic matrix:
(cid:88)
p (t)=1.
ij
j∈E
P(0) is the identity matrix: p (0)=1 and p (0)=0 if j (cid:54)=i.
ii ij
For every s,t ≥0, the matrices P(s) and P(t) commute and
P(t+s)=P(t)P(s).
The third property is called the semigroup property and the family of matrices is said
to be a transition semigroup. When written in terms of coordinates, this property is
(cid:88)
p (s+t)= p (s)p (t)
ij ik kj
k∈E
which is just the continuous-time version of the Chapman-Kolmogorov equations.
JayTaylor (ASU) APM504 Spring2015 8/55
Definitions
Oneconsequenceofthesemigrouppropertyisthatforanyt ≥0andanyn≥1,wehave
P(t)=P(t/n)n.
In fact, provided that the matrices P(t) depend continuously on t, it can be shown that
there exists a unique matrix Q such that for every t ≥0
∞
P(t)=eQt ≡(cid:88) 1tnQn.
n!
n=0
This matrix is called the rate matrix (or the generator matrix, infinitesimal matrix, or
Q-matrix) of the continuous-time Markov chain X. Rate matrices play a central role in
the description and analysis of continuous-time Markov chain and have a special
structure which is described in the next theorem.
JayTaylor (ASU) APM504 Spring2015 9/55
Definitions
Structure of the Q-matrix
Let Q =(q ) be the rate matrix of a continuous-time Markov chain X with transition
ij
semigroup (P(t):t ≥0). Then all of the row-sums of Q are equal to 0 and all of the
off-diagonal elements are non-negative:
q ≥ 0 if j (cid:54)=i
ij
(cid:88)
q = − q ,
ii ij
j(cid:54)=i
Furthermore, each transition probability p (t) is a differentiable function of t and
ij
p (t)−p (0)
q =p(cid:48)(0)= lim ij ij .
ij ij t→0 t
Remark: In matrix notation, these identities can be written as
Q =P(cid:48)(0)= lim 1(cid:0)P(t)−I(cid:1).
t→0 t
JayTaylor (ASU) APM504 Spring2015 10/55