Continuous-Time Markov Chains and Applications: A Two-Time-Scale Approach

451 Pages·2013·3.44 MB·English

by  G. George Yin, Qing Zhang (auth.)

Stochastic Modelling
Stochastic Mechanics
Random Media
and Applied Probability
Signal Processing
and Image Synthesis
(Formerly:
Mathematical Economics and Finance
ApplicationsofMathematics)
Stochastic Optimization
Stochastic Control
Stochastic Models in Life Sciences 37
Edited by B.Rozovski˘ı
P.W.Glynn
Advisory Board M.Hairer
I. Karatzas
F. Kelly
A. Kyprianou
Y. LeJan
B. Øksendal
G. Papanicolaou
E.Pardoux
E.Perkins
H.M.Soner
For further volumes:
http://www.springer.com/series/602
G. George Yin • Qing Zhang
Continuous-Time Markov
Chains and Applications
A Two-Time-Scale Approach
Second edition
123
G. George Yin Qing Zhang
Department of Mathematics Department of Mathematics
WayneState University University of Georgia
Detroit, Michigan Athens,Georgia
USA USA
Managing Editors
B. Rozovski˘ı Peter W. Glynn
Division of Applied Mathematics Instituteof Computational
Brown University and Mathematical Engineering
Providence, RI Stanford University
USA Stanford, CA
USA
ISSN 0172-4568
ISBN 978-1-4614-4345-2 ISBN 978-1-4614-4346-9 (eBook)
DOI 10.1007/978-1-4614-4346-9
Springer New York Heidelberg Dordrecht London
Library of Congress Control Number:2012945403
Mathematics Subject Classification (2010): 60J27, 93E20, 34E05
(cid:2)c SpringerScience+Business Media, LLC 2013
This work is subject to copyright. All rights are reserved by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation,
reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms
or in any other physical way, and transmission or information storage and retrieval,
electronic adaptation, computer software, or by similaror dissimilarmethodology now
known or hereafter developed. Exempted from this legal reservation are brief excerpts
inconnectionwithreviewsorscholarlyanalysisormaterialsuppliedspecificallyforthe
purpose of being entered and executed on a computer system, for exclusive use by the
purchaserofthework.Duplicationofthispublicationorpartsthereofispermittedonly
under the provisions of the Copyright Law of the Publisher’s location, in its current
version,andpermissionforusemustalwaysbeobtainedfromSpringer.Permissionsfor
usemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.
inthispublicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuch
names are exempt from the relevant protective laws and regulations and therefore free
forgeneraluse.
While the advice and informationinthis book arebelieved to be trueand accurate at
thedateofpublication,neithertheauthorsnortheeditorsnorthepublishercanaccept
any legal responsibility for any errors or omissions that may be made. The publisher
makes nowarranty,expressorimplied,withrespecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
To Our Mentors
Harold J. Kushner and Wendell H. Fleming

Contents
Preface xiii
Preface to the First Edition xv
Convention xvii
Notation xix
Part I: Prologue and Preliminaries 1
1 Introduction and Overview 3
1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 A Brief Survey . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.1 Markov Chains . . . . . . . . . . . . . . . . . . . . 10
1.2.2 Singular Perturbations . . . . . . . . . . . . . . . . 11
1.3 Outline of the Book . . . . . . . . . . . . . . . . . . . . . 12
2 Mathematical Preliminaries 17
2.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Piecewise-Deterministic Processes. . . . . . . . . . . . . . 21
2.4.1 Construction of Markov Chains . . . . . . . . . . . 21
2.5 Irreducibility and Quasi-Stationary Distributions . . . . . 23
vii
viii Contents
2.6 Gaussian Processes and Diffusions . . . . . . . . . . . . . 25
2.7 Switching Diffusions . . . . . . . . . . . . . . . . . . . . . 27
2.8 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3 Markovian Models 31
3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Birth and Death Processes. . . . . . . . . . . . . . . . . . 32
3.3 Finite-State Space Models . . . . . . . . . . . . . . . . . . 34
3.3.1 Queues with Finite Capacity . . . . . . . . . . . . 34
3.3.2 System Reliability . . . . . . . . . . . . . . . . . . 37
3.3.3 Competing Risk Theory . . . . . . . . . . . . . . . 39
3.3.4 Two-Time-Scale Cox Processes . . . . . . . . . . . 40
3.3.5 Random Evolutions . . . . . . . . . . . . . . . . . 41
3.3.6 Seasonal Variation Models . . . . . . . . . . . . . . 42
3.4 Stochastic Optimization Problems . . . . . . . . . . . . . 45
3.4.1 Simulated Annealing . . . . . . . . . . . . . . . . . 45
3.4.2 Continuous-Time Stochastic Approximation . . . . 46
3.4.3 Systems with MarkovianDisturbances . . . . . . . 48
3.5 Linear Systems with Jump Markov Disturbance . . . . . . 49
3.5.1 Linear Quadratic Control Problems . . . . . . . . 49
3.5.2 Singularly Perturbed LQ Systems with Wide-Band
Noise . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.5.3 Large-Scale Systems: Decomposition
and Aggregation . . . . . . . . . . . . . . . . . . . 51
3.6 Time-Scale Separation . . . . . . . . . . . . . . . . . . . . 53
3.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Part II: Two-Time-Scale Markov Chains 57
4 Asymptotic Expansions of Solutions for Forward
Equations 59
4.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2 Irreducible Case . . . . . . . . . . . . . . . . . . . . . . . 62
4.2.1 Asymptotic Expansions . . . . . . . . . . . . . . . 63
4.2.2 Outer Expansion . . . . . . . . . . . . . . . . . . . 66
4.2.3 Initial-Layer Correction . . . . . . . . . . . . . . . 69
4.2.4 Exponential Decay of ψk(·) . . . . . . . . . . . . . 72
4.2.5 Asymptotic Validation . . . . . . . . . . . . . . . . 74
4.2.6 Examples . . . . . . . . . . . . . . . . . . . . . . . 78
4.2.7 Two-Time-Scale Expansion . . . . . . . . . . . . . 81
4.3 Markov Chains with Multiple Weakly Irreducible Classes 84
4.3.1 Asymptotic Expansions . . . . . . . . . . . . . . . 88
4.3.2 Analysis of Remainder . . . . . . . . . . . . . . . . 101
4.3.3 Computational Procedure: User’s Guide . . . . . . 102
Contents ix
4.3.4 Summary of Results . . . . . . . . . . . . . . . . . 102
4.3.5 An Example . . . . . . . . . . . . . . . . . . . . . 104
4.4 Inclusion of Absorbing States . . . . . . . . . . . . . . . . 107
4.5 Inclusion of Transient States. . . . . . . . . . . . . . . . . 115
4.6 Remarks on Countable-State-Space Cases . . . . . . . . . 126
4.6.1 Countable-State Spaces: Part I . . . . . . . . . . . 126
4.6.2 Countable-State Spaces: Part II . . . . . . . . . . . 129
4.6.3 A Remark on Finite-Dimensional
Approximation . . . . . . . . . . . . . . . . . . . . 132
4.7 Remarks on Singularly Perturbed Diffusions . . . . . . . . 133
4.8 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5 Occupation Measures: Asymptotic Properties
and Ramification 141
5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 141
5.2 The Irreducible Case . . . . . . . . . . . . . . . . . . . . . 142
5.2.1 Occupation Measure . . . . . . . . . . . . . . . . . 143
5.2.2 Conditions and Preliminary Results . . . . . . . . 143
5.2.3 Exponential Bounds . . . . . . . . . . . . . . . . . 148
5.2.4 Asymptotic Normality . . . . . . . . . . . . . . . . 159
5.2.5 Extensions . . . . . . . . . . . . . . . . . . . . . . 169
5.3 Markov Chains with Weak and Strong Interactions . . . . 173
5.3.1 Aggregationof Markov Chains . . . . . . . . . . . 174
5.3.2 Exponential Bounds . . . . . . . . . . . . . . . . . 182
5.3.3 Asymptotic Distributions . . . . . . . . . . . . . . 191
5.4 Measurable Generators . . . . . . . . . . . . . . . . . . . . 213
5.5 Remarks on Inclusion of Transient
and Absorbing States . . . . . . . . . . . . . . . . . . . . 222
5.5.1 Inclusion of Transient States . . . . . . . . . . . . 222
5.5.2 Inclusion of Absorbing States . . . . . . . . . . . . 225
5.6 Remarks on a Stability Problem . . . . . . . . . . . . . . 229
5.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
6 Asymptotic Expansions of Solutions for
Backward Equations 235
6.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 235
6.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . 236
6.2.1 A Preliminary Lemma . . . . . . . . . . . . . . . . 236
6.2.2 Formulation . . . . . . . . . . . . . . . . . . . . . . 237
6.3 Construction of Asymptotic Expansions . . . . . . . . . . 238
6.3.1 Leading Term ϕ (t) and Zero-Order
0
Terminal-LayerTerm ψ (τ) . . . . . . . . . . . . . 241
0
6.3.2 Higher-Order Terms . . . . . . . . . . . . . . . . . 243
6.4 Error Estimates . . . . . . . . . . . . . . . . . . . . . . . . 246