Table Of Content

Stochastic Modelling Stochastic Mechanics Random Media and Applied Probability Signal Processing and Image Synthesis (Formerly: Mathematical Economics and Finance ApplicationsofMathematics) Stochastic Optimization Stochastic Control Stochastic Models in Life Sciences 37 Edited by B.Rozovski˘ı P.W.Glynn Advisory Board M.Hairer I. Karatzas F. Kelly A. Kyprianou Y. LeJan B. Øksendal G. Papanicolaou E.Pardoux E.Perkins H.M.Soner For further volumes: http://www.springer.com/series/602 G. George Yin • Qing Zhang Continuous-Time Markov Chains and Applications A Two-Time-Scale Approach Second edition 123 G. George Yin Qing Zhang Department of Mathematics Department of Mathematics WayneState University University of Georgia Detroit, Michigan Athens,Georgia USA USA Managing Editors B. Rozovski˘ı Peter W. Glynn Division of Applied Mathematics Instituteof Computational Brown University and Mathematical Engineering Providence, RI Stanford University USA Stanford, CA USA ISSN 0172-4568 ISBN 978-1-4614-4345-2 ISBN 978-1-4614-4346-9 (eBook) DOI 10.1007/978-1-4614-4346-9 Springer New York Heidelberg Dordrecht London Library of Congress Control Number:2012945403 Mathematics Subject Classification (2010): 60J27, 93E20, 34E05 (cid:2)c SpringerScience+Business Media, LLC 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similaror dissimilarmethodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts inconnectionwithreviewsorscholarlyanalysisormaterialsuppliedspecificallyforthe purpose of being entered and executed on a computer system, for exclusive use by the purchaserofthework.Duplicationofthispublicationorpartsthereofispermittedonly under the provisions of the Copyright Law of the Publisher’s location, in its current version,andpermissionforusemustalwaysbeobtainedfromSpringer.Permissionsfor usemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations areliabletoprosecutionundertherespectiveCopyrightLaw. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc. inthispublicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuch names are exempt from the relevant protective laws and regulations and therefore free forgeneraluse. While the advice and informationinthis book arebelieved to be trueand accurate at thedateofpublication,neithertheauthorsnortheeditorsnorthepublishercanaccept any legal responsibility for any errors or omissions that may be made. The publisher makes nowarranty,expressorimplied,withrespecttothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) To Our Mentors Harold J. Kushner and Wendell H. Fleming Contents Preface xiii Preface to the First Edition xv Convention xvii Notation xix Part I: Prologue and Preliminaries 1 1 Introduction and Overview 3 1.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 A Brief Survey . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.1 Markov Chains . . . . . . . . . . . . . . . . . . . . 10 1.2.2 Singular Perturbations . . . . . . . . . . . . . . . . 11 1.3 Outline of the Book . . . . . . . . . . . . . . . . . . . . . 12 2 Mathematical Preliminaries 17 2.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 Martingales . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.3 Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . 18 2.4 Piecewise-Deterministic Processes. . . . . . . . . . . . . . 21 2.4.1 Construction of Markov Chains . . . . . . . . . . . 21 2.5 Irreducibility and Quasi-Stationary Distributions . . . . . 23 vii viii Contents 2.6 Gaussian Processes and Diffusions . . . . . . . . . . . . . 25 2.7 Switching Diffusions . . . . . . . . . . . . . . . . . . . . . 27 2.8 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3 Markovian Models 31 3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.2 Birth and Death Processes. . . . . . . . . . . . . . . . . . 32 3.3 Finite-State Space Models . . . . . . . . . . . . . . . . . . 34 3.3.1 Queues with Finite Capacity . . . . . . . . . . . . 34 3.3.2 System Reliability . . . . . . . . . . . . . . . . . . 37 3.3.3 Competing Risk Theory . . . . . . . . . . . . . . . 39 3.3.4 Two-Time-Scale Cox Processes . . . . . . . . . . . 40 3.3.5 Random Evolutions . . . . . . . . . . . . . . . . . 41 3.3.6 Seasonal Variation Models . . . . . . . . . . . . . . 42 3.4 Stochastic Optimization Problems . . . . . . . . . . . . . 45 3.4.1 Simulated Annealing . . . . . . . . . . . . . . . . . 45 3.4.2 Continuous-Time Stochastic Approximation . . . . 46 3.4.3 Systems with MarkovianDisturbances . . . . . . . 48 3.5 Linear Systems with Jump Markov Disturbance . . . . . . 49 3.5.1 Linear Quadratic Control Problems . . . . . . . . 49 3.5.2 Singularly Perturbed LQ Systems with Wide-Band Noise . . . . . . . . . . . . . . . . . . . . . . . . . 50 3.5.3 Large-Scale Systems: Decomposition and Aggregation . . . . . . . . . . . . . . . . . . . 51 3.6 Time-Scale Separation . . . . . . . . . . . . . . . . . . . . 53 3.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Part II: Two-Time-Scale Markov Chains 57 4 Asymptotic Expansions of Solutions for Forward Equations 59 4.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 59 4.2 Irreducible Case . . . . . . . . . . . . . . . . . . . . . . . 62 4.2.1 Asymptotic Expansions . . . . . . . . . . . . . . . 63 4.2.2 Outer Expansion . . . . . . . . . . . . . . . . . . . 66 4.2.3 Initial-Layer Correction . . . . . . . . . . . . . . . 69 4.2.4 Exponential Decay of ψk(·) . . . . . . . . . . . . . 72 4.2.5 Asymptotic Validation . . . . . . . . . . . . . . . . 74 4.2.6 Examples . . . . . . . . . . . . . . . . . . . . . . . 78 4.2.7 Two-Time-Scale Expansion . . . . . . . . . . . . . 81 4.3 Markov Chains with Multiple Weakly Irreducible Classes 84 4.3.1 Asymptotic Expansions . . . . . . . . . . . . . . . 88 4.3.2 Analysis of Remainder . . . . . . . . . . . . . . . . 101 4.3.3 Computational Procedure: User’s Guide . . . . . . 102 Contents ix 4.3.4 Summary of Results . . . . . . . . . . . . . . . . . 102 4.3.5 An Example . . . . . . . . . . . . . . . . . . . . . 104 4.4 Inclusion of Absorbing States . . . . . . . . . . . . . . . . 107 4.5 Inclusion of Transient States. . . . . . . . . . . . . . . . . 115 4.6 Remarks on Countable-State-Space Cases . . . . . . . . . 126 4.6.1 Countable-State Spaces: Part I . . . . . . . . . . . 126 4.6.2 Countable-State Spaces: Part II . . . . . . . . . . . 129 4.6.3 A Remark on Finite-Dimensional Approximation . . . . . . . . . . . . . . . . . . . . 132 4.7 Remarks on Singularly Perturbed Diffusions . . . . . . . . 133 4.8 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 5 Occupation Measures: Asymptotic Properties and Ramification 141 5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 141 5.2 The Irreducible Case . . . . . . . . . . . . . . . . . . . . . 142 5.2.1 Occupation Measure . . . . . . . . . . . . . . . . . 143 5.2.2 Conditions and Preliminary Results . . . . . . . . 143 5.2.3 Exponential Bounds . . . . . . . . . . . . . . . . . 148 5.2.4 Asymptotic Normality . . . . . . . . . . . . . . . . 159 5.2.5 Extensions . . . . . . . . . . . . . . . . . . . . . . 169 5.3 Markov Chains with Weak and Strong Interactions . . . . 173 5.3.1 Aggregationof Markov Chains . . . . . . . . . . . 174 5.3.2 Exponential Bounds . . . . . . . . . . . . . . . . . 182 5.3.3 Asymptotic Distributions . . . . . . . . . . . . . . 191 5.4 Measurable Generators . . . . . . . . . . . . . . . . . . . . 213 5.5 Remarks on Inclusion of Transient and Absorbing States . . . . . . . . . . . . . . . . . . . . 222 5.5.1 Inclusion of Transient States . . . . . . . . . . . . 222 5.5.2 Inclusion of Absorbing States . . . . . . . . . . . . 225 5.6 Remarks on a Stability Problem . . . . . . . . . . . . . . 229 5.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 6 Asymptotic Expansions of Solutions for Backward Equations 235 6.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . 235 6.2 Problem Formulation . . . . . . . . . . . . . . . . . . . . . 236 6.2.1 A Preliminary Lemma . . . . . . . . . . . . . . . . 236 6.2.2 Formulation . . . . . . . . . . . . . . . . . . . . . . 237 6.3 Construction of Asymptotic Expansions . . . . . . . . . . 238 6.3.1 Leading Term ϕ (t) and Zero-Order 0 Terminal-LayerTerm ψ (τ) . . . . . . . . . . . . . 241 0 6.3.2 Higher-Order Terms . . . . . . . . . . . . . . . . . 243 6.4 Error Estimates . . . . . . . . . . . . . . . . . . . . . . . . 246