Markov Chain Models — Rarity and Exponentiality

198 Pages·1979·8.677 MB·English

by  Julian Keilson (auth.), Julian Keilson (eds.)

Applied
Mathematical
Sciences
EDITORS
Fritz John Lawrence Sirovich
Courant Institute 01 Division 01
Mathematical Sciences Applied Mathematics
New York University Brown University
New York, N.Y. 10012 Providence, R.I. 02912
Joseph P. LaSalle Gerald B. Whitham
Division of Applied Mathematics
Applied Mathematics Firestone Laboratory
Brown University California Institute 01 Technology
Providence, R.I. 02912 Pasadena,CA.91125
EDITORIAL STATEMENT
The mathematization of all sciences, the fading of traditional scientific bounda
ries, the impact of computer technology, the growing importance of mathematical
computer modelling and the necessity of scientific planning all create the need both
in education and research for books that are introductory to and abreast of these
developments.
The purpose of this series is to provide such books, suitable for the user of
mathematics, the mathematician interested in applications, and the student scientist.
In particular, this series will provide an outlet for material less formally presented and
more anticipatory of needs than finished texts or monographs, yet of immediate in
terest because of the novelty of its treatment of an application or of mathematics
being applied or lying close to applications.
The aim of the series is, through rapid publication in an attractive but inexpen
sive format, to make material of current interest widely accessible. This implies the
absence of excessive generality and abstraction, and unrealistic idealization, but with
quality of exposition as a goal.
Many of the books will originate out of and will stimulate the development of
new undergraduate and graduate courses in the applications of mathematics. Some
of the books will present introductions to new areas of research, new applications
and act as signposts for new directions in the mathematical sciences. This series will
often serve as an intermediate stage of the publication of material which, through
exposure here, will be further developed and refined. These will appear in conven
tional format and in hard cover.
MANUSCRIPTS
The Editors welcome all inquiries regarding the submission of manuscripts for
the series. Final preparation of all manuscripts will take place in the editorial offices
of the series in the Division of Applied Mathematics, Brown University, Providence,
Rhode Island.
SPRINGER-VERLAG NEW YORK INC., 175 Fifth Avenue, New York, N. Y. 10010
I
Applied Mathematical Sciences Volume 28
Julian Keilson
Markov Chain Models
Rarity
and Exponentiality
Springer-Verlag New York Heidelberg Berlin
Julian Keilson
The University of Rochester
Rochester, New York 14627
USA
AMS Subject Classification: 60 J 10
Library of Congress Cataloging In Publication Data
Keilson, Julian.
Markov chain models-rarity and exponentiality.
(Applied mathematical sciences; v. 28)
Bibliography: p.
Includes index.
1. Markov processes. I. Title. II. Series.
QAI.A647 vol. 28 [QA274.7] 510' .8s [519.2'33]79-10967
All rights reserved.
No part of this book may be translated or reproduced in any form
without written permission from Springer-Verlag.
© 1979 by Springer-Verlag New York Inc.
9 8 7 6 543 2 1
ISBN-13: 978-0-387-90405-4 e-ISBN-IJ: 978-1-4612-6200-8
001: 10.IOO7!978-1-46I2-6200-8
Foreword and Acknowledgment
The work here offered builds on ideas and results dev
eloped over the last ten years. These ideas have been motiva
ted by questions arising in the study of neuron firing, human
metabolism, congestion, inventories and system reliability. A
series of informal lectures on these ideas was presented during
the academic year 1973-74 while visiting the Department of
Statistics at Stanford University and the Department of Mathe
matics at the University of Helsinki. The lectures were dev
eloped further during a visit to the Operations Research Center
at Berkeley in the summer of 1974 into a somewhat tidier and
more extensive form, and subsequent changes have been minor.
The object of the effort is the development and pres
entation of the working tools needed to quantify the ergodic
and transient behavior of systems of many degrees of freedom
such as arise, for example, in the study of system reliabil
ity. The structure of passage time densities and exit time
densities to and from subsets of the state space is discussed
at length, with special emphasis on the exponentiality and re
lated structural properties present in these densities when
the sets visited are seen infrequently. The underlying themes
of reversibility in time and complete monotonicity ar,e of
particular importance. They are at the heart of much of the
tractability. It is assumed that the reader is familiar with
the elements of probability theory and stochastic processes
as found, for example, in Feller, Vol. I, and parts of Feller,
Vol. II. By and large, the material is self-contained.
The effort needed to present the material would not
have been made without the encouragement of Professor R. E.
v
vi
Barlow of Berkeley, whose infectious enthusiasm and stimu
lating discussions are gratefully acknowledged.
The labor involved was supported in large part by the
technical and editorial effort of many associates and, in
particular, of M. Heikkila, A. Kester, U. Sum ita , and Y.
Hayashi. Their help has been essential to the end product.
The editorial assistance of L. Ziegenfuss in the final pre
paration of the manuscript is gratefully acknowledged.
Appreciation is expressed for the contributions of
Professor S. C. Graves of M.l.T., Professor D. R. Smith of
Columbia University, and Professor E. Arjas of Oulu. Thanks
are also tended to Professor M. Brown of the City University
of New York, Professor R. Syski of Maryland, and Professor
W. Whitt of Yale University, for helpful discussions in re
cent years. Finally, I would like to express my appreciation
to the Office of Naval Research for its support, direct and
indirect, of this work.
Notation
!!:" £.' E., etc. column vectors
T T T
a£., E. , etc. row vectors
1 column vector with all components 1
a, b, P, etc. square matrices
aT , b T , P T , etc. transposed matrices
I identity matrix
state space
Markov chain in discrete time
a single-step transition probability
matrix for Nk
k-step transition probability from m
to n
state probability vector after k steps
Markov chain in continuous time
transition rate matrix for NCt) where
vmn is the transition rate from m to n
transition probability matrix for N(t)
for elapsed time T
E.(t) state probability vector for N(t) at t
ergodic probability vector
diagonal matrix with the n-th diagonal
component en of e
diagonal matrix with the n-th diagonal
component el/2
n
dyadic matrix, i.e. , a rank one matrix
with C-ab-T) mn ambn
/1'= G+B partition of the state space into a
set G (good) and a set B (bad)
submatrix of a on GxG
£G subvector of E. on G
vector of l's with cardinality of G
~
E [T], T, VT expectation of the random variable T
vii
viii
TK K-th moment of T
a2 variance of T
T
5T (t) probability density function of T
Stet) cumulative density function of T, i.e.,
f:""
Stet) = sT(x)dx
survival function of T, i.e.,
~T(t) = f"" sT(x)dx
t
the Laplace transform of Stet), i.e.,
f""
aT(s) = e-stsT(t)dt
o
'Ir(s) .s!'{p(t)} the Laplace transform of the matrix
pet), i.e.,
('Ir(s)) = f"" e-st(p(t)) dt, all t
mn 0 mn
a(t)*b(t) convolution of aCt) and bet), i.e.,
f~""
a(t)*b(t) a(x)b(t-x)dx
f:
or a(t)*b(t) = a(x)b(t-x)dx
when appropriate
aCt) -bet), t .... "" aCt) is asymptotically equal to bCt)
as t .... "", i.e., a(t)/b(t) .... I as
t .... ""
Contents
Page
FOREWORD AND ACKNOWLEDGMENT................ . . • • . . . . . . . . v
NOTATION .......••........•........•....•...•...•••...•. vii
CHAPTER O. INTRODUCTION AND SUMMARY. ..................• 1
CHAPTER 1. DISCRETE TIME MARKOV CHAINS; REVERSIBILITY
IN TIME..................................... 15
§1.00. Introduction ....... ; ................ 15
§1.0. Notation, Transition Laws .••.•.....• 15
§l.l. Irreducibility, Aperiodicity,
Ergodicity; Stationary Chains ....••• 16
§1.2. Approach to Ergodicity; Spectral
Structure, Perron-Romanovsky-
Frobenius Theorem ....•..••..•....•.. 17
§1.3. Time-Reversible Chains •••.••.••.••.• 18
CHAPTER 2. MARKOV CHAINS IN CONTINUOUS TIME; UNIFORMIZA
TION; REVERSIBILITy......................... 20
§2.00. Introduction .••.....• ,.............. 20
§2.0. Notation, Transition Laws; A Review. 20
§2.l. Uniformizable Chains - A Bridge
Between Discrete and Continuous Time
Chains. ..•...•....•.... •... . ..•.•.•• 22
§2.2. Advantages and Prevalence of Uni
formizable Chains .............••.... 24
§2.3. Ergodicity for Continuous Time
Chains.. ... .•. .. . . .•••• ...••. .. .•. . . 25
§2.4. Reversibility for Ergodic Markov
Chains in Continuous Time ......•.•.. 26
§2.5. Prevalence of Time-Reversibility •..• 27
CHAPTER 3. MORE ON TIME-REVERSIBILITY; POTENTIAL COEFF
ICIENTS; PROCESS MODIFICATION ...•...••••.... 31
§3.00. Introduction........................ 31
§3.l. The Advantages of Time-Reversibility 32
§3.2. The Spectral Representation ......... 32
ix
x
Page
§3.3. Potentials; Spectral Representa-
tion ............................... . 35
B.4. More General Time-Reversible Chains. 38
§3. 5. Process Modifications Preserving
Reversi bi1i ty ................•...... 38
§3.6. Replacement Processes .............. . 41
CHAPTER 4. POTENTIAL THEORY, REPLACEMENT, AND COMPENSA-
TION.. . . ... .. . . .. .. .. . . ... .. .... . . ..... .. .. . 43
§4.00. Introduction........................ 43
§4.1. The Green Potential................. 44
§4.2. The Ergodic Distribution for a Re-
placement Process................... 45
§4.3. The Compensation Method............. 47
§4.4. Notation for the Homogeneous Random
Walk................................ 47
§4.5. The Compensation Method Applied to
the Homogeneous Random Walk Modified
by Boundaries. ...................... 49
§4.6. Advantages of the Compensation
Method. An Illustrative Example..... 51
§4.7. Exploitation of the Structure of the
Green Potential for the Homogeneous
Random Walk. . . . . . . .. . . . . . .. . . . . . .. . . 53
§4.8. Similar Situations.................. 56
CHAPTER 5. PASSAGE TIME DENSITIES IN BIRTH-DEATH
PROCESSES; DISTRIBUTION STRUCTURE........... 57
§5.00. Introduction........................ 57
§5.1. Passage Time Densities for Birth-
Death Processes..................... 57
Passage Time Moments for a Birth-
§5~2.
Death Process....................... 61
§5.3. PF=, Complete Monotonicity, Log
Concavity and Log-Convexity......... 63
§S.4. Complete Monotonicity and Log-
Convexi ty. . . . . . . . . . . . . . . . . . . . . . . . . . . 66
§5.5. Complete Monotonicity in Time-
Reversible Processes................ 67