A first course in stochastic models

482 Pages·2003·2.071 MB·English

by  Henk C. Tijms

A First Course
in Stochastic Models
Henk C. Tijms
Vrije Universiteit, Amsterdam, The Netherlands
Copyright(cid:1)c 2003 JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,
WestSussexPO198SQ,England
Telephone[phone]
Email(forordersandcustomerserviceenquiries):[e-mail]
VisitourHomePageonwww.wileyeurope.comorwww.wiley.com
AllRightsReserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystemor
transmittedinanyformorbyanymeans,electronic,mechanical,photocopying,recording,scanning
orotherwise,exceptunderthetermsoftheCopyright,DesignsandPatentsAct1988orunderthe
termsofalicenceissuedbytheCopyrightLicensingAgencyLtd,90TottenhamCourtRoad,London
W1T4LP,UK,withoutthepermissioninwritingofthePublisher.RequeststothePublishershould
beaddressedtothePermissionsDepartment,JohnWiley&SonsLtd,TheAtrium,SouthernGate,
Chichester,WestSussexPO198SQ,England,[e-mail],orfaxedto(+44)
1243770620.
Thispublicationisdesignedtoprovideaccurateandauthoritativeinformationinregardtothesubject
mattercovered.ItissoldontheunderstandingthatthePublisherisnotengagedinrendering
professionalservices.Ifprofessionaladviceorotherexpertassistanceisrequired,theservicesofa
competentprofessionalshouldbesought.
OtherWileyEditorialOffices
JohnWiley&SonsInc.,111RiverStreet,Hoboken,NJ07030,USA
Jossey-Bass,989MarketStreet,SanFrancisco,CA94103-1741,USA
Wiley-VCHVerlagGmbH,Boschstr.12,D-69469Weinheim,Germany
JohnWiley&SonsAustraliaLtd,33ParkRoad,Milton,Queensland4064,Australia
JohnWiley&Sons(Asia)PteLtd,2ClementiLoop#02-01,JinXingDistripark,Singapore129809
JohnWiley&SonsCanadaLtd,22WorcesterRoad,Etobicoke,Ontario,CanadaM9W1L1
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappears
inprintmaynotbeavailableinelectronicbooks.
LibraryofCongressCataloging-in-PublicationData
Tijms,H.C.
Afirstcourseinstochasticmodels/HenkC.Tijms.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN0-471-49880-7(acid-freepaper)—ISBN0-471-49881-5(pbk.:acid-freepaper)
1.Stochasticprocesses.I.Title.
QA274.T462003
(cid:2)
519.23—dc21
2002193371
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
ISBN0-471-49880-7(Cloth)
ISBN0-471-49881-5(Paper)
Typesetin10/12ptTimesfromLATEXfilessuppliedbytheauthor,byLaserwordsPrivateLimited,
Chennai,India
PrintedandboundinGreatBritainbyTJInternationalLtd,Padstow,Cornwall
Thisbookisprintedonacid-freepaperresponsiblymanufacturedfromsustainableforestry
inwhichatleasttwotreesareplantedforeachoneusedforpaperproduction.
Contents
Preface ix
1 The Poisson Process and Related Processes 1
1.0 Introduction 1
1.1 ThePoissonProcess 1
1.1.1 TheMemorylessProperty 2
1.1.2 MergingandSplittingofPoissonProcesses 6
1.1.3 TheM/G/∞ Queue 9
1.1.4 ThePoissonProcessand theUniformDistribution 15
1.2 CompoundPoissonProcesses 18
1.3 Non-StationaryPoissonProcesses 22
1.4 MarkovModulatedBatchPoissonProcesses 24
Exercises 28
BibliographicNotes 32
References 32
2 Renewal-Reward Processes 33
2.0 Introduction 33
2.1 Renewal Theory 34
2.1.1 TheRenewal Function 35
2.1.2 TheExcess Variable 37
2.2 Renewal-RewardProcesses 39
2.3 TheFormulaofLittle 50
2.4 PoissonArrivalsSeeTimeAverages 53
2.5 ThePollaczek–KhintchineFormula 58
2.6 A ControlledQueuewithRemovableServer 66
2.7 An Up-AndDowncrossingTechnique 69
Exercises 71
BibliographicNotes 78
References 78
3 Discrete-Time Markov Chains 81
3.0 Introduction 81
3.1 TheModel 82
vi CONTENTS
3.2 TransientAnalysis 87
3.2.1 AbsorbingStates 89
3.2.2 Mean First-PassageTimes 92
3.2.3 TransientandRecurrent States 93
3.3 TheEquilibriumProbabilities 96
3.3.1 Preliminaries 96
3.3.2 TheEquilibriumEquations 98
3.3.3 TheLong-runAverageReward perTimeUnit 103
3.4 ComputationoftheEquilibriumProbabilities 106
3.4.1 MethodsforaFinite-StateMarkovChain 107
3.4.2 GeometricTail Approachforan InfiniteStateSpace 111
3.4.3 Metropolis—HastingsAlgorithm 116
3.5 Theoretical Considerations 119
3.5.1 StateClassification 119
3.5.2 ErgodicTheorems 126
Exercises 134
BibliographicNotes 139
References 139
4 Continuous-Time Markov Chains 141
4.0 Introduction 141
4.1 TheModel 142
4.2 TheFlowRateEquationMethod 147
4.3 ErgodicTheorems 154
4.4 MarkovProcesses onaSemi-InfiniteStrip 157
4.5 TransientStateProbabilities 162
4.5.1 TheMethodofLinear DifferentialEquations 163
4.5.2 TheUniformizationMethod 166
4.5.3 FirstPassage TimeProbabilities 170
4.6 TransientDistributionof CumulativeRewards 172
4.6.1 TransientDistributionofCumulativeSojournTimes 173
4.6.2 TransientRewardDistributionfortheGeneral Case 176
Exercises 179
BibliographicNotes 185
References 185
5 Markov Chains and Queues 187
5.0 Introduction 187
5.1 TheErlang DelayModel 187
5.1.1 TheM/M/1 Queue 188
5.1.2 TheM/M/c Queue 190
5.1.3 TheOutputProcessand TimeReversibility 192
5.2 LossModels 194
5.2.1 TheErlangLossModel 194
5.2.2 TheEngset Model 196
5.3 Service-SystemDesign 198
5.4 Insensitivity 202
5.4.1 A ClosedTwo-nodeNetworkwithBlocking 203
5.4.2 TheM/G/1 QueuewithProcessor Sharing 208
5.5 A PhaseMethod 209
CONTENTS vii
5.6 QueueingNetworks 214
5.6.1 OpenNetworkModel 215
5.6.2 ClosedNetworkModel 219
Exercises 224
BibliographicNotes 230
References 231
6 Discrete-Time Markov Decision Processes 233
6.0 Introduction 233
6.1 TheModel 234
6.2 ThePolicy-ImprovementIdea 237
6.3 TheRelativeValueFunction 243
6.4 Policy-IterationAlgorithm 247
6.5 Linear ProgrammingApproach 252
6.6 Value-IterationAlgorithm 259
6.7 ConvergenceProofs 267
Exercises 272
BibliographicNotes 275
References 276
7 Semi-Markov Decision Processes 279
7.0 Introduction 279
7.1 TheSemi-MarkovDecisionModel 280
7.2 Algorithmsforan OptimalPolicy 284
7.3 ValueIterationand FictitiousDecisions 287
7.4 OptimizationofQueues 290
7.5 One-StepPolicyImprovement 295
Exercises 300
BibliographicNotes 304
References 305
8 Advanced Renewal Theory 307
8.0 Introduction 307
8.1 TheRenewal Function 307
8.1.1 TheRenewal Equation 308
8.1.2 ComputationoftheRenewal Function 310
8.2 AsymptoticExpansions 313
8.3 AlternatingRenewalProcesses 321
8.4 RuinProbabilities 326
Exercises 334
BibliographicNotes 337
References 338
9 Algorithmic Analysis of Queueing Models 339
9.0 Introduction 339
9.1 BasicConcepts 341
viii CONTENTS
9.2 TheM/G/1 Queue 345
9.2.1 TheStateProbabilities 346
9.2.2 TheWaiting-TimeProbabilities 349
9.2.3 BusyPeriodAnalysis 353
9.2.4 WorkinSystem 358
9.3 TheMX/G/1 Queue 360
9.3.1 TheStateProbabilities 361
9.3.2 TheWaiting-TimeProbabilities 363
9.4 M/G/1 QueueswithBoundedWaitingTimes 366
9.4.1 TheFinite-BufferM/G/1 Queue 366
9.4.2 AnM/G/1 QueuewithImpatientCustomers 369
9.5 TheGI/G/1 Queue 371
9.5.1 Generalized ErlangianServices 371
9.5.2 Coxian-2Services 372
9.5.3 TheGI/Ph/1 Queue 373
9.5.4 ThePh/G/1 Queue 374
9.5.5 Two-momentApproximations 375
9.6 Multi-ServerQueueswithPoissonInput 377
9.6.1 TheM/D/c Queue 378
9.6.2 The M/G/c Queue 384
9.6.3 TheMX/G/c Queue 392
9.7 TheGI/G/c Queue 398
9.7.1 TheGI/M/c Queue 400
9.7.2 TheGI/D/c Queue 406
9.8 Finite-CapacityQueues 408
9.8.1 TheM/G/c/c+N Queue 408
9.8.2 A BasicRelationfortheRejectionProbability 410
9.8.3 TheMX/G/c/c+N QueuewithBatchArrivals 413
9.8.4 Discrete-TimeQueueingSystems 417
Exercises 420
BibliographicNotes 428
References 428
Appendices 431
Appendix A. Useful Tools in Applied Probability 431
Appendix B. Useful Probability Distributions 440
Appendix C. Generating Functions 449
Appendix D. The Discrete Fast Fourier Transform 455
Appendix E. Laplace Transform Theory 458
Appendix F. Numerical Laplace Inversion 462
Appendix G. The Root-Finding Problem 470
References 474
Index 475
Preface
The teaching of applied probability needs a fresh approach. The field of applied
probability has changed profoundly in the past twenty years and yet the textbooks
in use today do not fully reflect the changes. The development of computational
methods has greatly contributed to a better understanding of the theory. It is my
conviction that theory is better understood when the algorithms that solve the
problems the theory addresses are presented at the same time. This textbook tries
to recognize what the computer can do without letting the theory be dominated
by the computational tools. In some ways, the book is a successor of my earlier
book StochasticModelingandAnalysis. However, the set-up of the present text is
completelydifferent.Thetheoryhasamorecentralplaceandprovidesaframework
inwhichtheapplicationsfit.Withoutasolidbasisintheory,noapplicationscanbe
solved. The book is intended as a first introduction to stochastic models for senior
undergraduate students in computer science, engineering, statistics and operations
research, among others. Readers of this book are assumed to be familiar with the
elementary theory of probability.
I am grateful to my academic colleagues Richard Boucherie, Avi Mandelbaum,
Rein Nobel and Rien van Veldhuizen for their helpful comments, and to my stu-
dentsGayaBranderhorst,TonDieker,BorusJungbackerandSanneZwartfortheir
detailed checking of substantial sections of the manuscript. Julian Rampelmann
and Gloria Wirz-Wagenaar were helpful in transcribing my handwritten notes into
a nice Latex manuscript.
Finally, users of the book can find supporting educational software for Markov
chains and queues on my website http://staff.feweb.vu.nl/tijms.
CHAPTER 1
The Poisson Process and
Related Processes
1.0 INTRODUCTION
The Poisson process is a counting process that counts the number of occurrences
of some specific event through time. Examples include the arrivals of customers
at a counter, the occurrences of earthquakes in a certain region, the occurrences
of breakdowns in an electricity generator, etc. The Poisson process is a natural
modellingtoolinnumerousappliedprobabilityproblems.Itnotonlymodelsmany
real-world phenomena, but the process allows for tractable mathematical analysis
as well.
The Poisson process is discussed in detail in Section 1.1. Basic properties are
derived including the characteristic memoryless property. Illustrative examples are
given to show the usefulness of the model. The compound Poisson process is
dealt with in Section 1.2. In a Poisson arrival process customers arrive singly,
while in a compound Poisson arrival process customers arrive in batches. Another
generalization of the Poisson process is the non-stationary Poisson process that is
discussed in Section 1.3. The Poisson process assumes that the intensity at which
eventsoccuristime-independent.Thisassumptionisdroppedinthenon-stationary
Poisson process. The final Section 1.4 discusses the Markov modulated arrival
process in which the intensity at which Poisson arrivals occur is subject to a
random environment.
1.1 THE POISSON PROCESS
ThereareseveralequivalentdefinitionsofthePoissonprocess.Ourstartingpointis
a sequence X ,X ,... of positive, independent random variables with a common
1 2
probabilitydistribution.ThinkofXn asthetimeelapsedbetweenthe(n−1)thand
nth occurrence of some specific event in a probabilistic situation. Let
(cid:1)n
S0 =0 and Sn = Xk, n=1,2,... .
k=1
AFirstCourseinStochasticModelsH.C.Tijms
(cid:1)c 2003JohnWiley&Sons,Ltd.ISBNs:0-471-49880-7(HB);0-471-49881-5(PB)
2 THEPOISSONPROCESSANDRELATEDPROCESSES
Then Sn is the epoch at which the nth event occurs. For each t ≥ 0, define the
random variable N(t) by
N(t)=the largest integer n≥0 for which Sn ≤t.
The random variable N(t) represents the number of events up to time t.
Definition 1.1.1 The counting process {N(t), t ≥ 0} is called a Poisson process
with rate λ if the interoccurrence times X ,X ,... have a common exponential
1 2
distributionfunction
P{Xn ≤x}=1−e−λx, x ≥0.
The assumption of exponentially distributed interoccurrence times seems to be
restrictive, but it appears that the Poisson process is an excellent model for many
real-world phenomena. The explanation lies in the following deep result that is
onlyroughlystated;seeKhintchine(1969)forthepreciserationaleforthePoisson
assumptioninavarietyofcircumstances(thePalm–Khintchinetheorem).Suppose
that at microlevel there are a very large number of independent stochastic pro-
cesses, where each separate microprocess generates only rarely an event. Then
at macrolevel the superposition of all these microprocesses behaves approximately
as a Poisson process. This insightful result is analogous to the well-known result
that the number of successes in a very large number of independent Bernoulli
trials with a very small success probability is approximately Poisson distributed.
The superposition result provides an explanation of the occurrence of Poisson
processes in a wide variety of circumstances. For example, the number of calls
received at a large telephone exchange is the superposition of the individual calls
ofmanysubscriberseachcallinginfrequently.Thustheprocessdescribingtheover-
all number of calls can be expected to be close to a Poisson process. Similarly, a
Poisson demand process for a given product can be expected if the demands are
the superposition of the individual requests of many customers each asking infre-
quently for that product. Below it will be seen that the reason of the mathematical
tractability of the Poisson process is its memoryless property. Information about
the time elapsed since the last event is not relevant in predicting the time until the
next event.
1.1.1 The Memoryless Property
In the remainder of this section we use for the Poisson process the terminology of
‘arrivals’ instead of ‘events’. We first characterize the distribution of the counting
variable N(t). To do so, we use the well-known fact that the sum of k inde-
pendent random variables with a common exponential distribution has an Erlang
distribution. That is,