Markov Chains : Models, Algorithms and Applications

258 Pages·2013·1.938 MB·English

by  Wai-Ki Ching, Ximin Huang, Michael K. Ng, Tak-Kuen Siu

International Series in Operations
Research & Management Science
Volume 189
SeriesEditor
FrederickS.Hillier
StanfordUniversity,CA,USA
SpecialEditorialConsultant
CamilleC.Price
StephenF.AustinStateUniversity,TX,USA
Forfurthervolumes:
http://www.springer.com/series/6161

Wai-Ki Ching • Ximin Huang
Michael K. Ng (cid:129) Tak-Kuen Siu
Markov Chains
Models, Algorithms and Applications
Second Edition
123
Wai-KiChing XiminHuang
DepartmentofMathematics CollegeofManagement
TheUniversityofHongKong GeorgiaInstituteofTechnology
HongKong,SAR Atlanta,Georgia,USA
MichaelK.Ng Tak-KuenSiu
DepartmentofMathematics CassBusinessSchool
HongKongBaptistUniversity CityUniversityLondon
KowloonTong London
HongKongSAR UnitedKingdom
ISSN0884-8289
ISBN978-1-4614-6311-5 ISBN978-1-4614-6312-2(eBook)
DOI10.1007/978-1-4614-6312-2
SpringerNewYorkHeidelbergDordrechtLondon
LibraryofCongressControlNumber:2013931264
©SpringerScience+BusinessMediaNewYork2006,2013
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
To Mandyand myParents
Wai-KiChing
To myParents
XiminHuang
To Anna,Ceciliaand myParents
MichaelK. Ng
To CandyandmyParents
Tak-Kuen Siu

Preface
TheaimofthisbookistooutlinetherecentdevelopmentofMarkovchainmodels
andtheir applicationsin queueingsystems, manufacturingsystems, remanufactur-
ing systems, inventory systems, ranking the importance of a web site, and also
financialriskmanagement.
Thisbookconsistsofeightchapters.InChapter1, we givea briefintroduction
to the classical theory on both discrete and continuous time Markov chains. The
relationship between Markov chains of finite states and matrix theory will also
behighlighted.Someclassicaliterativemethodsforsolvinglinearsystemswillbe
introducedforfindingthe stationarydistributionof a Markovchain.We then give
thebasictheoriesandalgorithmsforhiddenMarkovmodels(HMMs)andMarkov
decisionprocesses(MDPs).
Chapter2discussestheapplicationsofcontinuoustimeMarkovchainstomodel
queueingsystemsanddiscretetimeMarkovchainsforcomputingthePageRank,a
rankingoftheimportanceofawebsiteintheInternet.Chapter3studiesMarkovian
models for manufacturing and remanufacturing systems. We present closed form
solutions and fast numerical algorithms for solving the captured systems. In
Chapter4,wepresentasimplehiddenMarkovmodel(HMM)withfastnumerical
algorithmsforestimatingthemodelparameters.Wethenpresentanapplicationof
theHMMforcustomerclassification.
Chapter 5 discusses Markov decision processes for customer lifetime values.
Customerlifetimevalues(CLV)isanimportantconceptandquantityinmarketing
management.WepresentanapproachbasedonMarkovdecisionprocessesforthe
calculationofCLVusingrealdata.
In Chapter6,we considerhigher-orderMarkovchainmodels.Inparticular,we
discuss a class of parsimonioushigher-orderMarkov chain models. Efficient esti-
mationmethodsformodelparametersbasedonlinearprogrammingarepresented.
Contemporary research results on applications to demand predictions, inventory
control, and financial risk measurement are presented. In Chapter 7, a class of
parsimoniousmultivariateMarkovmodelsisintroduced.Again,efficientestimation
methods based on linear programming are presented. Applications to demand
predictions,inventorycontrolpolicy,andmodelingcreditratingsdataarediscussed.
vii
viii Preface
InChapter8,werevisithiddenMarkovmodels.Weproposeanewclassofhidden
Markov models with efficient algorithms for estimating the model parameters.
Applicationstomodelinginterestrate,creditratings,anddefaultdataarediscussed.
TheauthorswouldliketothankOperationalResearchSociety,OxfordUniversity
Press, Palgrave,Taylor& Francis’, Wiley & Sons, JournalofCredit Risk Incisive
Media, Incisive Financial Publishing Limited, and Yokohama Publishers for their
permission to reproduce the material in this book. The authors would also like
to thank Werner Fortmann, Gretel Fortmann, and Mimi Lui for their help in the
preparationofthisbook.
HongKongSAR Wai-KiChing
Atlanta,Georgia XiminHuang
Kowloon,HongKongSAR MichaelK.Ng
Sydney,Australia Tak-KuenSiu
Contents
1 Introduction .................................................................. 1
1.1 MarkovChains.......................................................... 1
1.1.1 ExamplesofMarkovChains................................... 2
1.1.2 Thenth-StepTransitionMatrix................................ 5
1.1.3 IrreducibleMarkovChainandClassificationsofStates...... 7
1.1.4 AnAnalysisoftheRandomWalk ............................. 8
1.1.5 SimulationofMarkovChainswithEXCEL................... 10
1.1.6 BuildingaMarkovChainModel............................... 11
1.1.7 StationaryDistributionofaFiniteMarkovChain............. 13
1.1.8 ApplicationsoftheStationaryDistribution ................... 18
1.2 ContinuousTimeMarkovChainProcess.............................. 19
1.2.1 AContinuousTwo-StateMarkovChain....................... 21
1.3 IterativeMethodsforSolvingLinearSystems......................... 22
1.3.1 SomeResultsonMatrixTheory ............................... 23
1.3.2 SplittingofaMatrix............................................ 24
1.3.3 ClassicalIterativeMethods..................................... 26
1.3.4 SpectralRadius ................................................. 28
1.3.5 SuccessiveOver-Relaxation(SOR)Method .................. 29
1.3.6 ConjugateGradientMethod.................................... 30
1.3.7 ToeplitzMatrices ............................................... 34
1.4 HiddenMarkovModels................................................. 35
1.5 MarkovDecisionProcess............................................... 37
1.5.1 StationaryPolicy................................................ 41
1.6 Exercises................................................................. 42
2 QueueingSystemsandtheWeb............................................ 47
2.1 MarkovianQueueingSystems.......................................... 47
2.1.1 AnM/M/1=n(cid:2)2QueueingSystem ........................... 48
2.1.2 AnM/M/s=n(cid:2)s(cid:2)1QueueingSystem....................... 49
2.1.3 AllocationoftheArrivalsinaSystem
ofM/M/1/1Queues ........................................... 51
ix