Finite Markov Chains and Algorithmic Applications

123 Pages·2002·0.56 MB·English

by  Olle Häggström

Finite Markov Chains and
Algorithmic Applications
OLLE H(cid:128)GGSTR(cid:133)M
London Mathematical Society
Student Texts 00
Finite Markov Chains
and
Algorithmic Applications
OlleHa¨ggstro¨m
Matematiskstatistik,Chalmerstekniskaho¨gskolaochGo¨teborgsuniversitet
PUBLISHED BY CAMBRIDGE UNIVERSITY PRESS (VIRTUAL PUBLISHING)
FOR AND ON BEHALF OF THE PRESS SYNDICATE OF THE UNIVERSITY OF
CAMBRIDGE
The Pitt Building, Trumpington Street, Cambridge CB2 IRP
40 West 20th Street, New York, NY 10011-4211, USA
477 Williamstown Road, Port Melbourne, VIC 3207, Australia

http://www.cambridge.org

© Cambridge University Press 2002
This edition © Cambridge University Press (Virtual Publishing) 2003

First published in printed format 2002

A catalogue record for the original printed book is available
from the British Library and from the Library of Congress
Original ISBN 0 521 81357 3 hardback
Original ISBN 0 521 89001 2 paperback

ISBN 0 511 01941 6 virtual (netLibrary Edition)
Contents
Preface pagevii
1 Basicsofprobabilitytheory 1
2 Markovchains 8
3 ComputersimulationofMarkovchains 17
4 IrreducibleandaperiodicMarkovchains 23
5 Stationarydistributions 28
6 ReversibleMarkovchains 39
7 MarkovchainMonteCarlo 45
8 FastconvergenceofMCMCalgorithms 54
9 Approximatecounting 64
10 ThePropp–Wilsonalgorithm 76
11 Sandwiching 84
12 Propp–Wilsonwithread-oncerandomness 93
13 Simulatedannealing 99
14 Furtherreading 108
References 110
Index 113
v
This Page Intentionally Left Blank
Preface
The first version of these lecture notes was composed for a last-year under-
graduatecourseatChalmersUniversityofTechnology,inthespringsemester
2000. I wrote a revised and expanded version for the same course one year
later. Thisisthethirdandfinal(?) version.
Thenotesareintendedtobesufficientlyself-containedthattheycanberead
withoutanysupplementarymaterial,byanyonewhohaspreviouslytaken(and
passed)somebasiccourseinprobabilityormathematicalstatistics,plussome
introductorycourseincomputerprogramming.
The core material falls naturally into two parts: Chapters 2–6 on the basic
theory of Markov chains, and Chapters 7–13 on applications to a number of
randomizedalgorithms.
Markovchainsareaclassofrandomprocessesexhibitingacertain“mem-
oryless property”, and the study of these – sometimes referred to as Markov
theory – is one of the main areas in modern probability theory. This area
cannotbeavoidedbyastudentaimingatlearninghowtodesignandimplement
randomizedalgorithms, becauseMarkovchainsareafundamentalingredient
inthestudyofsuchalgorithms. Infact,anyrandomizedalgorithmcan(often
fruitfully)beviewedasaMarkovchain.
I have chosen to restrict the discussion to discrete time Markov chains
with finite state space. One reason for doing so is that several of the most
important ideas and concepts in Markov theory arise already in this setting;
these ideas are more digestible when they are not obscured by the additional
technicalities arising from continuous time and more general state spaces. It
can also be argued that the setting with discrete time and finite state space is
the most natural when the ultimate goal is to construct algorithms: Discrete
time is natural because computer programs operate in discrete steps. Finite
state space is natural because of the mere fact that a computer has a finite
amount of memory, and therefore can only be in a finite number of distinct
vii
viii Preface
“states”. Hence, the Markov chain corresponding to a randomized algorithm
implementedonarealcomputerhasfinitestatespace.
However, IdonotclaimthatmoregeneralMarkovchainsareirrelevantto
the study of randomized algorithms. For instance, an infinite state space is
sometimesusefulasanapproximationto(andeasiertoanalyzethan)afinite
but very large state space. For students wishing to dig into the more gen-
eral Markov theory, the final chapter provides several suggestions for further
reading.
Randomized algorithms are simply algorithms that make use of random
numbergenerators. InChapters7–13,theMarkovtheorydevelopedinprevi-
ouschaptersisappliedtosomespecificrandomizedalgorithms. TheMarkov
chain Monte Carlo (MCMC) method, studied in Chapters 7 and 8, is a class
of algorithms which provides one of the currently most popular methods for
simulatingcomplicatedstochasticsystems. InChapter9,MCMCisappliedto
theproblemofcountingthenumberofobjectsinacomplicatedcombinatorial
set. Then, in Chapters 10–12, we study a recent improvement of standard
MCMC,knownasthePropp–Wilsonalgorithm.Finally,Chapter13dealswith
simulatedannealing,whichisawidelyusedrandomizedalgorithmforvarious
optimizationproblems.
It should be noted that the set of algorithms studied in Chapters 7–13
constitutes only a small (and not particularly representative) fraction of all
randomizedalgorithms. Forabroaderviewofthewidevarietyofapplications
of randomization in algorithms, consult some of the suggestions for further
readinginChapter14.
The following diagram shows the structure of (essential) interdependence
betweenChapters2–13.
3 8 9
2 5 6 7 10 11
4 13 12
Howthechaptersdependoneachother.
Regardingexercises: Mostchaptersendwithanumberofproblems. These
areofgreatlyvaryingdifficulty.Toguidethestudentinthechoiceofproblems
toworkon, andtheamountoftimetoinvestintosolvingtheproblems, each
problem has been equipped with a parenthesized number between (1) and
Preface ix
(10)toranktheapproximatesizeanddifficultyoftheproblem. (1)means
thattheproblemamountssimplytocheckingsomedefinitioninthechapter(or
somethingsimilar),andshouldbedoableinacoupleofminutes. Attheother
endofthescale,(10)meansthattheproblemrequiresadeepunderstanding
of the material presented in the chapter, and at least several hours of work.
Some of the problems require a bit of programming; this is indicated by an
asterisk,asin(7*).
(cid:1)(cid:1)(cid:1)(cid:1)
I am grateful to Sven Erick Alm, Nisse Dohrne´r, Devdatt Dubhashi, Mihyun
Kang,DanMattsson,JesperMøllerandJeffSteif,whoallprovidedcorrections
toandconstructivecriticismofearlierversionsofthismanuscript.
This Page Intentionally Left Blank
1
Basics of probability theory
Themajorityofreaderswillprobablybebestoffbytakingthefollowingpiece
ofadvice:
Skipthischapter!
Those readers who have previously taken a basic course in probability or
mathematicalstatisticswillalreadyknoweverythinginthischapter,andshould
move right on to Chapter 2. On the other hand, those readers who lack such
background will have little or no use for the telegraphic exposition given
here,andshouldinsteadconsultsomeintroductorytextonprobability. Rather
thanbeingread,thepresentchapterisintendedtobeacollectionof(mostly)
definitions,thatcanbeconsultedifanythingthatlooksunfamiliarhappensto
appearinthecomingchapters.
(cid:1)(cid:1)(cid:1)(cid:1)
Let (cid:2) be any set, and let (cid:3) be some appropriate class of subsets of (cid:2),
satisfyingcertainassumptionsthatwedonotgofurtherinto(closednessunder
certainbasicsetoperations). Elementsof(cid:3)arecalledevents. For A⊆(cid:2),we
write Ac forthecomplementof Ain(cid:2),meaningthat
Ac ={s ∈(cid:2): s (cid:6)∈ A}.
Aprobabilitymeasureon(cid:2)isafunctionP:(cid:3) →[0,1],satisfying
(i) P(∅)=0.
(ii) P(Ac)=1−P(A)foreveryevent A.
(iii) If AandBaredisjointevents(meaningthat A∩B =∅),thenP(A∪B)=
P(A) + P(B). More generally, if A ,A ,... is a countable sequence
1 2
1