A Natural Introduction to Probability Theory

201 Pages·2008·1.44 MB·english

by  Ronald Meester

Ronald Meester
A Natural Introduction
to Probability Theory
Second Edition
Birkhäuser
Basel Boston Berlin
(cid:115) (cid:115)
Author:
Ronald Meester
Faculteit der Exacte Wetenschappen
Vrije Universiteit
De Boelelaan 1081a
1081 HV Amsterdam
The Netherlands
e-mail: [email protected]
2000 Mathematics Subject Classification 60-01
Library of Congress Control Number: 2007942638
Bibliographic information published by Die Deutsche Bibliothek
Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie;
detailed bibliographic data is available in the Internet at <http://dnb.ddb.de>.
ISBN 978-3-7643-8723-5 Birkhäuser Verlag, Basel – Boston – Berlin
This work is subject to copyright. All rights are reserved, whether the whole or part of the material is
concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting,
reproduction on microfilms or in other ways, and storage in data banks. For any kind of use permission
of the copyright owner must be obtained.
First edition 2003
© 2008 Birkhäuser Verlag AG
Basel · Boston · Berlin
P.O. Box 133, CH-4010 Basel, Switzerland
Part of Springer Science+Business Media
Printed on acid-free paper produced from chlorine-free pulp. TCF (cid:100)
Printed in Germany
ISBN 978-3-7643-8723-5 e-ISBN 978-3-7643-8724-2
9 8 7 6 5 4 3 2 1 www.birkhauser.ch
Contents
Preface to the First Edition viii
Preface to the Second Edition x
1 Experiments 1
1.1 Definitions and Examples . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Counting and Combinatorics . . . . . . . . . . . . . . . . . . . . . 6
1.3 Properties of Probability Measures . . . . . . . . . . . . . . . . . . 10
1.4 Conditional Probabilities. . . . . . . . . . . . . . . . . . . . . . . . 12
1.5 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.6 A First Law of Large Numbers . . . . . . . . . . . . . . . . . . . . 26
1.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2 Random Variables and Random Vectors 35
2.1 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.3 Expectation and Variance . . . . . . . . . . . . . . . . . . . . . . . 43
2.4 Random Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.5 Conditional Distributions and Expectations . . . . . . . . . . . . . 57
2.6 Generating Functions. . . . . . . . . . . . . . . . . . . . . . . . . . 62
2.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3 Random Walk 71
3.1 Random Walk and Counting . . . . . . . . . . . . . . . . . . . . . 71
3.2 The Arc-Sine Law . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4 Limit Theorems 81
4.1 The Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . . 81
4.2 The Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . 85
4.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
vi Contents
I Intermezzo 89
I.1 Uncountable Sample Spaces . . . . . . . . . . . . . . . . . . . . . . 89
I.2 An Event Without a Probability?! . . . . . . . . . . . . . . . . . . 90
I.3 Random Variables on Uncountable Sample Spaces . . . . . . . . . 92
5 Continuous Random Variables and Vectors 93
5.1 Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.2 Properties of Probability Measures . . . . . . . . . . . . . . . . . . 98
5.3 Continuous Random Variables. . . . . . . . . . . . . . . . . . . . . 100
5.4 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
5.5 Random Vectors and Independence . . . . . . . . . . . . . . . . . . 109
5.6 Functions of Random Variables and Vectors . . . . . . . . . . . . . 113
5.7 Sums of Random Variables . . . . . . . . . . . . . . . . . . . . . . 117
5.8 More About the Expectation; Variance . . . . . . . . . . . . . . . . 118
5.9 Random Variables Which are Neither Discrete Nor Continuous . . 122
5.10 Conditional Distributions and Expectations . . . . . . . . . . . . . 124
5.11 The Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . . 131
5.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
6 Infinitely Many Repetitions 137
6.1 Infinitely Many Coin Flips and Random Points in (0,1] . . . . . . 138
6.2 A More General Approach to Infinitely Many Repetitions . . . . . 140
6.3 The Strong Law of Large Numbers . . . . . . . . . . . . . . . . . . 142
6.4 Random Walk Revisited . . . . . . . . . . . . . . . . . . . . . . . . 146
6.5 Branching Processes . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
7 The Poisson Process 153
7.1 Building a Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.2 Basic Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
7.3 The Waiting Time Paradox . . . . . . . . . . . . . . . . . . . . . . 162
7.4 The Strong Law of Large Numbers . . . . . . . . . . . . . . . . . . 164
7.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
8 Limit Theorems 167
8.1 Weak Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
8.2 Characteristic Functions . . . . . . . . . . . . . . . . . . . . . . . . 169
8.3 Expansion of the Characteristic Function . . . . . . . . . . . . . . 173
8.4 The Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . . 176
8.5 The Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . 179
8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
9 Extending the Probabilities 183
9.1 General Probability Measures . . . . . . . . . . . . . . . . . . . . . 183
Contents vii
A Interpreting Probabilities 187
B Further Reading 191
C Answers to Selected Exercises 193
Index 195
Preface to the First Edition
According to Leo Breiman (1968), probability theory has a right and a left hand.
The righthandrefers to rigorousmathematics,andthe left handrefersto ‘proba-
bilistic thinking’. The combination of these two aspects makes probability theory
one of the most exciting fields in mathematics. One can study probability as a
purely mathematical enterprise, but even when youdo that, all the concepts that
arisedohaveameaningontheintuitivelevel.Forinstance,wehavetodefinewhat
we mean exactly by independent events as a mathematical concept, but clearly,
we all know that when we flip a coin twice, the event that the first gives heads is
independent of the event that the second gives tails.
Why have I written this book? I have been teaching probability for more
than fifteen years now, and decided to do something with this experience. There
are already many introductory texts about probability, and there had better be a
good reason to write a new one. I will try to explain my reasons now.
The typical target that I have in mind is a first year student who wants or
needs to learn about probability at a stage where he or she has not seen measure
theory as yet. The usual and classical way to set things up in this first year, is
to introduce probability as a measure on a sigma-algebra, thereby referring to
measuretheoryforthe necessarydetails.This meansthat the firstyearstudentis
confrontedwitha theoryofprobabilitythathe orshe cannotpossiblyunderstand
at that point. I am convinced that this is not necessary.
I do not (of course) claim that one should not base probability theory on
measuretheorylaterinthestudy,butIdonotreallyseeareasontodothisinthe
first year. One can – as I will show in this book – study discrete and continuous
randomvariablesperfectlywell,andwithmathematicalprecision,withintherealm
of Riemann integration.
It is not particularly difficult to write rigorously about discrete probability,
but it is harder to do this more generally. There are several texts available which
do promise this (no measure theory and rigorous) but I don’t think that any of
thesetexts canreallysaytohavekeptits promise.I haveachievedprecisionwith-
out measure theory, by deviating from the classical route of defining probability
measures on sigma-algebras. In this book, probabilities of events are defined as
soonas a certain(Riemann) integralexists. This is, as I will show, a very natural
thing to do. As a result, everything in this book can be followed with no more
backgroundthan ordinary calculus.
As a result of my approach, it will become clear where the limits of this
approach are, and this in fact forms the perfect motivation to study measure
theoryandprobabilitytheorybasedonmeasuretheorylaterinthe study.Indeed,
by the end ofthe book, the student shouldbe dying to learnmoreabout measure
theory.
Hence this approach to probability is fully consistent with the way math-
ematics works: first there is a theory and you try to see how far this gets you,
andwhenyouseethatcertain(advanced)things cannotbe treatedinthis theory,
Preface ix
youhave to extend the theory. As such, by reading this book, one not only learns
a lot about probability, but also about the way mathematics is discovered and
developed.
AnotherreasontowritethisbookisthatIthinkthatitisveryimportantthat
whenstudents startto learnaboutprobability,they come tointerestingresults as
soonaspossible,withoutbeingdisturbedbyunnecessarytechnicalcomplications.
Probability theory is one of those fields where you can derive very interesting,
important and surprising results with a minimum of technical apparatus. Doing
this first is not only interesting in itself. It also makes clear what the limits are of
this elementaryapproach,thereby motivatingfurther developments.Forinstance,
the first four chapters will be concerned with discrete probability, which is rich
enoughastosayeverythingaboutafinitenumberofcoinflips.Theverysurprising
arc-sine law for random walks can already be treated at this point. But it will
becomeclearthatinordertodescribeaninfinite sequenceofsuchflips,oneneeds
to introduce more complicated models. The so called weak law of large numbers
can be perfectly well stated and proved within the realm of discrete probability,
but the strong law of large numbers cannot.
Finally,probabilitytheoryisoneofthemostusefulfieldsinmathematics.As
such,it is extremely importantto pointout what exactly we do when we model a
particular phenomenon with a mathematical model involving uncertainty. When
can we safely do this, and how do we know that the outcomes of this model do
actually say something useful? These questions are addressed in the appendix. I
think that such a chapter is a necessary part of any text on probability, and it
provides a link between the left and right hand of Leo Breiman.
A few words about the contents of this book. The first four chapters deal with
discrete probability, wherethe possibleoutcomes ofanexperimentformafinite or
countable set. The treatment includes an elementary account on random walks,
theweaklawoflargenumbersandaprimitiveversionofthecentrallimittheorem.
We have also included a number of confusing examples which make clear that it
is sometimes dangerous to trust your probabilistic intuition.
After that, in the Intermezzo, we explain why discrete probability is insuffi-
cientto dealwitha numberofbasic probabilisticconcepts.Discrete probabilityis
unable to deal with infinitely fine operations, such as choosing a point on a line
segment,andinfinitely many repetitions of anoperation,like infinitely many coin
flips.
After the Intermezzo, Chapter 5 deals with these infinitely fine operations,
a subject which goes under the name continuous probability. In Chapter 6 we
continuewithinfinitelymanyrepetitions,withapplicationstobranchingprocesses,
random walk and strong laws of large numbers.
Chapter 7 is devoted to one of the most important stochastic processes, the
Poisson process, where we shall investigate a very subtle and beautiful interply
between discrete and continuous random variables.
In Chapter 8, we discuss a number of limit theorems based on characteristic
x Preface
functions. A full proof of the central limit theorem is available at that point.
Finally, in Chapter 9, we explore the limitations of the current approach.We will
outline how we can extend the current theory using measure theory. This final
chapter provides the link between this book and probability based on measure
theory.
Previous versions of this book were read by a number of people, whose
comments were extremely important. I would very much like to thank Hanneke
de Boer, Lorna Booth, Karma Dajani, Odo Diekmann, Massimo Franceschetti,
Richard Gill, Klaas van Harn, Rob Kooijman, Rein Nobel, Corrie Quant, Rahul
Roy,JeffreySteif,FreekSuyver,AadvanderVaartandDmitriZnamenskifortheir
valuable comments.Inparticular,I wouldalso like to thank Shimrit Abrahamfor
doing a large number of exercises.
Ronald Meester, Amsterdam, Summer 2003
Preface to the Second Edition
Teaching from a book leads to many remarks and complaints. Despite the care
given to the first edition, it turned out that it still contained a fair number of
small mistakes. I thank Corrie Quant and Karma Dajani (again), Rob van den
Berg, Misja Nuyens and Jan Hoogendijk for their corrections and other remarks
that helped me very much in preparing this second edition. Apart from correct-
ing mistakes and typos, I also added a significant number of exercises and some
examples. Finally, here and there I slightly rearranged the material. There are,
however,no essential differences between the first and second edition.
Ronald Meester, Amsterdam, Fall 2007