The Laws of Large Numbers

170 Pages·1967·7.194 MB·English

by  Pál Révész, Z. W. Birnbaum and E. Lukacs (Auth.)

Probability and Mathematical Statistics
A Series of Monographs and Textbooks
Edited by
Z. W. Birnbaum E. Lukacs
University of Washington Catholic University
Seattle, Washington Washington, D.C.
1. Thomas Ferguson. Mathematical Statistics: A Decision Theoretic
Approach. 1967
2. Howard Tucker. A Graduate Course in Probability. 1967
3. K. R. Parthasarathy. Probability Measures on Metric Spaces. 1967
4. P. Révész. The Laws of Large Numbers. 1968
In preparation
B. V. Gnedenko, Yu. K. Belyayev, and A. D. Solovyev. Mathematical
Methode of Reliability Theory
THE LAWS
OF
LARGE NUMBERS
by
PÂL RÉVËSZ
Mathematical Institute
Hungarian Academy of Sciences
Budapest, Hungary
1968
ACADEMIC PRESS
NEW YORK AND LONDON
) AKADÉMIAI KIADO, BUDAPEST IN HUNGARY 1967
ALL RIGHTS RESERVED.
NO PART OF THIS BOOK MAY BE REPRODUCED IN ANY FORM,
BY PHOTOSTAT, MICROFILM, OR ANY OTHER MEANS, WITHOUT
WRITTEN PERMISSION FROM THE PUBLISHERS.
ACADEMIC PRESS INC.
Ill Fifth Avenue, New York, New York 10003
United Kingdom Edition published by
ACADEMIC PRESS INC. (LONDON) LTD.
Berkeley Square House, London W.l
LIBRARY OF CONGRESS CATALOG CARD NUMBER: 68-26629
JOINT EDITION PUBLISHED BY
ACADEMIC PRESS, NEW YORK AND LONDON
AND
AKADÉMIAI KIADO, BUDAPEST
PRINTED IN THE UNITED STATES OF AMERICA
INTRODUCTION
For most people the idea of probability is closely related to that of
relative frequency. Therefore, it is natural that an attempt was made
to construct a mathematical theory of probability using this concept.
Such an approach was tried for instance by MISES [1], [2] in the
first three decades of this century. Unfortunately it did not produce
sufficiently deep results,1 and it was necessary to find another ap-
proach.
This was accomplished in Kolmogorov's axiomatic treatment of
probability. However, if we wish to be sure that this theory adequately
represents our natural ideas of probability, then we must investigate
the relationships between probability and relative frequency. The
results of this investigation are called the laws of large numbers.
Similarly, the relationships between other theoretical and practi-
cal concepts of probability theory, e.g. the concepts of expectation
and sample mean, can also be investigated by using the laws of large
numbers.
These remarks show the theoretical importance of this field; its
practical importance is no less. If we want to estimate the unknown
probability of a random event or the unknown expectation of a ran-
dom variable, then we have to show that the relative frequency or
the sample mean converges to the probability or to the expectation.
The statistical application of the laws of large numbers raises two
other questions: what is the rate of convergence of the relative fre-
quency or of the sample mean and what is the behaviour of the
sample mean if the elements of the sample are not necessarily inde-
pendent and identically distributed random variables. If we are
working with a class of stochastic processes then it is necessary to
know how to estimate the parameters of this process, or in other words,
it is necessary to give the corresponding laws of large numbers.
To define exactly the field of the laws of large numbers seems to be
very difficult. We can say, in an attempt to obtain a definition, that
1This way was tried again very recently by KOLMOGOROV [3] and MARTIN -
LÖF [1], Their results are based on very deep mathematical methods and in
some sense can solve the problem of MISES.
8 INTRODUCTION
a law of large numbers asserts the convergence, in a certain sense, of
the average
_ξ ξ +... + ξ
1 + 2 π
Ίη —
n
of the random variables ξ ξ , . . . to a random variable η. Actually,
ν 2
this class of theorems contains many theorems which are not related
to probability, for example the classical theorems of Fejér about the
Cesaro summability of Fourier series. Therefore, we will say that laws
of large numbers are theorems stating the convergence of η which
η
are interesting from the point of view of probability. Evidently it is
very difficult to say exactly what theorems are of interest in probabi-
lity theory. It may happen that the author investigates some problems
in which he is especially interested but which are not that closely
related to probability theory and he omits others, more closely related.
If we study the convergence of η , making use of different modes
η
of convergence, then we obtain different types of the laws of large
numbers. In this book we consider three types of laws of large numbers
according to the following kinds of convergence:1
1°. Stochastic convergence: P (| η — η | ^> ε) —► 0,
η
2°. Mean convergence: Ε[(η — η)2] -> 0,
η
3°. Convergence with probability 1 : Ρ(η —► η) = 1.
η
In connection with the resulting types of laws of large numbers we
will investigate the rate of convergence. In the first two cases the
definition of the rate of convergence is clear. In the third case the rate
of convergence will be characterized by the * 'largest' ' function f(n)
for which P(/(w)| η — η \ -+ θ) = 1. More precisely, we investigate
η
the class of functions f(n) for which the last mentioned formula holds.
Hence, the laws of the iterated logarithm will also be treated in this
book.
oo
Theorems on the convergence of a series of the form Σ c £ ,
k k
k=l
where, {S } is a sequence of random variables and { c } is a sequence
k k
of real numbers cannot be considered as a law of large numbers.
However, this class of theorems will occasionally be studied here,
because the convergence of a series of the above form immediately
implies a law of large numbers by Theorem 1.2.2.
Let us mention some types of theorems which could be considered
as laws of large numbers but which will not be investigated in this
book.
First, the theory of large deviations will be completely omitted,
though these theorems also give results about the rate of convergence
in case 1°. In general, we intend to distinguish the laws of large num-
bers and the limit theorems. Of course, it is not possible to succeed in
every case.
1 The exact definitions are given in § 1.1.
INTRODUCTION 9
Another class of theorems which will not be treated here is the laws
of large numbers for stochastic processes with a continuous param-
eter.
Likewise, the problems of double sequences are not investigated.
This question was recently treated in several papers (see e.g. GNE-
DENKO-KOLMOGOROV [1] and CHOW [1]).
The aim of the author is to give a general survey of the results and
the most important methods of proof in this field. Occasionally, when
the proof of a theorem requires very special methods, the proof will
be omitted. For instance, one can prove some laws of large numbers
for certain classes of stochastic processes making use only of general
methods of proof, but at times it is necessary to apply the deeper
special properties of the class of stochastic process in question. In
such cases we will not give the proof, because we do not intend to
study the special methods of stochastic processes. For example, the
theory of martingales which, can be applied as a method of proof to
obtain laws of large numbers, will be omitted.
In Chapter 0 we have collected the most important definitions and
theorems which are applied in this book. We emphasize that it is not
the aim of this chapter to give a systematic treatment. The reader
should be familiar with the most fundamental results and concepts
of probability (Fisz [1], GNEDENKO [1], LOÈVE [1], RÉNYI [1]),1
stochastic processes (DOOB [1]), measure theory (HALMOS [1]), ergodic
theory (HALMOS [2], or JACOBS [1], [2]), functional analysis (RIESZ-
SZ.-NAGY [1], YOSIDA [1], etc. The aim of Chapter 0 is to give some
help to the reader by presenting the necessary preliminary material.
Chapter 1 deals with the special concepts and general theorems of
the laws of large numbers. Chapters 2, 3, 4, 6, 7 discuss the laws
of large numbers of different classes of stochastic processes. Chapter
5 gives laws of large numbers for subsequences of sequences of
random variables. Chapter 8 contains some general laws of large
numbers which are not related to any concrete class of stochastic
processes. Chapters 9 and 10 treat some special questions.
In Chapter 11 we give some examples of the applications of the
theorems of this book. It is evident that many more quite different
applications can be found.
I am deeply indebted to Professors A. CSÂSZÂR, A. RÉNYI and
K. TANDORI for their valuable advice and criticism. My thanks are
due to Mr. G. EAGLESON for correcting the text linguistically and for
his valuable remarks.
1 The numbers in brackets refer to some books containing the necessary
fundamental knowledge. (See the ''References" at the end of the book.)
CHAPTER 0
MATHEMATICAL BACKGROUND
§ 0.1· Measure theory
The fundamental concept of measure theory is that of a measure
space.
Let X be an arbitrary abstract (non-empty) set, sometimes called
the basic space. A class «9* of the subsets of X is called a a-algebra if
χ&
AB^S* whenever A ζ& and B 6&
and
2 A^S* whenever A^S? (i = l,2, ...).
Here (and in the following) the product AB of two sets A and B de-
notes their common part (intersection). The union of two sets A and
B will be denoted by addition A + B. Finally B is the complement
of B, i.e. B contains those points of X which do not belong to B.
The elements of «5^ are called measurable sets. A set function μ
defined on S? is called a measure if
μ(Α) ^> 0 whenever Αζ£?
and
μ I 2 A\ = 2 M Ai) whenever AÇ.S? and AA = 0 (if ί+ /)
t i j
where 0 denotes the empty set. A measure μ is called σ-finite if there
exists a sequence A A , . . . of measurable sets such that
v 2
X= 2 Ai and μ(Α) < oo (i = 1, 2,. . . ).
ί
i=l
The ordered pair {X, &*} is called a measurable spaœe. The ordered
triple {X, ^Ι μ} (where X is an abstract non-empty set, *$^is a or-al-
gebra of the subsets of X (X £ S?) and μ is a measure defined on ^)
is called a measure space.
11
12 CHAPTER 0
A class (Q of the subsets of X is called an algebra if
x^m
AB^m whenever A Ç m and B Ç <£
and
.4 + J5 Ç « whenever ,4 Ç ^ and £ Ç «.
Evidently, for every algebra <Tl there is a unique smallest σ-algebra
S? =. S? (fR) containing Φ,. ("Smallest" means: if &** is a σ-algebra
containing <Tl then J?7* also contains S^(fQ).)
A real valued function f(x) defined on X is called measurable (with
respect to S?) if for each real z
{x:f(x)<z}i^.
Let / ^ ), / (#), · · · be a finite or infinite sequence of measurable
2
functions defined on a measurable space {X, S?}.£S(f / , . . .) g «5^
v 2
denotes the smallest o-algebra with respect to which the functions f / ,...
v 2
are measurable. If the function g(x) is measurable with respect to the
G-algebra e$(/j,/ , · · ·) then there exists a function A(., ., . . .) such
2
that g = h (/ , / , . . . ). 7>i particular, if f /.,, ... is a /mite sequence
3 2 v
of n elements then h is a Borel-measurable1 function defined on the n-di-
mensional Cartesian space, g is called a Baire-function of f / , . . ..
v 2
A measurable function f(x) is called integrable if it is integrable over
the space X (with respect to the measure μ) in the sense of Lebesgue,
and its integral is finite. The integral of f(x) over the whole space X
will be denoted in the following different ways:
X X
Among the properties of the Lebesgue integral we mention here
only two theorems.
THEOREM 0.1.1 (LEBESGUE THEOREM). // f / , . . . is a sequence of
v 2
integrable functions and there exists an integrable function b(x) such
that | /„ | <; b(x) (n = 1, 2, . . .) then
lim j7„ = J lim/n
n-*oo ri-»-«»
provided that lim / e:mte almost everywhere (i.e. for each x except a set
n
Π—*ΌΟ
of measure 0).
1 The Borel sets of the n-dimensional Cartesian space are the elements of
the smallest σ-algebra containing the n-dimensional intervals. A function
g{x x> · · ·> xn) m called Borel-measurable if it is measurable with respect to
19 2
this (7-algebra.
MATHEMATICAL BACKGROUND 13
THEOREM 0.1.2 (BEPPO-LEVI THEOREM). // fvf29. . . is a sequence
oo
of non-negative integrable functions for which Σ f fn < °° then the
oo n=l
series Σ f is convergent almost everywhere and
n
n=l
r» oo oo /»
\Σ fn= Σ /„·
Jn=l n=\J
A set function λ (defined on 5?) is called a signed measure if
λί j? Λ·1= J?*MÎ) whenever ^ ζ^ and AiAj = 0 (iff =4= 7)
and λ(^4) can take on at most one of the values + °° and — 00.
The signed measure λ is called σ-finite if there exists a sequence
A A , . . . of measurable sets such that
v 2
X = Σ Ai and I λ(Αΐ) I < °° (i = 1, 2,. . . ).
i-l
Henceforth we assume that all measures and all signed measures
are or-finite.
We can characterize signed measures by
THEOREM 0.1.3. // λ is a signed measure then there exist two measures
λ+ and λ~ such that
λ{Α) = λ+(Α) — λ-{Α) if A^S?.
The measures λ+ (resp. λ~) are called the upper (resp. lower) varia-
tion of λ. The measure | λ \ (A) = λ + (Α) + λ~(Α) is called the total
variation of λ.
We will say that a signed measure v is absolutely continuous with
respect to a signed measure μ (in symbols v <^ μ) if v(A) = 0 for
every measurable set A for which \ μ \ (A) = 0.
We mention only two fundamental results of measure theory.
THEOREM 0.1.4 (EXTENSION THEOREM). 2/ we have a a-additivef and
o-finite non-negative set function^ μ defined on an algebra ΠΙ then there
exists a unique measure μ* defined on S^{(V) such that
μ*(Α) = μ(A) whenever ΑζΠΙ.
This theorem implies that if two measures μ and μ are equal to
λ 2
each other on an algebra 02 then they are equal on S?(fQ).
1 That is μ (Σ A) = Σ μ (ΑΛ whenever A; £ #, Σ A: £ 01 and A A; = 0
t t
i=l i=l i=l
(i =(=i), and there exists a sequence A A . . . of elements of 01 such that
l9 if
00
Σ A, = X and μ (ΑΛ < oo.
/=1
14 CHAPTER 0
THEOREM 0.1.5 (RADON-NIKODYM THEOREM). If the signed measure
v is absolutely continuous with respect to a signed measure μ then there
exists a measurable function f, (on X), such that
k
whenever E £ S?. The function f is unique in the sense that if also v(E) =
= J g(x^ then μ {x : f(x) =(= &(x)} = 0.
E
The function f(x) is called the Radon-Nikodym derivative of v
with respect to μ. In symbols:
αμ
We say that the sequence f / , . . . of measurable functions con-
v 2
verges in measure to a function / if
lim/i{3:|/ -/|^e}=0
n
n-*oo
for any ε > 0. In symbols f => f.
n
In connection with the properties of almost everywhere convergence
we have
THEOREM 0.1.6 (EGOROV'S THEOREM). 7/ fn(x) -+f{x), almost every-
where on a space X of finite measure (i.e. f (x) ->/(#) for allx except a
n
set of measure 0), then for any ε > 0 there exists a set F Ç S? such that
μ(Ρ) < ε and f converges to f uniformly on F, i.e. for any δ > 0 there
n
exists an n = η (δ) such that
0
Α*{*:|/π-/|<Μ€*ΐ}=Μ*')
if n ^> n .
0
We now define the Cartesian product of two, or more, measure spaces.
Let {X, <9*, μ} and {Y, &", v} be two measure spaces. The product
X x Y means the set of the ordered pairs (x, y) (x Ç X, y\Y). If
A cz X and Bcz Y then AxBczXxY is defined by
AxB={(x,y):xtA,y£B}.
The product S?x ^ of the or-algebras & and &" is the smallest
σ-algebra containing the sets A X B where A £ &* and B Ç &*. We
can define the product measure μ χ v on & χ S" by
μΧν(ΑχΒ)=μ(Α)ν(Β) (ACS? Βζ<9~).
The measure of other elements of S? χ S" can be obtained by exten-
sion. The measure space {X X Y, S^x &" μ X v) is called the
y
Cartesian product of {X, &*, μ} and {Y, S~, v}.
If {X 3^, μ } {Χ , ό? , μ }, . . . is a sequence of measure spaces
v 1 ί 2 2 2
with μάΧι) = 1 (i = 1, 2, . . .) then the Cartesian product of them
can be defined as follows: