Table Of ContentAPPLIED PROBABILITY
A Series of the Applied Probability Trust
Editors
J. Gani C. C. Heyde
Robert Azencott
Didier Dacunha-Castelle
Series of
Irregular Observations
Forecasting and Model Building
Springer-Verlag
New York Berlin Heidelberg Tokyo
Robert Azencott Didier Oacunha-Castelle
Universite de Paris-Sud Univcrsitc de Paris-Sud
Equipe de Recerche Associee Equipe de Recerchc Associee
au C.N.R.S. au C.N.R.S. 532
Statistique Appliquee Mathematique Statislique Appliquee Malhemalique
91405 Orsay Cedex 91405 Orsay Cedex
France France
Series Editors
J. Gani c. C. Heyde
Statistics Program Department of Statistics
Department of Mathematics Institute of Advanced Studies
University of Califomia 1lK: Australian National University
Santa Barbara, CA 93106 Canberra, ACT 2601
U.S.A. Australia
AMS Classificalion 62-01, 62MIO, 62MI5
Library of Congress Cataloging-in-Publication Data
Azencou, Robert.
Series of irregular observations.
(Applied probability)
Bibliography: p.
Includes index..
I. Stochastic processes. I. Dacunha-Castelle,
Didier. II. Title. III. Series.
QA274.A94 1986 519.2 86-1834
French Edition, Series d'Observations Irregufieres, © Masson. Editeur, Paris, 1984.
© 1986 by Springer-Verlag New York Inc.
Soflcover reprint of the hardcover 1 st edition 1986
All rights reserved. No part of this book may be translated or reproduced in any fonn
without written pennission from Springer-Verlag, 175 Fifth Avenue, New York, New
York 10010, U.S.A.
9 8 7 654 3 2 I
ISBN-13: 978-1-4612-9357-6 e-ISBN-13: 978-1-4612-4912-2
DOl: 10.1007/978-1-4612-4912-2
CONTENTS
Introduction
CHAPTER I
Discrete Time Random Processes 3
I. Random Variables and Probability Spaces 3
2. Random Vectors 4
3. Random Processes 4
4. Second-Order Process 8
CHAPTER II
Gaussian Processes JO
I. The Use (and Misuse) of Gaussian Models JO
2. Fourier Transform: A Few Basic Facts II
3. Gaussian Random Vectors 12
4. Gaussian Processes 15
CHAPTER III
Stationary Processes 18
1. Stationarity and Model Building 18
2. Strict Stationarity and Second-Order Stationarity 20
3. Construction of Strictly Stationary Processes 21
4. Ergodicity 22
5. Second-Order Stationarity: Processes with Countable Spectrum 24
CHAPTER IV
Forecasting and Stationarity 25
I. Linear and Nonlinear Forecasting 25
2. Regular Processes and Singular Processes 26
vi Contents
3. Regular Stationary Processes and Innovation 28
4. Prediction Based on a Finite Number of Observations 31
5. Complements on Isometries 34
CHAPTER V
Random Fields and Stochastic Integrals 37
I. Random Measures with Finite Support 37
2. Uncorrelated Random Fields 38
3. Stochastic Integrals 43
CHAPTER VI
Spectral Representation of Stationary Processes 46
1. Processes with Finite Spectrum 46
2. Spectral Measures 47
3. Spectral Decomposition 50
CHAPTER VII
Linear Filters 55
1. Often Used Linear Filters 55
2. Multiplication of a Random Field by a Function 57
3. Response Functions and Linear Filters 58
4. Applications to Linear Representations 62
5. Characterization of Linear Filters as Operators 64
CHAPTER VIII
ARMA Processes and Processes with Rational Spectrum 67
1. ARMA Processes 67
2. Regular and Singular Parts of an ARM A Process 69
3. Construction of ARMA Processes 74
4. Processes with Rational Spectrum 76
5. Innovation for Processes with Rational Spectrum 81
CHAPTER IX
Nonstationary ARMA Processes and Forecasting 83
1. Nonstationary ARMA Models 83
2. Linear Forecasting and Processes with Rational Spectrum 91
3. Time Inversion and Estimation of Past Observations 95
4. Forecasting and Nonstationary ARMA Processes 97
CHAPTER X
Empirical Estimators and Periodograms 101
1. Empirical Estimation 101
2. Periodograms 103
3.- Asymptotic Normality and Periodogram 107
4. Asymptotic Normality of Empirical Estimators III
5. The Toeplitz Asymptotic Homomorphism 113
Contents vii
CHAPTER XI
Empirical Estimation of the Parameters for ARMA Processes with
Rational Spectrum 118
1. Empirical Estimation and Efficient Estimation 118
2. Computation of the a. and Yule-Walker Equations 119
3. Computation of the bl and of 0'2 120
4. Empirical Estimation of the Parameters When p, q are Known 123
5. Characterization of p and q 125
6. Empirical Estimation of d for an ARIMA (p,d,q) Model 128
7. Empirical Estimation of (p,q) 131
8. Complement: A Direct Method of Computation for the b. 135
9. The ARMA Models with Seasonal Effects 136
10. A Technical Result: Characterization of Minimal Recursive Identities 136
11. Empirical Estimation and Identification 138
CHAPTER XII
Effecient Estimation for the Parameters of a Process with
Rational Spectrum 141
1. Maximum Likelihood 141
2. The Box-Jenkins Method to Compute (d,b) 146
3. Computation of the Information Matrix 150
4. Convergence of the Backforecasting Algorithm 155
CHAPTER XIII
Asymptotic Maximum Likelihood 162
1. Approximate Log-Likelihood 162
2. Kullback Information 164
3. Convergence of Maximum Likelihood Estimators 167
4. Asymptotic Normality and Efficiency 169
CHAPTER XIV
Identification and Compensated Likelihood 181
1. Identification 181
2. Parametrization 182
3. Compensated Likelihood 184
4. Mathematical Study of Compensated Likelihood 185
5. Noninjective Parametrization 190
6. Almost Sure Bounds for the Maximal Log-Likelihood 193
7. Law of the Interated Logarithm for the Periodogram 203
CHAPTER XV
A Few Problems not Studied Here 223
1. Tests of Fit for ARMA Models 223
2. Nonlinearity 225
Appendix 227
Bibliography 231
Index 234
I NTRODUCTI ON
For the past thirty years, random stationary processes have
played a central part in the mathematical modelization of
numerous concrete phenomena.
Their domains of application include, among others,
signal theory (signal transmission in the presence of noise,
modelization of human speech, shape recognition, etc. .. . ),
the prediction of economic quantities (prices, stock
exchange fluctuations, etc.), meterology (analysis of
sequential climatic data), geology (modelization of the
dependence between the chemical composition of earth
samples and their locations), medicine (analysis of
electroencephalograms, electrocardiograms, etc . ... ).
Three mathematical points of view currently define the use
of stationary processes: spectral analysis, linked to Fourier
transforms and widely popularized by N. Wiener; Markov
representations, particularly efficient in automatic linear
control of dynamic systems, as shown by Kalman-Bucy's
pioneering work; finite autoregressive and moving average
schemes (ARMA processes) an early technique more recently
adapted for computer use and vulgarized by Box-Jenkins.
We have sought to present, in compact and rigorous
fashion, the essentials of spectral analysis and ARMA
modelization. We have deliberately restricted the scope of the
book to one-dimensional processes, in order to keep the basic
concepts as transparent as possible.
2 Introduction
At the university level, in probability and statistics
departments or electrical engineering departments, this book
contains enough material for a graduate course, or even for
an upper-level undergraduate course if the asymptotic studies
are reduced to a minimum. The prerequisites for most of
the chapters (l - 12) are fairly limited: the elements of
Hilbert space theory, and the basics of axiomatic probability
theory including L 2-spaces, the notions of distributions,
random variables and bounded measures.
The standards of precision, conciseness, and mathematical
rigour which we have maintained in this text are in clearcut
contrast with the majority of similar texts on the subject.
The main advantage of this choice should be a considerable
gain of time for the noninitiated reader, provided he or she
has a taste for mathematical language.
On the other hand, being fully aware of the usefulness of
ARMA models for applications, we present carefully and in
full detail the essential algorithms for practical modelling and
identification of ARMA processes. The experience gained
from several graduate courses on these themes (Universities of
Paris-Sud and of Paris-7) has shown that the mathematical
material included here is sufficient to build reasonable
computer programs of data analysis by ARMA modelling.
To facilitate the reading, we have inserted a
bibliographical guide at the end of each chapter and,
indicated by stars (* ... *), a few intricate mathematical points
which may be skipped over by nonspecialists.
On the mathematical level, this book has benefited from
two seminars on time series, organized by the authors at the
University of Paris-Sud and at the Ecole Normale Superieure
(rue d'Ulm) Paris. We clarify several points on which many
of the "classics" in this field remain evasive or erroneous:
structure and nonstationarity of ARMA and ARMA seasonal
processes, stationary and nonstationary solutions of the
general ARMA equation, convergence of the celebrated
Box-Jenkins back forecasting algorithms, asymptotic behaviour
of ARMA estimators, etc.
We would like to thank the early readers of the French
edition, particularly E. J. Hannan for his detailed and
crucial comments, as well as M. Bouaziz and L. Elie for their
thoughtful remarks.
Chapter I
DISCRETE TIME RANDOM PROCESSES
I. Random Variables and Probability Spaces
The experimental description of any random phenomenon
involves a family of numbers Xt ' t E T. Since Kolmogorov,
it has been mathematically convenient to summarize the
impact of randomness through the stochastic choice of a
point in an adequate set n (space of trials) and to consider
the random variables X as well determined functions on n
t
with values in IR.
The quantification of randomness then reduces to
specifying the family B of subsets of n which represent
relevant events and, for each event A E B, the probability
P(A) E [0,1] of its occurrence.
Mathematically, B is a a-algebra, i.e. a family left stable by
complements, countable unions and intersections; P is a
positive measure on (0,8) with p(n) = 1, and the real valued
random variables (r.v. for short) are the measurable functions
y: n ~ IR , that are such that y.l(J) E B for any Borel subset J
of IR.
The main object here is the family Xt ' t E T, of r.v.
accessible through experiment, and its statistical properties.
The probability space (n,B,P) is generally built up with these
statistical properties as a starting point, and in the present
context plays only a formal part, useful for the rigor of the
statements, but with no impact on actual computations.
4 I. Discrete Time Random Processes
In this text it will olten be possible at lirst reading, to
completely ignore a-algebras and measurability. This should
not hinder the understanding of the main results and their
applications.
2 Random Vectors
For every numerical r.v. Y on (o,B,P), we write
for the expectation of Y, when this number is well defined.
If X = (X1 .. .xk), where the coordinates Xk are numerical
r.v., we shall say that X is a (real) random vector. The law of
X, also called the joint law of Xr .. x is the probability n on
k
IRk defined by n(A) = P(X E A) for every Borel subset A of IRk.
One also says that n is the image of P by X. This is
equivalent to
fw f
I dn = n loX dP = E[f(X)]
for every function I: W IR such that one of the three terms
-+
is well defined. Replacing IR by cr, one defines simularly the
complex random vectors.
The probability distribution of X is said to have a density
cp: W -+ IR+ (with respect to Lebesgue measure) when
fw
E[f(X)] = f~ I dn = I(x)cp(x)dx
for all I as above.
3. Random Processes
3.1. Definitions.
Let T be an arbitrary set. A random process X indexed by T
is an arbitrary family Xt ' t E T, of random vectors defined
over the same probability space (o,B,p), with values in the
same space E = W or crk, called the state space of X. The set
T often represents time; in particular when T = IN or T = 7l,
