Stochastic Processes: Inference Theory

655 Pages·2000·24.12 MB·English

by  M. M. Rao (auth.)

Stochastic Processes
Mathematics and Its Applications
Managing Editor:
M. HAZEWINKEL
Centre for Mathematics and Computer Science, Amsterdam, The Netherlands
Volume 508
Stochastic Processes
Inference Theory
by
M.M. Rao
University of California,
Riverside, California, U.S.A.
SPRINGER-SCIENCE+BUSINESS MEDIA, B.V.
A C.I.P. Catalogue record for this book is available from the Library of Congress.
ISBN 978-1-4419-4832-8 ISBN 978-1-4757-6596-0 (eBook)
DOI 10.1007/978-1-4757-6596-0
Printed on acid-free paper
All Rights Reserved
© 2000 Springer Science+Business Media Dordrecht
Originally published by Kluwer Academic Publishers in 2000
No part of the material protected by this copyright notice may be reproduced or
utilized in any form or by any means, electronic or mechanical,
including photocopying, recording or by any information storage and
retrieval system, without written permission from the copyright owner.
To Professors
Vlf Grenander and Tom S. Pitcher
whose fundamental and deep contributions
shaped stochastic inference
CONTENTS
Preface Xl
Chapter I: Introduction and Preliminaries 1
1.1 The problem of inference 1
1.2 Testing a hypothesis 4
1.3 Distinguishability of hypotheses 7
1.4 Estimation of parameters 10
1.5 Inference as a decision problem 12
1.6 Complements and exercises 15
Bibliographical notes 17
Chapter II: Some Principles of Hypothesis Testing 19
2.1 Testing simple hypotheses ....... 19
2.2 Reduction of composite hypotheses 38
2.3 Composite hypotheses with iterated weights 42
2.4 Bayesian methodology for applications 48
2.5 Further results on composite hypotheses 55
2.6 Complements and exercises 67
Bibliographical notes 70
Chapter III: Parameter Estimation and Asymptotics 73
3.1 Loss functions of different types . . . . . 73
3.2 Existence and other properties of estimators . 75
3.3 Some principles of estimation . . . . . 88
3.4 Asymptotics in estimation methodology 107
3.5 Sequential estimation 117
3.6 Complements and exercises 125
Bibliographical notes 130
vii
viii Contents
Chapter IV: Inferences for Classes of Processes 133
4.1 Testing methods for second order processes 133
4.2 Sequential testing of processes ..... 155
4.3 Weighted unbiased linear least squares prediction 179
4.4 Estimation in discrete parameter models 196
4.5 Asymptotic properties of estimators 200
4.6 Complements and exercises 213
Bibliographical notes 219
Chapter V: Likelihood Ratios for Processes 223
5.1 Sets of admissible signals or translates 223
5.2 General Gaussian processes .... 247
5.3 Independent increment and jump Markov processes 277
5.4 Infinitely divisible processes 298
5.5 Diffusion type processes 314
5.6 Complements and exercises 327
Bibliographical notes 335
Chapter VI: Sampling Methods for Processes 339
6.1 Kotel'nikov-Shannon methodology 339
6.2 Band limited sampling ..... 348
6.3 Analyticity of second order processes 353
6.4 Periodic sampling of processes and fields 358
6.5 Remarks on optional sampling 374
6.6 Complements and exercises 375
Bibliographical notes 380
Chapter VII: More on Stochastic Inference 383
7.1 Absolute continuity of families of probability measures 383
7.2 Likelihood ratios for families of non Gaussian measures 405
7.3 Extension to two parameter families of measures 413
7.4 Likelihood ratios in statistical communication theory 429
7.5 The general Gaussian dichotomy and Girsanov's theorem 435
7.6 Complements and exercises 452
Bibliographical notes 462
Chapter VIII: Prediction and Filtering of Processes 465
8.1 Predictors and projections . . . . . . . . . . . 465
8.2 Least squares prediction: the Cramer-Hida approach . 480
Contents ix
8.3 Linear filtering: Bochner's formulation 488
8.4 Kalman-Bucy filters: the linear case 509
8.5 Kalman-Bucy filters: the nonlinear case 535
8.6 Complements and exercises 549
Bibliographical notes 553
Chapter IX: Nonparametric Estimation for Processes 557
9.1 Spectra for classes of second order processes 557
9.2 Asymptotically unbiased estimation of bispectra 560
9.3 Resampling procedure and consistent estimation 563
9.4 Associated spectral estimation for a class of processes 572
9.5 Limit distributions of (bi)spectral function estimators 583
9.6 Complements and exercises 593
Bibliographical notes 597
Bibliography 601
Notation index 627
A uthor index 633
Subject index 639
Preface
The material accumulated and presented in this volume can be ex
plained easily. At the start of my graduate studies in the early 1950s, I
came across Grenander's (1950) thesis, and was much attracted to the
entire subject considered there. I then began preparing for the neces
sary mathematics to appreciate and possibly make some contributions
to the area. Thus after a decade of learning and some publications
on the way, I wanted to write a modest monograph complementing
Grenander's fundamental memoir. So I took a sabbatical leave from
my teaching position at the Carnegie-Mellon University, encouraged
by an Air Force Grant for the purpose, and followed by a couple of
years more learning opportunity at the Institute for Advanced Study
to complete the project. As I progressed, the plan grew larger needing
a substantial background material which was made into an independent
initial volume in (1979). In its preface I said: "My intension was to
present the following material as the first part of a book treating the In
ference Theory of stochastic processes, but the latter account has now
receded to a distant future," namely for two more decades! Meanwhile,
a much enlarged second edition of that early work has appeared (1995),
and now I am able to present the main part of the original plan. In fact,
while this effort took on the form of a life's project, and developing all
the necessary backup material during the long gestation period, I have
written some seven books and directed several theses on related topics
that helped me appreciate the main subject much better. It is now
termed 'stochastic inference' as an abbreviation as well as a homage to
Grenander's "Stochastic processes and statistical inference" .
Let me explain the method adapted in preparing this work. At
the outset, it became clear that there can be no compromise with the
mathematics of inference theory. One observes that, broadly speak
ing, inference has theoretical, practical, philosophical, and interpreta
tive aspects. But these components are also present in other scientific
studies. However, for inference theory all these parts are founded on
sound mathematical principles, a violation of which leads to unintended
xi
xii Preface
controversies. Thus the primary concern here is on mathematical ram
ifications of the subject, and the work is illustrated with a number of
important examples, many of independent interest.
It is noted that, as a basis of the classical statistical inference, two
original sources are visible. The crucial idea on hypothesis testing is
founded in the formulation of the Neyman-Pearson lemma which itself
has a firm backing of the calculus of variations. All later developments
of the subject are extensions of this result. Similarly, the basic idea
of estimating a parameter of a distribution is founded on Fisher's for
mulation of the maximum likelihood (ML). The classical inference the
ory (for finite samples) grew out of these two fundamental principles.
But the subject of stochastic processes deals with infinite collections
of random variables, and there is a real barrier here to cross in or
der to apply the classical ideas. The necessary new accomplishment is
the formulation by Grenander who showed that the (abstract) Radon
Nikodym derivative must replace the likelihood function of the finite
sample case, leading the way to stochastic inference. Now the deter
mination of the general likelihood ratio (or the RN-derivative) involves
an intricate analysis for which one has to employ several different tech
nical tools. This was successfully done by Grenander himself and it
was advanced further by Pitcher. Thus for a proper understanding of
the subject, a greater preparation and a concerted effort are needed,
and to aid this,the previous concepts are restated at various places. I
shall now indicate, with some outlines of the material [more detailed
summaries are at the beginning of chapters], how my original plan is
executed.
The first three chapters contain the basic inference theory, as de
scribed above, from the point of view of adapting it to stochastic pro
cesses. Thus the work in Chapter I, begins with the question of dis
tinguishability (or 'identifiability') of a pair of probability measures or
distributions, before a hypothesis testing question can be raised. It is
better to know this fact in the theory, since, as shown by example, there
exist unequal distributions which cannot be distinguished. So condi
tions are presented for distinguishability. Then the inference problem
and its decision theoretic setup as a unifying formulation of both testing
and estimation are discussed. Chapter II is devoted to detailed anal
ysis, applications, and extensions of the Neyman-Pearson theory, and