Table Of ContentAdvanced Textbooks in Control and Signal Processing
Springer-Verlag London Ltd.
SeriesEditors
ProfessorMichaelJ.Grimble,ProfessorofIndustriaiSystemsandDirector
ProfessorMichaelA.Johnson,ProfessorofControlSystemsandDeputyDirector
IndustrialControlCentre,DepartmentofElectronicandElectricalEngineering,
UniversityofStrathclyde,GrahamHillsBuilding,50GeorgeStreet,GlasgowGI 1QE,U.K.
Othertitlespublishedin thisseries:
GeneticAlgorithms: ConceptsandDesigns
K.F. Man,K.S.TangandS.Kwong
ModelPredictiveControl
E. F.CamachoandC.Bordons
Discrete-TimeSignalProcessing
D.Williamson
PublicationDueSeptember1999
J.
E. W. Kamen and K. Su
Introduction to
Optimal Estimation
With 43 Figures
, Springer
E. W. Kamen, PhD
School of Electrical and Computer Engineering, Georgia Institute of Technology,
Atlanta, GA 30332-0250
J. K. Su, PhD
Telecommunications Laboratory, University of Erlangen-Nurnberg, Cauerstrasse 7,
D-91058 Erlangen, Germany
ISBN 978-1-85233-133-7
British Library Cataloguing in Publication Data
Kamen, Edward
Introduction to optimal estimation. -(control and signal
processing)
1.Signal processing -Digital techniques 2.Estimation
theory
I.Title II.Su, jonathan
621.3'822
ISBN 978-1-85233-133-7
Library of Congress Cataloging-in-Publication Data
Kamen, Edward W.
Introduction to optimal estimation I Edward Kamen and jonathan Su.
p. cm. --(Advanced textbooks in control and signal
processing)
Includes bibliographical references (p. ).
ISBN 978-1-85233-133-7 ISBN 978-1-4471-0417-9 (eBook)
DOI 10.1007/978-1-4471-0417-9
1. Signal processing. 2. Estimation theory. 3. Mathematical
optimaization. 1. Su, jonathan, 1969- . II. Title. III. Series.
TKS102.9.K36 1999
621.382'2--dc21 99-13005
Apart from any fair dealing for the purposes of research or private study, or criticism or review, as
permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the
publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued
by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be
sent to the publishers.
© Springer-Ve rlag London 1999
Originally published by Springer-Ve rlag London Limited in 1999
MA TLAB~ is the registered trademark ofThe Math Works, Inc., htţp:/Iwww.mathworks.com
The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a
specific statement, that such names are exempt from the relevant laws and regulations and therefore free
for general use.
The publisher makes no representation, express or implied, with regard to the accuracy of the
information contained in this book and cannot accept any legal responsibility or liability for any errors
or omissions that may be made,
Ta R. E. J(alman
-EWK
Ta Jennifer and J(endall
-JKS
Series Editors' Foreword
The topics of control engineering and signal processing
continue to flourish and develop. In common with general scientific
investigation, new ideas, concepts and interpretations emerge quite
spontaneously and these are then discussed, used, discarded or subsumed into
the prevailing subject paradigm. Sometimes these innovative concepts coalesce
into a newsub-discipline within the broad subject tapestryofcontrol and signal
processing. This preliminarybatde between old and new usually takes place at
conferences, through the Internet and in the journals ofthe discipline. After a
litde more maturity has been acquiredhas been acquired by the new concepts
thenarchivalpublicationasascientificorengineeringmonographmayoccur.
Anewconceptincontrolandsignalprocessingisknown tohavearrived
when sufficientmaterial hasdevelopedfor the topic to be taughtas aspecialised
tutorial workshop or as a course to undergraduates, graduates or industrial
engineers. The Advanced Textbooks in Control and Signal Processing Series is
designedas avehicle for the systematic presentation ofcourse material for both
popular and innovative topics in the discipline. It is hoped that prospective
authors will welcome the opportunity to publish a structured presentation of
either existing subject areas or some ofthe newer emerging control and signal
processingtechnologies.
Out of the 1940's came Wiener mtering and from the 1960's, and
KaIman, emerged the state-space system description, the KaIman mter and the
basis ofthe optimal linear gaussian regulator. This new technology dominated
the control research activities of the 1960's and 1970's. Many of the control
achievements of that era pervade the control curriculum today. The discrete
KaIman mter was a major achievement for the field ofoptimal estimation. It is
right therefore that this is the centrepiece of the new textbook Introduction to
OptimalEstimation byEdward Kamen, and Jonathan Su ofthe Georgia Institute
ofTechnology, U.S.A. In this textbook there is an introductory trio ofchapters
covering the basics of optimal estimation, an allegro of a chapter on Wiener
filtering and asolid concluding quartet of chapters on the theoretical and
applications aspects of the KaIman mter. The thorough and complete
development presented is adaptable for graduate courses, self-studyorcan even
be usedasagoodreference text.
M.].GrimbleandM.A. ]ohnson
IndustrialControlCentre
Glasgow,Scotland,U.K.
]une, 1999
Preface
This book began as a set oflecture notes prepared by the first-named author
for a senior elective on estimation taught at the University of Florida some
years ago. The notes were then expanded with a substantial amount of ma
terial added by the second-named author and used for a first-year graduate
course on estimation taught in the School of Electrical Engineering at the
Georgia Institute ofTechnology. Over the past few years, we have continued
to develop and refine the notes based in part on several teachings of the es
timation course at Georgia Tech, with the result being the present version of
the text. We have also developed a number of examples in the book using
MATLAB, and some of the homework problems require the use of MATLAB.
The primary objective in writing this book is to provide an introductory,
yet comprehensive, treatmentofboth Wiener and Kaimanfiltering along with
a development of least-squares estimation, maximum likelihood estimation,
and maximum aposteriori estimation based on discrete-time measurements.
Although this is a fairly broad range ofestimation techniques, it is possible to
cover all ofthem in some depth in a single textbook, which is precisely what
we have attempted to do here. We have also placed a good deal ofemphasis
on showing how these different approaches to estimationfit together to form a
systematicdevelopmentofoptimalestimation. It is possible to cover the bulk
of material in the book in a one-semester course, and in fact, the book has
been written to be used in a single course on estimation for seniors or first
year graduate students. The book can also be used for a one-quarter course,
although in this case, some material must be omitted due to the shorter time
period.
The background required for reading this book consists of a standard
course on probability and random variables and one or more courses on sig
nals and systems including a development ofthe state space theory oflinear
systems. It is helpful, hut not necessary, to have had some exposure to ran
dom signals and the study ofdeterministic systems driven by random signals
with random initial conditions. A summary treatment ofthis material which
is needed in the book is given in Chapter 2. In teachings ofthe course based
on the text material at Georgia Tech, we typically devote four or five 50-
x
minute lectures to the material in Chapter 2, so the students in the dass are
on somewhat the same level of proficiency in working with random variables
and signals. In this chapter and in other parts of the book we emphasize
the difference between formulations based on sampie realizations of random
signals and formulations based on random signals. This brings out the dif
ference between the issue of actually computing estimates versus the issue of
characterizing the properties ofestimates viewed as random variables.
The book begins in Chapter 1 with the description of the estimation
problem in a deterministic framework. Signal estimation is illustrated us
ing a frequency-domain approach and state estimation is approached using
~he least squares methodlogy. Then the treatment ofestimation in a stochas
tic framework begins in Chapter 2 with a summary of the theory of random
variables, random signals, and systems driven by random signals. In Chap
ter 3, different versions ofthe optimal signal estimation problem are studied,
with maximumlikelihood(ML), maximumaposterior'i (MAP), and minimum
rnean square error (MMSE) estimation covered. The case of linear MMSE
estimation leads to the Wiener filter, which is developed in Chapter 4. The
finite impulse response (FIR) Wiener filter, the noncausal infinite impulse
response (UR) Wiener filter, and the causal UR Wiener filter are all derived
in Chapter 4.
Chapter 5 begins the development ofthe KaIman filter for estimating the
state ofa linear system specified by astate model. The filter is derived using
the orthogonality principle. The innovations approach to the derivation of
the Kaiman filter is given in Chapter 6. Chapter 6 also contains results on
the time-varying case, robustness of the filter to model errors, the KaIman
predictor, and the KaIman smoother. Applications of the Kaiman filter to
target lracking, system identification, and the case of nonwhite noise are con
sidered in Chapter I. The last chapter focuses on the case when the system
state model is nonlinear, beginning with the derivation ofthe extended Kai
man filter (EhF). A new measurement update, which is more accurate in
general than the EKF measurement update, is derived using the Levenburg
Marquardt (LM) algorithm. Then applications of nonlinear filtering are con
sidered induding the identification of nonlinear systems modeled by neural
networks, FM demodulation, target tracking based on polar-coordinate mea
surements, and multiple t.arget tracking. The book also contains appendices
on the state model formulation, the z-transform, and expanded developments
of the properties of the Kaiman filter.
The authors wish to thank the following individuals for their suggestions
anc! comments on various drafts of the text: Louis Bellaire, Yong Lee, Brent
Romine, Chellury Sast.ry, Jeff Schodorf, and Jim Sills. Thanks also go to
t.he many students who have taken the course at Georgia Tech based on the
material in the book and who have offered helpful comments.
EWK, JKS
Contents
1 Introduction 1
1.1 Signal Estimation . 1
1.2 State Estimation . 9
1.3 Least Squares Estimation 13
Problems . . . . . . 22
2 Random Signals and Systems with Random Inputs 27
2.1 Random Variables . . . . . . . . . . . . . . . 27
2.2 Random Discrete-Time Signals . . . . . . . . 44
2.3 Discrete-Time Systems with Random Inputs. 51
Problems . . . . . . . . . . . . . . . . . . . . . . . 61
3 Optimal Estimation 69
3.1 Formulating the Problem 69
3.2 Maximum Likelihood and Maximum aposteriori Estimation 73
3.3 Minimum Mean-Square Error Estimation 80
3.4 Linear MMSE Estimation . . . . . . 87
3.5 Comparison ofEstimation Methods . 94
Problems . . . . . . . . . . . . . . . . . . 96
4 The Wiener Filter 101
4.1 Linear Time-Invariant MMSE Filters 101
4.2 The FIR Wiener Filter. . . . . . 105
4.3 The Noncausal Wiener Filter . . . . 114
4.4 Toward the Causal Wiener Filter . . 119
4.5 Derivation ofthe Causal Wiener Filter 130
4.6 Summary ofWiener Filters 139
Problems . 141
XII Contents
5 Recursive Estimation and the KaIman Filter 149
5.1 Estimation with Growing Memory 1.50
5.2 Estimation ofa Constant Signal .. 154
5.3 The Recursive Estimation Problem 160
5.4 The Signal/Measurement Model 160
5.5 Derivation of the KaIman Filter .. 163
5.6 Summary ofKaiman Filter Equations 169
.).7 Kaiman Filter Properties 171
5.8 The Steady-state Kaiman Filter ... 17.5
5.9 The SSKF as an Unbiased Estimator 182
5.10 Summary 184
Problems 18.5
6 Further Development of the KaIman Filter 191
6.1 The Innovations ..... 191
6.2 Derivation ofthe Kaiman Filter from the Innovations 198
6.3 Time-varying State Model and Nonstationary Noises 200
6.4 Modeling Errors . 205
6.5 Multistep Kaiman Prediction 210
6.6 Kaiman Smoothing . 211
Problems . 219
7 KaIman Filter Applications 225
7.1 Target Tracking . 22.5
7.2 Colored Process Noise 235
7.3 Correlated Noises 24.5
7.4 Colored Measurement Noise 2.52
7..) Target Tracking with Polar Measurements 253
7.6 System Identification 257
Problems 263
8 Nonlinear Estimation 269
8.1 The Extended Kaiman Filter 269
8.2 An Alternate Measurement Update 275
8.3 Nonlinear System Identification Using Neural Networks 281
8.4 Frequency Demodulation .. 28.5
8..5 Target Tracking Using the EKF . 288
8.6 Multiple Target Tracking 293
Problems 307
