Nonparametric system identification

319 Pages·2008·6.131 MB·English

by  Wlodzimierz Greblicki, Miroslaw Pawlak

Nonparametric System Identification
W.Greblicki M.Pawlak
April 27, 2007
ii
To my wife, Helena, and my chidren, Jerzy, Maria, and Magdalena
To my parents and family and those who I love
Contents
1 Introduction 3
2 Discrete-time Hammerstein systems 5
2.1 The system 5
2.2 Nonlinear subsystem 6
2.3 Dynamic subsystem identification 8
2.4 Bibliographic notes 10
3 Kernel algorithms 11
3.1 Motivation 11
3.2 Consistency 12
3.3 Applicable kernels 13
3.4 Convergence rate 16
3.5 Local error 18
3.6 Simulation example 19
3.7 Lemmas and proofs 22
3.8 Bibliographic notes 25
4 Semi-recursive kernel algorithms 27
4.1 Introduction 27
4.2 Consistency and convergence rate 28
4.3 Simulation example 30
4.4 Proofs and lemmas 31
4.5 Bibliographic notes 37
5 Recursive kernel algorithms 39
5.1 Introduction 39
5.2 Relation to stochastic approximation 39
5.3 Consistency and convergence rate 41
5.4 Simulation example 42
5.5 Auxiliary results, lemmas and proof 45
5.6 Bibliographic notes 51
6 Orthogonal series algorithms 53
6.1 Introduction 53
6.2 Fourier series estimate 55
6.3 Legendre series estimate 57
6.4 Laguerre series estimate 58
6.5 Hermite series estimate 60
6.6 Wavelet estimate 61
6.7 Local and global errors 62
iii
iv CONTENTS
6.8 Simulation example 62
6.9 Lemmas and proofs 64
6.10 Bibliographic notes 68
7 Algorithms with ordered observations 69
7.1 Introduction 69
7.2 Kernel estimates 69
7.3 Orthogonal series estimates 73
7.4 Lemmas and proofs 76
7.5 Bibliographic notes 84
8 Continuous-time Hammerstein systems 85
8.1 Identification problem 85
8.2 Kernel algorithm 87
8.3 Orthogonal series algorithms 89
8.4 Lemmas and proofs 90
8.5 Bibliographic notes 93
9 Discrete-time Wiener systems 95
9.1 The system 95
9.2 Nonlinear subsystem 96
9.3 Dynamic subsystem identification 100
9.4 Lemmas 101
9.5 Bibliographic notes 102
10 Kernel and orthogonal series algorithms 103
10.1 Kernel algorithms 103
10.2 Orthogonal series algorithms 105
10.3 Simulation example 108
10.4 Lemmas and proofs 108
10.5 Bibliographic notes 118
11 Continuous-time Wiener system 119
11.1 Identification problem 119
11.2 Nonlinear subsystem 120
11.3 Dynamic subsystem 121
11.4 Bibliographic notes 122
12 Other block-oriented nonlinear systems 123
12.1 Series-parallel block-oriented systems 123
12.2 Block-oriented systems with nonlinear dynamics 139
12.3 Concluding remarks 171
12.4 Bibliographical remarks 173
13 Multivariate nonlinear block-oriented systems 175
13.1 Multivariate nonparametric regression 175
13.2 Additive modeling and regression analysis 179
13.3 Multivariate systems 189
13.4 Concluding remarks 193
13.5 Bibliographic notes 194
CONTENTS v
14 Semiparametric identification 197
14.1 Introduction 197
14.2 Semiparametric models 198
14.3 Statistical inference for semiparametric models 201
14.4 Statistical inference for semiparametric Wiener models 207
14.5 Statistical inference for semiparametric Hammerstein models 223
14.6 Statistical inference for semiparametric parallel models 224
14.7 Direct estimators for semiparametric systems 226
14.8 Concluding remarks 241
14.9 Auxilary results, lemmas and proofs 241
14.10 Bibliographical remarks 246
A Convolution and kernel functions 247
A.1 Introduction 247
A.2 Convergence 247
A.3 Applications to probability 254
A.4 Lemmas 255
B Orthogonal functions 257
B.1 Introduction 257
B.2 Fourier series 258
B.3 Legendre series 264
B.4 Laguerre series 268
B.5 Hermite series 273
B.6 Wavelets 277
C Probability and statistics 281
C.1 White noise 281
C.2 Convergence of random variables 282
C.3 Stochastic approximation 285
C.4 Order statistics 285
vi CONTENTS
Preface
The aim of this book is to show that the nonparametric regression can be successfully applied to system
identification and how much can be achieved in this way. It gathers what has been done in the area so far,
presents main ideas, results, and some new recent developments.
The study of nonparametric regression estimation began with works published by Cencov, Watson, and
Nadarayaintheearlysixtiesofthepastcentury. Thehistoryofnonparametricregressioninsystemidentifica-
tionbeganabouttenyearslater. Suchmethodshavebeenappliedtotheidentificationofcompositesystems
consistingofnonlinearmemorylesssystemsandlineardynamicones. Thereforetheapproachisstrictlycon-
nectedwithsocalledblock-orientedmethodsdevelopedatleastsinceNarendraandGallmanworkpublished
in 1966. Hammerstein and Wiener structures are most popular and have received the greatest attention
with numerous applications. Fundamental for nonparametric methods is the observation that the unknown
characteristic of the nonlinear subsystem or its inverse can be represented as regression functions.
In terms of the a priori information, standard identification methods and algorithms work when it is
parametric,i.e.,whenourknowledgeaboutthesystemisratherlarge,e.g.,whenweknowthatthenonlinear
subsystem has a polynomial characteristic. In this book, the information is much smaller, nonparametric.
The mentioned characteristic can be, e.g., any integrable or bounded or, even, any Borel function.
It can thus be said that this book associates block oriented system identification with nonparametric
regression estimation, shows how to identify nonlinear subsystems, i.e., to recover their characteristics when
the a priori information is small. Owe to this, the approach should be of interest not only for researchers
but also for people interested in applications.
Chapters 2–7 are devoted to discrete-time Hammerstein systems. Chapter 2 presents the basic ideas
behind nonparametric methods. The kernel algorithm is presented in Chapter 3, its semirecursive versions
are examined in Chapter 4 while Chapter 5 deals with fully recursive modifications derived from the idea
of stochastic approximation. Then, next chapter is on the orthogonal series method. Algorithms using
trigonometric, Legendre, Laguerre and Hermite series are investigated. Some place is devoted to estimation
methods based on wavelets. Algorithms based on ordered observations are presented and examined in
Chapter 7. Chapter 8 discusses the algorithms when applied to continuous-time systems.
The Wiener system is identified in Chapters 9–11. Chapter 9 presents the motivation for nonparametric
algorithms which are studied in the next two chapters devoted to the discrete and continuous-time Wiener
systems, respectively. Chapter 12 is concerned with the generalization of our theory to other block-oriented
nonlinear systems. This includes, among others, parallel models, cascade-parallel models, sandwich models,
and generalized Hammerstein systems possessing local memory. In Chapter 13 the multivariate versions
of block-oriented systems are examined. The common problem of multivariate systems, i.e., the curse of
dimensionality is cured by using low-dimensional approximations. With respect to this issue models of
the additive form are introduced and examined. In Chapter 14 we develop identification algorithms for
a semiparametric class of block-oriented systems. Such systems are characterized by a mixture of finite
dimensional parameters and nonparametric functions, this normally being a set of univariate functions.
The reader is encouraged to look into appendices in which fundamental information about tools used in
the book is presented in detail. Appendix A is strictly related to kernel algorithms while B is tied with the
orthogonal series ones. Appendix C recalls some facts from probability theory and presents results from the
theory of order statistics used extensively in Chapter 7.
Over the years, our work has benefited greatly from the advice and support of a number of friends and
colleagues with interest in ideas of non-parametric estimation, pattern recognition and nonlinear system
modeling. There are too many names to list here, but special mention is due to Adam Krzyz˙ak, as well
1
2 CONTENTS
as Danuta Rutkowska, Leszek Rutkowski, Alexander Georgiev, Simon Liao, Pradeepa Yahampath, Vu-Luu
Nguyen and Yongqing Xin our past Ph.D. students, now professors at universities in Canada, the United
States, and Poland. Cooperation with them has been a great pleasure and given us a lot of satisfaction. We
are deeply indebted to Zygmunt Hasiewicz, Ewaryst Rafajl(cid:32)owicz, Uli Stadtmu¨ller, Ewa Rafajl(cid:32)owicz, Hajo
Holzmann, Andrzej Kozek who have contributed greatly to our research in the area of nonlinear system
identification, pattern recognition, and nonparametric inference.
Finally, but by no means least, we would like to thank Mount-first Ng for helping us with a number of
typesetting problems. Ed Shwedyk and January Gnitecki have provided support for correcting English
grammar.
We also thank Ms. Anna Littlewood, from Cambridge Press, for being a very supportive and patient editor.
Research presented in this monograph was partially supported by research grants from Wrocl(cid:32)aw University
of Technology, Wrocl(cid:32)aw, Poland and NSERC of Canada.
Wroc(cid:32)law, Winnipeg Wl(cid:32)odzimierz Greblicki, Miro(cid:32)lsaw Pawlak
April, 2007
Chapter 1
Introduction
System identification, as a particular process of statistical inference, exploits two types of information. The
firstisexperiment,theother,calledapriori,isknownbeforemakinganymeasurements. Inawidesense,the
a priori informationconcernsthesystemitselfandsignalsenteringthesystem. Elementsoftheinformation
are, e.g.:
• the nature of the signals which may be random or nonrandom, white or correlated, stationary or not,
their distributions can be known in full or partially (up to some parameters) or completely unknown,
• general information about the system which can be, e.g., continuous or discrete in the time domain,
stationary or not,
• the structure of the system which can be of the Hammerstein or Wiener type, or other,
• the knowledge about subsystems, i.e., about nonlinear characteristics and linear dynamics.
In other words, the a priori information is related to the theory of the phenomena taking place in the
system (a real physical process) or can be interpreted as a hypothesis (if so, results of the identification
should be necessarily validated) or can be abstract in nature.
This book deals with systems consisting of nonlinear memoryless and linear dynamic subsystems, i.e.,
Hammerstein and Wiener ones. With respect to them the a priori information is understood in a narrow
sense since it relates to the subsystems only and concerns the a priori knowledge about their descriptions.
The characteristic of the nonlinear subsystem is recovered with the help of nonparametric regression
estimates. The kernel and orthogonal series methods are used. Ordered statistics are also applied. Both
off-line and on-line algorithms are investigated. We examine only these estimation methods and nonlinear
models for which we are able to deliver fundamental results in terms of consistency and convergence rates.
Therearetechniques, e.g., neuralnetworks, whichmayexhibitapromisingperformancebuttheirstatistical
accuracy is mostly unknown.
ForthenonparametricregressionassuchthereaderisreferredtoGy¨orfi,Kohler,Krzyz˙akandWalk[204],
H¨ardle [219], Prakasa Rao [340], Simonoff [388] or Wand and Jones [425]. Wavelet estimates are discussed
in Walter [424].
Parametric methods are beyond the scope of this book, nevertheless, we mention Brockwell and Davies
[38], Ljung [288], Norton [315], Zhu [451] or S¨oderstrom and Stoica [391].
Nonlinear system identification within the parametric framework is studied by Nells [312], Westwick
and Kearney [432] , Marmarelis and Marmarelis [299], and Bendat [18]. A comprehensive list of references
concerningnonlinearsystemidentificationandapplicationshasbeengivenbyGiannakisandSerpendin[120].
A modern statistical inference for nonlinear time series is presented in Fan and Yao [105].
Itshouldbestressedthatnonparametricandparametricmethodsaresupposedtobeappliedindifferent
situations. The first are used when the a priori information is nonparametric. Clearly, in such a case,
parametriconescanonlyapproximate,butnotestimate,theunknowncharacteristics. Whentheinformation
is parametric, parametric methods are the natural choice. If, however, the unknown characteristic is a
complicatedfunctionofparametersconvergenceanalysisgetsdifficultand. Moreover,seriouscomputational
3
4 CHAPTER 1. INTRODUCTION
problems can occur. In such circumstances one can resort to nonparametric algorithms since, from the
computationalviewpoint,theyarenotdiscouraging. Onthecontrary,theyaresimplebut,however,consume
computermemory, since, e.g., kernelestimatesrequirealldatatobestored. Neverthelessitcanbesaidthat
the two approaches do not compete with each other since they are designed to be applied in quite different
situations. The situations differ from each other by the amount of the a priori information about the
identified system. A compromise between these two separate worlds can be, however, made by restricting
a class of nonparametric models to such which consist of a finite dimensional parameter and nonlinear
characteristicswhichrunthroughanonparametricclassofunivariatefunctions. Suchsemiparametricmodels
can be efficiently identified and the theory of semiparametric identification is examined in this book.
For two number sequences a and b , a = O(b ) means that a /b is bounded in absolute value as
n n n n n n
n→∞. In particular, a =O(1) denotes that a is bounded, i.e., that sup |a |<∞. Writing a ∼b we
n n n n n n
mean that a /b has a nonzero limit as n→∞.
n n
Throughout the book, “almost everywhere” means “almost everywhere with respect to the Lebesgue
measure” while “almost everywhere (µ)” means “almost everywhere with respect to the measure µ”.