Principles of adaptive filters and self-learning systems

397 Pages·2005·2.27 MB·English

by  Anthony Zaknich

Advanced Textbooks in Control and Signal Processing
SeriesEditors
ProfessorMichaelJ.Grimble,ProfessorofIndustrialSystemsandDirector
ProfessorEmeritusMichaelA.Johnson,ProfessorofControlSystemsandDeputyDirector
IndustrialControlCentre,DepartmentofElectronicandElectricalEngineering,
UniversityofStrathclyde,GrahamHillsBuilding,50GeorgeStreet,GlasgowG11QE,U.K.
Othertitlespublishedinthisseries:
GeneticAlgorithms
K.F.Man,K.S.TangandS.Kwong
NeuralNetworksforModellingandControlofDynamicSystems
M.Nørgaard,O.Ravn,L.K.HansenandN.K.Poulsen
ModellingandControlofRobotManipulators(2ndEdition)
L.SciaviccoandB.Siciliano
FaultDetectionandDiagnosisinIndustrialSystems
L.H.Chiang,E.L.RussellandR.D.Braatz
SoftComputing
L.Fortuna,G.Rizzotto,M.Lavorgna,G.Nunnari,M.G.XibiliaandR.Caponetto
StatisticalSignalProcessing
T.Chonavel
Discrete-timeStochasticProcesses(2ndEdition)
T.Söderström
ParallelComputingforReal-timeSignalProcessingandControl
M.O.Tokhi,M.A.HossainandM.H.Shaheed
MultivariableControlSystems
P.AlbertosandA.Sala
ControlSystemswithInputandOutputConstraints
A.H.GlattfelderandW.Schaufelberger
AnalysisandControlofNon-linearProcessSystems
K.Hangos,J.BokorandG.Szederkényi
ModelPredictiveControl(2ndEdition)
E.F.CamachoandC.Bordons
DigitalSelf-tuningControllers
V.Bobál,J.Böhm,J.FesslandJ.Macháˇcek
ControlofRobotManipulatorsinJointSpace
R.Kelly,V.SantibáñezandA.Loría
PublicationdueJuly2005
RobustControlDesignwithMATLAB®
D.-W.Gu,P.Hr.PetkovandM.M.Konstantinov
PublicationdueJuly2005
ActiveNoiseandVibrationControl
M.O.Tokhi
PublicationdueNovember2005
A. Zaknich
Principles of
Adaptive Filters and
Self-learning
Systems
With95Figures
123
AnthonyZaknich,PhD
SchoolofEngineeringScience,RockinghamCampus,
MurdochUniversity,SouthStreet,Murdoch,WA6150,Australia
and
CentreforIntelligentInformationProcessingSystems,
SchoolofElectrical,ElectronicandComputerEngineering,
TheUniversityofWesternAustralia,
35StirlingHighway,Crawley,WA6009,Australia
InstructorsSolutionsManualinPDFcanbedownloadedfromthebook’spage
atspringeronline.com
BritishLibraryCataloguinginPublicationData
Zaknich,Anthony
Principlesofadaptivefiltersandself-learningsystems.
(Advancedtextbooksincontrolandsignalprocessing)
1.Adaptivefilters2.Adaptivesignalprocessing3.System
analysis
I.Title
621.3’815324
ISBN-10:1852339845
LibraryofCongressControlNumber:2005923608
Apartfromanyfairdealingforthepurposesofresearchorprivatestudy,orcriticismorreview,as
permittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,
stored or transmitted, in any form or by any means, with the prior permission in writing of the
publishers,orinthecaseofreprographicreproductioninaccordancewiththetermsoflicencesissued
bytheCopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethosetermsshouldbe
senttothepublishers.
AdvancedTextbooksinControlandSignalProcessingseriesISSN1439-2232
ISBN-10 1-85233-984-5
ISBN-13 978-1-85233-984-5
SpringerScience+BusinessMedia
springeronline.com
©Springer-VerlagLondonLimited2005
Theuseofregisterednames,trademarks,etc.inthispublicationdoesnotimply,evenintheabsenceof
aspecificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandtherefore
freeforgeneraluse.
Thepublishermakesnorepresentation,expressorimplied,withregardtotheaccuracyoftheinfor-
mationcontainedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsor
omissionsthatmaybemade.
Typesetting:Camerareadybyauthor
Production:LE-TEXJelonek,Schmidt&VöcklerGbR,Leipzig,Germany
PrintedinGermany
69/3141-543210Printedonacid-freepaperSPIN10978566
Franica,Nikola,Iris,NelliandPiPi
Series Editors’ Foreword
The topics of control engineering and signal processing continue to flourish and
develop. In common with general scientific investigation, new ideas, concepts and
interpretations emerge quite spontaneously and these are then discussed, used,
discarded or subsumed into the prevailing subject paradigm. Sometimes these
innovative concepts coalesce into a new sub-discipline within the broad subject
tapestry of control and signal processing. This preliminary battle between old and
new usually takes place at conferences, through the Internet and in the journals of
the discipline. After a little more maturity has been acquired by the new concepts
then archival publication as a scientific or engineering monograph may occur.
A new concept in control and signal processing is known to have arrived when
sufficient material has evolved for the topic to be taught as a specialised tutorial
workshop or as a course to undergraduate, graduate or industrial engineers.
Advanced Textbooks in Control and Signal Processing are designed as a vehicle
for the systematic presentation of course material for both popular and innovative
topics in the discipline. It is hoped that prospective authors will welcome the
opportunity to publish a structured and systematic presentation of some of the
newer emerging control and signal processing technologies in the textbook series.
This new advanced course textbook for the control and signal processing series,
Principles of Adaptive Filtering and Self-learning Systems by Anthony Zaknich,
presents a bridge from classical filters like the Wiener and Kalman filters to the
new methods that use neural networks, fuzzy logic and genetic algorithms. This
links the classification-based adaptive filtering methods to the innovative non-
classical techniques, and both are presented in a unified manner. This eliminates
the dichotomy of many textbooks which focus on either classical methods or non-
classical methods.
The textbook is divided into six parts: Introduction, Modelling, Classical Filters
and Spectral Analysis, (Classical) Adaptive Filters, Non-Classical Adaptive
Systems and finally Adaptive Filter Applications. As befits an advanced course
textbook there are many illustrative examples and problem sections. An outline
Solutions Manual complete with a typical course framework and with specimen
examination papers is also available to tutors to download from
springeronline.com.
Solid foundations for a possible adaptive filtering course are laid in the
Introduction (Part I) with an overview chapter and a linear systems and stochastic
processes chapter of nearly 60 pages. All the main basic terms and definitions are
found in this introductory part.
viii Series Editors’ Foreword
Signal models and optimization principles are covered in Part II. In the two
chapters of this part are found concepts like the pseudo-inverse, matrix singular
value decompositions, least squares estimation and Prony’s method.
Filters proper emerge in Part III which covers the classical Wiener filter, the
Kalman filter and power spectral density analysis methods. The chapter on the
Kalman filter is nicely presented since it includes examples and an assessment of
the advantages and disadvantages of the Kalman filter method.
In Part IV, adaptation and filtering are united to yield a set of chapters on
adaptive filter theory. Since many of the techniques are used by control engineers
it is pleasing to have a chapter devoted to adaptive control systems (Chapter 11).
In fact the way that the author keeps linking the specifics of filtering theory to the
broader fields of filter implementation, practical applications and control systems
is a real strength of this book.
Neural networks, fuzzy logic and genetic algorithms are the constituent
techniques of the non-classical methods presented in Part V. Each technique is
given a chapter-length presentation and each chapter is full of reviews,
perspectives and applications advice. In all three chapters links are made to similar
applications in the field of control engineering. This gives credence to the idea that
twin adaptive filtering and digital control systems courses would be powerful
reinforcing strategy in any advanced systems postgraduate qualification.
The final part of the book comprises two chapters of adaptive filter applications
(Part VI). Whilst the range of applications presented is not exhaustive, fields like
speech encoding, event detection, data transmission and discussing both classical
and non-classical filter solution methods are covered.
In summary Anthony Zaknich’s is a particularly welcome entry to the
Advanced Textbooks in Control and Signal Processing series. Graduate students,
academics and industrial engineers will find the book is a constructive introduction
to adaptive filtering with many of the chapters appealing to a wider control,
electronic and electrical engineering readership.
M.J. Grimble and M.A. Johnson
Industrial Control Centre
Glasgow, Scotland, U.K.
January 2005
Preface
This book can be used as a textbook for a one semester undergraduate or
postgraduate introductory course on adaptive and self-learning systems for signal
processing applications. The topics are introduced and discussed sufficiently to
givethereaderadequatebackgroundtobeabletoconfidentlypursuethematdepth
in more advanced literature. Subject material in each chapter is covered concisely
butwithadequatediscussionandexamplestodemonstratetheimportantideas.Key
chapters include exercises at the end of the chapter to help provide a deeper topic
understanding. It is strongly recommended that the exercises be attempted first
beforemakingreferencetotheanswers,whichareavailableinaseparateSolutions
Manual. The Solutions Manual also includes a possible course outline using this
bookasthetextbook,plussampleassignmentsandrepresentativepastexamination
paperswithsolutionsthatmayaidinthedesignandconductofsuchacourse.
Topicsarepresentedinaprogressivesequencestartingwithashortintroduction
to adaptive filters and systems, linear systems and stochastic process theory.
UnavoidablythefirstChapterreferstosomeadvancedconceptsthataremorefully
describedinlaterChapters.InthefirstreadingofthisfirstChapteritmaybebestto
gloss over these and just accept a general understanding of what is described to
gaininitialfamiliaritywiththeterminologyandideas.
The introductory part of the book is followed by the detailed developments of
system and signal modelling theory, classical Wiener filter theory, Kalman filter
theory,spectralanalysistheory,classicaladaptivelinearandnonlinearfiltertheory,
adaptivecontrolsystems,nonclassicaladaptivesystemstheory,throughtoadaptive
filter application issues. Although the book concentrates on the more established
adaptive filter theory, introductions to artificial neural networks, fuzzy logic and
genetic algorithms are also included to provide a more generic perspective of the
topic ofadaptive learning. Asignificantfurtherofferingofthebookisamethodto
seamlessly combine a set of both classical and/or nonclassical adaptive systems to
form a powerful self-learning engineering solution method that is capable of
solving verycomplex nonlinear problems typical of the underwater acoustic signal
processingenvironment,aswellasotherequallydifficultapplicationdomains.
The concepts of system adaptation and self-learning are quite general and can
conjure up all sorts of ideas. In this book, these concepts have a very specific
meaning. Theysignifythata systemcanbe configured insucha waythatallowsit
toinsomesenseprogressivelyorganiseitselftowardsalearnedstateinresponseto
input signals. All learning has to be with respect to some appropriate context and
x Preface
suitable constraints. A human designer hoping to achieve some meaningful
functionality will of necessity initially supply the required context and constraints.
The systems of interest here start out with predetermined structures of some sort.
However, these structures have sufficient inherent flexibility to be able to adapt
their parametersand componentsto achieve specific solutionsformed fromclasses
of relationships predetermined by those structures. Sometimes all the signals
involved will come exclusively out of the system’s environment and sometimes
some of the signals will be supplied by a human supervisor, but in all cases the
systemmustbeabletoeventuallyachievecoherentsolutionswithinthatcontextby
itsownadaptiveorlearningprocesses.
Thedifferencebetweenanadaptivesystemandalearningsystemisinprinciple
slight. Itisto do withhistoryand convention,butmoreimportantlyitistodowith
thedegreeofflexibilityallowedbythesystemmodel.Classicallinearandnonlinear
adaptivefilterstypicallyhavelessflexibilityinthewaytheycanchangethemselves
and are generally referred to as adaptive. On the other hand nonclassical adaptive
systems such as Artificial Neural Networks (ANN), Adaptive Fuzzy Logic (FL),
GeneticAlgorithms(GA),andothermachinelearningsystemshaveamuchgreater
flexibility inherent within their structures and therefore can be seen more as
learningsystems.
The field ofnonclassicallearningsystemsisoftenreferredtoasComputational
Intelligence (CI). However, the word intelligence can also conjure up unintended
meanings. Intelligent methods are often referred to as model-free and are mostly
basedontheexamplesignals(ordata)ratherthanontheconstraintsimposedbythe
model itself (Haykin and Kosko 2001). In this special context, “more intelligent”
implies more able to extract system information from the example data alone and
belessdependentonapriorienvironmentalandsysteminformation.Itisfairtosay
that no limited physical system can be absolutely model free. Although some
models like ANNs can be made to be very flexible, having a huge number of
possibleconfigurationsorstates,itisreallyamatterofdegree.Beforetheadventof
recent finite data based statistical learning theories (Cherkassky and Mulier 1998,
Vapnik 1998, 2001) it was commonplace to limit the flexibility of learning
machines down to a sufficient degree in order to force some regularization or
smoothnessinalocalsense.Thisissomewhatlikeclassicaladaptivesystemsdoby
keepingthenumberoftheirmodelparameters(modelorder)toaslowasnecessary
in order to achieve good generalization results for the chosen problem. The
learning has to have a degree of local smoothness such that close input states are
close to their corresponding output states; else generalization oflearningwould be
impossible. The higher the order of the model with respect to the order of the
problem the more difficult it is for the adaptive system to maintain adequate
performance.
AuniquesituationisapplicabletoGAswithrespecttoCIandmachinelearning
inthattheyhavebeenabletoconsistentlycreatenumerousandvariedprogrammed
solutions automatically, starting from a high level statement of what needs to be
done (Koza et al 2003). Using a common generic approach they have produced
parameterized topologies for a vast number of complex problems. In that sense
GAs are exhibiting what Turing called Machine Intelligence (MI). To himMI was