Intelligent Signal Processing

595 Pages·2001·38.642 MB·English

by  Haykin, Simon, Kosko, Bart(eds.)

INTELLIGENT SIGNAL PROCESSING
detidE yb
nomiS nikyaH
McMaster ytisrevinU
Hamilton, Ontario, Canada
Bart Kosko
ytisrevinU of Southern ainrofilaC
Los Angeles, ,AC USA
A Selected Reprint Volume
IEEE
PRESS
ehT Institute 6f Electrical and Electronics Engineers, Inc., New York
This book and other books may be purchased at a discount
from the publisher when ordered in bulk quantities. Contact:
IEEE Press Marketing
Attn: Special Sales
544 Hoes Lane
P.O. Box 1331
Piscataway, NJ 08855-1331
Fax: 1÷ 732 189 9334
For more information about IEEE Press products, visit the
IEEE Online Catalog and Store: http://www.ieee.org/store.
© 2001 by the Institute of Electrical and Electronics Engineers, Inc.
3 Park Avenue, 17th Floor, New York, NY 10016-5997.
All rights reserved. No part of this book may be reproduced in any form,
nor may it be stored in a retrieval system or transmitted in any form,
without written permission from the publisher.
Printed in the United States of America.
01 9 8 7 6 5 4 3 2 1
ISBN 0-7803-6010-9
IEEE Order No. PC5860
Library of Congress Cataloging-in-Publication Data
Intelligent signal processing / edited by Simon Haykin, Bart Kosko.
p. cm.
"A selected reprint volume."
Includes index.
ISBN 0-7803-6010-9
.1 Signal processingmDigital techniques. .2 Intelligent control systems. .3 Adaptive
signal processing. I. Haykin, Simon ,.S 1931- II. Kosko, Bart.
TK5102.9.I5455 2000
621.382'2mdc21
00-061369
This book is dedicated to Bernard Widrow
for laying the foundations of adaptive filters
Preface
This expanded reprint volume is the first book devoted to ISP has emerged recently in signal processing in much
the new field of intelligent signal processing (ISP). It grew the same way that intelligent control has emerged from
out of the November 1998 ISP special issue of the IEEE standard linear control theory. Researchers have guessed
Proceedings that the two of us coedited. This book contains less at equations to model a complex system's throughput
new ISP material and a fuller treatment of the articles that and have instead let so-called "intelligent" or "model-
appeared in the ISP special issue. free" techniques guess more for them.
Adaptive neural networks have been'the most popular
black box tools in ISP. Multilayer perceptrons and radial-
basis function networks extend adaptive linear combiners
TAHW Is ISP? to the nonlinear domain but require vastly more computa-
tion. Other ISP techniques include fuzzy rule-based sys-
ISP uses learning and other "smart" techniques to extract tems, genetic algorithms, and the symbolic expert systems
as much information as possible from incoming signal and of artificial intelligence. Both neural and fuzzy systems can
noise data. It makes few if any assumptions about the learn with supervised and unsupervised techniques. Both
statistical structure of signals and their environment. ISP are (like polynomials) universal function approximators:
seeks to let the data set tell its own story rather than They can uniformly approximate any continuous function
to impose a story on the data in the form of a simple on a compact domain, but this may not be practical in many
mathematical model. real-world cases. The property of universal approximation
Classical signal processing has largely worked with math- justifies the term "model free" to describe neural and fuzzy
ematical models that are linear, local, stationary, and systems even though equations describe their own
Gaussian. These assumptions stem from the precomputer throughput structure. They are one-size-fits-all approxima-
age. They have always favored closed-form tractability tors that can model any process if they have access to
over real-world accuracy, and they are no less extreme enough training data.
because they are so familiar. But ISP tools face new problems when we apply them
But real systems are nonlinear except for a vanishingly to more real-world problems that are nonlinear, nonlocal,
small set'of linear systems. Almost all bell-curve probabil- nonstationary, non-Gaussian, and of high dimension. Prac-
ity densities have infinite variance and infinite higher order tical neural systems may require prohibitive computation
moments. The set of bell-curve densities itself is a vanishin- to tune the values of their synaptic weights for large sets
gly small set in the space of all probability densities. Real- of high-dimensional data. New signal data may require
world systems are often highly nonlinear and can depend total retraining or may force the neural network's vast and
on many partially correlated variables. The systems can unfathomable set of synapses to forget some of the signal
have an erratic or impulsive statistical structure that varies structure it has learned. Blind fuzzy approximators need
in time in equally erratic ways. Small changes in the signal a number of if-then rules that grows exponentially with
or noise structure can lead to qualitative global changes the dimension of the training data. This volume explores
in how the system filters noise or maintains stability. how the ISP tools can address these problems.
xvii
ecaferP
ORGANIZATION OF THE VOLUME .9 Yann LeCun, L6on Bottou, Yoshua Bengio, and Pat-
rick Haffner review ways that gradient-descent learn-
The 51 chapters in this book give a representative sample ing can train a multilayer perceptron for handwritten
of current research in ISP and each has helped extend the character recognition.
ISP frontier. Each chapter passed through a full peer- .01 Shigeru Katagiri, Biing-Hwang Juang, and Chin-Hui
review filter: Lee show how to use the new technique of generalized
probabilistic gradients to solve problems in pattern rec-
.1 Steve Mann describes a novel technique that lets one ognition.
include human intelligence in the operation of a wear-
.11 Lee A. Feldkamp, Timothy M. Feldkamp, and Danil
able computer.
V. Prokhorov present an adaptive classification scheme
.2 Sanya Mitaim and Bart Kosko present the noise pro- that combines both supervised and unsupervised
cessing technique of stochastic resonance in a signal learning.
processing framework and then show how neural or
.21 J. Scott Goldstein, J.R. Guescin and I.S. Reed describe
fuzzy or other model-free systems can adaptively add
an algebraic procedure based on reduced rank model-
many types of noise to nonlinear dynamical systems
ing as a basis for intelligent signal processing
to improve their signal-to-noise ratios.
.31 Simon Haykin and David J. Thomson discuss an adap-
.3 Malik Magdon-Ismail, Alexander Nicholson, and
tive procedure for the difficult task of detecting a non-
Yaser .S Abu-Mostafa explore how additive noise af-
stationary target signal in a nonstationary background
fects information processing in problems of financial
with unknown statistics.
engineering.
.41 Robert D. Dony and Simon Haykin describe an image
.4 Partha Niyogi, Fredrico Girosi, and Tomaso Poggio
segmentation system based on a mixture of principal
show how prior knowledge and virtual sampling can
components.
expand the size of a data set that trains a generalized
.51 Aapo Hyv~irinen, Patrik Hoyar and Erkki Oja discuss
supervised learning system.
how sparse coding can denoise images.
.5 Kenneth Rose reviews how the search technique of
deterministic annealing can optimize the design of un-
These chapters show how adaptive systems can solve a
supervised and supervised learning systems.
wide range of difficult tasks in signal processing that arise
.6 Jose C. Principe, Ludong Wang, and Mark A. Motter in highly diverse fields. They are a humble but important
use the neural self-organizing map as a tool for the first step on the road to truly intelligent signal processing.
local modeling of a nonlinear dynamical system.
.7 Lee A. Feldkamp and Gintaras V. Puskorius describe
how time-lagged recurrent neural networks can per- Simon Haykin
form difficult tasks of nonlinear signal processing. McMaster University
.8 Davide Mattera, Francesco Palmieri, and Simon Hay- Hamilton, Ontario, Canada
kin describe a semiparametric form of support vector
machine for nonlinear model estimation that uses prior Bart Kosko
knowledge that comes from a rough parametric model University of Southern California
of the system under study. Los Angeles, CA
xviii
List of Contributors
retpahC 1 retpahC 4
Steve Mann Partha Niyogi
University of Toronto Massachusetts Institute of Technology
Department of Electrical and Computer Engineering Center for Biological and Computational Learning
01 King's College Road, S.F. 1002 Cambridge, MA 92120
Toronto, Ontario, S5M 3G4 CANADA
Fredrico Girosi
Massachusetts Institute of Technology
retpahC 2 Center for Biological and Computational Learning
Bart Kosko Cambridge, MA 92120
Signal and Image Processing Institute
Tomaso Poggio
Department of Electrical Engineering-Systems
Massachusetts Institute of Technology
University of Southern California
Center for Biological and Computational Learning
Los Angeles, California 4652-98009
Cambridge, MA 92120
Sanya Mitaim
Signal and Image Processing Institute retpahC 5
Department of Electrical Engineering-Systems Kenneth Rose
University of Southern California Department of Electrical and Computer Engineering
Los Angeles, California 4652-98009 University of California
Santa Barbara, _AC 60139
retpahC 3 retpahC 6
Malik Magdon-Ismail Jose .C Principe
Department of Electrical Engineering Computational NeuroEngineering Laboratory
California Institute of Technology University of Florida
39-631 Pasadena, CA 52119 Gainsville, FL 11623
Alexander Nicholson Uudong Wang
Department of Electrical Engineering Computational NeuroEngineering Laboratory
California Institute of Technology University of Florida
39-631 Pasadena, CA 52119 Gainsville, FL 11623
Yaser .S Abu-Mostafa Mark A. Motter
Department of Electrical Engineering Computational NeuroEngineering Laboratory
California Institute of Technology University of Florida
39-631 Pasadena, CA 52119 Gainsville, FL 11623
xix
INTELLIGENT SIGNAL PROCESSING
detidE yb
nomiS nikyaH
McMaster ytisrevinU
Hamilton, Ontario, Canada
Bart Kosko
ytisrevinU of Southern ainrofilaC
Los Angeles, ,AC USA
A Selected Reprint Volume
IEEE
PRESS
ehT Institute 6f Electrical and Electronics Engineers, Inc., New York
This book and other books may be purchased at a discount
from the publisher when ordered in bulk quantities. Contact:
IEEE Press Marketing
Attn: Special Sales
544 Hoes Lane
P.O. Box 1331
Piscataway, NJ 08855-1331
Fax: 1÷ 732 189 9334
For more information about IEEE Press products, visit the
IEEE Online Catalog and Store: http://www.ieee.org/store.
© 2001 by the Institute of Electrical and Electronics Engineers, Inc.
3 Park Avenue, 17th Floor, New York, NY 10016-5997.
All rights reserved. No part of this book may be reproduced in any form,
nor may it be stored in a retrieval system or transmitted in any form,
without written permission from the publisher.
Printed in the United States of America.
01 9 8 7 6 5 4 3 2 1
ISBN 0-7803-6010-9
IEEE Order No. PC5860
Library of Congress Cataloging-in-Publication Data
Intelligent signal processing / edited by Simon Haykin, Bart Kosko.
p. cm.
"A selected reprint volume."
Includes index.
ISBN 0-7803-6010-9
.1 Signal processingmDigital techniques. .2 Intelligent control systems. .3 Adaptive
signal processing. I. Haykin, Simon ,.S 1931- II. Kosko, Bart.
TK5102.9.I5455 2000
621.382'2mdc21
00-061369
This book is dedicated to Bernard Widrow
for laying the foundations of adaptive filters
Preface
This expanded reprint volume is the first book devoted to ISP has emerged recently in signal processing in much
the new field of intelligent signal processing (ISP). It grew the same way that intelligent control has emerged from
out of the November 1998 ISP special issue of the IEEE standard linear control theory. Researchers have guessed
Proceedings that the two of us coedited. This book contains less at equations to model a complex system's throughput
new ISP material and a fuller treatment of the articles that and have instead let so-called "intelligent" or "model-
appeared in the ISP special issue. free" techniques guess more for them.
Adaptive neural networks have been'the most popular
black box tools in ISP. Multilayer perceptrons and radial-
basis function networks extend adaptive linear combiners
TAHW Is ISP? to the nonlinear domain but require vastly more computa-
tion. Other ISP techniques include fuzzy rule-based sys-
ISP uses learning and other "smart" techniques to extract tems, genetic algorithms, and the symbolic expert systems
as much information as possible from incoming signal and of artificial intelligence. Both neural and fuzzy systems can
noise data. It makes few if any assumptions about the learn with supervised and unsupervised techniques. Both
statistical structure of signals and their environment. ISP are (like polynomials) universal function approximators:
seeks to let the data set tell its own story rather than They can uniformly approximate any continuous function
to impose a story on the data in the form of a simple on a compact domain, but this may not be practical in many
mathematical model. real-world cases. The property of universal approximation
Classical signal processing has largely worked with math- justifies the term "model free" to describe neural and fuzzy
ematical models that are linear, local, stationary, and systems even though equations describe their own
Gaussian. These assumptions stem from the precomputer throughput structure. They are one-size-fits-all approxima-
age. They have always favored closed-form tractability tors that can model any process if they have access to
over real-world accuracy, and they are no less extreme enough training data.
because they are so familiar. But ISP tools face new problems when we apply them
But real systems are nonlinear except for a vanishingly to more real-world problems that are nonlinear, nonlocal,
small set'of linear systems. Almost all bell-curve probabil- nonstationary, non-Gaussian, and of high dimension. Prac-
ity densities have infinite variance and infinite higher order tical neural systems may require prohibitive computation
moments. The set of bell-curve densities itself is a vanishin- to tune the values of their synaptic weights for large sets
gly small set in the space of all probability densities. Real- of high-dimensional data. New signal data may require
world systems are often highly nonlinear and can depend total retraining or may force the neural network's vast and
on many partially correlated variables. The systems can unfathomable set of synapses to forget some of the signal
have an erratic or impulsive statistical structure that varies structure it has learned. Blind fuzzy approximators need
in time in equally erratic ways. Small changes in the signal a number of if-then rules that grows exponentially with
or noise structure can lead to qualitative global changes the dimension of the training data. This volume explores
in how the system filters noise or maintains stability. how the ISP tools can address these problems.
xvii