Mathematical Methods and Algorithms for Signal Processing

978 Pages·2006·38.97 MB·English

Mathematical Methods and Algorithms
for
Signal Processing

Mathematical Methods and Algorithms
for
Signal Processing
Todd I(. Moon
Utah State University
Wynn C. Stirling
Brigham Young University
PRENTICE HALL
Upper Saddle River, NJ 07458
Library of Congress Calaloging-in-Publication Data
Moon. Todd K
Mathematical methods and algorithms for signal processing 1
Todd K Moon, Wynn C Stirllng
P cm
Includes bibliographical references and index
ISBN 0-201 -361 86-8
I Srgnal procerslng-Mathematics 2 Algonthm'i I Stirl~ng
Wynn C I1 Title
TK5102 9 M63 1999
621 382'2'0151-dc21
99-3 1038
CIP
Publisher Marsha Horron
Editorial D~rector Tom Robbz~zs
Production editor Brirrney Cori /gun-Mc Elroy
Managrng editor VInce O 'Brren
Assistant managing ed~torE ileen Clurk
Art director Kevin Bern]
Cover deslgn Karl Mljajlmu
Manufacturing manager Trudy Pt rclottl
Asslrtant vice presrdent of production and manufactunng David W Riccardr
@ 2000 by Prentlce Hall
Prentice-Hall, Inc.
Upper Saddle River. New Jersey 07458
All rightc reserked No part of this book may be reproduced rn any form or by any means, wlthout
permlssron In wnting from the publisher
The author and publi\her of thls book have used their best efforts In preparing thls book These ef-
forts include the development, research, and testing of the theorres and programs to determine their
effect~venessT he author and pubhsher make no warranty of any k~nd.e xpressed or implled, wrth
regard to thece programs or the documentation contaned In this book The author and publisher shall
not be liable in any event for incidental or consequential damages in connection. or arlslng out of, the
furn~shingp. erformance. or use of these programs
Pnnted in the Unlted States of America
1 0 9 8 7 6 5 4
ISBN 0-201-361 86-8
PRENTICE-H~LIILU TER~\~A'IIO~(AULK ) LIMITED Londorl
PREYTICC-HAOLFL 4L1STRAL14P TY LIMITED, Sydnei
PRE\TICE-HALCLA NADIAN C . Toronto
PREYTICE-HALHLIS PA%OAMERICAS 'AU A. M, exico
PREYTICE-HAL01L- I NDIA PRIVATLEI MITFDN, ~MD.~ lhi
PREYTIC-HI ALLO F JAPANI. ~ cT o k ~
PRFNTICF-HA1 1 (SIN(~APOj PRTIE- 1-TD Srrzyuporr
EDITOR4 PKFZTICF-HAIL10I BRA\II LIL)A Rlo de J<~rioro
Contents
I Introduction and Foundations
1 Introduction and Foundations
1.1 What is signal processing? .........................................
1.2 Mathematical topics embraced by signal processing ...................
1.3 Mathematical models ..............................................
1.4 Models for linear systems and signals ...............................
1.4.1 Linear discrete-time models .................................
1.4.2 Stochastic MA and AR models .............................
1.4.3 Continuous-time notation ..................................
1.4.4 Issues and applications ....................................
1.4.5 Identification of the modes .................................
1.4.6 Control of the modes ......................................
1.5 Adaptive filtering ................................................
1.5.1 System identification ......................................
1 5.2 Inverse system identification ...............................
1 .5 .3 Adaptive predictors .......................................
1 .5 .4 Interference cancellation ...................................
1.6 Gaussian random variables and random processes ...................
1.6.1 Conditional Gaussian densities .............................
1.7 Markov and Hidden Markov Models ...............................
1.7.1 Markov models ...........................................
1.7.2 Hidden Markov models ....................................
1.8 Some aspects of proofs ...........................................
1.8.1 Proof by computation: direct proof .........................
1 3.2 Proof by contradiction .....................................
1.8.3 Proof by induction ........................................
1.9 An application: LFSRs and Massey's algorithm .....................
1.9.1 Issues and applications of LFSRs ...........................
1.9.2 Massey's algorithm .......................................
1.9.3 Characterization of LFSR length in Massey's algorithm .......
1.10 Exercises ........................................................
1.11 References ......................................................
I1 Vector Spaces and Linear Algebra
2 Signal Spaces
2.1 Metric spaces ....................................................
2.1.1 Some topological terms ....................................
2.1.2 Sequences, Cauchy sequences. and completeness .............
vi Contents
2.1.3 Technicalities associated with the L, and L, spaces .........
Vectorspaces ....................................................
2.2.1 Linear combinations of vectors ............................
2.2.2 Linear independence .....................................
2.2.3 Basis and dimension .....................................
2.2.4 Finite-dimensional vector spaces and matrix notation ........
Norms and normed vector spaces ..................................
2.3.1 Finite-dimensional normed linear spaces ...................
Inner products and inner-product spaces ............................
2.4.1 Weak convergence .......................................
Inducednorrns ...................................................
The Cauchy-Schwarz inequality ..................................
Direction of vectors: Orthogonality ...............................
Weighted inner products .........................................
2.8.1 Expectation as an inner product ..........................
Hilbert and Banach spaces .......................................
Orthogonal subspaces ...........................................
Linear transformations: Range and nullspace .......................
Inner-sum and direct-sum spaces .................................
Projections and orthogonal projections ............................
2.13.1 Projection matrices .....................................
The projection theorem ..........................................
Orthogonalization of vectors .....................................
Some final technicalities for infinite dimensional spaces .............
Exercises .......................................................
References .....................................................
3 Representation and Approximation in Vector Spaces
3.1 The approxirnation problem in Hilbert space .......................
3.1. I The Gram~nianm atrix ..................................
3.2 The orthogonality principle ......................................
3.2.1 Representations in infinite-dimensional space ..............
3.3 Error mininlization via gradients ..................................
3.4 Matrix representations of least-squares problems ...................
3.4.1 Weighted least-squares ..................................
3.4.2 Statistical properties of the least-squares estimate ..........
3.5 Minimum error in Hilbert-space approximations ...................
Applications of the orthogonality theorem
3.6 Approximation by continuous polynomials ........................
3.7 Approxiination by discrete polynomials ...........................
3.8 Linear regression ...............................................
3.9 Least-squares filtering ...........................................
3.9.1 Least-squares prediction and AR spectrum
. .
estimation .............................................
3.10 Miniruun~m ean-square estimation ................................
3.1 1 Minimum mean-squared error (MMSE) filtering ...................
3.12 Comparison of least squares and rniniii~urnm ean squares ............
3.13 Frequency-domain optimal filtering ...............................
3.13.1 Brief review of stochastic processes and
I~placetr ansforms .....................................
Contents vii
3.13.2 Two-sided Laplace transforms and their
. .
decompos~tlon.s ........................................
3.13.3 The Wiener-Hopf equation ..............................
3.13.4 Solution to the Wiener-Hopf equation ....................
3.13.5 Examples of Wiener filtering ............................
3.13.6 Mean-square error ......................................
3.13.7 Discrete-time Wiener filters ..............................
3.14 A dual approximation problem ...................................
3.15 Minimum-norm solution of underdetermined equations .............
3.16 Iterative Reweighted LS (IRLS) for L, optimization ................
3.17 Signal transformation and generalized Fourier series ................
3.18 Sets of complete orthogonal functions .............................
3.18.1 Trigonometric functions .................................
3.1 8.2 Orthogonal polynomials .................................
3.18.3 Sinc functions ..........................................
3.18.4 Orthogonal wavelets ....................................
3.19 Signals as points: Digital communications .........................
3.19.1 The detection problem ..................................
3.19.2 Examples of basis functions used in digital
communications ........................................
3.19.3 Detection in nonwhite noise .............................
3.20 Exercises .......................................................
3.2 1 References .....................................................
4 Linear Operators and Matrix Inverses
4.1 Linear operators ................................................
4.1.1 Linear functionals ......................................
4.2 Operator norms .................................................
4.2.1 Bounded operators ......................................
4.2.2 The Neumann expansion ................................
4.2.3 Matrix norms ..........................................
4.3 Adjoint operators and transposes .................................
4.3.1 A dual optimization problem .............................
4.4 Geometry of linear equations .....................................
4.5 Four fundamental subspaces of a linear operator ....................
4.5.1 The four fundamental subspaces with
non-closed range .......................................
4.6 Some properties of matrix inverses ................................
4.6.1 Tests for invertibility of matrices .........................
4.7 Some results on matrix rank ......................................
4.7.1 Numeric rank ..........................................
4.8 Another look at least squares .....................................
4.9 Pseudoinverses .................................................
4.10 Matrix condition number ........................................
4.1 1 Inverse of a small-rank adjustment ................................
4.1 1.1 An application: the RLS filter ............................
4.1 1.2 Two RLS applications ...................................
4.12 Inverse of a block (partitioned) matrix .............................
4.12.1 Application: Linear models ..............................
4.13 Exercises .......................................................
4.14 References .....................................................
viii Contents
5 Some Imporbnt Matrix Factorizations
5.1 The LU factorization ............................................
5.1.1 Computing the determinant using the LU factorization .....
5.1.2 Computing the LU factorization ........................
5.2 The Cholesky factorization .......................................
5.2.1 Algorithms for computing the Cholesky factorjzation .......
5.3 Unitary matrices and the QR factorization .........................
5.3.1 Unitary matrices ........................................
5.3.2 The QR factorization ....................................
5.3.3 QR factorization and least-squares filters ..................
5.3.4 Computing the QR factorization ..........................
5.3.5 Householder transformations ............................
5.3.6 Algorithms for Householder transformations ..............
5.3.7 QR factorization using Givens rotations ...................
5.3.8 Algorithms for QR factorization using Givens rotations .....
5.3.9 Solving least-squares problems using Givens rotations ......
5.3.10 Givens rotations via CORDIC rotations ...................
5.3.1 1 Recursive updates to the QR factorization .................
5.4 Exercises .......................................................
5.5 References .....................................................
6 Eigenvalues and Eigenvectors
6.1 Eigenvalues and linear systems ...................................
6.2 Linear dependence of eigenvectors ................................
6.3 Diagonalization of a matrix ......................................
6.3.1 The Jordan form ........................................
6.3.2 Diagonalization of self-adjoint matrices ...................
6.4 Geometry of invariant subspaces ..................................
6.5 Geometry of quadratic forms and the minimax principle ............
6.6 Extremai quadratic forms subject to linear constraints ...............
6.7 The Gershgorin circle theorem ...................................
Application of Eigendecomposition methods
6.8 Karhunen-Lokve low-rank approximations and principal methods ....
6.8.1 Principal component methods ............................
6.9 Eigenfilters .....................................................
6.9.1 Eigenfilters for random signals ...........................
6.9.2 Eigenfilter for designed spectral response .................
6.9.3 Constrained eigenfilters .................................
6.10 Signal subspace techniques ......................................
6.10.1 The signal model .......................................
6.10.2 The noise model ........................................
6.10.3 Pisarenko harmonic decomposition .......................
6.i0.4 MUSIC ................................................
6.1 1 Generalized eigenvalues .........................................
6.1 1.1 An application: ESPRIT .................................
6.12 Characteristic and minimal polynomials ...........................
6.1 2.1 Matrix polynomials .....................................
6.12.2 Minimal polynomials ...................................
6.13 Moving the eigenvalues around: Introduction to linear control .......
6.14 Noiseless constrained channel capacity ............................