Foundations of Signal Processing (free version)

661 Pages·2014·11.324 MB·English

by  Martin Vetterli, Jelena Kovacevic, Vivek K Goyal

Foundations of Signal Processing
Martin Vetterli
E´cole Polytechnique F´ed´erale de Lausanne
Jelena Kovaˇcevi´c
Carnegie Mellon University
Vivek K Goyal
Massachusetts Institute of Technology & Boston University
May 31, 2014
Copyright (c) 2014 Martin Vetterli, Jelena Kovaˇcevi´c, and Vivek K Goyal.
These materials are protected by copyright under the
Attribution-NonCommercial-NoDerivs 3.0 Unported License
from Creative Commons.
Contents
Quick reference page xv
Acknowledgments xxi
Preface xxiii
1 On rainbows and spectra 1
2 From Euclid to Hilbert 9
2.1 Introduction 10
2.2 Vectorspaces 18
2.2.1 Definition and properties 18
2.2.2 Innerproduct 23
2.2.3 Norm 27
2.2.4 Standard spaces 30
2.3 Hilbert spaces 35
2.3.1 Convergence 36
2.3.2 Completeness 37
2.3.3 Linear operators 40
2.4 Approximations, projections, and decompositions 50
2.4.1 Projection theorem 51
2.4.2 Projection operators 54
2.4.3 Direct sums and subspacedecompositions 60
2.4.4 Minimum mean-squared error estimation 63
2.5 Bases and frames 69
2.5.1 Bases and Riesz bases 69
2.5.2 Orthonormal bases 76
2.5.3 Biorthogonal pairs of bases 86
2.5.4 Frames 101
2.5.5 Matrix representations of vectors and linear operators 109
2.6 Computational aspects 119
2.6.1 Cost, complexity,and asymptotic notations 120
2.6.2 Precision 123
2.6.3 Conditioning 126
2.6.4 Solving systems of linear equations 129
2.A Elements of analysis and topology 135
2.A.1 Basic definitions 135
2.A.2 Convergence 136
2.A.3 Interchangetheorems 138
2.A.4 Inequalities 139
2.A.5 Integration byparts 140
2.B Elements of linear algebra 141
2.B.1 Basic definitionsand properties 141
2.B.2 Special matrices 147
2.C Elements of probability 151
2.C.1 Basic definitions 151
2.C.2 Standard distributions 154
2.C.3 Estimation 155
2.D Basis concepts 159
Chapterat a glance 161
Historical remarks 162
Furtherreading 162
Exercises with solutions 163
Exercises 169
3 Sequences and discrete-time systems 181
3.1 Introduction 182
3.2 Sequences 185
3.2.1 Infinite-lengthsequences 185
3.2.2 Finite-length sequences 192
3.2.3 Two-dimensional sequences 193
3.3 Systems 195
3.3.1 Discrete-time systems and their properties 195
3.3.2 Differenceequations 202
3.3.3 Linear shift-invariant systems 205
3.4 Discrete-time Fourier transform 216
3.4.1 Definition of the DTFT 216
3.4.2 Existence and convergence of theDTFT 218
3.4.3 Properties of theDTFT 221
3.4.4 Frequencyresponse of filters 227
3.5 z-transform 233
3.5.1 Definition of the z-transform 234
3.5.2 Existence and convergence of thez-transform 235
3.5.3 Properties of thez-transform 240
3.5.4 z-transform of filters 249
3.6 Discrete Fourier transform 252
3.6.1 Definition of the DFT 252
3.6.2 Properties of theDFT 255
3.6.3 Frequencyresponse of filters 259
3.7 Multirate sequences and systems 264
3.7.1 Downsampling 265
3.7.2 Upsampling 268
3.7.3 Combinations of downsampling and upsampling 270
3.7.4 Combinations of downsampling, upsampling, and filtering 272
3.7.5 Polyphase representation 278
3.8 Stochastic processes and systems 285
3.8.1 Stochastic processes 285
3.8.2 Systems 288
3.8.3 Discrete-time Fourier transform 292
3.8.4 Multirate sequences and systems 294
3.8.5 Minimum mean-squared error estimation 300
3.9 Computational aspects 303
3.9.1 Fast Fouriertransforms 303
3.9.2 Convolution 307
3.9.3 Multirate operations 311
3.A Elements of analysis 313
3.A.1 Complex numbers 313
3.A.2 Differenceequations 315
3.A.3 Convergence of theconvolution sum 316
3.A.4 Dirac delta function 316
3.B Elements of algebra 318
3.B.1 Polynomials 318
3.B.2 Vectorsand matrices of polynomials 321
3.B.3 Kroneckerproduct 324
Chapterat a glance 325
Historical remarks 328
Furtherreading 328
Exercises with solutions 329
Exercises 336
4 Functions and continuous-time systems 343
4.1 Introduction 344
4.2 Functions 345
4.2.1 Functionson thereal line 345
4.2.2 Periodic functions 351
4.3 Systems 351
4.3.1 Continuous-timesystems and theirproperties 352
4.3.2 Differential equations 355
4.3.3 Linear shift-invariant systems 355
4.4 Fourier transform 359
4.4.1 Definition of the Fourier transform 359
4.4.2 Existence and inversion of theFourier transform 360
4.4.3 Properties of theFourier transform 365
4.4.4 Frequencyresponse of filters 373
4.4.5 Regularity and spectral decay 374
4.4.6 Laplace transform 379
4.5 Fourier series 380
4.5.1 Definition of the Fourier series 381
4.5.2 Existence and convergence of theFourier series 383
4.5.3 Properties of theFourier series 385
4.5.4 Frequencyresponse of filters 394
4.6 Stochastic processes and systems 395
4.6.1 Stochastic processes 395
4.6.2 Systems 397
4.6.3 Fourier transform 399
Chapterat a glance 401
Historical remarks 403
Furtherreading 403
Exercises with solutions 404
Exercises 406
5 Sampling and interpolation 411
5.1 Introduction 412
5.2 Finite-dimensional vectors 420
5.2.1 Sampling and interpolation with orthonormal vectors 421
5.2.2 Sampling and interpolation with nonorthogonal vectors 425
5.3 Sequences 429
5.3.1 Sampling and interpolation with orthonormal sequences 430
5.3.2 Sampling and interpolation for bandlimited sequences 437
5.3.3 Sampling and interpolation with nonorthogonal sequences 442
5.4 Functions 447
5.4.1 Sampling and interpolation with orthonormal functions 449
5.4.2 Sampling and interpolation for bandlimited functions 452
5.4.3 Sampling and interpolation with nonorthogonal functions 470
5.5 Periodic functions 477
5.5.1 Sampling and interpolation with orthonormal periodic functions 477
5.5.2 Sampling and interpolation for bandlimited periodic functions 481
5.6 Computational aspects 489
5.6.1 Projection onto convex sets 491
Chapterat a glance 496
Historical remarks 498
Furtherreading 498
Exercises with solutions 500
Exercises 504
6 Approximation and compression 507
6.1 Introduction 508
6.2 Approximation of functions on finiteintervals by polynomials 513
6.2.1 Least-squares approximation 514
6.2.2 Lagrange interpolation: Matching points 517
6.2.3 Taylor series expansion: Matching derivatives 520
6.2.4 Hermiteinterpolation: Matching points and derivatives 522
6.2.5 Minimax polynomial approximation 523
6.2.6 Filter design 529
6.3 Approximation of functions by splines 537
6.3.1 Splinesand spline spaces 538
6.3.2 Bases for uniform spline spaces 541
6.3.3 Strang–Fix condition for polynomial representation 548
6.3.4 Continuous-timeoperatorsinsplinespacesimplementedwithdiscrete-
time processing 554
6.4 Approximation of functions and sequencesby series truncation 560
6.4.1 Linear and nonlinear approximations 560
6.4.2 Linear approximation of random vectors and stochastic processes 566
6.4.3 Linear and nonlinear diagonal estimators 571
6.5 Compression 576
6.5.1 Lossless compression 577
6.5.2 Scalar quantization 579
6.5.3 Transform coding 584
6.6 Computational aspects 591
6.6.1 Huffmanalgorithm for lossless code design 591
6.6.2 Iterativedesign of quantizers 593
6.6.3 Estimating from quantized samples 594
Chapterat a glance 597
Historical remarks 598
Furtherreading 599
Exercises with solutions 601
Exercises 607
7 Localization and uncertainty 615
7.1 Introduction 616
7.2 Localization for functions 619
7.2.1 Localization in time 620
7.2.2 Localization in frequency 622
7.2.3 Uncertaintyprinciple for functions 624
7.3 Localization for sequences 627
7.3.1 Localization in time 629
7.3.2 Localization in frequency 630
7.3.3 Uncertaintyprinciple for sequences 633
7.3.4 Uncertaintyprinciple for finite-length sequences 637
7.4 Tiling thetime–frequency plane 638
7.4.1 Localization for structured sets of functions 638
7.4.2 Localization for structured sets of sequences 640
7.5 Examples of local Fourier and wavelet bases 645
7.5.1 Local Fourier and wavelet bases for functions 645
7.5.2 Local Fourier and wavelet bases for sequences 651
7.6 Recap and a glimpse forward 656
7.6.1 Tools 657
7.6.2 Adaptingtools to real-world problems 658
7.6.3 Music analysis, communications, and compression 659
Chapterat a glance 666
Historical remarks 667
Furtherreading 667
Exercises with solutions 669
Exercises 671
Image and quote attribution 673
References 675
Index 681
Quick reference
Abbreviations
AR Autoregressive
ARMA Autoregressive movingaverage
AWGN Additivewhite Gaussian noise
BIBO Bounded input,boundedoutput
CDF Cumulative distribution function
DCT Discrete cosine transform
DFT Discrete Fouriertransform
DTFT Discrete-time Fourier transform
DWT Discrete wavelet transform
FFT Fast Fourier transform
FIR Finite impulse response
i.i.d. Independentand identically distributed
IIR Infiniteimpulse response
KLT Karhunen–Lo`evetransform
LMMSE Linear minimum mean-squared error
LPSV Linear periodically shift-varying
LSI Linear shift-invariant
MA Moving average
MAP Maximum a posteriori probability
ML Maximum likelihood
MMSE Minimum mean-squared error
MSE Mean-squared error
PDF Probability density function
PMF Probability mass function
POCS Projection onto convexsets
rad Radians
ROC Region of convergence
SNR Signal-to-noise ratio
SVD Singular value decomposition
WSCS Wide-sense cyclostationary
WSS Wide-sense stationary
Abbreviations used in tables and captions but not in the text
FT Fourier transform
FS Fourier series
LT Laplace transform
Sets
natural numbers N 0, 1, ...
integers Z ..., 1, 0, 1, ...
positive integers Z+ 1, 2,−...
rational numbers Q p/q, p,q Z, q=0
∈ 6
real numbers R ( , )
positive real numbers R+ (0−,∞ )∞
∞
complex numbers C a+jb or rejθ with a,b,r,θ R
∈
a generic index set
I
a generic vector space V
a generic Hilbert space H
closure of set S S
Real and complex analysis
sequence x argument n is an integer, n Z
n
∈
function x(t) argument t is continuous-valued,t R
∈
ordered sequence (x )
n n
set containing x x
n n n
{ }
vector x with x as elements [x ]
n n
Kroneckerdelta sequence δ δ =1 for n=0; δ =0 otherwise
n n n
∞
Dirac delta function δ(t) x(t)δ(t)dt=x(0) for x continuous at 0
Z−∞
indicator function of intervalI 1 1 (t)=1 for t I;1 (t)=0 otherwise
I I I
∈
integration byparts udv=uv vdu
−
Z Z
complex number z a+jb, rejθ, a,b R, r [0, ),θ [0, 2π)
∈ ∈ ∞ ∈
conjugation z∗ a jb, re−jθ
−
real part of ( ) (a+jb)=a, a, b R
ℜ · ℜ ∈
imaginary part of ( ) (a+jb)=b, a, b R
ℑ · ℑ ∈
conjugation of coefficients X (z) X∗(z∗)
principal root of unity W∗N e−j2π/N
Asymptotic notation
big O x O(y) 0 x γy for all n n ;
n n 0
∈ ≤ ≤ ≥
some n and γ >0
0
little o x o(y) 0 x γy for all n n ;
n n 0
∈ ≤ ≤ ≥
some n , any γ >0
0
Omega x Ω(y) x γy for all n n ;
n n 0
∈ ≥ ≥
some n and γ >0
0
Theta x Θ(y) x O(y) and x Ω(y)
∈ ∈ ∈
asymptotic equivalence x y lim x /y =1
n n n
≍ →∞
Standard vector spaces
Hilbert space of square-summable
ℓ2(Z) x:Z C x 2 < with
sequences ( → (cid:12) | n| ∞)
(cid:12)Xn
inner produc(cid:12)(cid:12)t hx,yi= nxnyn∗
(cid:12)
Hilbert space of square-integrable P
2(R) x:R C x(t)2dt< with
functions L → | | ∞
(cid:26) (cid:12)Z (cid:27)
inner produc(cid:12)(cid:12)t hx,yi= x(t)y(t)∗dt
(cid:12)
R
normed vectorspace of sequences
ℓp(Z) x:Z C x p < with
with finite ℓp norm, 1≤p<∞ (norm x→ =(cid:12)(cid:12)(cid:12)Xn | nx| p 1∞/p)
k kp (cid:12)(cid:12) n| n|
normed vectorspace of functions (cid:0)P (cid:1)
p(R) x:R C x(t)pdt< with
with finite p norm, 1 p< L → | | ∞
L ≤ ∞ (cid:26) (cid:12)Z (cid:27)
norm x =(cid:12) x(t)pdt 1/p
k kp (cid:12)(cid:12) | |
normed vectorspace of bounded (cid:0)R (cid:1)
sequences with supremum norm ℓ∞(Z) x:Z→C sup|xn|<∞ with
(cid:26) (cid:12) n (cid:27)
norm kxk∞ =(cid:12)(cid:12) supn|xn|
(cid:12)
normed vectorspace of bounded
∞(R) x:R C esssup x(t) < with
functions with supremum norm L → | | ∞
(cid:26) (cid:12) t (cid:27)
norm x =(cid:12) esssup x(t)
k k∞ (cid:12) t| |
(cid:12)
Bases and frames for sequences
standard basis e e =1 for n=k; e =0 otherwise
k k,n k,n
{ }
vector, element of basis or frame ϕ when applicable, a column vector
basis or frame Φ set of vectors ϕ
k
{ }
operator Φ concatenation of ϕ in a linear
k
{ }
operator: [ϕ ϕ ... ϕ ]
0 1 N 1
−
vector, element of dual basis or frame ϕ when applicable, a column vector
Φ set of vectors ϕ
k
{ }
operator Φe concatenation of ϕ in a linear
k
{ }
e operator: [ϕ0 ϕe1 ... ϕN 1]
−
expansion with a basis or frame xe=ΦΦ∗x e
e e e
e
Discrete-time signal processing
Sequence x signal, vector
n
Convolution
linear h x x h
k n k
∗ −
XNk∈Z1
circular −
h⊛x x h
(N-periodic sequences) k (n−k)modN
Xk=0
(h x) convolution result at n
n
∗
Eigensequence v eigenvector
n
infinitetime v =ejωn h v=H(ejω)v
n
∗
finitetime v =ej2πkn/N h⊛v=H v
n k
Frequency response eigenvalue corresponding to v
n
infinitetime H(ejω) hne−jωn
nNX∈Z1 N 1
finitetime Hk − hne−j2πkn/N = − hnWNkn
n=0 n=0
X X
Continuous-time signal processing
Function x(t) signal
Convolution
∞
linear h x x(τ)h(t τ)dτ
∗ −
Z−∞
circular T/2
h⊛x x(τ)h(t τ)dτ
(T-periodicfunctions) −
Z−T/2
(h x)(t) convolution result at t
∗
Eigenfunction v(t) eigenvector
infinitetime v(t)=ejωt h v=H(ω)v
∗
finitetime v(t)=ej2πkt/T h⊛v=H v
k
Frequency response eigenvalue corresponding to v(t)
infinitetime H(ω) ∞ h(t)e−jωtdt
Z−∞
T/2
finitetime Hk h(τ)e−j2πkτ/Tdτ
Z−T/2