Digital and Statistical Signal Processing

559 Pages·2019·8.56 MB·English

by  Anastasia Veloni

Digital and Statistical
Signal Processing
Anastasia Veloni
Nikolaos I. Miridakis
Erysso Boukouvala
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2019 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
International Standard Book Number-13: 978-1-138-58006-0 (Hardback)
Library of Congress Cataloging‑in‑Publication Data
Names: Veloni, Anastasia, author. | Miridakis, Nikolaos, author. |
Boukouvala, Erysso, author.
Title: Digital and statistical signal processing / Anastasia Veloni, Nikolaos
Miridakis, and Erysso Boukouvala.
Description: Boca Raton : Taylor & Francis, a CRC title, part of the Taylor &
Francis imprint, a member of the Taylor & Francis Group, the academic
division of T&F Informa, plc, 2018. | Includes bibliographical references.
Identifiers: LCCN 2018027136| ISBN 9781138580060 (hardback : acid-free paper)
| ISBN 9780429507526 (ebook)
Subjects: LCSH: Signal processing--Digital techniques.
Classification: LCC TK5102.9 .V45 2018 | DDC 621.382/2--dc23
LC record available at https://lccn.loc.gov/2018027136
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Preface ...........................................................................................................................................xiii
Authors ...........................................................................................................................................xv
List of Acronyms ........................................................................................................................xvii
Section I Topics on Digital Signal Processing
1. Introduction.............................................................................................................................3
1.1 Introduction ...................................................................................................................3
1.2 Advantages of Digital Signal Processing ...................................................................4
1.3 Digitization Steps of Analog Signals .........................................................................5
1.3.1 Sampling ............................................................................................................5
1.3.2 Quantization .....................................................................................................7
1.3.3 Coding................................................................................................................9
1.4 Sampling and Reconstruction of Sinusoidal Signals .............................................10
1.4.1 Proof of the Sampling Theorem and a Detailed Discussion ....................12
1.5 Physical Sampling .......................................................................................................17
1.6 Sampling and Holding ...............................................................................................20
1.7 Non-Accurate Reconstruction of Analog Signals...................................................21
1.8 Solved Problems ..........................................................................................................22
2. Discrete-Time Signals and Systems ..................................................................................45
2.1 Discrete-Time Signals .................................................................................................45
2.2 Basic Discrete-Time Signals .......................................................................................45
2.2.1 Impulse Function ...........................................................................................45
2.2.2 Unit Step Function .........................................................................................47
2.2.3 Ramp Function ...............................................................................................47
2.2.4 Unit Rectangular Function (Pulse Function) .............................................48
2.2.5 Exponential Function .....................................................................................48
2.2.6 The Sinusoidal Sequence ...............................................................................50
2.3 Even and Odd Discrete-Time Signals ......................................................................51
2.4 Energy and Power of a Discrete-Time Signal..........................................................53
2.5 Conversion of the Independent and Dependant Variable.....................................54
2.6 Discrete-Time Systems ...............................................................................................54
2.7 Categories of Discrete-Time Systems .......................................................................55
2.7.1 Linear Discrete Systems ................................................................................55
2.7.2 Time-Invariant Discrete Systems .................................................................56
2.7.3 Discrete Systems with Memory ....................................................................57
2.7.4 Invertible Discrete Systems ...........................................................................57
2.7.5 Casual Discrete Systems ................................................................................57
2.7.6 Stable Discrete Systems .................................................................................58
2.8 System Connections ....................................................................................................58
2.9 Convolution ..................................................................................................................59
2.10 Deconvolution ..............................................................................................................62
2.11 Correlation — Autocorrelation ..................................................................................63
2.12 Difference Equations ..................................................................................................64
2.13 Discrete-Time Systems of Finite Impulse Response ...............................................65
2.14 Solved Problems ..........................................................................................................66
3. z-Transform ..........................................................................................................................113
3.1 Introduction ...............................................................................................................113
3.2 From Laplace Transform to z-Transform ................................................................113
3.2.1 Comparison of the s- and z-Planes into the Region of Convergence .....116
3.3 Properties of z-Transform .........................................................................................117
3.3.1 Time Shift ......................................................................................................117
3.3.2 Linearity.........................................................................................................118
3.3.3 Time Reversal ................................................................................................118
3.3.4 Convolution ...................................................................................................119
3.3.5 Differentiation in z-Plane ............................................................................119
3.3.6 Multiplication by an Exponential Sequence .............................................120
3.3.7 Conjugation of a Complex Sequence .........................................................120
3.3.8 Initial and Final Value Theorem ................................................................121
3.3.9 Correlation of Two Sequences ....................................................................121
3.4 Inverse z-Transform ..................................................................................................121
3.4.1 Method of Power Series Expansion (Division Method) .........................122
3.4.2 Method of Partial Fraction Expansion ......................................................122
3.4.3 Method of Complex Integration .................................................................123
3.5 z-Transform in System Analysis..............................................................................124
3.5.1 Transfer Function of Discrete-Time Signal ..............................................124
3.5.2 Causality of Discrete-Time Systems ..........................................................124
3.5.3 Stability of Discrete-Time Systems ............................................................125
3.5.4 Transfer Function of Connected Systems .................................................126
3.5.5 Transfer Function of Discrete-Time Systems ...........................................127
3.6 Formula Tables ...........................................................................................................129
3.7 Solved Problems ........................................................................................................130
4. Structures for the Realization of Discrete-Time Systems ..........................................179
4.1 Introduction ...............................................................................................................179
4.2 Block Diagrams .........................................................................................................179
4.3 Realization Structures ..............................................................................................181
4.3.1 Implementation Structures of IIR Discrete Systems ...............................183
4.3.2 Implementation Structures of FIR Discrete Systems ..............................186
4.4 Signal Flow Graphs ...................................................................................................188
4.4.1 Mason’s Gain Formula .................................................................................189
4.5 Solved Problems ........................................................................................................190
5. Frequency Domain Analysis ............................................................................................211
5.1 Introduction ...............................................................................................................211
5.2 Discrete-Time Fourier Transform (DTFT) ..............................................................212
5.3 Discrete Fourier Series (DFS) ...................................................................................214
5.3.1 Periodic Convolution ...................................................................................215
5.3.2 The Relation of the DFS Components and the DTFT over a Period .....216
5.4 Discrete Fourier Transform ......................................................................................216
5.4.1 Properties of the DFT ...................................................................................218
5.4.1.1 Linearity .........................................................................................218
5.4.1.2 Circular Shift.................................................................................218
5.4.1.3 Circular Convolution ...................................................................219
5.4.1.4 Multiplication of Sequences ........................................................220
5.4.1.5 Parseval’s Theorem .......................................................................220
5.5 Fast Fourier Transform .............................................................................................221
5.5.1 FFT Equations ...............................................................................................221
5.5.2 Computation of the IDFT Using FFT .........................................................228
5.5.3 Fast Convolution ..........................................................................................228
5.5.3.1 Overlap and Add Method ...........................................................228
5.5.3.2 Overlap and Save Method ...........................................................229
5.6 Estimation of Fourier Transform through FFT .....................................................229
5.7 Discrete Cosine Transform ......................................................................................229
5.8 Wavelet Transform ....................................................................................................231
5.8.1 Wavelet Transform Theory .........................................................................233
5.9 Solved Problems ........................................................................................................236
6. Design of Digital Filters ....................................................................................................287
6.1 Introduction ...............................................................................................................287
6.2 Types of Digital Filters ..............................................................................................288
6.3 Digital Filter Design Specifications .........................................................................288
6.4 Design of Digital IIR Filters .....................................................................................290
6.5 Indirect Methods of IIR Filter Design ....................................................................292
6.5.1 The Impulse Invariant Method ..................................................................292
6.5.2 Step Invariant Method (or z-Transform Method with Sample
and Hold) .......................................................................................................293
6.5.3 Backward Difference Method ....................................................................295
6.5.4 Forward Difference Method .......................................................................296
6.5.5 Bilinear or Tustin Method ...........................................................................297
6.5.6 Matched Pole-Zero Method ........................................................................299
6.6 Direct Methods of IIR Filter Design .......................................................................300
6.6.1 Design of H(ejω)2 Method ........................................................................300
6.6.2 The Method of Calculating h[n] .................................................................301
6.7 IIR Filter Frequency Transformations ....................................................................301
6.8 FIR Filters ....................................................................................................................303
6.9 FIR Linear Phase Filters ............................................................................................304
6.10 Stability of FIR Filters ...............................................................................................307
6.11 Design of FIR Filters ..................................................................................................307
6.12 The Moving Average Filters ....................................................................................307
6.13 FIR Filter Design Using the Frequency Sampling Method .................................309
6.14 FIR Filter Design Using the Window Method .......................................................311
6.15 Optimal Equiripple FIR Filter Design ....................................................................317
6.16 Comparison of the FIR Filter Design Methods .....................................................319
6.17 Solved Problems ........................................................................................................320
Section II Statistical Signal Processing
7. Statistical Models ................................................................................................................383
7.1 The Gaussian Distribution and Related Properties .............................................383
7.1.1 The Multivariate Gaussian Distribution...................................................385
7.1.2 The Central Limit Theorem ........................................................................387
7.1.3 The Chi-Squared RV Distribution .............................................................388
7.1.4 Gamma Distribution ....................................................................................388
7.1.5 The Non-Central Chi-Squared RV Distribution.......................................389
7.1.6 The Chi-Squared Mixed Distribution .......................................................389
7.1.7 The Student’s t-Distribution .......................................................................390
7.1.8 The Fisher-Snedecor F-Distribution ..........................................................390
7.1.9 The Cauchy Distribution .............................................................................391
7.1.10 The Beta Distribution ..................................................................................391
7.2 Reproducing Distributions ......................................................................................392
7.3 Fisher-Cochran Theorem .........................................................................................392
7.4 Expected Value and Variance of Samples ..............................................................393
7.5 Statistical Sufficiency ................................................................................................395
7.5.1 Statistical Sufficiency and Reduction Ratio ..............................................396
7.5.2 Definition of Sufficient Condition .............................................................397
7.5.3 Minimal Sufficiency .....................................................................................399
7.5.4 Exponential Distributions Category ..........................................................402
7.5.5 Checking Whether a PDF Belongs to the Exponential
Distribution Category ..................................................................................404
8. Fundamental Principles of Parametric Estimation ......................................................405
8.1 Estimation: Basic Components ................................................................................405
8.2 Estimation of Scalar Random Parameters .............................................................406
8.2.1 Estimation of Mean Square Error (MSE) ..................................................407
8.2.2 Estimation of Minimum Mean Absolute Error .......................................409
8.2.3 Estimation of Mean Uniform Error (MUE) ..............................................411
8.2.4 Examples of Bayesian Estimation ..............................................................413
8.3 Estimation of Random Vector Parameters .............................................................421
8.3.1 Squared Vector Error ...................................................................................421
8.3.2 Uniform Vector Error...................................................................................422
8.4 Estimation of Non-Random (Constant) Parameters .............................................422
8.4.1 Scalar Estimation Criteria for Non-Random Parameters .......................423
8.4.2 The Method of Statistical Moments for Scalar Estimators.....................426
8.4.3 Scalar Estimators for Maximum Likelihood ............................................429
8.4.4 Cramer-Rao Bound (CRB) in the Estimation Variance ...........................433
8.5 Estimation of Multiple Non-Random (Constant) Parameters .............................441
8.5.1 Cramer-Rao (CR) Matrix Bound in the Covariance Matrix ...................442
8.5.2 Methods of Vector Estimation through Statistical Moments ................446
8.5.3 Maximum Likelihood Vector Estimation .................................................447
8.6 Handling of Nuisance Parameters .........................................................................452
9. Linear Estimation ................................................................................................................455
9.1 Constant MSE Minimization, Linear and Affine Estimation .............................455
9.1.1 Optimal Constant Estimator of a Scalar RV .............................................456
9.2 Optimal Linear Estimator of a Scalar Random Variable .....................................456
9.3 Optimal Affine Estimator of a Scalar Random Variable θ ..................................458
9.3.1 Superposition Property of Linear/Affine Estimators .............................459
9.4 Geometric Interpretation: Orthogonality Condition and Projection Theorem .....459
9.4.1 Reconsideration of the Minimum MSE Linear Estimation ....................460
9.4.2 Minimum Affine MSE Estimation ............................................................462
9.4.3 Optimization of the Affine Estimator for the Linear Gaussian Model ....462
9.5 Optimal Affine Vector Estimator ............................................................................463
9.5.1 Examples of Linear Estimation ..................................................................464
9.6 Non-Statistical Least Squares Technique (Linear Regression) ...........................467
9.7 Linear Estimation of Weighted LLS ........................................................................473
9.8 Optimization of LMWLS in Gaussian Models .....................................................477
10. Fundamentals of Signal Detection ..................................................................................479
10.1 The General Detection Problem ..............................................................................484
10.1.1 Simple and Composite Hypotheses ..........................................................485
10.1.2 The Decision Function ................................................................................486
10.2 Bayes Approach to the Detection Problem ............................................................488
10.2.1 Assign a Priori Probabilities .......................................................................488
10.2.2 Minimization of the Average Risk ............................................................488
10.2.3 The Optimal Bayes Test Minimizes [C] .................................................489
10.2.4 Minimum Probability of the Error Test ....................................................490
10.2.5 Evaluation of the Performance of Bayes Likelihood Ratio Test ............490
10.2.6 The Minimax Bayes Detector .....................................................................491
10.2.7 Typical Example ...........................................................................................493
10.3 Multiple Hypotheses Tests.......................................................................................496
10.3.1 A Priori Probabilities ...................................................................................498
10.3.2 Minimization of the Average Risk .............................................................498
10.3.3 Disadvantages of Bayes Approach ............................................................501
10.4 Frequentist Approach for Detection .......................................................................502
10.4.1 Case of Simple Hypotheses: θ ∈ {θ ,θ} .....................................................502
0 1
10.5 ROC Curves for Threshold Testing ..............................................................................506
Appendix I: Introduction to Matrix Algebra and Application to Signals
and System ..................................................................................................................................517
Appendix II: Solved Problems in Statistical Signal Processing ......................................527
Bibliography ................................................................................................................................543
Index .............................................................................................................................................545
Preface
The goal of this textbook is to support the teaching of digital and statistical signal
processing in higher education. Particular attention is paid to the presentation of the fun-
damental theory; key topics are outlined in a comprehensible way, and all areas of the
subject are discussed in a fashion that aims at simplification without sacrificing accuracy.
The book is divided into two sections.
In the first section, we aim at a deep understanding of the subject of Digital Signal
Processing and provide numerous examples and solved problems often dealt with the use
of MATLAB®.
The second section covers Statistical Signal Processing. The basic principles of statistical
inference are discussed and their implementation in practical signal and system condi-
tions are analyzed. The discussion is strongly supported by examples and solved prob-
lems in this section, as well.
The content of the book is developed in ten chapters and two appendices as follows:
Chapter 1 contains introductory concepts for a basic understanding on the field of
Digital Signal Processing, with particular emphasis on Digitization and Reconstruction of
continuous-time signals.
Chapter 2 refers to the characteristics and properties of discrete-time signals and
systems.
Chapter 3 refers to z-Transform, which is a basic mathematical tool for studying
discrete-time systems.
Chapter 4 analyzes the implementation of the filters FIR and IIR in various forms (Direct
Form I and II, Cascade form, Parallel form).
Chapter 5 describes the Analysis of Discrete Systems in the Frequency domain. We
analyze the Discrete Time Fourier Transform, Discrete Fourier Series, Discrete Fourier
Transform, Fast Fourier Transform and Discrete Wavelet Transform.
Chapter 6 deals with the design of the IIR and FIR digital filters. Indirect design meth-
ods (Invariant Impulse Response, Invariant Step Response, Differential Imaging, Bilinear
Transform, Pole and Zero Position Matching) and direct methods of designing IIR filters are
analyzed. The design of FIR filters through sampling in the frequency domain and with the use
of Windows (Rectangular, Bartlett, Hanning, Hamming, Blackman, Kaiser) is also described.
Chapter 7 describes comprehensively and develops the most important statistical mod-
els used in stochastic signal processing.
Chapter 8 deals with the concept of parametric estimation of stochastic signals when
processed for both one- and multi-dimensional cases.
Chapter 9 analyzes the particular problem of linear estimation. This problem is wide-
spread in many practical applications due to its attractively low computational complexity
in processing stochastic signals.
Chapter 10 hosts the fundamental principles of stochastic signal detection. The most well-
known theoretical approaches and practical techniques for detecting signals are analyzed.
Appendix I summarizes the basic principles in the Algebra of Vectors and Matrices,
with a particular emphasis on their application to stochastic signals and systems.
Finally, Appendix II presents numerous solved problems in Statistical Signal Processing,
aiming at further understanding and consolidation of the theoretical concepts of the sec-
ond section of the book.
Acronyms
Acronym Technical Term
A/D analog-to-digital
ADC analog-to-digital converter
BIBO bounded-input, bounded-output
BLRT Bayes likelihood ratio test
CDF cumulative distribution function
CLT central limit theorem
CME conditional mean estimator
CmE conditional median estimator
CRB or CR Cramer-Rao bound
CWT continuous wavelet transform
DAC digital-to-analog converter
DCT discrete cosine transform
DFS discrete Fourier series
DFT discrete Fourier transform
DIF decimation in frequency
DIT decimation in time
DSP digital signal processing
DTFT discrete-time Fourier transform
DWT discrete wavelets transform
FFT fast Fourier transform
FIR finite impulse response
FSR full scale range
FT Fourier transform
HPF high-pass filter
IDCT inverse discrete cosine transform
IDFS inverse discrete Fourier series
IDFT inverse discrete Fourier transform
IDWT inverse discrete wavelets transform
IFFT inverse fast Fourier transform
IHPF inverse high-pass filter
IID independent and identically distributed (RV)
IIR infinite impulse response
ILPF inverse low-pass filter
IUD independent uniformly distributed random variables
IZT inverse z-transform
LCM least common multiple
LLS linear least squares
LMMSE linear minimum mean squared error
LMWLS linear minimum weighted least squares
LPF low-pass filter
LRT likelihoodratio test
(Continued)
Acronym Technical Term
LTI linear time invariant
MAE mean absolute error
MAP maximum a posteriori
ML maximum likelihood
MMSE minimum mean squared error
MP most powerful (test)
MP-LRT most powerful likelihood ratio test
MSE means quared error (estimator)
MUE mean uniform error
OOK On-off keying (technique)
PAM pulse amplitude modulation
PDF probability density function
RF radio frequency
ROC receiver operating characteristic (curve)
RR reduction ratio
RV random variable
SFG signal flow graph
SNR signal-to-noise ratio
STFT short time Fourier transform
TF transfer function
UMVU uniform minimum variance unbiased (estimator)
WSS wide-sense stationary
WSSE weighted sum of squared error
ZOH zero order hold
ZT z-transform