Digital Signal Processing with Matlab Examples, Volume 3 Model-Based Actions and Sparse Representation

438 Pages·2017·6.549 MB·english

by  Jose Maria Giron-Sierra

Jose Maria Giron-Sierra
Digital Signal Processing
with Matlab Examples,
Volume 3
Model-Based Actions and Sparse
Representation
123
Jose Maria Giron-Sierra
SystemsEngineeringandAutomaticControl
Universidad Complutense deMadrid
Madrid
Spain
ISSN 1860-4862 ISSN 1860-4870 (electronic)
Signals andCommunication Technology
ISBN978-981-10-2539-6 ISBN978-981-10-2540-2 (eBook)
DOI 10.1007/978-981-10-2540-2
LibraryofCongressControlNumber:2016951678
MATLAB® is a registered trademark of The MathWorks, Inc., and is used with permission. The
MathWorks does not warrant the accuracy of the text or exercises in this book. This book’s use or
discussionofMATLABsoftwareorrelatedproductsdoesnotconstituteendorsementorsponsorshipby
theMathWorksofaparticularpedagogicalapproachorparticularuseoftheMATLABsoftware.
©SpringerScience+BusinessMediaSingapore2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
foranyerrorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerNatureSingaporePteLtd.
Theregisteredcompanyaddressis:152BeachRoad,#22-06/08GatewayEast,Singapore189721,Singapore
Preface
This is the third book of a trilogy. As in the other books, a series of MATLAB
programs are embedded in the chapters for several purposes: to illustrate the
techniques, to provide implementation examples, to encourage for personal
exploration departing from a successful start.
The book has two parts, each having just one chapter. These chapters are long
and have a considerable number of bibliographic references.
When using a GPS on a car, sometimes it is not possible to keep contact with
satellites,likeforinstanceinsidetunnels.Inthiscase,amodelofthecarmotion—a
dynamic model—can be used for data substitution. The adequate combination of
measurementsandmodelsisthekeyideaoftheKalmanfilter,whichisthecentral
topic of the first part of the book. This filter was formulated for linear conditions.
Therearemodificationsfornonlinearconditions,liketheextendedKalmanfilter,or
the unscented Kalman filter. A new idea is to use particle filters. These topics are
covered in the chapter under an important perspective: Bayesian filtering.
Compressed sensing has emerged as a promising idea. One of the intended
applications is networked devices or sensors, which are becoming a reality. This
topic is considered in the second part of the book. Some experiments that
demonstrate image denoising applications were included.
For easier reading of the book, the longer programs have been put in an
appendix.Andasecondappendixonoptimizationhasbeenaddedtosupportsome
contents of the last chapter.
The reader is invited to discover the profound interconnections and common-
alities that exist behind the variety of topics in this book. This common ground
would become surely the humus for the next signal processing future.
As said in the preface of the other books, our particular expertise on signal
processing has two main roots: research and teaching. I belong to the Faculty of
Physics, University Complutense of Madrid, Spain. During our experimental
research on autonomous vehicles, maritime drones, satellite control, etc., we
practiced main methods of digital signal processing, for the use of a variety of
sensors and for prediction of vehicle motions. From years ago, I teach Signal
Processingina Master of Biomedical Physics,and aMasteronNew technologies.
v
vi Preface
The style of the programs included in the book is purposively simple enough.
Thereaderisinvitedtotypesettheprogramsincludedinthebook,foritwouldhelp
forcatchingcodingdetails.Anyway,allprogramsareavailablefromthebookweb
page: www.dacya.ucm.es/giron/SPBook3/Programs.
A lot of different materials have been used to erect this book: articles, blogs,
codes, experimentation. I tried to cite with adequate references all the pieces that
have been useful. If someone has been forgotten, please contact me. Most of the
references cited in the text are available from Internet. We have to express our
gratitude to the public information available in this way.
Please, send feedback and suggestions for further improvement and support.
Acknowledgments
ThankstomyUniversity, mycolleagues and mystudents. Since this andtheother
book required a lot of time taken from nights, weekends and holidays, I have to
sincerely express my gratitude to my family.
Madrid, Spain Jose Maria Giron-Sierra
Contents
Part I Model-Based Actions: Filtering, Prediction, Smoothing
1 Kalman Filter, Particle Filter and Other Bayesian Filters..... .... 3
1.1 Introduction .... .... ..... .... .... .... .... .... ..... .... 3
1.2 Preliminaries.... .... ..... .... .... .... .... .... ..... .... 5
1.2.1 A Basic Example... .... .... .... .... .... ..... .... 7
1.2.2 Prediction with the Gauss-Markov Model.... ..... .... 10
1.2.3 Continuation of a Simple Example of Recursive
Wiener Filter. ..... .... .... .... .... .... ..... .... 12
1.3 Kalman Filter... .... ..... .... .... .... .... .... ..... .... 16
1.3.1 The Algorithm..... .... .... .... .... .... ..... .... 17
1.3.2 Evolution of Filter Variables.. .... .... .... ..... .... 21
1.3.3 Several Perspectives .... .... .... .... .... ..... .... 27
1.3.4 Some Connections.. .... .... .... .... .... ..... .... 29
1.3.5 Numerical Issues... .... .... .... .... .... ..... .... 30
1.3.6 Information Filter .. .... .... .... .... .... ..... .... 31
1.4 Nonlinear Conditions . ..... .... .... .... .... .... ..... .... 32
1.4.1 Propagation and Nonlinearity . .... .... .... ..... .... 32
1.4.2 Jacobian. Hessian. Change of Coordinates.... ..... .... 39
1.4.3 Local Linearization . .... .... .... .... .... ..... .... 41
1.4.4 Example of a Body Falling Towards Earth... ..... .... 43
1.5 Extended Kalman Filter (EKF)... .... .... .... .... ..... .... 52
1.5.1 The EKF Algorithm .... .... .... .... .... ..... .... 52
1.5.2 Assessment of the Linearized Approximation . ..... .... 58
1.6 Unscented Kalman Filter (UKF).. .... .... .... .... ..... .... 62
1.6.1 The Unscented Transform.... .... .... .... ..... .... 63
1.6.2 The Unscented Kalman Filter (UKF).... .... ..... .... 70
1.7 Particle Filter ... .... ..... .... .... .... .... .... ..... .... 76
1.7.1 An Implementation of the Particle Filter. .... ..... .... 77
1.7.2 Resampling Schemes.... .... .... .... .... ..... .... 82
1.7.3 Multinomial Resampling. .... .... .... .... ..... .... 84
vii
viii Contents
1.7.4 Systematic Resampling .. .... .... .... .... ..... .... 86
1.7.5 Stratified Resampling.... .... .... .... .... ..... .... 87
1.7.6 Residual Resampling.... .... .... .... .... ..... .... 87
1.7.7 Comparison.. ..... .... .... .... .... .... ..... .... 88
1.7.8 Roughening.. ..... .... .... .... .... .... ..... .... 88
1.7.9 Basic Theory of the Particle Filter.. .... .... ..... .... 89
1.7.10 Sequential Monte Carlo (SMC).... .... .... ..... .... 90
1.7.11 Proposal Importance Functions .... .... .... ..... .... 93
1.7.12 Particle Filter Variants... .... .... .... .... ..... .... 97
1.7.13 Marginalized Particle Filter (Rao-Blackwellized
Particle Filter)..... .... .... .... .... .... ..... .... 98
1.7.14 Regularized Particle Filters ... .... .... .... ..... .... 99
1.8 The Perspective of Numerical Integration... .... .... ..... .... 100
1.8.1 Geometry.... ..... .... .... .... .... .... ..... .... 101
1.8.2 Quadrature... ..... .... .... .... .... .... ..... .... 102
1.8.3 Other Approximations... .... .... .... .... ..... .... 105
1.8.4 Gaussian Filters.... .... .... .... .... .... ..... .... 108
1.8.5 Assumed Density. Expectation Propagation... ..... .... 112
1.9 Other Bayesian Filters ..... .... .... .... .... .... ..... .... 113
1.9.1 Ensemble Kalman Filter (EnKF)... .... .... ..... .... 114
1.9.2 Iterative Kalman Filter... .... .... .... .... ..... .... 115
1.9.3 Gaussian Particle Filter .. .... .... .... .... ..... .... 116
1.9.4 Divide and Conquer .... .... .... .... .... ..... .... 116
1.9.5 Combinations ..... .... .... .... .... .... ..... .... 118
1.9.6 Algorithms with Special Characteristics . .... ..... .... 118
1.10 Smoothing . .... .... ..... .... .... .... .... .... ..... .... 119
1.10.1 Optimal Prediction.. .... .... .... .... .... ..... .... 119
1.10.2 One-Stage Smoothing ... .... .... .... .... ..... .... 120
1.10.3 Three Types of Smoothers ... .... .... .... ..... .... 122
1.10.4 Bayesian Smoothing .... .... .... .... .... ..... .... 132
1.11 Applications of Bayesian Filters.. .... .... .... .... ..... .... 134
1.11.1 Navigation... ..... .... .... .... .... .... ..... .... 134
1.11.2 Tracking .... ..... .... .... .... .... .... ..... .... 134
1.11.3 Information Fusion . .... .... .... .... .... ..... .... 135
1.11.4 SLAM.. .... ..... .... .... .... .... .... ..... .... 135
1.11.5 Speech, Sounds.... .... .... .... .... .... ..... .... 137
1.11.6 Images.. .... ..... .... .... .... .... .... ..... .... 137
1.11.7 Earth Monitoring and Forecasting.. .... .... ..... .... 137
1.11.8 Energy and Economy ... .... .... .... .... ..... .... 138
1.11.9 Medical Applications.... .... .... .... .... ..... .... 138
1.11.10 Traffic .. .... ..... .... .... .... .... .... ..... .... 138
1.11.11 Other Applications . .... .... .... .... .... ..... .... 139
Contents ix
1.12 Frequently Cited Examples.. .... .... .... .... .... ..... .... 139
1.12.1 Bearings-Only Tracking . .... .... .... .... ..... .... 139
1.12.2 Other Tracking Cases ... .... .... .... .... ..... .... 140
1.12.3 Univariate Non-stationary Growth Model .... ..... .... 140
1.12.4 Financial Volatility Model.... .... .... .... ..... .... 141
1.12.5 Nonlinear Series ... .... .... .... .... .... ..... .... 141
1.12.6 The Pendulum..... .... .... .... .... .... ..... .... 142
1.13 Resources.. .... .... ..... .... .... .... .... .... ..... .... 142
1.13.1 MATLAB ... ..... .... .... .... .... .... ..... .... 142
1.13.2 Internet . .... ..... .... .... .... .... .... ..... .... 143
References.. .... .... .... ..... .... .... .... .... .... ..... .... 144
Part II Sparse Representation. Compressed Sensing
2 Sparse Representations... ..... .... .... .... .... .... ..... .... 151
2.1 Introduction .... .... ..... .... .... .... .... .... ..... .... 151
2.2 Sparse Solutions. .... ..... .... .... .... .... .... ..... .... 152
2.2.1 The Central Problem.... .... .... .... .... ..... .... 152
2.2.2 Norms and Sparsity. .... .... .... .... .... ..... .... 155
2.2.3 Solving Sparsity Optimization Problems. .... ..... .... 156
2.3 Compressed Sensing.. ..... .... .... .... .... .... ..... .... 182
2.3.1 Statement of the Approach ... .... .... .... ..... .... 182
2.3.2 Compression and Recovery. The Matrix A... ..... .... 184
2.3.3 Incoherence and Sensing. .... .... .... .... ..... .... 189
2.3.4 Stable and Robust Recovery .. .... .... .... ..... .... 190
2.3.5 Phase Transitions... .... .... .... .... .... ..... .... 191
2.3.6 Some Applications . .... .... .... .... .... ..... .... 192
2.4 Image Processing .... ..... .... .... .... .... .... ..... .... 193
2.4.1 Texture + Cartoon.. .... .... .... .... .... ..... .... 193
2.4.2 Patches . .... ..... .... .... .... .... .... ..... .... 202
2.4.3 Morphological Components... .... .... .... ..... .... 208
2.5 An Additional Repertory of Applicable Concepts
and Tools.. .... .... ..... .... .... .... .... .... ..... .... 215
2.5.1 Sparse Representation in MATLAB .... .... ..... .... 215
2.5.2 Diffusion in 2D.... .... .... .... .... .... ..... .... 219
2.5.3 Bregman-Related Algorithms.. .... .... .... ..... .... 228
2.6 Matrix Completion and Related Problems .. .... .... ..... .... 233
2.6.1 Matrix Completion . .... .... .... .... .... ..... .... 234
2.6.2 Decomposition of Matrices ... .... .... .... ..... .... 238
2.7 Experiments .... .... ..... .... .... .... .... .... ..... .... 241
2.7.1 Signal Denoising Based on Total Variation (TV) ... .... 241
2.7.2 Picture Reconstruction Based on Matrix Completion. .... 244
2.7.3 Text Removal ..... .... .... .... .... .... ..... .... 248
x Contents
2.8 Resources.. .... .... ..... .... .... .... .... .... ..... .... 250
2.8.1 MATLAB ... ..... .... .... .... .... .... ..... .... 250
2.8.2 Internet . .... ..... .... .... .... .... .... ..... .... 252
References.. .... .... .... ..... .... .... .... .... .... ..... .... 253
Appendix A: Selected Topics of Mathematical Optimization. ..... .... 263
Appendix B: Long Programs. ..... .... .... .... .... .... ..... .... 367
Index .... .... .... .... .... ..... .... .... .... .... .... ..... .... 429
List of Figures
Figure 1.1 Keeping the car at a distance from the road border.. ..... .. 7
Figure 1.2 Prediction (P), measurement (M) and update (U) PDFs.... .. 9
Figure 1.3 Variation of K in function of σy=σx . .... .... .... ..... .. 10
Figure 1.4 The algorithm is a cycle .. .... .... .... .... .... ..... .. 18
Figure 1.5 A two-tank system example ... .... .... .... .... ..... .. 18
Figure 1.6 System outputs (measurements). .... .... .... .... ..... .. 19
Figure 1.7 System states, and states estimated by the Kalman filter... .. 19
Figure 1.8 Error evolution .... ..... .... .... .... .... .... ..... .. 22
Figure 1.9 Evolution of the Kalman gains . .... .... .... .... ..... .. 23
Figure 1.10 Evolution of the state covariance.... .... .... .... ..... .. 24
Figure 1.11 The prediction step, from left to right.... .... .... ..... .. 25
Figure 1.12 The measurement... ..... .... .... .... .... .... ..... .. 25
Figure 1.13 Estimation of the next state.... .... .... .... .... ..... .. 26
Figure 1.14 Bayes net corresponding to Kalman filter . .... .... ..... .. 27
Figure 1.15 Satellite position under disturbances . .... .... .... ..... .. 33
Figure 1.16 Example of nonlinear function: arctan()... .... .... ..... .. 34
Figure 1.17 Original and propagated PDFs.. .... .... .... .... ..... .. 35
Figure 1.18 Propagation of a PDF through nonlinearity.... .... ..... .. 36
Figure 1.19 Propagated PDFs for sigma¼0:7;1;2 ... .... .... ..... .. 37
Figure 1.20 Propagation of a shifted PDF through nonlinearity .. ..... .. 38
Figure 1.21 Basic linear approximation using tangent . .... .... ..... .. 42
Figure 1.22 Falling body example .... .... .... .... .... .... ..... .. 43
Figure 1.23 System states (cross marks).... .... .... .... .... ..... .. 44
Figure 1.24 Distance measurement and drag .... .... .... .... ..... .. 45
Figure 1.25 The three non-zero components of the of=ox Jacobian ... .. 47
Figure 1.26 Propagation of ellipsoids (state N=43 -> 44)s.. .... ..... .. 49
Figure 1.27 System states (cross marks) under noisy conditions.. ..... .. 51
Figure 1.28 Distance measurement. Drag... .... .... .... .... ..... .. 52
Figure 1.29 System states (cross marks), and states estimated
by the EKF (continuous).. .... .... .... .... .... ..... .. 54
xi
xii ListofFigures
Figure 1.30 Error evolution .... ..... .... .... .... .... .... ..... .. 56
Figure 1.31 Evolution of matrix P .... .... .... .... .... .... ..... .. 57
Figure 1.32 Evolution of the altitude and velocity Kalman gains. ..... .. 57
Figure 1.33 Propagation of a PDF through nonlinearity and through
tangent... .... .... ..... .... .... .... .... .... ..... .. 58
Figure 1.34 Propagated PDFs for sigma¼0:2;0:4;0:6 .... .... ..... .. 60
Figure 1.35 Propagation of a shifted PDF through nonlinearity,
and through tangent. ..... .... .... .... .... .... ..... .. 61
Figure 1.36 Example of sigma points.. .... .... .... .... .... ..... .. 65
Figure 1.37 Example of sigma points.. .... .... .... .... .... ..... .. 66
Figure 1.38 Propagation of sigma points ... .... .... .... .... ..... .. 68
Figure 1.39 Uncertainty on the angle-radius plane (satellite example),
and sigma points... ..... .... .... .... .... .... ..... .. 69
Figure 1.40 The uncertainty on the Cartesian plane, and sigma points.. .. 69
Figure 1.41 The UT approximation and propagated data points.. ..... .. 70
Figure 1.42 System states (cross marks), and states estimated
by the UKF (continuous).. .... .... .... .... .... ..... .. 72
Figure 1.43 Error evolution .... ..... .... .... .... .... .... ..... .. 76
Figure 1.44 Evolution of matrix P .... .... .... .... .... .... ..... .. 76
Figure 1.45 Evolution of the altitude and velocity Kalman gains. ..... .. 77
Figure 1.46 System states (cross marks), and states estimated
by the particle filter (continuous).... .... .... .... ..... .. 79
Figure 1.47 Error evolution .... ..... .... .... .... .... .... ..... .. 82
Figure 1.48 Histogram of weights, resampling example.... .... ..... .. 83
Figure 1.49 Cumsum() of weights, resampling example.... .... ..... .. 84
Figure 1.50 Zoom on the cumsum() plot ... .... .... .... .... ..... .. 84
Figure 1.51 Histograms of prior and resampled particles ... .... ..... .. 85
Figure 1.52 Histograms of systematic, stratified, multinomial,
residual resampling example ... .... .... .... .... ..... .. 89
Figure 1.53 A possible situation of prior and likelihood PDFs... ..... .. 94
Figure 1.54 Using Gaussian kernel for filter regularization.. .... ..... .. 100
Figure 1.55 Organigram of the section. .... .... .... .... .... ..... .. 101
Figure 1.56 Approximations of Student’s T PDF: (top) Laplace's
method, (bottom) KLD minimization. .... .... .... ..... .. 107
Figure 1.57 Organigram of the section. .... .... .... .... .... ..... .. 113
Figure 1.58 System estimated states (cross marks), and fixed-interval
smoothed states (continuous)... .... .... .... .... ..... .. 124
Figure 1.59 System estimated states (cross marks), and fixed-lagl
smoothed states (continuous)... .... .... .... .... ..... .. 129
Figure 1.60 Smoothing of states with fixed-point smoothing .... ..... .. 132
Figure 1.61 Evolution of covariances in the fixed-point smoothing
example.. .... .... ..... .... .... .... .... .... ..... .. 132
Figure 1.62 Bearings only tracking scenario. .... .... .... .... ..... .. 139