Table Of ContentSignals and Communication Technology
Jose Maria Giron-Sierra
Digital Signal
Processing with
Matlab Examples,
Volume 3
Model-Based Actions and Sparse
Representation
Signals and Communication Technology
More information about this series at http://www.springer.com/series/4748
Jose Maria Giron-Sierra
Digital Signal Processing
with Matlab Examples,
Volume 3
Model-Based Actions and Sparse
Representation
123
Jose Maria Giron-Sierra
SystemsEngineeringandAutomaticControl
Universidad Complutense deMadrid
Madrid
Spain
ISSN 1860-4862 ISSN 1860-4870 (electronic)
Signals andCommunication Technology
ISBN978-981-10-2539-6 ISBN978-981-10-2540-2 (eBook)
DOI 10.1007/978-981-10-2540-2
LibraryofCongressControlNumber:2016951678
MATLAB® is a registered trademark of The MathWorks, Inc., and is used with permission. The
MathWorks does not warrant the accuracy of the text or exercises in this book. This book’s use or
discussionofMATLABsoftwareorrelatedproductsdoesnotconstituteendorsementorsponsorshipby
theMathWorksofaparticularpedagogicalapproachorparticularuseoftheMATLABsoftware.
©SpringerScience+BusinessMediaSingapore2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
foranyerrorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerNatureSingaporePteLtd.
Theregisteredcompanyaddressis:152BeachRoad,#22-06/08GatewayEast,Singapore189721,Singapore
Preface
This is the third book of a trilogy. As in the other books, a series of MATLAB
programs are embedded in the chapters for several purposes: to illustrate the
techniques, to provide implementation examples, to encourage for personal
exploration departing from a successful start.
The book has two parts, each having just one chapter. These chapters are long
and have a considerable number of bibliographic references.
When using a GPS on a car, sometimes it is not possible to keep contact with
satellites,likeforinstanceinsidetunnels.Inthiscase,amodelofthecarmotion—a
dynamic model—can be used for data substitution. The adequate combination of
measurementsandmodelsisthekeyideaoftheKalmanfilter,whichisthecentral
topic of the first part of the book. This filter was formulated for linear conditions.
Therearemodificationsfornonlinearconditions,liketheextendedKalmanfilter,or
the unscented Kalman filter. A new idea is to use particle filters. These topics are
covered in the chapter under an important perspective: Bayesian filtering.
Compressed sensing has emerged as a promising idea. One of the intended
applications is networked devices or sensors, which are becoming a reality. This
topic is considered in the second part of the book. Some experiments that
demonstrate image denoising applications were included.
For easier reading of the book, the longer programs have been put in an
appendix.Andasecondappendixonoptimizationhasbeenaddedtosupportsome
contents of the last chapter.
The reader is invited to discover the profound interconnections and common-
alities that exist behind the variety of topics in this book. This common ground
would become surely the humus for the next signal processing future.
As said in the preface of the other books, our particular expertise on signal
processing has two main roots: research and teaching. I belong to the Faculty of
Physics, University Complutense of Madrid, Spain. During our experimental
research on autonomous vehicles, maritime drones, satellite control, etc., we
practiced main methods of digital signal processing, for the use of a variety of
sensors and for prediction of vehicle motions. From years ago, I teach Signal
Processingina Master of Biomedical Physics,and aMasteronNew technologies.
v
vi Preface
The style of the programs included in the book is purposively simple enough.
Thereaderisinvitedtotypesettheprogramsincludedinthebook,foritwouldhelp
forcatchingcodingdetails.Anyway,allprogramsareavailablefromthebookweb
page: www.dacya.ucm.es/giron/SPBook3/Programs.
A lot of different materials have been used to erect this book: articles, blogs,
codes, experimentation. I tried to cite with adequate references all the pieces that
have been useful. If someone has been forgotten, please contact me. Most of the
references cited in the text are available from Internet. We have to express our
gratitude to the public information available in this way.
Please, send feedback and suggestions for further improvement and support.
Acknowledgments
ThankstomyUniversity, mycolleagues and mystudents. Since this andtheother
book required a lot of time taken from nights, weekends and holidays, I have to
sincerely express my gratitude to my family.
Madrid, Spain Jose Maria Giron-Sierra
Contents
Part I Model-Based Actions: Filtering, Prediction, Smoothing
1 Kalman Filter, Particle Filter and Other Bayesian Filters..... .... 3
1.1 Introduction .... .... ..... .... .... .... .... .... ..... .... 3
1.2 Preliminaries.... .... ..... .... .... .... .... .... ..... .... 5
1.2.1 A Basic Example... .... .... .... .... .... ..... .... 7
1.2.2 Prediction with the Gauss-Markov Model.... ..... .... 10
1.2.3 Continuation of a Simple Example of Recursive
Wiener Filter. ..... .... .... .... .... .... ..... .... 12
1.3 Kalman Filter... .... ..... .... .... .... .... .... ..... .... 16
1.3.1 The Algorithm..... .... .... .... .... .... ..... .... 17
1.3.2 Evolution of Filter Variables.. .... .... .... ..... .... 21
1.3.3 Several Perspectives .... .... .... .... .... ..... .... 27
1.3.4 Some Connections.. .... .... .... .... .... ..... .... 29
1.3.5 Numerical Issues... .... .... .... .... .... ..... .... 30
1.3.6 Information Filter .. .... .... .... .... .... ..... .... 31
1.4 Nonlinear Conditions . ..... .... .... .... .... .... ..... .... 32
1.4.1 Propagation and Nonlinearity . .... .... .... ..... .... 32
1.4.2 Jacobian. Hessian. Change of Coordinates.... ..... .... 39
1.4.3 Local Linearization . .... .... .... .... .... ..... .... 41
1.4.4 Example of a Body Falling Towards Earth... ..... .... 43
1.5 Extended Kalman Filter (EKF)... .... .... .... .... ..... .... 52
1.5.1 The EKF Algorithm .... .... .... .... .... ..... .... 52
1.5.2 Assessment of the Linearized Approximation . ..... .... 58
1.6 Unscented Kalman Filter (UKF).. .... .... .... .... ..... .... 62
1.6.1 The Unscented Transform.... .... .... .... ..... .... 63
1.6.2 The Unscented Kalman Filter (UKF).... .... ..... .... 70
1.7 Particle Filter ... .... ..... .... .... .... .... .... ..... .... 76
1.7.1 An Implementation of the Particle Filter. .... ..... .... 77
1.7.2 Resampling Schemes.... .... .... .... .... ..... .... 82
1.7.3 Multinomial Resampling. .... .... .... .... ..... .... 84
vii
viii Contents
1.7.4 Systematic Resampling .. .... .... .... .... ..... .... 86
1.7.5 Stratified Resampling.... .... .... .... .... ..... .... 87
1.7.6 Residual Resampling.... .... .... .... .... ..... .... 87
1.7.7 Comparison.. ..... .... .... .... .... .... ..... .... 88
1.7.8 Roughening.. ..... .... .... .... .... .... ..... .... 88
1.7.9 Basic Theory of the Particle Filter.. .... .... ..... .... 89
1.7.10 Sequential Monte Carlo (SMC).... .... .... ..... .... 90
1.7.11 Proposal Importance Functions .... .... .... ..... .... 93
1.7.12 Particle Filter Variants... .... .... .... .... ..... .... 97
1.7.13 Marginalized Particle Filter (Rao-Blackwellized
Particle Filter)..... .... .... .... .... .... ..... .... 98
1.7.14 Regularized Particle Filters ... .... .... .... ..... .... 99
1.8 The Perspective of Numerical Integration... .... .... ..... .... 100
1.8.1 Geometry.... ..... .... .... .... .... .... ..... .... 101
1.8.2 Quadrature... ..... .... .... .... .... .... ..... .... 102
1.8.3 Other Approximations... .... .... .... .... ..... .... 105
1.8.4 Gaussian Filters.... .... .... .... .... .... ..... .... 108
1.8.5 Assumed Density. Expectation Propagation... ..... .... 112
1.9 Other Bayesian Filters ..... .... .... .... .... .... ..... .... 113
1.9.1 Ensemble Kalman Filter (EnKF)... .... .... ..... .... 114
1.9.2 Iterative Kalman Filter... .... .... .... .... ..... .... 115
1.9.3 Gaussian Particle Filter .. .... .... .... .... ..... .... 116
1.9.4 Divide and Conquer .... .... .... .... .... ..... .... 116
1.9.5 Combinations ..... .... .... .... .... .... ..... .... 118
1.9.6 Algorithms with Special Characteristics . .... ..... .... 118
1.10 Smoothing . .... .... ..... .... .... .... .... .... ..... .... 119
1.10.1 Optimal Prediction.. .... .... .... .... .... ..... .... 119
1.10.2 One-Stage Smoothing ... .... .... .... .... ..... .... 120
1.10.3 Three Types of Smoothers ... .... .... .... ..... .... 122
1.10.4 Bayesian Smoothing .... .... .... .... .... ..... .... 132
1.11 Applications of Bayesian Filters.. .... .... .... .... ..... .... 134
1.11.1 Navigation... ..... .... .... .... .... .... ..... .... 134
1.11.2 Tracking .... ..... .... .... .... .... .... ..... .... 134
1.11.3 Information Fusion . .... .... .... .... .... ..... .... 135
1.11.4 SLAM.. .... ..... .... .... .... .... .... ..... .... 135
1.11.5 Speech, Sounds.... .... .... .... .... .... ..... .... 137
1.11.6 Images.. .... ..... .... .... .... .... .... ..... .... 137
1.11.7 Earth Monitoring and Forecasting.. .... .... ..... .... 137
1.11.8 Energy and Economy ... .... .... .... .... ..... .... 138
1.11.9 Medical Applications.... .... .... .... .... ..... .... 138
1.11.10 Traffic .. .... ..... .... .... .... .... .... ..... .... 138
1.11.11 Other Applications . .... .... .... .... .... ..... .... 139
Contents ix
1.12 Frequently Cited Examples.. .... .... .... .... .... ..... .... 139
1.12.1 Bearings-Only Tracking . .... .... .... .... ..... .... 139
1.12.2 Other Tracking Cases ... .... .... .... .... ..... .... 140
1.12.3 Univariate Non-stationary Growth Model .... ..... .... 140
1.12.4 Financial Volatility Model.... .... .... .... ..... .... 141
1.12.5 Nonlinear Series ... .... .... .... .... .... ..... .... 141
1.12.6 The Pendulum..... .... .... .... .... .... ..... .... 142
1.13 Resources.. .... .... ..... .... .... .... .... .... ..... .... 142
1.13.1 MATLAB ... ..... .... .... .... .... .... ..... .... 142
1.13.2 Internet . .... ..... .... .... .... .... .... ..... .... 143
References.. .... .... .... ..... .... .... .... .... .... ..... .... 144
Part II Sparse Representation. Compressed Sensing
2 Sparse Representations... ..... .... .... .... .... .... ..... .... 151
2.1 Introduction .... .... ..... .... .... .... .... .... ..... .... 151
2.2 Sparse Solutions. .... ..... .... .... .... .... .... ..... .... 152
2.2.1 The Central Problem.... .... .... .... .... ..... .... 152
2.2.2 Norms and Sparsity. .... .... .... .... .... ..... .... 155
2.2.3 Solving Sparsity Optimization Problems. .... ..... .... 156
2.3 Compressed Sensing.. ..... .... .... .... .... .... ..... .... 182
2.3.1 Statement of the Approach ... .... .... .... ..... .... 182
2.3.2 Compression and Recovery. The Matrix A... ..... .... 184
2.3.3 Incoherence and Sensing. .... .... .... .... ..... .... 189
2.3.4 Stable and Robust Recovery .. .... .... .... ..... .... 190
2.3.5 Phase Transitions... .... .... .... .... .... ..... .... 191
2.3.6 Some Applications . .... .... .... .... .... ..... .... 192
2.4 Image Processing .... ..... .... .... .... .... .... ..... .... 193
2.4.1 Texture + Cartoon.. .... .... .... .... .... ..... .... 193
2.4.2 Patches . .... ..... .... .... .... .... .... ..... .... 202
2.4.3 Morphological Components... .... .... .... ..... .... 208
2.5 An Additional Repertory of Applicable Concepts
and Tools.. .... .... ..... .... .... .... .... .... ..... .... 215
2.5.1 Sparse Representation in MATLAB .... .... ..... .... 215
2.5.2 Diffusion in 2D.... .... .... .... .... .... ..... .... 219
2.5.3 Bregman-Related Algorithms.. .... .... .... ..... .... 228
2.6 Matrix Completion and Related Problems .. .... .... ..... .... 233
2.6.1 Matrix Completion . .... .... .... .... .... ..... .... 234
2.6.2 Decomposition of Matrices ... .... .... .... ..... .... 238
2.7 Experiments .... .... ..... .... .... .... .... .... ..... .... 241
2.7.1 Signal Denoising Based on Total Variation (TV) ... .... 241
2.7.2 Picture Reconstruction Based on Matrix Completion. .... 244
2.7.3 Text Removal ..... .... .... .... .... .... ..... .... 248
