Adaptive Blind Signal and Image Processing: Learning Algorithms and Applications

587 Pages·2002·22.436 MB·English

by  Cichocki A., Amari Sh.-H.

Adaptive
Blind Signal
and
Image Processing
Learning Algorithms
and Applications
Andrzej CICHOCKI Shun-ichi AMARI includes CD

Contents
Preface xxix
1 Introduction to Blind Signal Processing: Problems and Applications 1
1.1 Problem Formulations – An Overview 2
1.1.1 Generalized Blind Signal Processing Problem 2
1.1.2 Instantaneous Blind Source Separation and
Independent Component Analysis 5
1.1.3 Independent Component Analysis for Noisy Data 11
1.1.4 Multichannel Blind Deconvolution and Separation 14
1.1.5 Blind Extraction of Signals 18
1.1.6 Generalized Multichannel Blind Deconvolution –
State Space Models 19
1.1.7 Nonlinear State Space Models – Semi-Blind Signal
Processing 21
1.1.8 Why State Space Demixing Models? 22
1.2 Potential Applications of Blind and Semi-Blind Signal
Processing 23
1.2.1 Biomedical Signal Processing 24
1.2.2 Blind Separation of Electrocardiographic Signals of
Fetus and Mother 25
1.2.3 Enhancement and Decomposition of EMG Signals 27
v
vi CONTENTS
1.2.4 EEG and Data MEG Processing 27
1.2.5 Application of ICA/BSS for Noise and Interference
Cancellation in Multi-sensory Biomedical Signals 29
1.2.6 Cocktail Party Problem 34
1.2.7 Digital Communication Systems 35
1.2.7.1 Why Blind? 37
1.2.8 Image Restoration and Understanding 37
2 Solving a System of Algebraic Equations and Related Problems 43
2.1 Formulation of the Problem for Systems of Linear Equations 44
2.2 Least-Squares Problems 45
2.2.1 Basic Features of the Least-Squares Solution 45
2.2.2 Weighted Least-Squares and Best Linear Unbiased
Estimation 47
2.2.3 Basic Network Structure-Least-Squares Criteria 49
2.2.4 Iterative Parallel Algorithms for Large and Sparse
Systems 49
2.2.5 Iterative Algorithms with Non-negativity Constraints 51
2.2.6 Robust Circuit Structure by Using the Interactively
Reweighted Least-Squares Criteria 54
2.2.7 Tikhonov Regularization and SVD 57
2.3 Least Absolute Deviation (1-norm) Solution of Systems of
Linear Equations 61
2.3.1 Neural Network Architectures Using a Smooth
Approximation and Regularization 62
2.3.2 Neural Network Model for LAD Problem Exploiting
Inhibition Principles 64
2.4 Total Least-Squares and Data Least-Squares Problems 67
2.4.1 Problems Formulation 67
2.4.1.1 A Historical Overview of the TLS Problem 67
2.4.2 Total Least-Squares Estimation 69
2.4.3 Adaptive Generalized Total Least-Squares 73
2.4.4 Extended TLS for Correlated Noise Statistics 75
2.4.4.1 Choice of R¯ in Some Practical Situations 77
NN
2.4.5 Adaptive Extended Total Least-Squares 77
2.4.6 An Illustrative Example - Fitting a Straight Line to a
Set of Points 78
2.5 Sparse Signal Representation and Minimum Fuel Consumption
Problem 79
CONTENTS vii
2.5.1 Approximate Solution of Minimum Fuel Problem
Using Iterative LS Approach 81
2.5.2 FOCUSS Algorithms 83
3 Principal/Minor Component Analysis and Related Problems 87
3.1 Introduction 87
3.2 Basic Properties of PCA 88
3.2.1 Eigenvalue Decomposition 88
3.2.2 Estimation of Sample Covariance Matrices 90
3.2.3 Signal and Noise Subspaces - AIC and MDL Criteria
for their Estimation 91
3.2.4 Basic Properties of PCA 93
3.3 Extraction of Principal Components 94
3.4 Basic Cost Functions and Adaptive Algorithms for PCA 98
3.4.1 The Rayleigh Quotient – Basic Properties 98
3.4.2 Basic Cost Functions for Computing Principal and
Minor Components 99
3.4.3 Fast PCA Algorithm Based on the Power Method 101
3.4.4 Inverse Power Iteration Method 104
3.5 Robust PCA 104
3.6 Adaptive Learning Algorithms for MCA 107
3.7 Unified Parallel Algorithms for PCA/MCA and PSA/MSA 110
3.7.1 Cost Function for Parallel Processing 111
3.7.2 Gradient of J(W) 112
3.7.3 Stability Analysis 113
3.7.4 Unified Stable Algorithms 116
3.8 SVD in Relation to PCA and Matrix Subspaces 118
3.9 Multistage PCA for BSS 119
Appendix A. Basic Neural Networks Algorithms for Real and
Complex-Valued PCA 122
Appendix B. Hierarchical Neural Network for Complex-valued
PCA 125
4 Blind Decorrelation and SOS for Robust Blind Identification 129
4.1 Spatial Decorrelation - Whitening Transforms 130
4.1.1 Batch Approach 130
4.1.2 Optimization Criteria for Adaptive Blind Spatial
Decorrelation 132
viii CONTENTS
4.1.3 Derivation of Equivariant Adaptive Algorithms for
Blind Spatial Decorrelation 133
4.1.4 Simple Local Learning Rule 136
4.1.5 Gram-Schmidt Orthogonalization 138
4.1.6 Blind Separation of Decorrelated Sources Versus
Spatial Decorrelation 139
4.1.7 Bias Removal for Noisy Data 139
4.1.8 Robust Prewhitening - Batch Algorithm 140
4.2 SOS Blind Identification Based on EVD 141
4.2.1 Mixing Model 141
4.2.2 Basic Principles: SD and EVD 143
4.3 Improved Blind Identification Algorithms Based on
EVD/SVD 148
4.3.1 Robust Orthogonalization of Mixing Matrices for
Colored Sources 148
4.3.2 Improved Algorithm Based on GEVD 153
4.3.3 Improved Two-stage Symmetric EVD/SVD Algorithm 155
4.3.4 BSS and Identification Using Bandpass Filters 156
4.4 Joint Diagonalization - Robust SOBI Algorithms 157
4.4.1 Modified SOBI Algorithm for Nonstationary Sources:
SONS Algorithm 160
4.4.2 Computer Simulation Experiments 161
4.4.3 Extensions of Joint Approximate Diagonalization
Technique 162
4.4.4 Comparison of the JAD and Symmetric EVD 163
4.5 Cancellation of Correlation 164
4.5.1 Standard Estimation of Mixing Matrix and Noise
Covariance Matrix 164
4.5.2 Blind Identification of Mixing Matrix Using the
Concept of Cancellation of Correlation 165
Appendix A. Stability of the Amari’s Natural Gradient and
the Atick-Redlich Formula 168
Appendix B. Gradient Descent Learning Algorithms with
Invariant Frobenius Norm of the Separating Matrix 171
Appendix C. JADE Algorithm 173
5 Sequential Blind Signal Extraction 177
5.1 Introduction and Problem Formulation 178
5.2 Learning Algorithms Based on Kurtosis as Cost Function 180
CONTENTS ix
5.2.1 A Cascade Neural Network for Blind Extraction of
Non-Gaussian Sources with Learning Rule Based on
Normalized Kurtosis 181
5.2.2 Algorithms Based on Optimization of Generalized
Kurtosis 184
5.2.3 KuicNet Learning Algorithm 186
5.2.4 Fixed-point Algorithms 187
5.2.5 Sequential Extraction and Deflation Procedure 191
5.3 On Line Algorithms for Blind Signal Extraction of
Temporally Correlated Sources 193
5.3.1 On Line Algorithms for Blind Extraction Using
Linear Predictor 195
5.3.2 Neural Network for Multi-unit Blind Extraction 197
5.4 Batch Algorithms for Blind Extraction of Temporally
Correlated Sources 199
5.4.1 Blind Extraction Using a First Order Linear Predictor 201
5.4.2 Blind Extraction of Sources Using Bank of Adaptive
Bandpass Filters 202
5.4.3 Blind Extraction of Desired Sources Correlated with
Reference Signals 205
5.5 Statistical Approach to Sequential Extraction of Independent
Sources 206
5.5.1 Log Likelihood and Cost Function 206
5.5.2 Learning Dynamics 208
5.5.3 Equilibrium of Dynamics 209
5.5.4 Stability of Learning Dynamics and Newton’s Method 210
5.6 Statistical Approach to Temporally Correlated Sources 212
5.7 On-line Sequential Extraction of Convolved and Mixed
Sources 214
5.7.1 Formulation of the Problem 214
5.7.2 Extraction of Single i.i.d. Source Signal 215
5.7.3 Extraction of Multiple i.i.d. Sources 217
5.7.4 Extraction of Colored Sources from Convolutive
Mixture 218
5.8 Computer Simulations: Illustrative Examples 219
5.8.1 Extraction of Colored Gaussian Signals 219
5.8.2 Extraction of Natural Speech Signals from Colored
Gaussian Signals 221
5.8.3 Extraction of Colored and White Sources 222
5.8.4 Extraction of Natural Image Signal from Interferences 223
x CONTENTS
5.9 Concluding Remarks 224
Appendix A. Global Convergence of Algorithms for Blind
Source Extraction Based on Kurtosis 225
Appendix B. Analysis of Extraction and Deflation Procedure 227
Appendix C. Conditions for Extraction of Sources Using
Linear Predictor Approach 228
6 Natural Gradient Approach to Independent Component Analysis 231
6.1 Basic Natural Gradient Algorithms 232
6.1.1 Kullback–Leibler Divergence - Relative Entropy as
Measure of Stochastic Independence 232
6.1.2 Derivation of Natural Gradient Basic Learning Rules 235
6.2 Generalizations of Basic Natural Gradient Algorithm 237
6.2.1 Nonholonomic Learning Rules 237
6.2.2 Natural Riemannian Gradient in Orthogonality
Constraint 239
6.2.2.1 Local Stability Analysis 240
6.3 NG Algorithms for Blind Extraction 242
6.3.1 Stiefel Manifolds Approach 242
6.4 Generalized Gaussian Distribution Model 243
6.4.1 The Moments of the Generalized Gaussian
Distribution 248
6.4.2 Kurtosis and Gaussian Exponent 249
6.4.3 The Flexible ICA Algorithm 250
6.4.4 Pearson Model 253
6.5 Natural Gradient Algorithms for Non-stationary Sources 254
6.5.1 Model Assumptions 254
6.5.2 Second Order Statistics Cost Function 255
6.5.3 Derivation of NG Learning Algorithms 255
Appendix A. Derivation of Local Stability Conditions for NG
ICA Algorithm (6.19) 258
Appendix B. Derivation of the Learning Rule (6.32) and
Stability Conditions for ICA 260
Appendix C. Stability of Generalized Adaptive Learning
Algorithm 262
Appendix D. Dynamic Properties and Stability of
Nonholonomic NG Algorithms 264
Appendix E. Summary of Stability Conditions 267
Appendix F. Natural Gradient for Non-square Separating
Matrix 268