Complex Valued Nonlinear Adaptive Filters: Noncircularity, Widely Linear and Neural Models

327 Pages·2009·14.716 MB·English

by  Danilo Mandic, Vanessa Goh

Complex Valued Nonlinear
Adaptive Filters
Complex Valued Nonlinear Adaptive Filters: Noncircularity, Widely Linear and Neural Models
Danilo P. Mandic and Vanessa Su Lee Goh
© 2009 John Wiley & Sons, Ltd. ISBN: 978-0-470-06635-5
www.it-ebooks.info
Complex Valued Nonlinear
Adaptive Filters
Noncircularity, Widely Linear and
Neural Models
DaniloP.Mandic
ImperialCollegeLondon,UK
VanessaSuLeeGoh
ShellEP,Europe
www.it-ebooks.info
Thiseditionfirstpublished2009
© 2009,JohnWiley&Sons,Ltd
Registeredoffice
JohnWiley&SonsLtd,TheAtrium,SouthernGate,Chichester,WestSussex,PO198SQ,UnitedKingdom
Fordetailsofourglobaleditorialoffices,forcustomerservicesandforinformationabouthowtoapplyfor
permissiontoreusethecopyrightmaterialinthisbookpleaseseeourwebsiteatwww.wiley.com.
TherightoftheauthortobeidentifiedastheauthorofthisworkhasbeenassertedinaccordancewiththeCopyright,
DesignsandPatentsAct1988.
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmitted,inany
formorbyanymeans,electronic,mechanical,photocopying,recordingorotherwise,exceptaspermittedbytheUK
Copyright,DesignsandPatentsAct1988,withoutthepriorpermissionofthepublisher.
Wileyalsopublishesitsbooksinavarietyofelectronicformats.Somecontentthatappearsinprintmaynotbe
availableinelectronicbooks.
Designationsusedbycompaniestodistinguishtheirproductsareoftenclaimedastrademarks.Allbrandnamesand
productnamesusedinthisbookaretradenames,servicemarks,trademarksorregisteredtrademarksoftheir
respectiveowners.Thepublisherisnotassociatedwithanyproductorvendormentionedinthisbook.This
publicationisdesignedtoprovideaccurateandauthoritativeinformationinregardtothesubjectmattercovered.Itis
soldontheunderstandingthatthepublisherisnotengagedinrenderingprofessionalservices.Ifprofessionaladvice
orotherexpertassistanceisrequired,theservicesofacompetentprofessionalshouldbesought.
LibraryofCongressCataloging-in-PublicationData
Mandic,DaniloP.
Complexvaluednonlinearadaptivefilters:noncircularity,widelylinear,andneuralmodels/byDaniloP.
Mandic,VanessaSuLeeGoh,ShellEP,Europe.
p.cm.
Includesbibliographicalreferencesandindex.
ISBN978-0-470-06635-5(cloth)
1. Functionsofcomplexvariables.2. Adaptivefilters–Mathematicalmodels.3. Filters(Mathematics)
4. Nonlineartheories.5. Neuralnetworks(Computerscience)I. Goh,VanessaSuLee.II. Holland,Shell.
III. Title.
TA347.C64.M362009
621.382’2–dc22 2009001965
AcataloguerecordforthisbookisavailablefromtheBritishLibrary.
ISBN:978-0-470-06635-5
Typesetin10/12ptTimesbyThomsonDigital,Noida,India
PrintedinGreatBritainbyCPIAntonyRowe,Chippenham,Wiltshire
www.it-ebooks.info
The real voyage of discovery consists
not in seeking new landscapes
but in having new eyes
Marcel Proust
www.it-ebooks.info
Contents
Preface xiii
Acknowledgements xvii
1 TheMagicofComplexNumbers 1
1.1 HistoryofComplexNumbers 2
1.1.1 HypercomplexNumbers 7
1.2 HistoryofMathematicalNotation 8
1.3 DevelopmentofComplexValuedAdaptiveSignalProcessing 9
2 WhySignalProcessingintheComplexDomain? 13
2.1 SomeExamplesofComplexValuedSignalProcessing 13
2.1.1 DualityBetweenSignalRepresentationsinRandC 18
2.2 ModellinginCisNotOnlyConvenientButAlsoNatural 19
2.3 WhyComplexModellingofRealValuedProcesses? 20
2.3.1 PhaseInformationinImaging 20
2.3.2 ModellingofDirectionalProcesses 22
2.4 ExploitingthePhaseInformation 23
2.4.1 SynchronisationofRealValuedProcesses 24
2.4.2 AdaptiveFilteringbyIncorporatingPhaseInformation 25
2.5 OtherApplicationsofComplexDomainProcessingofRealValuedSignals 26
2.6 AdditionalBenefitsofComplexDomainProcessing 29
3 AdaptiveFilteringArchitectures 33
3.1 LinearandNonlinearStochasticModels 34
3.2 LinearandNonlinearAdaptiveFilteringArchitectures 35
3.2.1 FeedforwardNeuralNetworks 36
3.2.2 RecurrentNeuralNetworks 37
3.2.3 NeuralNetworksandPolynomialFilters 38
3.3 StateSpaceRepresentationandCanonicalForms 39
www.it-ebooks.info
viii Contents
4 ComplexNonlinearActivationFunctions 43
4.1 PropertiesofComplexFunctions 43
4.1.1 SingularitiesofComplexFunctions 45
4.2 UniversalFunctionApproximation 46
4.2.1 UniversalApproximationinR 47
4.3 NonlinearActivationFunctionsforComplexNeuralNetworks 48
4.3.1 Split-complexApproach 49
4.3.2 FullyComplexNonlinearActivationFunctions 51
4.4 GeneralisedSplittingActivationFunctions(GSAF) 53
4.4.1 TheCliffordNeuron 53
4.5 Summary:ChoiceoftheComplexActivationFunction 54
5 ElementsofCRCalculus 55
5.1 ContinuousComplexFunctions 56
5.2 TheCauchy–RiemannEquations 56
5.3 GeneralisedDerivativesofFunctionsofComplexVariable 57
5.3.1 CRCalculus 59
5.3.2 LinkbetweenR-andC-derivatives 60
5.4 CR-derivativesofCostFunctions 62
5.4.1 TheComplexGradient 62
5.4.2 TheComplexHessian 64
5.4.3 TheComplexJacobianandComplexDifferential 64
5.4.4 GradientofaCostFunction 65
6 ComplexValuedAdaptiveFilters 69
6.1 AdaptiveFilteringConfigurations 70
6.2 TheComplexLeastMeanSquareAlgorithm 73
6.2.1 ConvergenceoftheCLMSAlgorithm 75
6.3 NonlinearFeedforwardComplexAdaptiveFilters 80
6.3.1 FullyComplexNonlinearAdaptiveFilters 80
6.3.2 DerivationofCNGDusingCRcalculus 82
6.3.3 Split-complexApproach 83
6.3.4 DualUnivariateAdaptiveFilteringApproach(DUAF) 84
6.4 NormalisationofLearningAlgorithms 85
6.5 PerformanceofFeedforwardNonlinearAdaptiveFilters 87
6.6 Summary:ChoiceofaNonlinearAdaptiveFilter 89
7 AdaptiveFilterswithFeedback 91
7.1 TrainingofIIRAdaptiveFilters 92
7.1.1 CoefficientUpdateforLinearAdaptiveIIRFilters 93
7.1.2 TrainingofIIRfilterswithReducedComputational
Complexity 96
www.it-ebooks.info
Contents ix
7.2 NonlinearAdaptiveIIRFilters:RecurrentPerceptron 97
7.3 TrainingofRecurrentNeuralNetworks 99
7.3.1 OtherLearningAlgorithmsandComputationalComplexity 102
7.4 SimulationExamples 102
8 FilterswithanAdaptiveStepsize 107
8.1 BenvenisteTypeVariableStepsizeAlgorithms 108
8.2 ComplexValuedGNGDAlgorithms 110
8.2.1 ComplexGNGDforNonlinearFilters(CFANNGD) 112
8.3 SimulationExamples 113
9 FilterswithanAdaptiveAmplitudeofNonlinearity 119
9.1 DynamicalRangeReduction 119
9.2 FIRAdaptiveFilterswithanAdaptiveNonlinearity 121
9.3 RecurrentNeuralNetworkswithTrainableAmplitudeofActivation
Functions 122
9.4 SimulationResults 124
10 Data-reusingAlgorithmsforComplexValuedAdaptiveFilters 129
10.1 TheData-reusingComplexValuedLeastMeanSquare(DRCLMS)
Algorithm 129
10.2 Data-reusingComplexNonlinearAdaptiveFilters 131
10.2.1 ConvergenceAnalysis 132
10.3 Data-reusingAlgorithmsforComplexRNNs 134
11 ComplexMappingsandMo¨biusTransformations 137
11.1 MatrixRepresentationofaComplexNumber 137
11.2 TheMo¨biusTransformation 140
11.3 ActivationFunctionsandMo¨biusTransformations 142
11.4 All-passSystemsasMo¨biusTransformations 146
11.5 FractionalDelayFilters 147
12 AugmentedComplexStatistics 151
12.1 ComplexRandomVariables(CRV) 152
12.1.1 ComplexCircularity 153
12.1.2 TheMultivariateComplexNormalDistribution 154
12.1.3 MomentsofComplexRandomVariables(CRV) 157
12.2 ComplexCircularRandomVariables 158
12.3 ComplexSignals 159
12.3.1 WideSenseStationarity,Multicorrelations,andMultispectra 160
12.3.2 StrictCircularityandHigher-orderStatistics 161
12.4 Second-orderCharacterisationofComplexSignals 161
12.4.1 AugmentedStatisticsofComplexSignals 161
12.4.2 Second-orderComplexCircularity 164
www.it-ebooks.info
x Contents
13 WidelyLinearEstimationandAugmentedCLMS(ACLMS) 169
13.1 MinimumMeanSquareError(MMSE)EstimationinC 169
13.1.1 WidelyLinearModellinginC 171
13.2 ComplexWhiteNoise 172
13.3 AutoregressiveModellinginC 173
13.3.1 WidelyLinearAutoregressiveModellinginC 174
13.3.2 QuantifyingBenefitsofWidelyLinearEstimation 174
13.4 TheAugmentedComplexLMS(ACLMS)Algorithm 175
13.5 AdaptivePredictionBasedonACLMS 178
13.5.1 WindForecastingUsingAugmentedStatistics 180
14 DualityBetweenComplexValuedandRealValuedFilters 183
14.1 ADualChannelRealValuedAdaptiveFilter 184
14.2 DualityBetweenRealandComplexValuedFilters 186
14.2.1 OperationofStandardComplexAdaptiveFilters 186
14.2.2 OperationofWidelyLinearComplexFilters 187
14.3 Simulations 188
15 WidelyLinearFilterswithFeedback 191
15.1 TheWidelyLinearARMA(WL-ARMA)Model 192
15.2 WidelyLinearAdaptiveFilterswithFeedback 192
15.2.1 WidelyLinearAdaptiveIIRFilters 195
15.2.2 AugmentedRecurrentPerceptronLearningRule 196
15.3 TheAugmentedComplexValuedRTRL(ACRTRL)Algorithm 197
15.4 TheAugmentedKalmanFilterAlgorithmforRNNs 198
15.4.1 EKFBasedTrainingofComplexRNNs 200
15.5 AugmentedComplexUnscentedKalmanFilter(ACUKF) 200
15.5.1 StateSpaceEquationsfortheComplexUnscentedKalman
Filter 201
15.5.2 ACUKFBasedTrainingofComplexRNNs 202
15.6 SimulationExamples 203
16 CollaborativeAdaptiveFiltering 207
16.1 ParametricSignalModalityCharacterisation 207
16.2 StandardHybridFilteringinR 209
16.3 TrackingtheLinear/NonlinearNatureofComplexValuedSignals 210
16.3.1 SignalModalityCharacterisationinC 211
16.4 SplitvsFullyComplexSignalNatures 214
16.5 OnlineAssessmentoftheNatureofWindSignal 216
16.5.1 EffectsofAveragingonSignalNonlinearity 216
16.6 CollaborativeFiltersforGeneralComplexSignals 217
16.6.1 HybridFiltersforNoncircularSignals 218
16.6.2 OnlineTestforComplexCircularity 220
www.it-ebooks.info
Contents xi
17 AdaptiveFilteringBasedonEMD 221
17.1 TheEmpiricalModeDecompositionAlgorithm 222
17.1.1 EmpiricalModeDecompositionasaFixedPointIteration 223
17.1.2 ApplicationsofRealValuedEMD 224
17.1.3 UniquenessoftheDecomposition 225
17.2 ComplexExtensionsofEmpiricalModeDecomposition 226
17.2.1 ComplexEmpiricalModeDecomposition 227
17.2.2 RotationInvariantEmpiricalModeDecomposition(RIEMD) 228
17.2.3 BivariateEmpiricalModeDecomposition(BEMD) 228
17.3 AddressingtheProblemofUniqueness 230
17.4 ApplicationsofComplexExtensionsofEMD 230
18 ValidationofComplexRepresentations–IsThisWorthwhile? 233
18.1 SignalModalityCharacterisationinR 234
18.1.1 SurrogateDataMethods 235
18.1.2 TestStatistics:TheDVVMethod 237
18.2 TestingfortheValidityofComplexRepresentation 239
18.2.1 ComplexDelayVectorVarianceMethod(CDVV) 240
18.3 QuantifyingBenefitsofComplexValuedRepresentation 243
18.3.1 ProsandConsoftheComplexDVVMethod 244
AppendixA:SomeDistinctivePropertiesofCalculusinC 245
AppendixB:Liouville’sTheorem 251
AppendixC:HypercomplexandCliffordAlgebras 253
C.1 DefinitionsofAlgebraicNotionsofGroup,RingandField 253
C.2 DefinitionofaVectorSpace 254
C.3 HigherDimensionAlgebras 254
C.4 TheAlgebraofQuaternions 255
C.5 CliffordAlgebras 256
AppendixD:RealValuedActivationFunctions 257
D.1 LogisticSigmoidActivationFunction 257
D.2 HyperbolicTangentActivationFunction 258
AppendixE:ElementaryTranscendentalFunctions(ETF) 259
AppendixF:TheONotationandStandardVectorandMatrixDifferentiation 263
F.1 TheONotation 263
F.2 StandardVectorandMatrixDifferentiation 263
www.it-ebooks.info
xii Contents
AppendixG:NotionsFromLearningTheory 265
G.1 TypesofLearning 266
G.2 TheBias–VarianceDilemma 266
G.3 RecursiveandIterativeGradientEstimationTechniques 267
G.4 TransformationofInputData 267
AppendixH:NotionsfromApproximationTheory 269
AppendixI:TerminologyUsedintheFieldofNeuralNetworks 273
AppendixJ:ComplexValuedPipelinedRecurrentNeuralNetwork(CPRNN) 275
J.1 TheComplexRTRLAlgorithm(CRTRL)forCPRNN 275
J.1.1 LinearSubsectionWithinthePRNN 277
AppendixK:GradientAdaptiveStepSize(GASS)AlgorithmsinR 279
K.1 GradientAdaptiveStepsizeAlgorithmsBasedon∂J/∂μ 280
K.2 VariableStepsizeAlgorithmsBasedon∂J/∂ε 281
AppendixL:DerivationofPartialDerivativesfromChapter8 283
L.1 Derivationof∂e(k)/∂w (k) 283
n
L.2 Derivationof∂e∗(k)/∂ε(k−1) 284
L.3 Derivationof∂w(k)/∂ε(k−1) 286
AppendixM:APosterioriLearning 287
M.1 APosterioriStrategiesinAdaptiveLearning 288
AppendixN:NotionsfromStabilityTheory 291
AppendixO:LinearRelaxation 293
O.1 VectorandMatrixNorms 293
O.2 RelaxationinLinearSystems 294
O.2.1 ConvergenceintheNormorStateSpace? 297
AppendixP:ContractionMappings,FixedPointIterationandFractals 299
P.1 HistoricalPerspective 303
P.2 MoreonConvergence:ModifiedContractionMapping 305
P.3 FractalsandMandelbrotSet 308
References 309
Index 321
www.it-ebooks.info