Coherence. In Signal Processing and Machine Learning

495 Pages·2022·5.81 MB·English

by  David Ramírez, Ignacio Santamaría, Louis Scharf

David Ramírez
Ignacio Santamaría
Louis Scharf
Coherence
In Signal Processing
and Machine Learning
Coherence
David Ramírez • Ignacio Santamaría (cid:129)
Louis Scharf
Coherence
In Signal Processing
and Machine Learning
DavidRamírez IgnacioSantamaría
UniversidadCarlosIIIdeMadrid UniversidaddeCantabria
Madrid,Spain Santander,Spain
LouisScharf
ColoradoStateUniversity
FortCollins,CO,USA
ISBN978-3-031-13330-5 ISBN978-3-031-13331-2 (eBook)
https://doi.org/10.1007/978-3-031-13331-2
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerland
AG2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
ToAnaBelén,Carmen,andMerche
DavidRamírez
ToIsabel,Cristina,Inés,andIgnacio
IgnacioSantamaría
Tothenextgeneration
LouisScharf
Preface
This book is directed to graduate students, practitioners, and researchers in signal
processingandmachinelearning.Abasicunderstandingoflinearalgebraandprob-
abilityisassumed.Thisbackgroundiscomplementedinthebookwithappendices
onmatrixalgebra,(complex)multivariatenormaltheory,andrelateddistributions.
The book begins in Chap.1 with a breezy account of coherence as it has com-
monly appeared in science, engineering, signal processing, and machine learning.
Chapter 2isacomprehensive account of classicalleastsquares theory,withafew
originalvariationsoncross-validationandmodel-orderdetermination.Compression
of ambient space dimension is analyzed by comparing multidimensional scaling
andarandomizedsearchalgorithminspiredbytheJohnson-Lindenstrausslemma.
But the central aim of the book is to analyze coherence in its many guises,
beginningwiththecorrelationcoefficientanditsmultivariateextensionsinChap.3.
Chapters 4–8 contain a wealth of results on maximum likelihood theory in the
complexmultivariatenormalmodelforestimatingparametersanddetectingsignals
in first- and second-order statistical models. Chapters 5 and 6 are addressed
to matched and adaptive subspace detectors. Particular attention is paid to the
geometries,invariances,andnulldistributionsofthesesubspacedetectors.Chapters
7and8extendtheseresultstodetectionofsignalsthatarecommontotwoormore
channels, and to detection of spatial correlation and cyclostationarity. Coherence
playsacentralroleinthesechapters.
Chapter9addressessubspaceaveraging,anemergingtopicofinterestinsignal
processing and machine learning. The motivation is to identify subspace models
(or centroids) for measurements so that images may be classified or noncoherent
communication signals may be decoded. The dimension of the average or central
subspace can also be estimated efficiently and applied to source enumeration
in array processing. In Chap.10, classical quadratic performance bounds on the
accuracyofparameterestimatorsarecomplementedwithanaccountofinformation
geometry. The motivation is to harmonize performance bounds for parameter
estimators with the corresponding geometry of the underlying manifold of log-
likelihood random variables. Chapter 11 concludes the book with an account of
other problems and methods in signal processing and machine learning where
coherenceisanorganizingprinciple.
This book is more research monograph than textbook. However, many of its
chapters would serve as complementary resource materials in a graduate-level
vii
viii Preface
courseonsignalprocessing,machinelearning,orstatistics.Theappendicescontain
comprehensive accounts of matrix algebra and distribution theory, topics that join
optimization theory to form the mathematical foundations of signal processing
and machine learning. Chapters 2–4 would complement textbooks on multivariate
analysisbycoveringleastsquares,linearminimummean-squarederrorestimation,
and hypothesis testing of covariance structure in the complex multivariate normal
model.Chapters5–8containanaccountofmatchedandadaptivesubspacedetectors
that would complement a course on detection and estimation, multisensor array
processing, and related topics. Chapters 9–11 would serve as resource materials
inacourseonadvancedtopicsinsignalprocessingandmachinelearning.
Itwouldbe futiletoattempt anacknowledgment toallof our students,friends,
and colleagues who have influenced our thinking and guided our educations.
But several merit mention for their contribution of important ideas to this book:
YuriAbramovich,PiaAbbaddo,JavierÁlvarez-Vizoso,AntonioArtés-Rodríguez,
Mahmood Azimi-Sadjadi, Carlos Beltrán, Olivier Besson, Pascal Bondon, Ron
Butler, Margaret Cheney, Yuejie Chi, Edwin Chong, Doug Cochran, Henry Cox,
Víctor Elvira, Yossi Francos, Ben Friedlander, Antonio García Marqués, Scott
Goldstein,ClaudeGueguen,AlfredHanssen,StephenHoward,JesúsIbáñez,Steven
Kay, Michael Kirby, Nick Klausner, Shawn Kraut, Ramdas Kumaresan, Roberto
López-Valcarce,MikeMcCloud,ToddMcWhorter,BillMoran,TomMullis,Danilo
Orlando, Pooria Pakrooh, Daniel P. Palomar, Jesús Pérez-Arriaga, Chris Peterson,
AliPezeshki,BernardPicinbono,JoséPríncipe,GiuseppeRicci,ChristRichmond,
Peter Schreier, Santiago Segarra, Songsri Sirianunpiboon, Steven Smith, John
Thomas, Rick Vaccaro, Steven Van Vaerenbergh, Gonzalo Vázquez-Vilar, Javier
Vía, Haonan Wang, and Yuan Wang. Javier Álvarez-Vizoso, Barry Van Veen, Ron
Butler,andStephenHowardwerekindenoughtoreviewchaptersandofferhelpful
suggestions.
David Ramírez wishes to acknowledge the generous support received from the
Agencia Española de investigación (AEI), the Deutsche Forschungsgemeinschaft
(DFG),theComunidaddeMadrid(CAM),andtheOfficeofNavalResearch(ONR)
Global.
Ignacio Santamaría gratefully acknowledges the various agencies that have
funded his research over the years, in particular, the Agencia Española de inves-
tigación (AEI) and the funding received in different projects of the Plan Nacional
deI+D+IoftheSpanishgovernment.
LouisScharfacknowledges,withgratitude,generousresearchsupportfromthe
USNationalScienceFoundation(NSF),OfficeofNavalResearch(ONR),AirForce
Office of Scientific Research (AFOSR), and Defense Advanced Research Projects
Agency(DARPA).
Madrid,Spain DavidRamírez
Santander,Spain IgnacioSantamaría
FortCollins,CO,USA LouisScharf
December2021
Contents
1 Introduction................................................................. 1
1.1 TheCohererofHertz,Branly,andLodge .......................... 2
1.2 Interference,Coherence,andtheVanCittert-ZernikeStory....... 2
1.3 HanburyBrown-TwissEffect ....................................... 4
1.4 ToneWobbleandCoherenceforTuning ........................... 6
1.5 BeampatternsandDiffractionofElectromagneticRadiation
byaSlit............................................................... 7
1.6 LIGOandtheDetectionofEinstein’sGravitationalWaves ....... 9
1.7 CoherenceandtheHeisenbergUncertaintyRelations............. 10
1.8 Coherence,Ambiguity,andtheMoyalIdentities .................. 13
1.9 Coherence,Correlation,andMatchedFiltering.................... 13
1.10 CoherenceandMatchedSubspaceDetectors....................... 18
1.11 WhatQualifiesasaCoherence?..................................... 21
1.12 WhyComplex? ...................................................... 21
1.13 WhatistheRoleofGeometry? ..................................... 24
1.14 MotivatingProblems................................................. 24
1.15 APreviewoftheBook .............................................. 26
1.16 ChapterNotes........................................................ 30
2 LeastSquaresandRelated ................................................ 33
2.1 TheLinearModel.................................................... 35
2.2 Over-DeterminedLeastSquaresandRelated ...................... 37
2.2.1 LinearPrediction .......................................... 40
2.2.2 OrderDetermination....................................... 41
2.2.3 Cross-Validation........................................... 43
2.2.4 WeightedLeastSquares................................... 44
2.2.5 ConstrainedLeastSquares ................................ 45
2.2.6 ObliqueLeastSquares..................................... 46
2.2.7 TheBLUE(orMVUBorMVDR)Estimator ............ 48
2.2.8 SequentialLeastSquares.................................. 50
2.2.9 TotalLeastSquares........................................ 51
2.2.10 Least Squares and Procrustes Problems for
ChannelIdentification..................................... 54
2.2.11 LeastSquaresModalAnalysis............................ 55
ix
x Contents
2.3 Under-determinedLeastSquaresandRelated...................... 56
2.3.1 Minimum-NormSolution ................................. 56
2.3.2 SparseSolutions ........................................... 57
2.3.3 MaximumEntropySolution............................... 61
2.3.4 MinimumMean-SquaredErrorSolution................. 63
2.4 MultidimensionalScaling ........................................... 63
2.5 TheJohnson-LindenstraussLemma ................................ 68
2.6 ChapterNotes........................................................ 75
Appendices................................................................... 76
2.A CompletingtheSquareinHermitianQuadraticForms ............ 76
2.A.1 GeneralizingtoMultipleMeasurementsandOther
CostFunctions............................................. 76
2.A.2 LMMSEEstimation ....................................... 77
3 Coherence,ClassicalCorrelations,andtheirInvariances ............. 79
3.1 CoherenceBetweenaRandomVariableandaRandomVector... 80
3.2 CoherenceBetweenTwoRandomVectors......................... 86
3.2.1 RelationshipwithCanonicalCorrelations................ 87
3.2.2 TheCirculantCase ........................................ 88
3.2.3 RelationshipwithPrincipalAngles....................... 88
3.2.4 DistributionofEstimatedSignal-to-NoiseRatioin
AdaptiveMatchedFiltering............................... 90
3.3 CoherenceBetweenTwoTimeSeries .............................. 91
3.4 Multi-ChannelCoherence........................................... 94
3.5 PrincipalComponentAnalysis...................................... 95
3.6 Two-ChannelCorrelation............................................ 98
3.7 MultistageLMMSEFilter........................................... 106
3.8 ApplicationtoBeamformingandSpectrumAnalysis.............. 110
3.8.1 TheGeneralizedSidelobeCanceller ..................... 111
3.8.2 CompositeCovarianceMatrix ............................ 111
3.8.3 Distributions of the Conventional and Capon
Beamformers............................................... 113
3.9 Canonicalcorrelationanalysis ...................................... 115
3.9.1 CanonicalCoordinates .................................... 116
3.9.2 DimensionReductionBasedonCanonicaland
Half-CanonicalCoordinates............................... 117
3.10 PartialCorrelation ................................................... 118
3.10.1 RegressingTwoRandomVectorsontoOne.............. 119
3.10.2 RegressingOneRandomVectorontoTwo............... 121
3.11 ChapterNotes........................................................ 123
4 CoherenceandClassicalTestsintheMultivariateNormal
Model ........................................................................ 125
4.1 HowLimitingIstheMultivariateNormalModel?................. 125
4.2 LikelihoodintheMVNModel...................................... 126
4.2.1 Sufficiency ................................................. 127
4.2.2 Likelihood.................................................. 127
Contents xi
4.3 HypothesisTesting................................................... 130
4.4 InvarianceinHypothesisTesting.................................... 132
4.5 TestingforSphericityofRandomVariables........................ 134
4.5.1 SphericityTest:ItsInvariancesandNullDistribution ... 134
4.5.2 Extensions.................................................. 136
4.6 Testingforsphericityofrandomvectors ........................... 138
4.7 TestingforHomogeneityofCovarianceMatrices.................. 139
4.8 TestingforIndependence............................................ 141
4.8.1 TestingforIndependenceofRandomVariables.......... 141
4.8.2 TestingforIndependenceofRandomVectors............ 143
4.9 Cross-ValidationofaCovarianceModel ........................... 145
4.10 ChapterNotes........................................................ 147
5 MatchedSubspaceDetectors.............................................. 149
5.1 SignalandNoiseModels............................................ 150
5.2 TheDetectionProblemandItsInvariances......................... 154
5.3 DetectorsinaFirst-OrderModelforaSignalinaKnown
Subspace ............................................................. 155
5.3.1 Scale-InvariantMatchedSubspaceDetector ............. 156
5.3.2 MatchedSubspaceDetector............................... 157
5.4 Detectors in a Second-Order Model for a Signal in a
KnownSubspace..................................................... 158
5.4.1 Scale-InvariantMatchedSubspaceDetector ............. 158
5.4.2 MatchedSubspaceDetector............................... 161
5.5 DetectorsinaFirst-OrderModelforaSignalinaSubspace
KnownOnlybyitsDimension...................................... 162
5.5.1 Scale-InvariantMatchedDirectionDetector ............. 162
5.5.2 MatchedDirectionDetector............................... 163
5.6 Detectors in a Second-Order Model for a Signal in a
SubspaceKnownOnlybyitsDimension........................... 164
5.6.1 Scale-InvariantMatchedDirectionDetector ............. 165
5.6.2 MatchedDirectionDetector............................... 168
5.7 FactorAnalysis ...................................................... 169
5.8 AMIMOVersionoftheReed-YuDetector......................... 171
5.9 ChapterNotes........................................................ 176
Appendices................................................................... 177
5.A VariationsonMatchedSubspaceDetectorsinaFirst-Order
ModelforaSignalinaKnownSubspace .......................... 177
5.A.1 Scale-Invariant, Geometrically Averaged,
MatchedSubspaceDetector............................... 177
5.A.2 Refinement:SpecialSignalSequences ................... 178
5.A.3 Rapprochement ............................................ 179
5.B Derivation of the Matched Subspace Detector in a
Second-OrderModelforaSignalinaKnownSubspace.......... 180