Table Of ContentSPRINGER BRIEFS IN COMPUTER SCIENCE
Xiaojun Yuan
Zhipeng Xue
Turbo Message
Passing
Algorithms
for Structured
Signal Recovery
123
SpringerBriefs in Computer Science
SeriesEditors
StanZdonik,BrownUniversity,Providence,RI,USA
ShashiShekhar,UniversityofMinnesota,Minneapolis,MN,USA
XindongWu,UniversityofVermont,Burlington,VT,USA
LakhmiC.Jain,UniversityofSouthAustralia,Adelaide,SA,Australia
DavidPadua,UniversityofIllinoisUrbana-Champaign,Urbana,IL,USA
XueminShermanShen,UniversityofWaterloo,Waterloo,ON,Canada
BorkoFurht,FloridaAtlanticUniversity,BocaRaton,FL,USA
V.S.Subrahmanian,UniversityofMaryland,CollegePark,MD,USA
MartialHebert,CarnegieMellonUniversity,Pittsburgh,PA,USA
KatsushiIkeuchi,Meguro-ku,UniversityofTokyo,Tokyo,Japan
BrunoSiciliano,UniversitàdiNapoliFedericoII,Napoli,Italy
SushilJajodia,GeorgeMasonUniversity,Fairfax,VA,USA
NewtonLee,InstituteforEducation,ResearchandScholarships,LosAngeles,
CA,USA
SpringerBriefs present concise summaries of cutting-edge research and practical
applicationsacrossawidespectrumoffields.Featuringcompactvolumesof50to
125pages,theseriescoversarangeofcontentfromprofessionaltoacademic.
Typicaltopicsmightinclude:
(cid:129) Atimelyreportofstate-of-theartanalyticaltechniques
(cid:129) A bridge between new research results, as published in journal articles, and a
contextualliteraturereview
(cid:129) Asnapshotofahotoremergingtopic
(cid:129) Anin-depthcasestudyorclinicalexample
(cid:129) Apresentationofcoreconceptsthatstudentsmustunderstandinordertomake
independentcontributions
Briefsallowauthorstopresenttheirideasandreaderstoabsorbthemwithminimal
time investment. Briefs will be published as part of Springer’s eBook collection,
withmillionsofusersworldwide.Inaddition,Briefswillbeavailableforindividual
print and electronic purchase. Briefs are characterized by fast, global electronic
dissemination, standard publishing contracts, easy-to-use manuscript preparation
and formatting guidelines, and expedited production schedules. We aim for pub-
lication 8–12 weeks after acceptance. Both solicited and unsolicited manuscripts
areconsideredforpublicationinthisseries.
**Indexing:ThisseriesisindexedinScopusandzbMATH**
Moreinformationaboutthisseriesathttp://www.springer.com/series/10028
Xiaojun Yuan (cid:129) Zhipeng Xue
Turbo Message Passing
Algorithms for Structured
Signal Recovery
XiaojunYuan ZhipengXue
CenterforIntelligentNetworking& SchoolofInformationScienceand
Communication Technology
UniversityofElectronicScience& ShanghaiTechUniversity
TechnologyofChina Shanghai,China
Chengdu,China
ISSN2191-5768 ISSN2191-5776 (electronic)
SpringerBriefsinComputerScience
ISBN978-3-030-54761-5 ISBN978-3-030-54762-2 (eBook)
https://doi.org/10.1007/978-3-030-54762-2
©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2020
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Messagepassingisanefficientiterativealgorithmforsolvinginference,optimiza-
tion, and satisfaction problems. It has found successful applications in numerous
areas, including channel coding (e.g., low-density parity check (LDPC) codes and
turbocodes),computervision(e.g.,stereoimages),andfreeenergyapproximation.
Recently,novelmessagepassingalgorithms,suchasapproximatemessagepassing
(AMP)andturbocompressedsensing(Turbo-CS),havebeendevelopedforsparse
signal reconstruction. While the AMP algorithm exhibits guaranteed performance
for sensing matrices with independent and identically distributed (i.i.d.) elements,
Turbo-CS achieves a much better performance when the sensing matrix is con-
structedbasedonfastorthogonaltransforms(suchasthediscreteFouriertransform
and the discrete cosine transform). Inspired by the success of Turbo-CS, a series
ofmessagepassingalgorithmshavebeendevelopedforsolvingvariousstructured
signalrecoveryproblemswithcompressedmeasurements.Wecallthesealgorithms
turbomessagepassingalgorithms.Inthisbook,wewillundertakeacomprehensive
study on turbo message passing algorithms for structured signal recovery, where
the structured signals include: (1) a sparse vector/matrix (which corresponds to
the compressed sensing (CS) problem), (2) a low-rank matrix (which corresponds
to the affine rank minimization (ARM) problem), and (3) a mixture of a sparse
matrixandalow-rankmatrix(whichcorrespondstotherobustprincipalcomponent
analysis (RPCA) problem). In particular, the book is divided into the following
parts: First, for the CS problem, we introduce a turbo message passing algorithm
termed denoising-based Turbo-CS (D-Turbo-CS). We show that with D-Turbo-
CS, signals without the knowledge of the prior distributions, such as images, can
be recovered from compressed measurements by using message passing with a
verylowmeasurementrate.Second,weintroduceaturbomessagepassing(TMP)
algorithm for solving the ARM problem. We further discuss the impact of the use
ofvariouslinearoperatorsontherecoveryperformance,suchasright-orthogonally
invariantlinear(ROIL)operatorsandrandomselectors.Third,weintroduceaTMP
algorithmforsolvingthecompressiveRPCAproblemwhichaimstorecoveralow-
rank matrix and a sparse matrix from their compressed mixture. We then apply
the TMP algorithm to the video background subtraction problem and show that
v
vi Preface
TMP achieves much better numerical and visual recovery performance than its
counterparts.Withthisbook,wewishtospurnewresearchesonapplyingmessage
passingtovariousinferenceproblems.
Chengdu,China XiaojunYuan
Shanghai,China ZhipengXue
June2020
Acknowledgements
Thisbookhasbecomearealitywiththekindsupportandhelpofmanyindividuals
inthepastfewyears.Wewouldliketoextendoursincerethankstoallofthem.
vii
Contents
1 Introduction .................................................................. 1
1.1 Background.............................................................. 1
1.1.1 CompressedSensing............................................ 2
1.1.2 Low-RankMatrixRecovery.................................... 3
1.1.3 CompressedRobustPrincipalComponentAnalysis.......... 3
1.2 Contributions............................................................ 4
1.3 Organization............................................................. 4
References..................................................................... 5
2 TurboMessagePassingforCompressedSensing......................... 7
2.1 Preliminaries ............................................................ 7
2.1.1 CompressedSensing............................................ 7
2.1.2 TurboCompressedSensing .................................... 8
2.2 Denoising-BasedTurbo-CS ............................................ 11
2.2.1 ProblemStatement.............................................. 11
2.2.2 ExtrinsicMessagesforaGenericDenoiser ................... 12
2.2.3 Denoising-BasedTurbo-CS.................................... 15
2.3 ConstructionofExtrinsicDenoisers ................................... 16
2.3.1 ExtrinsicSURE-LETDenoiser ................................ 16
2.3.2 OtherExtrinsicDenoisers...................................... 18
2.4 EvolutionAnalysisofD-Turbo-CS .................................... 19
2.4.1 MSEEvolution.................................................. 19
2.4.2 SparseSignalwithi.i.d.Entries................................ 20
2.4.3 SignalwithCorrelatedEntries ................................. 21
2.5 PerformanceComparisons.............................................. 23
2.6 Summary ................................................................ 26
References..................................................................... 27
3 Turbo-TypeAlgorithmforAffineRankMinimization .................. 29
3.1 ProblemDescription.................................................... 29
3.2 TheTARMAlgorithm.................................................. 30
3.2.1 AlgorithmDescription.......................................... 30
ix
x Contents
3.2.2 DeterminingtheParametersofTARM ........................ 32
3.3 Low-RankMatrixRecovery............................................ 34
3.3.1 Preliminaries.................................................... 34
3.3.2 ParameterDesign............................................... 35
3.3.3 StateEvolution.................................................. 37
3.3.4 NumericalResults .............................................. 39
3.4 MatrixCompletion...................................................... 43
3.4.1 Determiningμ ................................................. 43
t
3.4.2 Determiningα andc .......................................... 44
t t
3.4.3 NumericalResults .............................................. 45
3.5 MoreNumericalResults................................................ 48
3.5.1 Low-RankMatrixRecovery.................................... 48
3.5.2 MatrixCompletion ............................................. 52
3.6 Summary ................................................................ 52
References..................................................................... 64
4 Turbo Message Passing for Compressed Robust Principal
ComponentAnalysis......................................................... 67
4.1 ProblemDescription.................................................... 67
4.2 Turbo-TypeMessagePassingAlgorithms ............................. 68
4.2.1 TMPFramework................................................ 68
4.2.2 ModuleA:LinearEstimation.................................. 69
4.2.3 ModuleB:SparseMatrixEstimation.......................... 72
4.2.4 ModuleC:Low-RankMatrixEstimation ..................... 74
4.2.5 OverallAlgorithm .............................................. 75
4.3 SparseMatrixDenoisersforModuleB................................ 77
4.3.1 Preliminaries.................................................... 77
4.3.2 Soft-ThresholdingDenoiser.................................... 77
4.3.3 SURE-LETDenoiser........................................... 78
4.3.4 OutputMSEofModuleB...................................... 79
4.4 Low-RankDenoisersforModuleC.................................... 80
4.4.1 Singular-ValueSoft-ThresholdingDenoiser................... 80
4.4.2 Singular-ValueHard-ThresholdingDenoiser.................. 81
4.4.3 BestRank-r Denoiser .......................................... 82
4.4.4 OutputMSEofModuleC...................................... 83
4.5 ConvergenceandComplexityofTMP................................. 83
4.5.1 RevisitingAssumptions4.1and4.2 ........................... 83
4.5.2 DampingTechnique ............................................ 85
4.5.3 ComplexityofTMP ............................................ 86
4.6 NumericalResults....................................................... 86
4.6.1 TMPwithDifferentDenoisersandMSEEstimations........ 87
4.6.2 ConvergenceComparisonofTMPwithExistingAlgorithms 88
4.6.3 PhaseTransition ................................................ 90
4.6.4 VideoBackgroundandForegroundSeparation ............... 91
