Splitting Methods in Communication, Imaging, Science, and Engineering

822 Pages·2016·41.949 MB·English

by  Roland Glowinski, Stanley J. Osher, Wotao Yin (eds.)

Scientifi c Computation
Roland Glowinski
Stanley J. Osher
Wotao Yin Editors
Splitting Methods
in Communication,
Imaging, Science,
and Engineering
Splitting Methods in Communication, Imaging,
Science, and Engineering
Scientific Computation
EditorialBoard
J.-J.Chattot,Davis,CA,USA
P.Colella,Berkeley,CA,USA
R.Glowinski,Houston,TX,USA
P.Joly,LeChesnay,France
D.I.Meiron,Pasadena,CA,USA
O.Pironneau,Paris,France
A.Quarteroni,Lausanne,Switzerland
andPolitecnicoofMilan,Italy
J.Rappaz,Lausanne,Switzerland
R.Rosner,Chicago,IL,USA
P.Sagaut,Paris,France
J.H.Seinfeld,Pasadena,CA,USA
A.Szepessy,Stockholm,Sweden
M.F.Wheeler,Austin,TX,USA
M.Y.Hussaini,Tallahassee,FL,USA
Moreinformationaboutthisseriesathttp://www.springer.com/series/718
Roland Glowinski • Stanley J. Osher • Wotao Yin
Editors
Splitting Methods
in Communication, Imaging,
Science, and Engineering
123
Editors
RolandGlowinski StanleyJ.Osher
DepartmentofMathematics DepartmentofMathematics
UniversityofHouston UCLA
Houston,TX,USA LosAngeles,CA,USA
WotaoYin
DepartmentofMathematics
UCLA
LosAngeles,CA,USA
ISSN1434-8322 ISSN2198-2589 (electronic)
ScientificComputation
ISBN978-3-319-41587-1 ISBN978-3-319-41589-5 (eBook)
DOI10.1007/978-3-319-41589-5
LibraryofCongressControlNumber:2016951957
MathematicsSubjectClassification(2010):49-02,65-06,90-06,68U10,47N10
©SpringerInternationalPublishingSwitzerland2016
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
Preface
Operator-splittingmethodshavebeenaroundformorethanacentury,startingwith
theircommonancestor,theLiescheme,introducedbySophusLieinthemid-1870s.
ItseemshoweverthatonehadtowaitafterWW2toseethesemethodsjoiningthe
computationaland applied mathematics mainstream, the driving force being their
applicabilitytothesolutionofchallengingproblemsfromscienceandengineering
modeled by partial differential equations. The main actors of this renewed inter-
est in operator-splitting methods were mainly Douglas, Peaceman, Rachford, and
Wachpress in the USA with the alternating direction implicit (ADI) methods and
Dyakonov,Marchuk,and Yanenko in the USSR with the fractionalstep methods.
Thesebasicmethodologieshaveknownmanyvariantsandimprovementsandgen-
eratedaquiteimportantliteratureconsistingofmanyarticlesandfewbooks,ofthe-
oreticaland applied natures, with computationalmechanicsand physicsbeing the
mainsourcesofapplications.Inthemid-1970s,tightrelationshipsbetweentheaug-
mentedLagrangianmethodsofHestenesandPowellandADImethodswereiden-
tified,leadingtothe alternatingdirectionmethodsof multipliers(ADMM).Albeit
originatingfromproblemsfromcontinuummechanicsmodeledbypartialdifferen-
tialequationsandinequalities,itwasquicklyrealizedthattheADMMmethodology
appliestoproblemsoutsidetherealmofpartialdifferentialequationsandinequali-
ties,ininformationsciencesinparticular,anareawhereADMMhasenjoyedavery
fast-growingpopularity.Themainreasonofthispopularityisthatmostoftenlarge-
scale optimization problems have decomposition properties that ADMM can take
advantageof,leadingtomodularalgorithms,wellsuitedtoparallelization.Another
factorexplainingADMM’sgrowingpopularityduringthelastdecadewasthedis-
covery around 2007 of its many commonalitieswith the split-Bregman algorithm
widelyusedfirstinimageprocessingandthenincompressedsensing,amongother
applications.
Late 2012, the three editors of this book were participating in a conference in
Hong Kong, the main conference topics being scientific computing, image pro-
cessing, and optimization. Since most lectures at the conference had some rela-
tions with operator splitting, ADMM, and split-Bregman algorithms, the idea of
a book dedicated to these topics was explored, considering the following facts:
v
vi Preface
(i) The practitioners of the above methods have become quite specialized, form-
ingsubcommunitieswithveryfewinteractionsbetweenthem.(ii)Newapplications
of operator-splitting and related algorithms appear on an almost daily basis. (iii)
Thediversificationofthealgorithmsandtheirapplicationshasbecomesolargethat
avolumecontainingthecontributionsofarelativelylargenumberofexpertsisnec-
essaryinordertointerestalargeaudience;indeed,thelastreviewpublicationson
the above topics being quite specialized (as shown in Chapter 1), the editors did
theirverybesttoproducealargespectrumvolume.
FollowingaSpringeragreementtopublishabookonoperatorsplitting,ADMM,
split-Bregman, and related algorithms, covering both theory and applications, ex-
pertswereapproachedtocontributetothisvolume.Wearepleasedtosaythatmost
ofthementhusiasticallyagreedtobepartoftheproject.
Thisbookisdividedinchapterscoveringthehistory,foundations,applications,
aswellasrecentdevelopmentsofoperatorsplitting,ADMM,splitBregman,andre-
latedalgorithms.Duetosizeandtimeconstraints,manyrelevantinformationcould
notbeincludedinthebook:theeditorsapologizetothoseauthorswhosecontribu-
tionshavebeenpartiallyortotallyoverlooked.
Manythanksareinorder:
• First, to the organizersof the December 2012 Hong Kong conferenceon Ad-
vances in Scientific Computing, Imaging Sciences and Optimization. Indeed,
theinceptionofthisprojecttookplaceduringthismeeting.
• To Springerfor acceptingto publishthis volume.The editorsacknowledgein
particulartheassistanceprovidedbyAchiDosanjh;shewasinvolvedwiththe
projectfromday oneandneverlosther faithin it(andin the editors),despite
themany(unavoidable)delaysencounteredduringitscompletion.
• Totheauthorsofthevariouschaptersandtothosecolleagueswhoacceptedto
reviewthem.Theyarereallytheoneswhobroughtthisbookintoexistence.
• ToHengdaWenandTsorng-WhayPanfortheirassistanceonmanyissuesas-
sociatedwiththepreparationofthe“manuscript”(someofthemLATEXrelated).
Bothofthemsavedthedaymorethanonce.
• Tothevariousinstitutionssupportingtheauthors,theeditors,andthereviewers.
• ToIndhumathiatSPiGlobalforherleadershipintransforminga complicated
manuscriptintoabook
Wewouldliketothankalsoallthescientistswhocontributedintheirownwayto
operator-splittingandrelatedmethods;theymadethisbookpossible.Amongthem,
wewouldliketogiveaspecialtributetoErnieEsserandMichèleSchatzman;their
untimelydeparturewas a shock to their friendsand colleagues.Both of them had
outstanding contributions to various topics addressed in this book, for which we
thankthemanddedicatethisbooktotheirmemory.
Houston,TX,USA RolandGlowinski
LosAngeles,CA,USA StanleyJ.Osher
LosAngeles,CA,USA WotaoYin
February2016
Contents
1 Introduction................................................ 1
RolandGlowinski,StanleyJ.Osher,andWotaoYin
1 MotivationandBackground.................................... 1
2 Lie’sSchemes............................................... 3
3 OntheStrangSymmetrizedOperator-SplittingScheme............ 5
4 OntheSolutionoftheSub-initialValueProblems................. 6
5 FurtherCommentsonMultiplicativeOperator-SplittingSchemes.... 7
6 OnADIMethods............................................ 7
7 OperatorSplittinginOptimization.............................. 10
8 BregmanMethodsandOperatorSplitting........................ 14
References..................................................... 15
2 SomeFactsAboutOperator-SplittingandAlternatingDirection
Methods.................................................... 19
RolandGlowinski,Tsorng-WhayPan,andXue-ChengTai
1 Introduction................................................. 19
2 Operator-SplittingSchemesfortheTimeDiscretizationofInitial
ValueProblems.............................................. 21
2.1 Generalities............................................ 21
2.2 Time-Discretizationof(2.1)byLie’sScheme................ 22
2.3 Time-Discretizationof(2.1)byStrang’sSymmetrizedScheme.. 23
2.4 Time-Discretization of (2.1) by Peaceman-Rachford’s
AlternatingDirectionMethod............................. 25
2.5 Time-Discretization of (2.1) by Douglas-Rachford’s
AlternatingDirectionMethod ............................. 26
2.6 Time-Discretizationof(2.1)byaFractionalθ-Scheme........ 28
2.7 Two Applications: Smallest Eigenvalue Computation
andSolutionofanAnisotropicEikonalEquation............. 30
2.8 Time-Discretizationof(2.1)byaParallelSplittingScheme..... 33
vii
viii Contents
3 AugmentedLagrangianAlgorithmsandAlternatingDirection
MethodsofMultipliers........................................ 34
3.1 Introduction............................................ 34
3.2 Decomposition-CoordinationMethodsbyAugmented
Lagrangians............................................ 35
3.3 OntheRelationshipBetweenAlternatingDirection
MethodsandALG2,ALG3............................... 40
4 Operator-SplittingMethodsfortheDirectNumericalSimulation
ofParticulateFlow........................................... 41
4.1 Generalities.ProblemFormulation......................... 41
4.2 AFictitiousDomainFormulation.......................... 43
4.3 SolvingProblem(2.79)–(2.84)byOperator-Splitting.......... 46
4.4 NumericalExperiments.................................. 46
5 Operator-SplittingMethodsfortheNumericalSolutionofNonlinear
ProblemsfromCondensateandPlasmaPhysics................... 52
5.1 Introduction............................................ 52
5.2 OntheSolutionoftheGross-PitaevskiiEquation............. 52
5.3 OntheSolutionofZakharovSystems....................... 56
6 ApplicationsofAugmentedLagrangianandADMMAlgorithms
totheSolutionofProblemsfromImaging........................ 61
6.1 VariationalModelsforImageProcessing.................... 61
6.2 FastNumericalAlgorithmsforVariationalImageProcessing
Models Based on Operator- Splitting and Augmented
LagrangianMethods(ALM).............................. 68
7 FurtherCommentsandComplements........................... 81
References..................................................... 86
3 OperatorSplitting........................................... 95
ShevMacNamaraandGilbertStrang
1 Introduction................................................. 96
2 SplittingforOrdinaryDifferentialEquations..................... 97
2.1 GaininganOrderofAccuracybyTakinganAverage..........100
2.2 HigherOrderMethods...................................100
2.3 ConvectionandDiffusion.................................101
2.4 AReaction-DiffusionPDE:SplittingLinearfromNonlinear....102
2.5 StabilityofSplittingMethods.............................103
2.6 OrdinarySplittingDoesNOTPreservetheSteadyState.......105
3 BalancedSplitting:ASymmetricStrangSplittingThatPreserves
theSteadyState..............................................105
3.1 BalancedSplittingPreservestheSteadyState................106
3.2 SplittingFastfromSlow..................................107
4 AVerySpecialToeplitz-Plus-HankelSplitting....................107
4.1 AllMatrixFunctions f(K)AreToeplitz-Plus-Hankel.........109
4.2 TheWaveEquationIsToeplitz-Plus-Hankel.................111
References.....................................................112
Contents ix
4 ConvergenceRateAnalysisofSeveralSplittingSchemes...........115
DamekDavisandWotaoYin
1 Introduction.................................................116
1.1 Notation...............................................118
1.2 Assumptions...........................................118
1.3 TheAlgorithms.........................................119
1.4 BasicPropertiesofAveragedOperators.....................120
2 SummableSequenceLemma...................................120
3 IterativeFixed-PointResidualAnalysis..........................122
3.1 o(1/(k+1))FPRofAveragedOperators....................122
3.2 o(1/(k+1))FPRofRelaxedPRS.........................125
3.3 O(1/Λ2)ErgodicFPRofFejérMonotoneSequences.........126
k
4 SubgradientsandFundamentalInequalities.......................126
4.1 ASubgradientRepresentationofRelaxedPRS...............127
4.2 OptimalityConditionsofRelaxedPRS......................128
4.3 FundamentalInequalities.................................129
5 ObjectiveConvergenceRates..................................130
5.1 ErgodicConvergenceRates...............................130
5.2 NonergodicConvergenceRates............................132
6 OptimalFPRRateandArbitrarilySlowConvergence..............134
6.1 OptimalFPRRates......................................134
6.2 ArbitrarilySlowConvergence.............................136
7 OptimalObjectiveRates......................................137
7.1 ErgodicConvergenceofMinimizationProblems.............137
7.2 OptimalNonergodicObjectiveRates.......................139
8 FromRelaxedPRStoRelaxedADMM..........................144
8.1 PrimalObjectiveConvergenceRatesinADMM..............145
9 Conclusion..................................................146
References.....................................................146
A FurtherApplicationsoftheResultsofSection3...................149
1.1 o(1/(k+1)2)FPRofFBSandPPA........................149
1.2 o(1/(k+1)2)FPRofOneDimensionalDRS................151
B FurtherLowerComplexityResults..............................152
2.1 ErgodicConvergenceofFeasibilityProblems................152
2.2 OptimalObjectiveandFPRRateswithLipschitzDerivative....152
C ADMMConvergenceRateProofs..............................153
3.1 DualFeasibilityConvergenceRates........................155
3.2 ConvertingDualInequalitiestoPrimalInequalities...........156
3.3 Converting Dual Convergence Rates to Primal
ConvergenceRates......................................158
D Examples...................................................159
4.1 FeasibilityProblems.....................................159
4.2 ParallelizedModelFittingandClassification.................160
4.3 DistributedADMM......................................162