Statistical Image Processing and Multidimensional Modeling

478 Pages·2011·17.07 MB·English

by  Paul Fieguth (auth.)

EESEtatistical Image Processing
anE d Multidimensional Modeling
Information Science and Statistics
SSSeries Editors:
M. Jordan
Robert Nowak
Bernhard Schölkopf
Forothertitlespublishedinthisseries,goto
www.springer.com/series/3816
Paul Fieguth
Statistical Image Processing
and Multidimensional Modeling
Paul Fieguth
Department of Systems Design Engineering
Faculty of Engineering
University of Waterloo
Waterloo Ontario N2L-3G1
Canada
[email protected]

ISSN1613-9011
ISBN978-1-4419-7293-4 e-ISBN978-1-4419-7294-1
DOI10.1007/978-1-4419-7294-1
Springer New York Dordrecht Heidelberg London
Library of Congress Control Number: 2010938436
©SpringerScience + BusinessMedia,LLC2011
Allrightsreserved.Thisworkmaynotbetranslatedorcopiedinwholeorinpartwithoutthewritten
permissionofthepublisher(SpringerScience+BusinessMedia,LLC,233SpringStreet,NewYork,NY
10013,USA),exceptforbriefexcerptsinconnectionwithreviewsorscholarlyanalysis.Useinconnection
withanyformofinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdevelopedisforbidden.
Theuseinthispublicationoftradenames,trademarks,servicemarks,andsimilarterms,eveniftheyare
notidentifiedassuch,isnottobetakenasanexpressionofopinionastowhetherornottheyaresubject
toproprietaryrights.
Printedonacid-freepaper
Springer is part of Springer Science+Business Media (www.springer.com)
Preface
Asayoungprofessorin1997ItaughtmygraduatecourseinStochasticImagePro-
cessingforthefirsttime.Lookingbackonmyroughnotesfromthattime,thecourse
musthavebeenanearimpenetrabledisasterforthegraduatestudentsenrolled,with
alonglistoferrors,confusions,andbadnotation.
Witheveryrepetitionthecourseimproved,withsignificantchangestonotation,con-
tent, and flow. However, at the same time that a cohesive, large-scale form of the
course took shape, the absence of any textbook covering this material became in-
creasinglyapparent.Thereare countlesstextson the subjectsof image processing,
Kalmanfiltering,andsignalprocessing,howeverpreciouslittleforrandomfieldsor
spatialstatistics.ThefewtextsthatdocoverGibbsmodelsorMarkovrandomfields
tendto behighlymathematicalresearchmonographs,notwellsuitedasatextbook
foragraduatecourse.
Morethanjustagraduatecoursetextbook,thistextwasdevelopedwiththegoalof
being a useful reference for graduate students working in the areas of image pro-
cessing, spatialstatistics, and randomfields. In particular,thereare manyconcepts
whichareknownanddocumentedintheresearchliterature,whichareusefulforstu-
dentstounderstand,butwhichdonotappearinmanytextbooks.Thisperceptionis
drivenbymyownexperienceasaPhDstudent,whichwouldhavebeenconsiderably
simplifiedifIhadhadatextaccessibletomeaddressingsomeofthefollowinggaps:
• FFT-basedestimation(Section8.3)
• Anice,simple,cleardescriptionofmultigrid(Section9.2.5)
• Theinferenceofdynamicmodelsfromcross-statistics(Chapter10)
• A clear distinction and relationship between squared and unsquared kernels
(Chapter5)
V
VI Preface
• AgraphicalsummaryrelatingGibbsandMarkovmodels(Figure6.11)
Tofacilitatetheuseofthistextbookandthemethodsdescribedwithinit,Iammaking
availableonline(seepageXV)muchofthecodewhichIdevelopedforthistext.This
code,somecolourfigures,and(hopefullyfew)erratacanbefoundfromthisbook’s
homepage:
http://ocho.uwaterloo.ca/book
This text has benefited from the work, support, and ideas of a great many people.
I owe a debt of gratitude to the countless researchers upon whose work this book
isbuilt, andwhoare listed inthe bibliography.Please acceptmyapologiesforany
omissions.
The contents of this book are closely aligned with my research interests over the
pasttenyears.Consequentlytheworkofanumberofmyformergraduatestudents
appearsinsomeforminthisbook,andIwouldliketorecognizethecontributionsof
SimonAlexander,WesleyCampaigne,GabrielCarballo,MichaleJamieson,FuJin,
FakhryKhellah,YingLiu,andAzadehMohebi.
I would like to thank my Springer editor, John Kimmel, who was an enthusiastic
supporterof this text, and highly tolerantof my slow pace in writing. Thanksalso
to copyeditor Valerie Grecofor her carefulexaminationof grammarand punctua-
tion(andwhereanyremainingerrorsaremine,nothers).I wouldlike tothankthe
anonymousreviewers,whoreadthetextthoroughlyandwhoprovidedexceptionally
helpfulconstructivecriticism.Iwouldalsoliketothankthenon-anonymousreview-
ers,friendsandstudentswhogavethetextanotherlook:WernerFieguth,BettyPries,
AkshayaMishra,AlexanderWong,LiLiu,andGeraldMwangi.
I would like to thank Christoph Garbe and Michael Winckler, the two people who
coordinatedmystayattheUniversityofHeidelberg,wherethistextwascompleted.
MythankstotheDeutscherAkademischerAustauschDienst,theHeidelbergGradu-
ateSchool,andtotheHeidelbergCollaboratoryforImageProcessingforsupporting
myvisit.
ManythanksandappreciationtoBettyforencouragingthisproject,andtothekids
inAppendixCforjustbeingwhotheyare.
Waterloo,Ontario PaulFieguth
July,2010
Contents
Preface V
TableofContents VII
ListofExamples XIII
ListofCodeSamples XV
Nomenclature XVII
1 Introduction 1
PartI Inverse Problems andEstimation 11
2 InverseProblems 13
2.1 DataFusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 Posedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 Conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4 RegularizationandPriorModels . . . . . . . . . . . . . . . . . . . 29
2.4.1 DeterministicRegularization . . . . . . . . . . . . . . . . . 34
2.4.2 BayesianRegularization . . . . . . . . . . . . . . . . . . . 37
2.5 StatisticalOperations . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.5.1 CanonicalProblems . . . . . . . . . . . . . . . . . . . . . 40
2.5.2 PriorSampling . . . . . . . . . . . . . . . . . . . . . . . . 42
2.5.3 Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.5.4 PosteriorSampling . . . . . . . . . . . . . . . . . . . . . . 49
2.5.5 ParameterEstimation . . . . . . . . . . . . . . . . . . . . . 50
Application2:OceanAcousticTomography . . . . . . . . . . . . . . . . 50
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
ForFurtherStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3 StaticEstimationandSampling 57
3.1 Non-BayesianEstimation . . . . . . . . . . . . . . . . . . . . . . . 58
VII
VIII CONTENTS
3.2 BayesianEstimation . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.1 BayesianStaticProblem . . . . . . . . . . . . . . . . . . . 67
3.2.2 BayesianEstimationandPriorMeans . . . . . . . . . . . . 68
3.2.3 ApproximateBayesianEstimators . . . . . . . . . . . . . . 70
3.2.4 Bayesian/NonBayesianDuality . . . . . . . . . . . . . . . 73
3.3 StaticSampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.4 DataFusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
Application3:AtmosphericTemperatureInversion[282] . . . . . . . . . 78
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
ForFurtherStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4 DynamicEstimationandSampling 85
4.1 TheDynamicProblem . . . . . . . . . . . . . . . . . . . . . . . . 86
4.1.1 First-OrderGauss–MarkovProcesses . . . . . . . . . . . . 88
4.1.2 Static—DynamicDuality . . . . . . . . . . . . . . . . . . 89
4.2 KalmanFilterDerivation . . . . . . . . . . . . . . . . . . . . . . . 93
4.3 KalmanFilterVariations . . . . . . . . . . . . . . . . . . . . . . . 100
4.3.1 KalmanFilterAlgorithms . . . . . . . . . . . . . . . . . . 102
4.3.2 Steady-StateKalmanFiltering . . . . . . . . . . . . . . . . 107
4.3.3 KalmanFilterSmoother . . . . . . . . . . . . . . . . . . . 109
4.3.4 NonlinearKalmanFiltering . . . . . . . . . . . . . . . . . 114
4.4 DynamicSampling . . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.5 DynamicEstimationforDiscrete-StateSystems . . . . . . . . . . . 119
4.5.1 MarkovChains . . . . . . . . . . . . . . . . . . . . . . . . 119
4.5.2 TheViterbiAlgorithm . . . . . . . . . . . . . . . . . . . . 120
4.5.3 ComparisontoKalmanFilter . . . . . . . . . . . . . . . . 122
Application4:TemporalInterpolationofOceanTemperature[191] . . . . 125
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
ForFurtherStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
PartII Modelling ofRandom Fields 131
5 MultidimensionalModelling 133
5.1 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.2 CouplingandDimensionalityReduction . . . . . . . . . . . . . . . 135
5.3 SparseStorageandComputation . . . . . . . . . . . . . . . . . . . 139
5.3.1 SparseMatrices . . . . . . . . . . . . . . . . . . . . . . . . 139
5.3.2 MatrixKernels . . . . . . . . . . . . . . . . . . . . . . . . 141
5.3.3 Computation . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.4 Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
5.5 DeterministicModels . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.5.1 BoundaryEffects . . . . . . . . . . . . . . . . . . . . . . . 153
CONTENTS IX
5.5.2 DiscontinuityFeatures . . . . . . . . . . . . . . . . . . . . 155
5.5.3 Prior-MeanConstraints . . . . . . . . . . . . . . . . . . . . 156
5.6 StatisticalModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
5.6.1 AnalyticalForms . . . . . . . . . . . . . . . . . . . . . . . 160
5.6.2 AnalyticalFormsandNonstationaryFields . . . . . . . . . 164
5.6.3 Recursive/DynamicModels . . . . . . . . . . . . . . . . . 166
5.6.4 BandedInverse-Covariances . . . . . . . . . . . . . . . . . 167
5.7 ModelDetermination . . . . . . . . . . . . . . . . . . . . . . . . . 169
5.8 ChoiceofRepresentation . . . . . . . . . . . . . . . . . . . . . . . 172
Application5:SyntheticApertureRadarInterferometry[53] . . . . . . . 173
ForFurtherStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6 MarkovRandomFields 179
6.1 One-DimensionalMarkovianity . . . . . . . . . . . . . . . . . . . 180
6.1.1 MarkovChains . . . . . . . . . . . . . . . . . . . . . . . . 181
6.1.2 Gauss–MarkovProcesses . . . . . . . . . . . . . . . . . . . 181
6.2 MultidimensionalMarkovianity . . . . . . . . . . . . . . . . . . . 182
6.3 Gauss–MarkovRandomFields . . . . . . . . . . . . . . . . . . . . 185
6.4 CausalGauss–MarkovRandomFields . . . . . . . . . . . . . . . . 189
6.5 GibbsRandomFields . . . . . . . . . . . . . . . . . . . . . . . . . 192
6.6 ModelDetermination . . . . . . . . . . . . . . . . . . . . . . . . . 199
6.6.1 AutoregressiveModelLearning . . . . . . . . . . . . . . . 199
6.6.2 NoncausalMarkovModelLearning . . . . . . . . . . . . . 201
6.7 ChoicesofRepresentation . . . . . . . . . . . . . . . . . . . . . . 207
Application6:TextureClassification . . . . . . . . . . . . . . . . . . . . 208
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
ForFurtherStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
SampleProblems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
7 HiddenMarkovModels 215
7.1 HiddenMarkovModels . . . . . . . . . . . . . . . . . . . . . . . . 216
7.1.1 ImageDenoising . . . . . . . . . . . . . . . . . . . . . . . 216
7.1.2 ImageSegmentation . . . . . . . . . . . . . . . . . . . . . 219
7.1.3 TextureSegmentation . . . . . . . . . . . . . . . . . . . . 220
7.1.4 EdgeDetection . . . . . . . . . . . . . . . . . . . . . . . . 221
7.2 ClassesofJointMarkovModels . . . . . . . . . . . . . . . . . . . 222
7.3 ConditionalRandomFields . . . . . . . . . . . . . . . . . . . . . . 225
7.4 Discrete-StateModels . . . . . . . . . . . . . . . . . . . . . . . . . 227
7.4.1 LocalGibbsModels . . . . . . . . . . . . . . . . . . . . . 228
7.4.2 NonlocalStatistical-TargetModels . . . . . . . . . . . . . . 229
7.4.3 LocalJointModels . . . . . . . . . . . . . . . . . . . . . . 230
7.5 ModelDetermination . . . . . . . . . . . . . . . . . . . . . . . . . 231
Application7:ImageSegmentation . . . . . . . . . . . . . . . . . . . . . 233
ForFurtherStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237