Image Co-segmentation

231 Pages·2023·9.76 MB·English

by  Avik Hati, Rajbabu Velmurugan, Sayan Banerjee, Subhasis Chaudhuri

Studies in Computational Intelligence 1082
Avik Hati
Rajbabu Velmurugan
Sayan Banerjee
Subhasis Chaudhuri
Image
Co-segmentation
Studies in Computational Intelligence
Volume 1082
SeriesEditor
JanuszKacprzyk,PolishAcademyofSciences,Warsaw,Poland
The series “Studies in Computational Intelligence” (SCI) publishes new develop-
mentsandadvancesinthevariousareasofcomputationalintelligence—quicklyand
withahighquality.Theintentistocoverthetheory,applications,anddesignmethods
of computational intelligence, as embedded in the fields of engineering, computer
science, physics and life sciences, as well as the methodologies behind them. The
series contains monographs, lecture notes and edited volumes in computational
intelligence spanning the areas of neural networks, connectionist systems, genetic
algorithms,evolutionarycomputation,artificialintelligence,cellularautomata,self-
organizingsystems,softcomputing,fuzzysystems,andhybridintelligentsystems.
Ofparticularvaluetoboththecontributorsandthereadershiparetheshortpublica-
tiontimeframeandtheworld-widedistribution,whichenablebothwideandrapid
disseminationofresearchoutput.
IndexedbySCOPUS,DBLP,WTIFrankfurteG,zbMATH,SCImago.
AllbookspublishedintheseriesaresubmittedforconsiderationinWebofScience.
· · ·
Avik Hati Rajbabu Velmurugan Sayan Banerjee
Subhasis Chaudhuri
Image Co-segmentation
AvikHati RajbabuVelmurugan
DepartmentofElectronics DepartmentofElectricalEngineering
andCommunicationEngineering IndianInstituteofTechnologyBombay
NationalInstituteofTechnology Mumbai,Maharashtra,India
Tiruchirappalli
Tiruchirappalli,Tamilnadu,India SubhasisChaudhuri
DepartmentofElectricalEngineering
SayanBanerjee IndianInstituteofTechnologyBombay
DepartmentofElectricalEngineering Mumbai,Maharashtra,India
IndianInstituteofTechnologyBombay
Mumbai,Maharashtra,India
ISSN 1860-949X ISSN 1860-9503 (electronic)
StudiesinComputationalIntelligence
ISBN 978-981-19-8569-0 ISBN 978-981-19-8570-6 (eBook)
https://doi.org/10.1007/978-981-19-8570-6
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SingaporePteLtd.2023
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Image segmentation is a classical and well-known problem in image processing,
where an image is partitioned into non-overlapping regions. Such regions may be
objects or meaningful parts of a scene. It is usually a challenging task to perform
imagesegmentationandautomaticallyextractobjectswithouthigh-levelknowledge
oftheobjectcategory.Instead,ifwehavetwoormoreimagescontainingacommon
objectofinterest,jointlytryingtosegmenttheimagestoobtainthecommonobject
willhelpinautomatingthesegmentationprocess.Thisisreferredtoastheproblem
ofimageco-segmentation.Thismonographexploresseveralapproachestoperform
robustco-segmentationofimages.
Theproblemofco-segmentationisnotaswellresearchedassegmentation.For
us,themotivationforunderstandingimageco-segmentationarosefromconsidering
the problem of identifying videos with similar content and also retrieving images
by searching for a similar image, even before deep learning became popular. We
realizedthatearlierapproacheshadconsideredsaliencyofanobjectinanimageas
one of the cues for co-segmentation. However, realizing various restrictive issues
with this approach, we started exploring other methods that can perform robust
co-segmentation.Webelievethatagoodrepresentationfortheforegroundandback-
groundinanimageisessential,andhenceuseagraphrepresentationfortheimages,
whichhelpedinbothunsupervisedandsupervisedapproaches.Thiswaywecoulduse
andextendgraphmatchingalgorithmsthatcanbemademorerobust.Thiscouldalso
bedoneinthedeepneuralnetworkframework,extendingthestrengthofthemodelto
supervisedapproaches.Giventhatgraph-basedapproachesforco-segmentationhave
notsufficientlybeenexploredinliterature,wedecidedtobringoutthismonograph
onco-segmentation.
In this monograph, we present several methods for co-segmentation that were
developed over a period of seven years. Most of these methods use the power of
superpixelstorepresentimagesandgraphstorepresentconnectednessamongthem.
Such representations could exploit efficient graph matching algorithms that could
leadtoco-segmentation.However,therewereseveralchallengesindevelopingsuch
algorithmswhicharebroughtoutinthechaptersofthismonograph.Thechallenges
bothinformulatingandimplementingsuchalgorithmsareillustratedwithanalytical
v
vi Preface
andexperimentalresults.Intheunsupervisedframework,oneoftheanalyticalchal-
lengesrelatestothestatisticalmodedetectioninamultidimensionalfeaturespace.
While a solution is discussed in the monograph, this is one of the problems still
consideredtobeachallengeinmachinelearningalgorithms.
Afterpresentingunsupervisedapproaches,wepresentsupervisedapproachesto
solve the problem of co-segmentation. These methods lead to better performance
withsufficientlylabeledlargedatasetsofimages.However,withfewerimages,these
methodscouldnotdowell.Hence,inthemonograph,wepresentsomerecenttech-
niques such as few-shot learning to solve the problem of having access to fewer
samples during training for the co-segmentation problem. In most of the methods
presented, the problem of co-segmenting a single object across multiple images is
presented.However,theproblemofco-segmentingmultipleobjectsacrossmultiple
images is still a challenging problem. We believe the approaches presented in this
monographwillhelpresearcherstoaddresssuchco-segmentationproblemsandin
alessconstrainedsetting.
Mostofthemethodspresentedaregoodreferencestopracticingresearchers.In
addition,theprimarytargetgroupforthismonographisgraduatestudentsinelectrical
engineering,computerscience,ormathematicswhohaveinterestinimageprocessing
andmachinelearning.Sinceco-segmentationisanaturalextensionofsegmentation,
themonographbrieflydescribestopicsthatwouldberequiredforasmoothtransition
from segmentation problems. The later chapters in the monograph will be useful
for students in the area of machine learning, including a data-deprived method.
Overall,thechapterscanhelppractitionerstoconsidertheuseofco-segmentation
indevelopingefficientimageorvideoretrievalalgorithms.
We strongly believe that the monograph will be useful for several readers and
welcomeanysuggestionsorcomments.
Mumbai,India AvikHati
July2022 RajbabuVelmurugan
SayanBanerjee
SubhasisChaudhuri
Acknowledgements TheauthorswouldliketoacknowledgepartialsupportprovidedbyNational
CentreforExcellenceinInternalSecurity(NCETIS),IITBombay.Fundingsupportintheformof
JCGhoshFellowshiptothelastauthorisalsogratefullyacknowledged.Wealsoacknowledgethe
contributionsofDr.FerozAliandDivakarBhatindevelopingsomeofthemethodsdiscussedin
thismonograph.Theauthorsthankthepublisherforaccommodatingourrequestsandsupporting
thedevelopmentofthismonograph.Wealsothankourfamiliesfortheirsupportthroughoutthis
endeavor.
Contents
1 Introduction .................................................. 1
1.1 ImageCo-segmentation ................................... 2
1.2 ImageSaliencyandCo-saliency ............................ 7
1.3 BasicComponentsofCo-segmentation ...................... 10
1.3.1 TheProblem ...................................... 12
1.4 OrganizationoftheMonograph ............................ 14
1.4.1 Co-segmentationofanImagePair .................... 14
1.4.2 RobustCo-segmentationofMultipleImages ........... 16
1.4.3 Co-segmentationbySuperpixelClassification .......... 16
1.4.4 Co-segmentationbyGraphConvolutionalNeural
Network .......................................... 18
1.4.5 ConditionalSiameseConvolutionalNetwork ........... 18
1.4.6 Co-segmentationinFew-ShotSetting ................. 18
2 SurveyofImageCo-segmentation .............................. 21
2.1 UnsupervisedCo-segmentation ............................. 21
2.1.1 MarkovRandomFieldModel-BasedMethods ......... 21
2.1.2 Saliency-BasedMethods ............................ 22
2.1.3 OtherCo-segmentationMethods ..................... 23
2.2 SupervisedCo-segmentation ............................... 25
2.2.1 Semi-supervisedMethods ........................... 25
2.2.2 DeepLearning-BasedMethods ...................... 26
2.3 Co-segmentationDatasets ................................. 27
3 MathematicalBackground ..................................... 29
3.1 SuperpixelSegmentation .................................. 29
3.2 LabelPropagation ........................................ 32
3.2.1 Two-classLabelPropagation ........................ 33
3.2.2 MulticlassLabelPropagation ........................ 34
3.3 SubgraphMatching ....................................... 36
3.4 ConvolutionalNeuralNetwork ............................. 44
3.4.1 NonlinearActivationFunctions ...................... 45
vii
viii Contents
3.4.2 PoolinginCNN ................................... 47
3.4.3 RegularizationMethods ............................. 48
3.4.4 LossFunctions .................................... 50
3.4.5 OptimizationMethods .............................. 51
3.5 GraphConvolutionalNeuralNetwork ....................... 53
3.6 VariationalInference ...................................... 55
3.7 Few-shotLearning ....................................... 57
4 MaximumCommonSubgraphMatching ........................ 59
4.1 Introduction ............................................. 59
4.1.1 ProblemFormulation ............................... 59
4.2 Co-segmentationforTwoImages ........................... 60
4.2.1 ImageasAttributedRegionAdjacencyGraph .......... 60
4.2.2 MaximumCommonSubgraphComputation ........... 62
4.2.3 RegionCo-growing ................................ 65
4.2.4 CommonBackgroundElimination ................... 71
4.3 MultiscaleImageCo-segmentation ......................... 72
4.4 ExperimentalResults ..................................... 73
4.5 ExtensiontoCo-segmentationofMultipleImages ............ 81
5 MaximallyOccurringCommonSubgraphMatching ............. 89
5.1 Introduction ............................................. 89
5.2 ProblemFormulation ..................................... 90
5.2.1 MathematicalDefinition ............................ 90
5.2.2 Multi-imageCo-segmentationProblem ............... 91
5.2.3 OverviewoftheMethod ............................ 92
5.3 SuperpixelClustering ..................................... 93
5.3.1 FeatureComputation ............................... 94
5.3.2 Coarse-levelCo-segmentation ....................... 94
5.3.3 HoleFilling ....................................... 98
5.4 CommonObjectDetection ................................ 99
5.4.1 LatentClassGraph ................................. 100
5.4.2 RegionGrowing ................................... 103
5.5 ExperimentalResults ..................................... 110
5.5.1 QuantitativeandQualitativeAnalysis ................. 110
5.5.2 MultipleClassCo-segmentation ..................... 115
5.5.3 ComputationTime ................................. 121
6 Co-segmentationUsingaClassificationFramework .............. 123
6.1 Introduction ............................................. 123
6.1.1 ProblemDefinition ................................. 123
6.2 Co-segmentationAlgorithm ............................... 127
6.2.1 Mode Estimation in a Multidimensional
Distribution ....................................... 127
6.2.2 DiscriminativeSpaceforCo-segmentation ............. 130
6.2.3 SpatiallyConstrainedLabelPropagation .............. 136
Contents ix
6.3 ExperimentalResults ..................................... 142
6.3.1 QuantitativeandQualitativeAnalyses ................. 142
6.3.2 AblationStudy .................................... 143
6.3.3 AnalysisofDiscriminativeSpace .................... 146
6.3.4 ComputationTime ................................. 148
7 Co-segmentationUsingGraphConvolutionalNetwork ........... 151
7.1 Introduction ............................................. 151
7.2 Co-segmentationFramework ............................... 152
7.2.1 GlobalGraphComputation .......................... 152
7.3 GraphConvolution-BasedFeatureComputation .............. 154
7.3.1 GraphConvolutionFilters ........................... 156
7.3.2 AnalysisofFilterOutputs ........................... 157
7.4 NetworkArchitecture ..................................... 158
7.4.1 NetworkTrainingandTestingStrategy ................ 160
7.5 ExperimentalResults ..................................... 162
7.5.1 InternetDataset .................................... 162
7.5.2 PASCAL-VOCDataset ............................. 163
8 ConditionalSiameseConvolutionalNetwork ..................... 167
8.1 Introduction ............................................. 167
8.2 Co-segmentationFramework ............................... 171
8.2.1 ConditionalSiameseEncoder-DecoderNetwork ........ 171
8.2.2 SiameseMetricLearningNetwork ................... 173
8.2.3 DecisionNetwork .................................. 174
8.2.4 LossFunction ..................................... 174
8.2.5 TrainingStrategy .................................. 175
8.3 ExperimentalResults ..................................... 176
8.3.1 PASCAL-VOCDataset ............................. 176
8.3.2 InternetDataset .................................... 178
8.3.3 MSRCDataset .................................... 178
8.3.4 AblationStudy .................................... 181
9 Few-shotLearningforCo-segmentation ......................... 185
9.1 Introduction ............................................. 185
9.2 Co-segmentationFramework ............................... 186
9.2.1 ClassAgnosticMeta-Learning ....................... 187
9.2.2 DirectedVariationalInferenceCross-Encoder .......... 192
9.3 NetworkArchitecture ..................................... 193
9.3.1 Encoder-Decoder .................................. 193
9.3.2 ChannelAttentionModule(ChAM) .................. 194
9.3.3 SpatialAttentionModule(SpAM) .................... 194
9.4 ExperimentalResults ..................................... 194
9.4.1 PMFImplementationDetails ........................ 195
9.4.2 PerformanceAnalysis .............................. 195
9.4.3 AblationStudy .................................... 199