Computer Vision Analysis of Image Motion by Variational Methods

212 Pages·2014·3.651 MB·English

by  Amar Mitiche, J.K. Aggarwal (auth.)

Springer Topics in Signal Processing
Amar Mitiche
J.K. Aggarwal
Computer Vision
Analysis of
Image Motion
by Variational
Methods
Springer Topics in Signal Processing
Volume 10
For furthervolumes:
http://www.springer.com/series/8109
Amar Mitiche J. K. Aggarwal
•
Computer Vision Analysis
of Image Motion by
Variational Methods
123
AmarMitiche J.K.Aggarwal
INRS-Energie, Matériaux et Department of Electricaland Computer
Télécommunications Engineering
InstitutNational de laRecherche The Universityof Texas
Scientifique Austin, TX
Montreal, QC USA
Canada
ISSN 1866-2609 ISSN 1866-2617 (electronic)
ISBN 978-3-319-00710-6 ISBN 978-3-319-00711-3 (eBook)
DOI 10.1007/978-3-319-00711-3
SpringerChamHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2013939338
(cid:2)SpringerInternationalPublishingSwitzerland2014
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodology now known or hereafter developed. Exempted from this legal reservation are brief
excerpts in connection with reviews or scholarly analysis or material supplied specifically for the
purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe
work. Duplication of this publication or parts thereof is permitted only under the provisions of
theCopyright Law of the Publisher’s location, in its current version, and permission for use must
always be obtained from Springer. Permissions for use may be obtained through RightsLink at the
CopyrightClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Contents
1 Image Motion Processing in Visual Function. . . . . . . . . . . . . . . . . 1
1.1 Image Motion in Visual Function. . . . . . . . . . . . . . . . . . . . . . 1
1.2 Computer Vision Applications. . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Variational Processing of Image Motion. . . . . . . . . . . . . . . . . 6
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2 Background Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1 Curvature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.1 Curvature of a Parametric Curve . . . . . . . . . . . . . . . . 13
2.1.2 Curvature of an Implicit Curve . . . . . . . . . . . . . . . . . 15
2.2 Euler-Lagrange Equations. . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.1 Definite Integrals. . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.2 Variable Domain of Integration. . . . . . . . . . . . . . . . . 22
2.3 Differentiation Under the Integral Sign. . . . . . . . . . . . . . . . . . 31
2.4 Descent Methods for Unconstrained Optimization . . . . . . . . . . 31
2.4.1 Real Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4.2 Integral Functionals . . . . . . . . . . . . . . . . . . . . . . . . . 33
2.5 Level Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3 Optical Flow Estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2 The Optical Flow Constraint. . . . . . . . . . . . . . . . . . . . . . . . . 45
3.3 The Lucas-Kanade Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4 The Horn and Schunck Algorithm . . . . . . . . . . . . . . . . . . . . . 48
3.4.1 Discretization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.4.2 Gauss-Seidel and Jacobi Iterations. . . . . . . . . . . . . . . 50
3.4.3 Evaluation of Derivatives . . . . . . . . . . . . . . . . . . . . . 51
3.4.4 Ad hoc Variations to Preserve Motion Boundaries. . . . 52
3.5 Deriche–Aubert–Kornprobst Method . . . . . . . . . . . . . . . . . . . 54
v
vi Contents
3.6 Image-Guided Regularization . . . . . . . . . . . . . . . . . . . . . . . . 59
3.6.1 The Oriented-Smoothness Constraint . . . . . . . . . . . . . 61
3.6.2 Selective Image Diffusion. . . . . . . . . . . . . . . . . . . . . 61
3.7 Minimum Description Length . . . . . . . . . . . . . . . . . . . . . . . . 63
3.8 Parametric Estimation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.9 Variations on the Data and Smoothness Terms . . . . . . . . . . . . 70
3.10 Multiresolution and Multigrid Processing . . . . . . . . . . . . . . . . 70
3.10.1 Multiresolution Processing . . . . . . . . . . . . . . . . . . . . 71
3.10.2 Multigrid Computation. . . . . . . . . . . . . . . . . . . . . . . 72
3.11 Joint Estimation and Segmentation. . . . . . . . . . . . . . . . . . . . . 75
3.12 Joint Optical Flow and Disparity Estimation. . . . . . . . . . . . . . 83
3.13 State-of-the-Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
4 Motion Detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.2 Background Modelling and Point-Wise Background
Subtraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.2.1 Parametric Modelling: The Stauffer–Grimson
Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.2.2 Nonparametric Modelling . . . . . . . . . . . . . . . . . . . . . 102
4.2.3 Image Spatial Regularization. . . . . . . . . . . . . . . . . . . 102
4.3 Variational Background Subtraction. . . . . . . . . . . . . . . . . . . . 104
4.3.1 Probability Models. . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.3.2 Template Models. . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.4 Detection by Image Differencing. . . . . . . . . . . . . . . . . . . . . . 113
4.4.1 Region-Based Image Differencing Detection. . . . . . . . 114
4.4.2 MAP Image Differencing Detection. . . . . . . . . . . . . . 116
4.4.3 Boundary-Based Image Differencing Detection . . . . . . 117
4.5 Optical Flow Based Motion Detection . . . . . . . . . . . . . . . . . . 121
4.6 Motion Detection with a Moving Viewing System. . . . . . . . . . 125
4.6.1 Region Based Detection Normal Component
Residuals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.6.2 Detection by Optical Flow Residuals . . . . . . . . . . . . . 128
4.6.3 Detection by a Geodesic. . . . . . . . . . . . . . . . . . . . . . 130
4.6.4 A Contrario Detection by Displaced Frame
Differences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
4.7 Selective Detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.7.1 Connected Component Analysis. . . . . . . . . . . . . . . . . 133
4.7.2 Variational Integration of Object Features
in Motion Detection. . . . . . . . . . . . . . . . . . . . . . . . . 134
4.8 Motion Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Contents vii
5 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.2 Kernel-Based Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
5.2.1 Mean-Shift Density Mode Estimation. . . . . . . . . . . . . 148
5.2.2 Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
5.3 Active Curve Density Tracking . . . . . . . . . . . . . . . . . . . . . . . 152
5.3.1 The Bhattacharyya Flow. . . . . . . . . . . . . . . . . . . . . . 153
5.3.2 The Kullback-Leibler Flow. . . . . . . . . . . . . . . . . . . . 155
5.3.3 Level Set Equations. . . . . . . . . . . . . . . . . . . . . . . . . 156
5.4 Tracking by Region-Based Matching . . . . . . . . . . . . . . . . . . . 157
5.4.1 Basic Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . 157
5.4.2 Matching Local Statistics . . . . . . . . . . . . . . . . . . . . . 160
5.4.3 Using Shape and Motion. . . . . . . . . . . . . . . . . . . . . . 161
5.5 Tracking in the Spatiotemporal Domain . . . . . . . . . . . . . . . . . 165
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6 Optical Flow Three-Dimensional Interpretation . . . . . . . . . . . . . . 175
6.1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.2 Projection Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6.3 Ego-Motion in a Static Environment . . . . . . . . . . . . . . . . . . . 181
6.3.1 Indirect Interpretation. . . . . . . . . . . . . . . . . . . . . . . . 182
6.3.2 Direct Interpretation. . . . . . . . . . . . . . . . . . . . . . . . . 183
6.4 Non-Stationary Environment. . . . . . . . . . . . . . . . . . . . . . . . . 184
6.4.1 Point-Wise Interpretation . . . . . . . . . . . . . . . . . . . . . 185
6.4.2 Region-Based Interpretation . . . . . . . . . . . . . . . . . . . 188
6.5 Scene Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Chapter 1
Image Motion Processing in Visual Function
Retinal motion comes about whenever we move or look at moving objects. Small
involuntaryretinalmovementstakeplaceevenwhenwefixateonastationarytarget.
Processingofthisever-presentimagemotionplaysseveralfundamentalfunctional
rolesinhumanvision.Inmachinevisionaswell,imagemotionprocessingbycom-
putervisionalgorithmshasinmanyusefulapplicationsseveralessentialfunctions
reminiscentoftheprocessingbythehumanvisualsystem.Asthefollowingdiscus-
sionsetstopointout,computervisionmodellingofmotionhasaddressedproblems
similartosomethathaveariseninhumanvisionresearch,includingthoseconcerning
theearliestfundamentalquestionsandexplanationsputforthbyHelmholtzandby
Gibsonabouthumanmotionperception.However,computervisionmotionmodels
have evolved independently ofhuman perception concerns andspecificities,much
like the camera has evolved independently of the understanding of the human eye
biologyandfunction[1].
1.1 ImageMotion inVisualFunction
The most obvious role of image motion processing by the human visual system
is to perceive the motion of real objects. The scope and quality of this perception
varieswidelyaccordingtothevisualtaskperformed,rangingfromdetectionwhere
moving versus static labelling of objects in the visual field is sufficient, to event
interpretation where a characterization of motion by more detailed evaluation or
attributesisrequired.
Less evident a role is the perception of depth. Computational and experimental
investigationshaverevealedthelinkbetweentheimagemotionandthevariablesof
depthandthree-dimensional(3D)motion.Toemphasizethisroleofimagemotion,
Nakayama and Loomis [2] named kineopsis, by analogy to stereopsis, the process
ofrecoveringdepthand3Dmotionfromimagemotion.
A.MiticheandJ.K.Aggarwal,ComputerVisionAnalysisofImageMotionbyVariational 1
Methods,SpringerTopicsinSignalProcessing10,DOI:10.1007/978-3-319-00711-3_1,
©SpringerInternationalPublishingSwitzerland2014
2 1 ImageMotionProcessinginVisualFunction
Kineopsis:Theroleofmotionintheperceptionofdepth,andstructurethereof,has
beenknownforalongtime.InthewordsofHelmholtzforinstance([3],pp.297),
overahundredyearsagoinhisHandbookofPhysiologicalOptics,1910:
“Ifanybodywithtwogoodeyeswillcloseoneofthemandlookatunfamiliarobjectsof
irregularform,hewillbeapttogetawrong,oratanyrateanunreliable,ideaoftheirshape.
Buttheinstanthemovesabout,hewillbegintohavethecorrectapperceptions.”
Headdsthefollowingexplanationastotheoriginofthisperceptionofenviron-
mentalstructure,orapperceptionashecalledit:
“Inthevariationsoftheretinalimage,whicharetheresultsofmovements,theonlywayan
apperceptionofdifferencesofdistanceisobtainedisbycomparingtheinstantaneousimage
withthepreviousimagesintheeyethatareretainedinmemory.”
Thisisthefirstrecordedenunciationofstructure-from-motion,tyingthepercep-
tion of structure to image brightness variations. By distinguishing geometry from
photometry, Gibson elaborated on this Helmholtz view of structure-from-motion
andstatedinhisbookThePerceptionoftheVisualWorld,1950,thatimagemotion
wastheactualstimulusfortheperceptionofstructure,ratherthanimagevariations
asHelmholtzconjectured.Hewasquiteexplicitaboutitwhenhewrote([4],pp.119):
“Whenitissoconsidered,asaprojectionoftheterrainorastheprojectionofanarrayof
slantedsurfaces,theretinalimageisnotapictureofobjectsbutacomplexofvariations.Ifthe
relativemotionisanalyzedoutandisolatedfromthecomplexofothervariations,itproves
tobealawfulandregularphenomenon.Definedasagradient ofmotion,itispotentially
astimuluscorrelateforanexperienceofcontinuousdistanceonasurface,asweshallsee,
andonenolongerisrequiredtopostulateaprocessofunconsciousinferenceaboutisolated
objects.”
By gradient of motion Gibson meant not the spatial or temporal variations of
image motion but the image motion field itself, or optical flow, stating, when he
discussedtheexampleofthemotionfieldontheretinaofaflierlandingonarunway
([4],pp.128),that:
“Thegradientsofmotionareapproximatelyrepresentedbyasetofvectorsindicatingdirec-
tionandrateatvariouspoints.Allvelocitiesvanishatthehorizon”.
TheveracityofGibson’sstatementthatimagemotionisthestimulusfortheper-
ceptionofstructureisnotsomuchsurprisingwhenweobservethattheperception
ofthestructureofasurfaceinmotiondoesnotchangefordifferenttexturecover-
ingsofthissurface.Therehavebeenseveralexperimentsdesignedtodemonstrate
unequivocally this perception of structure-from-motion, first the landmark kinetic
deptheffect experimentofWallachandO’Connell[5]whichusedtheshadowofa
tiltedrodprojectedonatranslucentscreenwhichviewersobservedfromtheback.
ItwasalsodemonstratedbyGibsonetal.[6]whousedatextureofpaintsplashed
on two lined-up parallel transparent screens the shadows of which were presented
toviewersonafrontaltranslucentscreen.Themoststrikingdemonstrationsareper-
haps the experiments of Ullman [7] and of Rogers and Graham [8] with random
dot distributions. Random dots constitute stimuli void of any texture or geometric
arrangement.RogersandGraham’sdemonstration[8]istosomeextentamechan-
ical counterpart of Ullman’s experiment with computer-generated random dots on
rotating cylinders [7]. Ullman presented viewers with the orthographic projection
onacomputer screenofabout ahundred pointsoneach oftwoimaginarycoaxial
1.1 ImageMotioninVisualFunction 3
Fig.1.1 Ullman’srotatingcylinderssetupsimulatedbyacomputerprogram:Viewerswereshown
theorthographicprojectiononacomputerscreenofasetofaboutahundredrandompointsoneach
oftwocoaxialcylindersofdifferentradii.Thecylindersoutlinewasnotincludedinthedisplay
sothattheywereimaginarytotheviewersand,therefore,contributednocluetotheperception.
Lookingattherandomdotsimageonthescreenwhenthecylinderswerenotmovingaffordedno
perceptionofdepth.Butwhenthecylindersweremadetomove,byacomputerprogram,observers
reportedthevividperceptionoftworotatingcoaxialcylindersandwerealsoabletogiveagood
estimateoftheamountofrotation
cylindersofdifferentradii(Fig.1.1).Thecylinderswereimaginaryinthesensethat
theiroutlinewasnotpresentedinthedisplaysoasnottoofferviewersacuetothe
perceptionofstructure.Lookingattheimageonthescreenoftherandomdotson
staticcylindersaffordednoperceptionofdepth.Butwhenthecylindersweremade
tomove,byacomputerprogram,observersreportedperceivingvividlytworotating
coaxialcylindersandcouldalsoestimatetheamountofrotation.
TheviewofHelmholtzontheroleofimagevariationsintheperceptionofstruc-
ture,whichwequotedpreviously,isquitegeneralbutotherwisecorrect,becausethe
linkbetweenimagevariationsandimagemotionislawful,toemploythisexpression
oftenusedbyGibsontomeanarelationwhichcanbeformallyspecifiedbygoverning
laws.HornandSchunckprovideduswithsuchalaw[9]intheformofarelation,or
equation,deducedfromtheassumptionthattheimagesensedfromagivenpointof
asurfaceinspaceremainsunchangedwhenthesurfacemoves.Theequation,which
wewillinvestigatethoroughlyinChap.3anduserepeatedlyintheotherchapters,is
calledtheopticalflowconstraint,ortheHornandSchunckequation:
I u+I v+I =0, (1.1)
x y t
where I ,I ,I aretheimagespatiotemporalderivatives,t beingthetimeand x,y
x y t
theimagespatialcoordinates,and(u,v) = (dx,dy)istheopticalflowvector.The
dt dt
equationiswrittenforeverypointoftheimagepositionalarray.
As to the link between image motion and environmental motion and structure,
onecangetalaw,orequation,bydrawingaviewingsystemconfiguration model,
projecting points in three-dimensional space onto the imaging surface, and taking
thetimederivativeoftheprojectedpointscoordinates.UnderaCartesianreference