Efficient Feature Matching by Progressive Candidate Search

1.6 MB·

by  Sehyung Lee

Efficient Feature Matching by Progressive Candidate Search
SehyungLee JongwooLim IlHongSuh
HanyangUniversity HanyangUniversity HanyangUniversity
Seoul,Korea Seoul,Korea Seoul,Korea
[email protected] [email protected] [email protected]
7
1
Abstract itoftenfailsinchallengingsituations,suchasthepresence
0
2 of repetitive patterns, or large illumination or viewing di-
Wepresentanovelfeaturematchingalgorithmthatsys- rectionchanges.
n
a tematicallyutilizesthegeometricpropertiesoffeaturessuch Most local features provide the scale and orientation in
J as position, scale, and orientation, in addition to the con- addition to the feature position, and this information has

0 ventional descriptor vectors. In challenging scenes with beenusedinfeaturematching. Forexample,whenthefea-
2 thepresenceofrepetitivepatternsorwithalargeviewpoint ture descriptors are not discriminative enough, ambiguous
change,itishardtofindthecorrectcorrespondencesusing matchcandidatescanberemovedbysuchadditionalinfor-
]
V feature descriptors only, since the descriptor distances of mation. Several feature matching algorithms utilizing the
thecorrectmatchesmaynotbetheleastamongthecandi- additional pairwise information between neighboring fea-
C
datesduetoappearancechanges. Assumingthatthelayout tures have been proposed, and achieved more robust fea-
.
s ofthenearbyfeaturesdoesnotchangedmuch,wepropose ture matching than simple descriptor matching approach.
c
[ the bidirectional transfer measure to gauge the geometric One of the popular directions is the graph matching such
consistencyofapairoffeaturecorrespondences. Thefea- as the spectral clustering [13], Monte-carlo sampling [22],
1
ture matching problem is formulated as a Markov random and tensor decomposition [8]. Another direction is using
v
6 field(MRF)whichusesdescriptordistancesandrelativege- theMarkovrandomfields(MRF)suchasdualdecomposi-
7 ometric similarities together. The unmatched features are tion[23,16]orgraphcut[9].
6 explicitlymodeledintheMRFtominimizeitsnegativeim- In spite of success of these approaches, there are still
5 pact. For speed and stability, instead of solving the MRF
severallimitations. Whenthereexistunpairedfeatures,the
0 on the entire features at once, we start with a small set of
featureswhichdonothavetruematchesintheotherimage,
.
1 confident feature matches, and then progressively search
theerrorbytheirincorrectmatchcandidatesispropagated
0 the candidates in nearby features and expand the MRF
through the whole graph, especially in MRF models, and
7
with them. Experimental comparisons show that the pro-
1 affectstheestimationoftheotherfeaturematches. Another
posed algorithm finds better feature correspondences, i.e.
: drawbackofthegraphoptimizationalgorithmsistheirhigh
v morematcheswithhigherinlierratio,inmanychallenging
computational cost. The recent algorithms with less com-
i
X sceneswithmuchlowercomputationalcostthanthestate- plexity[28,27,4]arestillnotfastenoughforsolvinglarge
of-the-artalgorithms.
r scalematchingproblems. Thesearethemainreasonswhy
a
most existing algorithms [8, 7, 26, 23, 13, 22] deal with a
relativelysparseandsmallfeaturesetoffeatures.
1.Introduction
In this paper we propose a unified feature matching
Localfeaturesarewidelyusedinmanyapplicationsin- framework that considers both geometric and appearance
cluding 3D scene reconstruction, image retrieval, image properties of commonly-used local features (Figure 1).
stitching, and object recognition. Among them, SIFT [17] Similar to the MRF-based algorithms, the feature corre-
and SURF [2] are the most popular due to their invari- spondences are found by solving an MRF on the feature
ancetoilluminationandsmallaffinedeformation. Thefea- set,whoseunaryandbinaryenergytermsrepresentfeature
turecorrespondencesbetweentwoimagesareusuallyfound descriptordistancesandrelativegeometricinconsistencies.
by comparing the feature descriptors only. The incorrect Tomitigatenegativeimpactofunpairedfeatures,wein-
matchesarefilteredbyRANSAC[10],andtheinlierfeature troduce the ‘unmatched’ label for each node in MRF with
matches are kept for further processing. This framework properunaryandbinarycosts.Alsothe‘bidirectionaltrans-
hasbeensuccessfullyusedinmanyapplications, however, fer measure’ computes the relative geometric consistency
1
Our algorithm
Progressive
Compute initial Find new Update skeletal RANSAC or
Feature extraction min-sum message
seed features candidate features features other processing
passing
image pair correspondences
Figure1: Overviewoftheproposedalgorithm.
betweentwofeaturecorrespondencesmorerobustly. much smaller than the number of features. Note that the
The‘progressivecorrespondencesearch’process,which followingstagesafterfeaturematching,suchasRANSAC,
identifiescorrespondencesofmoreconfidentfeaturesfirst, furtherfilterinconsistentfeaturematches, thusitiscritical
then expands to other features, also helps reducing the tohavesufficientnumberofgoodfeaturecorrespondences
chanceofhavingincorrectlymatchedfeaturesinthegraph. inthefeaturematchingstage.
Ourprogressivecorrespondencesearchalsogreatlyreduces Our approach is related to several previous works.
search time by starting with good initial matches at each Leordeanu and Hebert [13] introduced pairwise geometric
stage.Inmatchingaroundtwothousandfeaturesineachim- informationtobuildtheaffinitymatrix.Torresanietal.[23]
age,theproposedalgorithmfindshigherqualityfeaturecor- designedacomplexcostfunctionbasedontheappearance
respondences within 0.4 s, whereas other algorithms take and spatial proximities which can be efficiently optimized
severalsecondstominutes. Extensiveexperimentalevalua- by dual decomposition. Cho et al. [6] employed the ag-
tiondemonstratesthatourapproacheffectivelyhandlesthe glomerativeclusteringalgorithminsteadofspectralcluster-
limits of conventional approaches and achieves significant ing[13].Theirworkintroducedvarioussolvers,reweighted
improvementinchallengingfeaturematchingproblems. randomwalks[7]andMonte-Carlosampling[22]. Inthese
Themaincontributionsofourworkaretwofold. algorithms,theaffinitymatrixisconstructedbycombining
• TheproposedMRFframeworkwiththeunmatchedla- the descriptor similarity and Euclidean distance between
bel and bidirectional transfer measure can handle the features. Zhou and Torre [27] factorized the affinity ma-
challenging scenes better and yields much improved trix as a product of smaller matrices. Because of this ma-
featurematches. trix factorization, there is no need to explicitly store the
• ProgressiveconstructionandoptimizationoftheMRF full-size affinity matrix. Zaragoza et al. [25] proposed a
basedonspatialneighborhoodandmultiplecandidacy techniquetocomputeanas-projective-as-possiblewarping
achieveshigheraccuracy,betterrobustness,andmuch that aims for the feature locations to be globally projec-
fasterprocessing. tive. This method is developed for image stitching appli-
cation in which the correspondence algorithm is based on
2.RelatedWork the multiple-homography fitting using their previous work
[5].In[15],proposedbyLinetal.,startingwithasetofini-
After SIFT [17] and SURF [2], many novel feature de- tialmatches,theiralgorithmcomputesanaffinemotionfield
tection algorithms such as BRISK [14], BRIEF [3], ORB betweentwoimagesbyexploitingthat eachmatchdefines
[21]andKAZE[1],weredevelopedtoimproveSIFT’sper- alocalaffinetransformation. Giventhemotionfield,more
formance. Inthesealgorithms,feature-to-featurematching correspondences are then recovered by finding the nearest
with descriptors is the most common method for comput- neighboringmatchindescriptorspaceforeachfeature.
ing correspondences. In spite of the distinctiveness of the Theproposedworkdiffersfromthesealgorithmsinsev-
proposeddescriptor,itisstillinsufficientformatchingfea- eral ways. We directly measure the geometric dissimilar-
turesinthepresenceofseveretransformation. Tocompen- itybetweencorrespondencesusingtheposition, scale, and
sateforthelimitationsofsimpledescriptormatching,many orientation provided by local features, instead of the dis-
algorithms have been proposed. We briefly review related tances between neighboring features and their correspond-
works to clarify the differences between the proposed and ing features [13], the length and direction differences of
existingmethods. correspondences[23]orhomographytransformationusing
To quickly find a good initial matches using feature additional affine region detector [6, 7, 22]. The proposed
descriptors, several heuristic algorithms have been devel- algorithmcanbeusedwithmostavailablelocalfeaturesas
oped. Thenearestneighboranddistanceratio(NNDR)al- long as the geometric properties are provided, not limited
gorithm [17] is based on the insight that the match is am- to a few specific local features. Also, unlike the previous
biguousifthebestandthesecond-bestmatchcandidatesof approaches [27, 8, 7, 23, 13] that perform estimation on
afeaturehavesimilardescriptors. Amatchisacceptedonly the entire feature set at once, our algorithm incrementally
iftheratioofthetwodescriptordistancesisbelowathresh- finds the geometrically reasonable matches in whole fea-
old.NNDReffectivelydiscardstheambiguousmatches,but turesetfromtheinitialsetoffeatures. Becauseofthis,the
as a consequence the number of feature matches are often proposed algorithm can handle a large number of features
𝐼𝑟 𝐼𝑡 𝐱 𝑠′|𝐜
Transfer 𝑇𝑡𝑇𝑠−1𝐱𝑠′ 𝜏 𝐜′;𝐜
𝑡 𝑠′ 𝐜′ 𝑡′
𝑠 𝑡′ 𝑠 Transfer 𝑇𝑡′𝑇𝑠−′1𝐱𝑠 𝐱 𝑠|𝐜′
𝜏 𝐜;𝐜′
𝑠′ 𝐜
𝑡
Figure 3: The bidirectional transfer measure. The color points
representthedetectedlocalfeatures. Thearrowsanddottedrect-
Figure2: TheMRFmodelforfeaturecorrespondence. Thecir- anglesaretheorientationsandscalesofthefeatures. Theneigh-
clesrepresentthedetectedfeatures. TheMRFisconstructedon boringfeatureiswarpedintotheoppositeimage,andthetransfer
thefeaturesinthereferenceimage(left),andthesolidlinesinthe errorismeasuredbythedistancebetweenoriginalandtransferred
reference image are the MRF edges. Each node i has its poten- locations,xˆs(cid:48)|candxˆt(cid:48).
tialcorrespondencec whichisshownasdottedarcsbetweentwo
i
views. The pairwise energy between (s ,t ) and (s ,t ) affects n
i i j j (cid:88)
theMRFcost,theredcirclesrepresentthefeaturesthattheirtrue Eφ(C)= eφ(ci), and (2)
correspondencesdonotexistinthetargetimage. i
(cid:26) (cid:107)d −d (cid:107) ift (cid:54)=∅
and produce more accurate matches with minimal compu- e (c )= si ti i , (3)
φ i α otherwise
tationalcost.
whered andd arethefeaturedescriptorvectorsofthe
si ti
3.ProblemFormulation features s and t . The unary energy term penalizes dis-
i i
similar feature correspondences, and the penalty term for
The feature matching task can be formulated as an en- unmatchedfeaturesαissetto0.5.
ergyminimizationproblemoftheMRFconstructedonthe
features.Giventwosetsoffeaturesdetectedinthereference Bidirectional Transfer Measure. In most local features
andtargetimages,theMRFmodelisconstructedonthefea- the scale and orientation of each feature point is available
turesinthereferenceimage. Thevariableineachnoderep- as a by-product in realizing the scale and orientation in-
resentsitscorrespondenceinthetargetimage,andtheentire variance. Although the scales, orientations, or positions
costoftheMRFmodelrepresentstheoverallconsistencyof of the features change according to camera pose or scene
thecurrentcorrespondencesbetweentwoimages. Figure2 changes, theirrelativedifferencetendstoremainsimilarif
visualizestheformulation. thefeaturesbelongtothesameobject.Weproposethebidi-
The energy function of MRF is comprised of the unary rectional transfer distances between neighboring features
andpairwisepotentials. TheunarypotentialE represents to measure the relative difference between their geometric
φ
theappearancemodeltomeasurethedescriptordissimilar- properties.
ity of the current correspondence. The pairwise potential Let T be the transformation that maps a unit patch,
i
E models the relative geometric inconsistency between whichiscenteredattheorigin,axis-aligned,andunit-sized,
ψ
the pairs of correspondences. Then, the best correspon- onto the patch of the feature i. In case of SIFT or SURF,
dencesetC∗ isobtainedthatminimizestheoverallenergy, each feature patch is represented as x-, y-position, scale,
whichiswrittenas andorientation(tx ,ty ,σ ,θ ),andthenT iswrittenasa
i i i i i
C∗ =argminE (C)+λE (C), (1) 2Dsimilaritytransform
φ ψ
C  
σ cosθ −σ sinθ tx
whereC ={c }isthesetoffeaturecorrespondences,c = i i i i i
(s ,t ),s isthiefeatureindexinthereferenceimageandi t Ti =σisinθi σicosθi tyi.
i i i i 0 0 1
iseitherthefeatureindexinthetargetimageor∅ifs isan
i
unmatched/unpairedfeature. λistheweightparameterfor Depending on the geometric properties provided by local
leveragingtheunaryandpairwisemodels(λissetto0.1). features,T canbeanytypeof2Dtransformationincluding
i
Toreducethesearchspace,foreachfeaturesweonlycon- similarity,affineorhomography.
siderthetop-κdescriptormatchesinthetargetimageasthe For a feature correspondences c = (s,t), the warped
candidate matches, i.e. c ∈ {(s ,t )} ∪{(s ,∅)} position of a nearby feature s(cid:48) in the target image can be
i i k k=1...κ i
(κissetto15). NotethatthesizeoftheMRFgraphisnot written as xˆ = T T−1x . If the relative geometry be-
s(cid:48)|c t s s(cid:48)
affectedbyκ,butbythenumberofneighborfeatures. tween two feature correspondences c and c(cid:48) = (s(cid:48),t(cid:48)) is
TheunarytermE measuresthedescriptorsimilaritybe- approximatelypreservedbetweentwoimages,theposition
φ
tweencorrespondingfeatures. ofthematchedfeatureinthetargetimagex mustbeclose
t(cid:48)
toxˆs(cid:48)|c. Thereforetheone-waytransferdistanceτ(c(cid:48);c)is Seed features Corresponding features
thendefinedas 𝐜2 𝐜4
τ(c(cid:48);c)=(cid:26) (cid:13)(cid:13)xˆs(cid:48)|c−xt(cid:48)(cid:13)(cid:13)2, ift(cid:54)=∅andt(cid:48) (cid:54)=∅ , 𝑠1 𝑠2 𝑠4 𝐜3 𝑡1 𝑡2 𝑡3 𝑡4
0 otherwise
𝑠3
𝐜1
andthepairwiseenergye (c,c(cid:48))isdefinedas
ψ
Consistency check
e (c,c(cid:48))=τ(c(cid:48);c)+τ(c;c(cid:48))+τ(c(cid:48)−1;c−1)+τ(c−1;c(cid:48)−1),
ψ
where c−1 = (t,s) represents the inverse correspondence 𝑠2 𝑠4
ofcdefinedbyflippingthereferenceandtargetfeatures. It 𝑠1 𝑠5
considers the differences not only by the forward warping 𝑠3
butalsobythebackwardwarping. andthepairwiseenergy
Eψ oftheMRFisasfollows: Extended MRF Inconsistent match Consistent match
n m Figure 4: Overview of progressive feature matching. After the
(cid:88) (cid:88)
E (C)= e (c ,c ), correspondences of seed features (blue circles) and their corre-
ψ ψ i j
spondences are computed (black dotted lines), our MRF model
i j∈N(i)
isexpanded.Bycomparingthecandidatematchesoftheextended
whereN(i)isthesetofneighborsofthei-thfeature. featureswiththenearbyseedcorrespondences,onlygeometrically
Ourbidirectionaltransfermodelisbasedonthesimilar- consistentmatchesareconsideredascandidatematchesintheex-
ity transform obtained by the relative changes of potential pandedMRFmodel.
matches. When affine-based features such as [18, 20] are
used,thebidirectionaltransfermodelcanbeeasilyextended riskandachievefasterconvergence,weproposeaprogres-
totheaffinetransformation.However,theyaremuchslower sivefeaturematchingscheme. Oncetheseedfeatureswith
thanotherwidelyusedalgorithms[14,1,21,3,17,2],and goodcorrespondencesareavailable,thentheycanguidethe
inthispaperweonlyusethesimilaritymodel. matching process of the nearby features from similar part
ofthescene. Thismotivatestheprogressivesearchoffea-
Belief Propagation. The problem of Equation 1 is NP- ture correspondence. Starting from a small set of highly
hard, thus in general it is not possible to find the optimal confident candidate matches, the BP algorithm finds the
solution. Insteadweemploythebeliefpropagation(BP)al- convergedcorrespondenceassignments,andtheunmatched
gorithmtofindtheapproximatesolution. Amongthemany features are excluded from the MRF. Then the MRF is it-
variantsofBPalgorithms,themin-summessagepassingis eratively expanded by nearby features around the existing
usedduetoitscomputationalefficiencycomparedtosum- features,followedbyBPoptimizationandfiltering,untilall
product or max-product methods. The message from the featuresareconsidered.
nodeitothenodej forthecorrespondencec is
j
Initial Seed Features. To reduce the chance of selecting
(cid:104) (cid:88) (cid:105)
m (c )=min e (c )+λe (c ,c )+ m (c ) . unpairedfeatures,weselectthefeatureswithhighdescrip-
i→j j φ i ψ i j k→i i
ci tor similarity scores as the initial seed features. Among
k∈N(i)\j
the features in the reference image whose NNDR ratio is
Afterconvergence,thebeliefofacorrespondenceciineach smallerthanθ,thetop-r featuresintermsofthedescriptor
nodeiiscomputedas score are used as the nodes of the initial MRF model, and
(cid:88) each node is connected with its K-nearest neighbors (we
b(c )=e (c )+ m (c ). (4)
i φ i k→i i use θ = 0.9, r = 100, and K = 5 in our experiments).
k∈N(i) Although we selected high-quality matches for the initial
The optimal feature correspondence set C∗ is determined MRF,aftertheBPprocess,theremayexistunmatchedfea-
byindividuallyselectingthelabelhavingminimumcostin tures (the cost of ‘unmatched’ is the smallest). These fea-
eachnode. turesareexcludedfromtheMRFgraphandputbacktothe
candidatefeatureset.
4.ProgressiveFeatureMatching
Progressive Candidate Search. The features in the MRF
ItispossibletobuildanMRFwithalldetectedfeatures of the previous iteration are called as the ‘seed’ features
atonceandtrytofindthesolutionofEquation1usingBP, in the current iteration. In our framework, their feature
but it often converges to a wrong solution and takes long correspondences are used as the guides to find the corre-
time to converge due to the large amount of incorrect or spondencesofneighboringfeatureswhentherearemultiple
confusing initial feature correspondences. To reduce this candidate matches with similar descriptor scores. Among
allfeaturesnotintheMRF,theK-nearestneighbor(KNN) Algorithm1:ProgressiveMin-SumMessagePassing
featuresofthe‘seed’featuresareselectedandaddedtothe
Input:seednodesS,seedcorrespondencesC ={p }.
S i
MRFasthe‘extended’nodes. TheKNNnodesofeachex-
Output:extendedcorrespondencesC .
S∪Q
tendednodesareconnectedwithedges. Intheprogressive Q←{} ...theextendedfeatureset.
iteration,thecorrespondencesoftheseedfeaturesarekept foreach f do
fixed,andtheseednodesonlyemitthemessagestowardthe if f ∈/ Sthen
neighboring extended nodes in the BP process. By com- FindK-nearestseednodesNS(f).
paringthecandidatematchesoftheextendedfeatureswith γf ←{∅} ...candidatematchsoff toconsider.
the nearby seed correspondences, geometrically inconsis- foreach candidatematch(f,gk)k=1...κoff do
ifmin (e ((f,g ),p ))<θ then
tent matches are quickly excluded in the message passing. n∈NS(f) ψ k n seed
Q←Q∪{f}
Forspeed-up,theinconsistentmatchcandidateswithanyof
γ ←γ ∪{g }
f f k
nearbyseednodesareexplicitlyexcludedfromtheconsid-
end
eration. By fixing the seed correspondences and reducing
end
the number of candidate matches of extended nodes, it is
else
possibletokeepthecomputationalcomplexityinareason- Addseedfeatureanditscorrespondence.
ablelevel.Theabove-mentionedprocessisdescribedinAl- Q←Q∪{s }
f
gorithm 1 and illustrated in Figure 4. We provide detailed γf ←γf ∪{tf}
descriptionofthealgorithminthenextparagraphs. end
GiventheseedsetS = {s }andtheircorrespondences end
i
C ={p =(s ,t )},everyleftoverfeaturef ∈/ Sistested BuildtheMRFusingQ.
S i i i
repeat
if it has any nearby seed correspondence that has similar
foreach extendednodef ∈Qdo
relativemotion,i.e.
foreach K-nearestneighbornodenoff do
(cid:0) (cid:1)
min min (e ((f,g ),p )) <θ , (5) foreach c=(f,g),g∈γ do
l=1...κ k∈NS(f) ψ l k seed f
m (c)← min (e (c )+
n→f φ n
where NS(f) is the KNN seed features of f, gl is the in- c(cid:80)n=(n,∗)
λe (c ,c)+ m (c ))
dexoftop-κcandidatematchesoff,andθseed issetto80. ψ n l∈N(n)\f l→n n
The extended feature set Q contains the features that sat- end
isfytheabovecondition. TheMRFisbuiltuponS∪Qby end
connectingeachnodeinQwithitsKNNnodes. Notethat end
foreach extendednodef ∈Qdo
the correspondence of the seed nodes are fixed, thus it is
foreach c=(f,g),g∈γ do
not necessary to perform any message passing to the seed (cid:80) f
b(c)←e (c)+ m (c)
nodes. Only the messages toward the extended nodes are φ l∈N(f) l→f
end
computedandupdated.
end
IntheMRFoftheinitialseedfeatures,allκcorrespon-
until convergence;
dencecandidatesareconsidered,butintheprogressivecan-
C ←C
didate search, only the candidates that are consistent with S∪Q S
foreach extendednodef ∈Qdo
anyoftheconnectedseednodes(Equation5)areinvolved if argmin (b(c))(cid:54)=(f,∅)then
c=(f,∗)
inBPprocess. Thisreducesthesizeofthemessagemi→j CS∪Q ←CS∪Q∪argminc=(f,∗)(b(c))
drasticallyfromκ2toκi×κj whereκiandκj arethenum- end
berofcandidatesofnodeiandj (includingunmatched∅) end
respectively. Ifanodeisaseednode, itonlyhasonecan-
didate, and in practice most extended nodes have around
5 candidates. The min-sum BP algorithm is then run on
• Computethecorrespondencesoftheseedfeaturesus-
theMRFtofindthebestfeaturecorrespondencesoftheex-
ingourMRFmodel.
tendedfeatures,excepttheunmatchedfeatures(thecostof
• Computenewchildfeaturesandtheirpossiblematch-
‘unmatched’ is less than those of any candidate matches).
ing hypotheses by comparing them with their parent
Since the time complexity of the BP algorithm scales lin-
features.
earlytothemessagesize,thereductionfromκ2 toκ ×κ
i j • Compute the correspondences of the child features
(κ ,κ (cid:28)κ)greatlyimprovestherunningspeed.
i j throughalgorithm1.
Overallalgorithmflow. Theproposedalgorithmproceeds • Updatetheparentfeatures,anditerateSteps3to5until
asfollow: newcandidatesarenotfound.
• Computetheinitialseedfeaturesbyselectingthefea- Once the feature correspondences are computed, further
tureshighlyrankedindescriptormatching. post-processingsuchasRANSACcanfollowasaconven-
100
NNDR1 NNDR2 BMF
90
MDLT Non-Prog Prog
80
70
60
50
40
30
20
10
0
L1 L2 L3 L4 L5 avg L1 L2 L3 L4 L5 avg L1 L2 L3 L4 L5 avg
PMR Precision MS
Figure5:Comparisonsoffeaturematchingalgorithms(unit:%).Itcanbeeasilyverifiedthattheproposedalgorithmoutperformstherest
byalargemargin.
tionalapproach. PMR L1 L2 L3 L4 L5 avg
NNDR1 40.39 32.53 23.00 13.34 6.92 23.24
NNDR2 52.71 45.44 35.68 25.48 17.61 35.38
BMF 59.22 49.41 37.97 27.16 16.07 37.96
MDLT 62.78 54.55 39.68 26.45 14.59 39.61
5.ExperimentalResults Non-Prog 53.11 45.98 35.71 26.15 18.01 35.79
Prog 66.88 58.16 45.98 32.63 21.67 45.07
Precision L1 L2 L3 L4 L5 avg
We conducted extensive experimental evaluation of the
NNDR1 97.91 96.58 93.06 88.29 77.13 90.60
proposedalgorithmonvariousdatasets. Thefirstsetofex-
NNDR2 88.56 83.71 74.07 61.08 44.70 70.42
periments shows the quantitative comparison of the sim- BMF 94.00 93.34 85.50 74.57 54.07 80.30
ple NNDR-based algorthms, the state-of-the-art methods MDLT 91.44 84.85 79.99 66.41 54.72 75.48
Non-Prog 91.17 85.43 75.98 64.95 50.07 73.52
[25, 15], and the proposed algorithm. In the second set
Prog 95.81 94.80 91.02 87.94 81.75 90.26
of experiments the improvement by the proposed method
MS L1 L2 L3 L4 L5 avg
on several popular local features. We urge readers to see
NNDR1 39.62 31.61 21.77 12.18 5.74 22.18
the supplementary material for complete experimental re- NNDR2 47.95 39.59 28.25 17.06 8.78 28.32
sults and detailed explanations. All experiments are con- BMF 56.80 47.20 33.37 20.75 9.09 33.44
MDLT 58.06 47.01 33.94 21.00 9.77 33.96
ducted in MATLAB on a computer with i7 4.0 GHz CPU
Non-Prog 49.09 40.43 28.81 18.26 9.56 29.23
and 32GB memory, and our algorithm is partially imple- Prog 64.52 55.36 42.33 29.29 18.12 41.92
mented as MEX functions. The source codes and dataset
Table1:PMR,Precision,andMSofBMF[15],MDLT[25],non-
willbeprovidedtopublic.
progressive (Non-Prog), and progressive (Prog) methods on the
Heinly et al. [12] proposes three different quantita- testdataset(unit: %). Thedataset[19]contains5differentlevels
tive evaluation metrics: putative match ratio, precision, ofgeometricandphotometrictransformation. Fromlefttoright,
theaverageperformanceofeachlevel(L1-L5),andtotalaverage
and matching score. The putative match ratio, PMR =
#putativematches, the ratio of computed matches to all detec- areshown.
#features
tions, represents the selectivity of the matchers. A less re-
5.1.ComparisontoExistingAlgorithms
strictive matching algorithm yields higher putative match
ratios,whereasveryrestrictivematchermaydiscardpoten-
Theproposedmethodiscomparedwiththreealgorithms,
tially valid matches and end up with a low putative match
NNDRwithtwodifferentthresholds(0.9forNNDR1,and
ratio. Theprecision,Precision= #inliermatches ,denotesthe
#putativematches 0.8 for NNDR2), MDLT [25], and BMF [15] using their
ratio of correct matches out of the putative matches (the implementations1. InthisexperimentASIFT[20]featureis
inlier ratio). It measures the ‘quality’ of feature match-
used since BMF only works with ASIFT. Evaluation with
ing, i.e. how many are correct among the declared feature
otherfeaturesarediscussedinthenextsubsection.
matches. Forexample,averyrestrictivematchermayhave
For quantitative comparison, the well-known dataset
a low putative match ratio but the precision can be high if
constructedbyMikolajczyketal.[19]isusedsinceitpro-
most incorrect matches are filtered. The matching score,
videstheground-truthhomographytransformation. Itcon-
MS = #inliermatches, which is equivalent to PMR × Preci-
#features sists of 8 sets of 6 images, one reference and 5 target im-
sion, show how many true matches are selected among all
ages, whose geometric and/or photometric properties are
featuredetections.Wecannowquantifyhowmanyfeatures
werematched(PMR),andhowmanyaretruematches(Pre- 1BMF[15]:http://mmcheng.net/bfun/
cisionandMS). MDLT[25]:https://cs.adelaide.edu.au/jzaragoza/doku.php?id=mdlt
#matches: 287, #inliers: 185, ratio: 0.6446
SIFT Nearest NNDR1 NNDR2 Prog
PMR 100 23.67 9.49 20.49
Precision 2.52 16.94 42.91 45.49
MS 2.52 4.72 4.70 10.13
SURF Nearest NNDR1 NNDR2 Prog
PMR 100 25.25 8.36 26.95
Precision 3.01 11.36 27.81 31.54
MS 3.01 3.40 2.93 9.51
KAZE Nearest NNDR1 NNDR2 Prog
#matches: 341, #inliers: 199, ratio: 0.5836 PMR 100 20.67 8.12 36.29
Precision 5.60 22.99 44.18 35.23
MS 5.60 5.68 4.13 13.60
Overall Nearest NNDR1 NNDR2 Prog
PMR 100 23.20 8.65 27.91
Precision 3.71 17.10 38.30 37.42
MS 3.71 4.60 3.92 11.08
Table2: Thefeaturematchingperformanceonvariouslocalfea-
tures: SIFT[17], SURF[2], andKAZE[1](unit: %). Thepro-
#matches: 900, #inliers: 745, ratio: 0.8278
posedmethodand3basicmatchingcriteriaareusedforcompar-
ison. The quantitative evaluation is based on averages of PMR,
Precision,andMSmetrics.
1078SIFTfeaturesare0.54, 4.82, 0.62, and0.21seconds,
respectively.Inthetestwithmorenumberoffeatures(2430
and 2196), the processing times are 3.92, 9.81, 7.27, and
0.36seconds,respectively. Theproposedalgorithmismore
Figure 6: From top to bottom, the feature matching results of efficient and effective than conventional feature matching
BMF,MDLT,andProgontheL5pairof‘graf’inthedataset[25]. algorithmsandthenon-progressiveversion.
Theproposedalgorithmfindsmoreandbetterfeaturematchesthan
others.
5.2.ExperimentswithVariousLocalFeatures
gradually changed. The label from L1 to L5 denotes the Anotheradvantageoftheproposedmethodisitsgener-
amountofdeformationordegradation.Thefeaturematches ality - it can be used with any local features. In this ex-
areconsideredascorrectifthedistancetothewarpedpoint periment, SIFT [17], SURF [2], and KAZE [1] features in
bytheground-truthhomographyissmallerthan10pixels. OpenCV3.0aretested. Wecomparetheproposedmethod
Figure5andTable1showtheaveragedperformancein withNNDRofthreedifferentthresholds: 1.0(Nearest),0.9
eachlevel(L1-L5)andtheaverageoveralllevels. Thepro- (NNDR1), and 0.8 (NNDR2). Similar to the previous ex-
posed method (Prog) outperforms BMF and MDLT by a periments,PMR,Precision,andMSofthefeaturematching
large margin in PMR and MS metrics in all levels, and in resultsarecompared.
Precisionitperformsslightlybetterthantheothers.Wealso In addition to Heinly’s dataset [12], we constructed an
testanotherversionofouralgorithm(Non-Prog)whichruns extended dataset of 40 images with planar and non-planar
withouttheprogressivecorrespondencesearchtoshowthe sceneswithlargegeometricandphotometricvariations.Es-
effectivenessofourprogressivealgorithm. pecially,theimagepairscontainingmanyrepetitivepatterns
In many tasks using features, having more inliers is suchaswindowsanddecorativeelementsareincluded. For
as important as having high inlier ratio. For example, in quantitative evaluation, the inlier matches are found using
structure-from-motion,theinlierratiodeterminesthenum- the RANSAC algorithm with 8-point algorithm [11], fol-
berofRANSACiterationsandthenumberofinliersaffects lowedbymanualeliminationofafewobviousoutliers.Fig-
thequalityandfidelityofthe3Dreconstruction.Thebenefit ure 7 displays one of the test images and the matching re-
ofutilizingrelativegeometricinformationbecomessignif- sults using SIFT features. As shown in these images, our
icantwhenthereexistsalargedeformation(inL5orL4)- methodproducessignificantlybettercorrespondenceswith
allmetricsincludingPrecisionoftheproposedmethodare higher precision (162 inliers and 32.93% precision vs. 49
much higher. Figure 6 shows a representative example of inliersand25%precisionofNNDR2). Weencourageread-
featurematchingbydifferentalgorithms. erstoseemorematchingresultsinthesupplementaryma-
Alsointermsoftheprocessingtime,theproposedalgo- terial.
rithmismuchfasterthantheotheralgorithms.Theprocess- Figure8summarizestheexperimentalresults(averaged
ingtimesofBMF,MDLT,Non-ProgandProgon1115and over the 40 image pairs). One can see significant perfor-
#matches: 650, #inliers: 43, ratio: 0.0662
#matches: 196, #inliers: 49, ratio: 0.2500
Figure9: Theresultsonfeaturematchinginthepresenceofin-
#matches: 492, #inliers: 162, ratio: 0.3293 dependentmotions. TheMRFedgesacrossmotionboundaryare
effectlydisabledautomaticallybythealgorithm.Theyarebyfea-
turematchingonlywithoutanypostprocessing.
veryhighprecisionandMSatthesametime.
5.3.FeatureMatchinginIndependentMotion
Onemaywonderiftheproposedrelativegeometriccon-
Figure 7: ThefeaturecorrespondenceresultsonSIFTfeatures.
straint may cause problem when there exist independent
EachsetofimagesshowtheresultsofNNDR1,NNDR2,andour
motions. Although the relative geometry between features
method. Theproposedalgorithmproducedmuchmoreinliercor-
in the same object will remain similar, the features from
respondences.Simplematchingmethodsfailtofindgoodmatches
duetomanyrepetitivepatternsandlargeviewpointchange. different objects would not have such geometric consis-
tency.Intheproposedalgorithm,ifanedgeisonthemotion
boundary,themutualcontributionofthenodestotheother
50 100 100 100 100
Nearest nodesbecomesveryweakduetothemotioninconsistency.
45 NNDR1 Thisineffectcutstheedgesonthemotionboundaries,thus
40 NNDR2
Prog onlythefeatureswithsimilarmotioncontributestothees-
35
timationofcandidatematchscore.
30
Theproposedalgorithmistestedwiththeimagesofpub-
25
licdataset[24],andtworepresentativeresultsaredisplayed
20
inFigure9(moreresultsareinthesupplementarymaterial).
15
Fromtheexperimentalresultsitisshownthattheproposed
10
algorithmcanhandletheindependentmotions.
5
0
SIFT SURF KAZEOverall SIFT SURF KAZEOverall SIFT SURF KAZEOverall 6.ConclusionandFutureWork
PMR Precision MS
Figure 8: The feature matching performance on various local In this paper, we propose a feature matching algorithm
features: SIFT [17], SURF [2], and KAZE [1] (unit: %). The thatutilizesbothgeometricpropertiesanddescriptoroflo-
proposedmethodandNNDRwith3differentthresholdsarecom- cal features. The feature matching problem is formulated
pared.Refertothetextformoredetail. as an MRF model, and the solution is found by the min-
sum BP algorithm. Bidirectional transfer measure com-
mance improvement by the proposed algorithm compared putes the relative geometric consistency of a pair of fea-
to basic approaches. The proposed algorithm achieves the ture correspondences, and it is used to filter the similar-
best MS in all local features, finding 2-4 times more in- looking match candidates in the target image. Progressive
liermatchescomparedtoanyofNNDR-basedmethods. In correspondencesearchisintroducedtofindthesolutionef-
termsofprecision,theproposedmethodshowssimilarper- ficiently and robustly. Through extensive experiments, we
formancetoNNDR2. ConsideringthatNNDR2yieldsvery quantitatively evaluated that the proposed method outper-
low PMR, which means it only accepts the matches with forms both graph-based and descriptor-based conventional
very distinct descriptors, the proposed algorithm achieves featurematchingalgorithms.
References [16] K.Liu,J.Zhang,K.Huang,andT.Tan. Deformableobject
matchingviadeformationdecompositionbased2dlabelmrf.
[1] P.F.Alcantarilla, A.Bartoli, andA.J.Davison. Kazefea-
InComputerVisionandPatternRecognition(CVPR),2014
tures. In Computer Vision–ECCV 2012, pages 214–227.
IEEEConferenceon,pages2321–2328.IEEE,2014.
Springer,2012.
[17] D. G. Lowe. Distinctive image features from scale-
[2] H.Bay, T.Tuytelaars, andL.VanGool. Surf: Speededup
invariant keypoints. International journal of computer vi-
robustfeatures.InComputervision–ECCV2006,pages404–
sion,60(2):91–110,2004.
417.Springer,2006.
[18] K.MikolajczykandC.Schmid. Scale&affineinvariantin-
[3] M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, terestpointdetectors. Internationaljournalofcomputervi-
C. Strecha, and P. Fua. Brief: Computing a local binary sion,60(1):63–86,2004.
descriptor very fast. Pattern Analysis and Machine Intelli-
[19] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman,
gence,IEEETransactionson,34(7):1281–1298,2012.
J. Matas, F. Schaffalitzky, T. Kadir, and L. Van Gool. A
[4] H.-Y. Chen, Y.-Y. Lin, and B.-Y. Chen. Robust feature comparisonofaffineregiondetectors. Internationaljournal
matching with alternate hough and inverted hough trans- ofcomputervision,65(1-2):43–72,2005.
forms.InComputerVisionandPatternRecognition(CVPR),
[20] J.-M.MorelandG.Yu. Asift: Anewframeworkforfully
2013IEEEConferenceon,pages2762–2769.IEEE,2013.
affineinvariantimagecomparison. SIAMJournalonImag-
[5] T.-J.Chin,J.Yu,andD.Suter.Acceleratedhypothesisgener- ingSciences,2(2):438–469,2009.
ationformultistructuredataviapreferenceanalysis. Pattern
[21] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. Orb:
Analysis and Machine Intelligence, IEEE Transactions on, an efficient alternative to sift or surf. In Computer Vi-
34(4):625–638,2012. sion(ICCV),2011IEEEInternationalConferenceon,pages
[6] M.ChoandJ.Lee. Featurecorrespondenceanddeformable 2564–2571.IEEE,2011.
object matching via agglomerative correspondence cluster- [22] Y.Suh,M.Cho,andK.M.Lee.Graphmatchingviasequen-
ing.InComputerVision,2009IEEE12thInternationalCon- tial monte carlo. In Computer Vision–ECCV 2012, pages
ferenceon,pages1280–1287.IEEE,2009. 624–637.Springer,2012.
[7] M.Cho,J.Lee,andK.M.Lee. Reweightedrandomwalks [23] L. Torresani, V. Kolmogorov, and C. Rother. Feature cor-
forgraphmatching. InComputerVision–ECCV2010,pages respondence via graph matching: Models and global opti-
492–505.Springer,2010. mization. InComputerVision–ECCV2008,pages596–609.
[8] O.Duchenne,F.Bach,I.-S.Kweon,andJ.Ponce. Atensor- Springer,2008.
based algorithm for high-order graph matching. Pattern [24] H.S.Wong,T.-J.Chin,J.Yu,andD.Suter.Dynamicandhi-
Analysis and Machine Intelligence, IEEE Transactions on, erarchicalmulti-structuregeometricmodelfitting. InCom-
33(12):2383–2395,2011. puter Vision (ICCV), 2011 IEEE International Conference
[9] O. Duchenne, A. Joulin, and J. Ponce. A graph-matching on,pages1044–1051.IEEE,2011.
kernelforobjectcategorization.InComputerVision(ICCV), [25] J. Zaragoza, T.-J. Chin, Q.-H. Tran, M. S. Brown, and
2011IEEEInternationalConferenceon,pages1792–1799. D. Suter. As-projective-as-possible image stitching with
IEEE,2011. moving dlt. Pattern Analysis and Machine Intelligence,
[10] M. A. Fischler and R. C. Bolles. Random sample consen- IEEETransactionson,36(7):1285–1298,2014.
sus: aparadigmformodelfittingwithapplicationstoimage [26] R. Zass and A. Shashua. Probabilistic graph and hyper-
analysisandautomatedcartography. Communicationsofthe graph matching. In Computer Vision and Pattern Recog-
ACM,24(6):381–395,1981. nition,2008.CVPR2008.IEEEConferenceon,pages1–8.
[11] R.Hartleyetal.Indefenseoftheeight-pointalgorithm.Pat- IEEE,2008.
ternAnalysisandMachineIntelligence,IEEETransactions [27] F. Zhou and F. De la Torre. Factorized graph matching.
on,19(6):580–593,1997. InComputerVisionandPatternRecognition(CVPR),2012
[12] J.Heinly,E.Dunn,andJ.-M.Frahm.Comparativeevaluation IEEEConferenceon,pages127–134.IEEE,2012.
ofbinaryfeatures. InComputerVision–ECCV2012,pages [28] F. Zhou and F. De la Torre. Deformable graph matching.
759–773.Springer,2012. InComputerVisionandPatternRecognition(CVPR),2013
[13] M.LeordeanuandM.Hebert.Aspectraltechniqueforcorre- IEEEConferenceon,pages2922–2929.IEEE,2013.
spondenceproblemsusingpairwiseconstraints.InComputer
Vision,2005.ICCV2005.TenthIEEEInternationalConfer-
enceon,volume2,pages1482–1489.IEEE,2005.
[14] S. Leutenegger, M. Chli, and R. Y. Siegwart. Brisk: Bi-
nary robust invariant scalable keypoints. In Computer Vi-
sion(ICCV),2011IEEEInternationalConferenceon,pages
2548–2555.IEEE,2011.
[15] W.-Y. D. Lin, M.-M. Cheng, J. Lu, H. Yang, M. N. Do,
andP.Torr. Bilateralfunctionsforglobalmotionmodeling.
InComputerVision–ECCV2014,pages341–356.Springer,
2014.