ChESS - Quick and Robust Detection of Chess-board Features

2.1 MB·English

by  Stuart Bennett

ChESS – Quick and Robust Detection of Chess-board Features
StuartBennettandJoanLasenby∗
December5,2012
Abstract
system.Forsuchusefeatureextractionmustbehighlyautomated
3 andfast.
1 Localization of chess-board vertices is a common task in com- Wewillpresentarobustprocessspecificallytargetingthede-
0
putervision,underpinningmanyapplications,butrelativelylittle tectionofchess-boardpatternvertices,whichratherthanrequir-
2
workfocussesondesigningaspecificfeaturedetectorthatisfast, ing a binary vertex/not-vertex threshold, provides a measure of
n
accurateandrobust. Inthispaperthe“Chess-boardExtractionby strengthsimilarinoutputtothemuch-usedHarrisandStephens
a
J SubtractionandSummation”(ChESS)featuredetector,designed [1988]cornerdetectorinthesameproblem-space. Thispermits
3 toexclusivelyrespondtochess-boardvertices,ispresented. The deferralofinclusiondecisionstoalaterstagewhereoneisbetter
2 methodproposedisrobustagainst noise, poorlightingandpoor abletoexploitspatialandgeometricconsiderations.
contrast, requiresnopriorknowledgeoftheextentofthechess- Theprocessisalsocomputationallyefficientandwelldisposed

] board pattern, is computationally very efficient, and provides a toavarietyofparallelprocessingtechniques,withareferenceim-
V
strength measure of detected features. Such a detector has sig- plementationcapableofthroughputofover700VGAresolution
C
nificantapplicationbothinthekeyfieldofcameracalibration,as framespersecondoncommodityPChardware.
.
s well as in Structured Light 3D reconstruction. Evidence is pre-
c sented showing its robustness, accuracy, and efficiency in com-
[
parisontoothercommonlyuseddetectorsbothundersimulation 2 Related work

1 andinexperimental3Dreconstructionofflatplateandcylindri-
v calobjects. There are various published techniques used for finding the in-
1
tersections in chess-board patterns, typically employed during a
9
4 Keywords: Chess-board corner detection; Feature extraction; cameracalibrationroutine,thoughrelativelylittleworkfocusses
5 Patternrecognition; Cameracalibration; Structuredlightsurface onanoptimaldetectorforsuchcommonlyusedfeatures. Asob-
. measurement;Photogrammetricmarkerdetection servedinSohetal.[1997],regardingcameracalibration:“itisof-
1
0 tenassumedthatthedetectionofsuchchartsormarkerswhichare
3 designedtoenhancetheirdetectabilityistrivial”. Theycontinue
1 Introduction
1 thatthisassumptionisill-advised,asgenericapproaches,suchas
:
v thatofCanny[1986], donotmakeuseofthespecificproperties
i Many applications in machine vision depend on having accu- of the features and are likely to suffer under sub-optimal condi-
X
rately localized the vertices of a chess-board pattern, since such tionsoflightingandobjectpose,andfurthermoretheyhighlight
ar patternsarecommonlyusedincameracalibration. Theavailable therisksofemployinglossy“remedies,suchaswidekernelfilter-
methodsforthisprocesstendtosufferinthefaceofsevereoptical ing,whicharenotoriousfordegradingthepositionalinformation
distortionandperspectiveeffects, andoftenrequirehand-tuning andshapeofcriticalfeatures”.
of parameters, depending on lighting and pattern scale. Manual
Sohetal. goontodescribeagridprocessingschemeusinga
interventionistimeconsuming,requiringoperatorskillandpro-
chain of Sobel operators, local thresholding, non-maximal sup-
hibitingautomateduse.
pression, edge joining, geometric constraints and finally taking
Anotherapplicationneedingprecisevertexdetectionis3Dsur- thecentresofgravityofthefoundsquares. AsdelaEscaleraand
face reconstruction, where a chess-board pattern is employed as Armingol [2010] noted, such localization methods were aban-
part of a simple yet accurate structured light projector-camera doned since the centres of gravity do not coincide with the cen-
tresofthesquaresduetoperspectiveeffects. DelaEscaleraand
∗S. Bennett and J. Lasenby are with Cambridge University Engineering
ArmingolalsomakeasimilarpointtoSohetal.,thatlessatten-
Department, Trumpington Street, Cambridge, CB2 1PZ, United Kingdom.
email:{sb476,jl221}@cam.ac.uk tion has been paid to locating the points used in calibration al-
1
There is a similar method, whose details are unpublished,
4 4 4 4 4 4 4 4 4
included in the Parallel Tracking and Mapping for Small AR
4 3 3 3 3 3 3 3 4
L1 Workspaces(PTAM)referenceimplementation(KleinandMur-
4 3 2 2 2 2 2 3 4
ray 2007). In this the circular sampling ring from the FAST
4 3 2 1 1 1 2 3 4
4 3 2 1 0 1 2 3 4 L2 detector (Rosten and Drummond 2005; Rosten and Drummond
4 3 2 1 1 1 2 3 4 2006)isused,andupperandlowerthresholdsareformedwhich
4 3 2 2 2 2 2 3 4 L3 are a fixed intensity distance from the mean of the sixteen sam-
4 3 3 3 3 3 3 3 4 pledpoints. Proceedingaroundtheringthenumberoftransitions
4 4 4 4 4 4 4 4 4 pastthesethresholdsarecounted,andiffoursuchtransitionsare
(b)Thelinearizedlayers1-3,start- foundthecentrepointisflaggedasapossiblegridcorner.
(a) The rectangular sampling win-
ing on the top horizontal row and The results of a variety of basic detection and refinement
dow,withlayersnumbered
workingclockwise
schemesareemployedinmanycameracalibrationpapers,awell
knownexamplebeingthatofZhang[2000],butareviewofsuch
Figure 1: Illustration of the Sun et al. [2008] sampling window
publicationsisbeyondthescopeofthiswork.
andlayerscheme
In general terms the Harris and Stephens detector is the one
encountered most frequently in the literature, other notable pa-
persemployingitincludingShuetal.[2003]andDouskosetal.
gorithmsthantothecalibrationalgorithmsthemselves–adeficit
[2007]. SeveralpaperssuchasthatofLuccheseandMitra[2002]
thisworkaimstoredress.
exist, but these detail refinement strategies to the features given
DelaEscaleraandArmingol’sdetectionschemeusestheHar-
by a Harris and Stephens detector. This paper aims to offer a
risandStephenscornerdetectortolocateagrid,beforeemploy-
competingsolutiontotheuseoftheHarrisandStephensdetector
ing the Hough transform on the image to enforce linearity con-
in chess-board applications which is intrinsically more accurate
straints and discard responses from the corner detector which
(yet also amenable to the use of similar subsequent refinement
do not lie along strong linear features. This then restricts the
strategiesiftheapplicationdemandsit).
method’s use in applications where the grid is potentially dis-
torted,beitduetoopticaldistortionoranon-planarsurface.They
discounttheuseofcornerpositionsalone(inasituationwherethe 3 Sampling strategy
gridboundaryisunknown)duetotheexcessivenumberofnon-
gridcornerslikelytobefoundbyHarrisandStephens’general-
Whenweconsideranoutlineforanefficientchess-boardcorner
purposealgorithmelsewhereinascene.
detector, an assumption of the squares of the pattern being ap-
YuandPeng[2006]describeanalternativemethodoffinding proximatelyaxis-alignedwiththecamera’ssensorwouldleadto
features,whichattemptstopattern-matchasmallimageofanin- adesignofverylowcomplexity,butthisisobviouslyanexcessive
tersectionbymeasuringthecorrelationofthispatternoverallthe restriction. Afurtherstepupthecomplexityscalewouldsuggest
capturedimage. Unlessanumberofsuchsmallimagesaretested analyzing the image for overall feature directionality and then
however,thismethodisclearlyatadisadvantagewhenthegridis proceed with a detector whose axes are aligned to the detected
rotatedrelativetotheintersectionviewstoredinthepattern. globalorientation,insomerespectssimilartodelaEscaleraand
FinallySunetal.[2008]detailamethodwheretheypassarect- Armingol’sscheme,butasnotedearlier,opticaldistortion,oruse
angularorcircularwindowoverthecapturedimageandforeach ofanon-planarsurfaceforthepattern,willleadtothegridbend-
position transform the 2D points distribution along the perime- ingsignificantly.Apartfromtheconsequencethattheremaythen
ter of this window into a 1D vector. For each ring concentric be no strong global orientation found, the wider implication is
withtheperimeteranothervectorisformedsimilarly,eachvector thatageneralpurposedetectormustcopewithfeaturesatallori-
being termed a ‘layer’, as numbered in Figure 1a and linearized entations. Intheinterestofconsistency,itisobviousthatsucha
in Figure 1b. The layers are binarized using a locally adaptive detectormuststrivetoawardthesamestrength(insomesense)to
threshold,openandclosemorphologicaloperationsapplied(Har- featureswhichareidenticalinallrespectsapartfromorientation.
alicketal.1987),andthepositionswheresomeproportionofthe For a rotationally invariant detector, we must sample enough
layers have four regions (when each layer is viewed as a ring) directionsawayfromthecentreofthefeaturetoproducearesult
are determined to be chess-board vertices. Sun et al. claim the reliable at any angle. In the instance of a chess-board vertex,
techniqueworkswell,butnotethatitproducesfalsecornersfrom thefeaturemayminimallybedescribedbyfindingapointwhere
noiseandisratherslow. Theschemealsoreliesonthethreshold- twosamplestakeninopposingdirectionsfromthefeaturecentre
ingproducinganacceptablebinaryresult. areofonesense(say,black)andanothertwosamplestakenata
2
ninetydegreerotationtothefirstpairhavetheoppositesense(say,
white). Thisgivesafourpointsamplingpattern,asinFigure2a,
whichmaybeviewedasacrosswithintersectiononthefeature
centre(thehatchedsquaresdenotingsampledpixels).
Atthispointwealsoconsiderthenatureofthedatacommonly
expected from a typical camera. The sampled pixels around the
edges of grid squares often take middling intensity values rela-
(a) Four point sampling: optimal(b) Four point sampling: pessimal
tivetotheextremalintensitiesofpixelssamplingtheinteriorsof
whensamplinginsidegridsquares whensamplinggridsquareedges
squares. Acoupleofreasonsforthisphenomenonaregivenbe-
low.
• Opticalblurduetobothimperfectfocusandimperfectop-
tics.
• Pixelquantization–asingleintensityvalueisassignedto
anareaof2Dopticalsignal, andtheedgesofpixelsonthe
(c)Eightpointsampling: adequate
sensor are seldom perfectly aligned with the edges of the
foranyvertexorientation
incominggridimage.
Figure 2: Illustration of lower bound on number of samples re-
It is clear therefore that should the sampling cross coincide
quiredtoidentifyvertex
with the edges of the grid squares the result will not be reliable
(Figure2b). Insuchacaseanothercrossin-fillingthefirstmust
be used to get a result, leading to a combined sampling pattern
ofeightpoints,asillustratedinFigure2c. Clearlythough,aver- Item1aboveinformsourdecisiontoensuredistancesfromthe
tex response in such a case is liable to, by some metric, have centre are approximately equal – if blur and quantization issues
half the magnitude of a response where the grid edges lie be- attenuatewithdistancefromthefeaturecentreitwouldbeunfair
tweenthearmsofthesamplingcrossesandalleightsamplescon- to have certain directions sampled further out than others, vio-
tributeconstructively. Samplingmoredirectionsamelioratesthis lating the condition that a feature’s response ought not to vary
unevenness,asmoresamplingpointsareliabletobeinanareaof with rotation. Approximately equal distances and equal angular
solidintensityratherthanonagridsquareedge,butatincreased spacingconstrainthesamplingpointstobearrangedinacircle,
computationalcostofprocessingtheextrasamples. centredonthefeaturecentre. Furthermore,takingthetwoitems
Havingobtainedalowerbound(ofeight;weassumenoFAST- above together, it is apparent that the radius of this circle ought
like per-pixel sampling decisions) on sampling points, the issue to be minimized (to avoid aliasing on to other grid squares, and
of sample positioning becomes relevant. In order for the fea- allowtheuseofmoredensepatternsifdesired), butbigenough
ture response (however it may be calculated) at any rotation to to escape the central region of blurriness. In some respects this
be approximately constant, it is vital for the sampling points to circle resembles Sun et al.’s outer ‘layer’ when using a circular
be spaced at equal angles incrementally about the feature cen- window, and indeed is quite similar to that used in the PTAM
tre. Then,consideringdistanceawayfromthefeaturecentre,two code.
thingsareapparent: Empirically, for the majority of data considered from VGA
(640 × 480 pixels) resolution cameras, a ring of radius 5 pix-
1. Pixelsclosetothevertexaremorelikelytocontainanedge
els(px)withsixteensamplesgivesagoodresponsewithoutcon-
thanthosefurtherout,asthecentralanglesubtendedbythe
straining the minimum chess-board square size unduly, and at
quasi-segment(ofapseudo-circlecentredonthevertex)en-
low computational expense. This circle also has the desirable
closing the pixel is much greater. Further away from the
property that the angular sample spacing closely approximates
vertex,pixelsaremorelikelytobeinareasofevenintensity,
the22.5degreeoptimalspacingofasixteensegmentcircle,with
intheinteriorsofthegridsquares. Notingthepreviousob-
samplingpointsspacedbyeither21.8°or23.2°(shownasα and
servationsonpixelsneargridsquareedges,samplingpixels
β inFigure3).
closetothecentrewillleadtoaweakerresponse.
Thesameangularspacingmaybeachievedwitharadius10px
2. Going too far away from centre risks sampling pixels from circle,anduseofsuchacirclemaybeappropriateinthecaseof
squares not forming the current feature, leading to a con- highlyblurredimages. Theultimatesizingoftheringisdepen-
fusedresponse. dentontheapplication’sopticalsystem. Withoutlossofgeneral-
3
α
β
Figure5: Thecaseofasimpleedge
Figure3: Similarityofr=5circle’ssamplingangles of points 90° out of phase on the circle should be of very dif-
ferent intensity to those at 0° and 180° phase, while being sim-
ilar to each other, as previously illustrated in Figure 2a. Tak-
ing I as the nth sampling point proceeding around the sam-
n
pling ring from some arbitrary starting point I , the magnitude
0
of (I +I )−(I +I ) should be very large when sam-
n n+8 n+4 n+12
pling around a vertex. The sum response (SR), so called due to
thesummationofoppositesamples,isthengivenby
3
SR= ∑|(I +I )−(I +I )| (1)
n n+8 n+4 n+12
n=0
andislargeatavertexpoint.
Figure4: Smallerringsinsidetheblurredregioncontributerela-
Themostcommonclassoffalsepositiveswhenusingadetec-
tivelylittletoimprovingtheresponse
torsimplyemployingthesumresponseisthatofthosethatoccur
along edges, though these are typically much smaller in magni-
ityaradius5pxringwillbeconsideredhenceforth,unlessstated tude than vertex responses. The origin of these may be simply
orillustratedotherwise. understoodbyimaginingacasewhereoneofthefoursampling
It is worth noting that employing inner rings, as in Figure 4, terms in the sum response, say In+8, is one, and the rest zero.
Thesesamplesareeasilyseentobeconsistentwithanedge,and
combiningaconcentricradius3samplingcirclewiththeradius5
a positive response still results (though half the magnitude that
circle,isnotuseful,asassumingtheoutermostringhasbeensized
wouldoccurifI werealsoone,beingthevertexcase).
appropriatelyfortheexpectedimageblurring,anyinnerringwill n
Noting that for a simple edge such as that shown in Figure 5
be sampling the blurry area and have little beneficial response.
(wherewithoutlossofgeneralityaradiusthreecircleisusedfor
Furthermore, such techniques slow the processing of the image.
illustrativepurposes),pointsonoppositesidesofthesamplecir-
With this observation our design departs significantly from Sun
cleshouldgenerallybeofdifferingintensities; thereforethediff
etal.’s,inthatitdoesnothavemultiple‘layers’.
response(DR),whichmaybeexpressedas
7
4 Detection algorithm DR= ∑|I −I |, (2)
n n+8
n=0
Rather than use the Sun et al./PTAM approach of performing a shouldbelargealongedges.
computationally intensive locally thresholded binarization, and Subtractingthediffresponsefromthesumresponseformsan
thenmakingaharddecisionwhetherasetofsamplesappearsto intermediate response with a much improved signal-to-noise ra-
beacornerornot,somewayofmeasuringsimilaritytoacorner tio. Considering the common example described above (one in
isdesirableinordertoprovidemoreinformationtolaterfeature four samples vastly different to the others) it may be seen that
consumers. This provides an output more similar to that of the theeffectofsubtractionistototallycancelthecontributionofthe
widelyusedHarrisandStephensdetectorthanthatofFAST.The sumresponse,givinganintuitivelycorrectintermediateresponse
calculationdetailedbelowprovidesthiscontinuousquantity. of zero. Later we will consider how the sum and diff responses
The initial grid vertex response is given by the sum response. maybeinterpretedusingananalogytotheDFT.
When centred on a chess-board vertex, points on opposite sides A final major false positive elimination is to remove the case
ofthesamplecircleshouldbeofsimilarintensities,andthepair where the sample circle covers a solid stripe, as in Figure 6b.
4
(a) A corner: high response desir-
(b)Astripe:rejectionrequired
able
Figure6:Twoverydifferentfeaturesthathavethesameresponse
onthesamplingring
(a)SecondDFTco-efficientcorner(b)FirstDFTco-efficientedgecor-
Observe that the circle’s samples for the corner feature shown
correlation relation
inFigure6awillbeexactlythesameasthoseforthestripe. The
twocasesmayonlybedistinguishedbytakingsampleselsewhere Figure7: 1DDFTcorrelationwithsamplecirclevectors
– a good location being at the centre of the ring, exploiting the
aforementionedexpectationofaregionofintermediateintensity
resultingfromblur. Bycomputingalocalintensitymeanwhich WetermthisalgorithmtheChESS(Chess-boardExtractionby
considersafew(say,foraradius5pxcircle,5)pixelsatthecentre SubtractionandSummation)detector.
ofthesamplingcircle,andalargerspatialmeanofallsamplesin
thering(neighbourmean),anabsolutedifferenceofmeans,mean
4.1 DFT based interpretation of the ChESS
response,canbefound:
detector
meanresponse=|neighbourmean−localmean|. (3)
Ifthesixteensamplestakenbythesamplingcirclearelinearized
This will be large in the stripe case: using Figure 6 as an ex- into a 1D data vector, it is seen that the FFT of this vector has
ample,theneighbourmeanwillbealightgreyinbothcases,as ahighabsolutevalueforthesecondco-efficientwhenthecircle
will the local mean in (a) – leading to a small mean response, is centred on a grid intersection, with the first co-efficient high
whereas for (b) the local mean will be much darker leading to whencentredoveranedge(thezerothtermbeingtheDCterm).
a large absolute difference. The mean response, multiplied by This makes intuitive sense when considered graphically. In
the number of sampled pixels (16), may be subtracted from the Figure7aacornerfeatureislinearizedclockwisefromthetopleft
existingresponse,yieldingtheoverallresponseR: sample,andtheintensityvaluescorrelatewellwithtwocyclesof
acosinewaveofsomephase,whichisakintotheDFT’ssecond
R=sumresponse−diffresponse−16×meanresponse. (4)
oscillatoryterm.LikewiseinFigure7bonecosinecycleissimilar
The factor of sixteen ensures a zero overall response in the to the intensity vector formed from an edge. By inspection it is
undesirablecase; thatwheresaysamplesI andI havevalue apparent that any rotation of the feature described will merely
n n+8
oneandI andI havevaluezero, asdoesthemeanofthe resultinachangeofphaseinthematchingcosine.
n+4 n+12
pixelscentredinthecircle(i.e. theFigure6bcase). Itmaynowbeseenthatthesumresponseattemptstoperforma
Thisoverallresponseisnotclaimedtobeperspective-invariant twocyclecosine-likematchtofindgridintersections,accumulat-
assuchaclaimwouldmakenosense–thedetectordoesnotknow ingoverfourphases. Similarly, thediffresponsematchesedges
if the image contains perspectively distorted chess-board inter- byamethodnotdissimilartoonematchingasinglecyclecosine
sections or merely features looking like perspectively distorted overeightphases.
chess-boardintersections. Henceahighlydistortedintersection,
whetherdistortedbyperspectiveeffectsorotherwise,willstillbe
assigned a strength, albeit one lower than that if the candidate 5 Feature selection
vertexwereviewed‘faceon’.
Anadditionalstagetoprovidelocalizedcontrast/responseen- As noted in the introduction to this paper, deciding which re-
hancementindarkerareasoftheimage,suchasbyper-pixeldi- sponsesaretobetreatedastruefeaturesislefttobedetermined
vision by the neighbour mean, is not employed in this detector, by application specific constraints. A few particular steps may
asthenoiseamplificationinlowintensityareasistoogreat. beofuseinmostinstanceshowever,exploitinglargerscalespa-
5
pixel axes (returning to the previous DFT analogy, this is deter-
miningaquantizedvalueforthetwo-cyclecosine’sphase). This
labellinghasmanypotentialuses,butatrivialexampleisthatin
finding chess-board vertices the orientation labels of connected
Figure 8: Responses with feature rotation of 0°, 11.25°, 22.5°, verticesoughttobeinapproximateanti-phase.Thedetailsofthis
33.75°,and45° labellingareexplainedbelow,separatelytothemaindetector,as
whiletheprocessescouldbeconductedsimultaneouslyitisoften
moreefficienttoonlyperformthelabellinghavingselectedaset
tialconstraintstoeliminatefalsepositivefeatures, andtheseare ofcandidatefeatures.
givenbelow. Thesamemeasure(M)asusedinthedetector’ssumresponse
is employed; M =(I +I )−(I +I ). |M| gives four
n n n+8 n+4 n+12
• Positiveresponsethreshold–discardresponsepixelswith
uniquevalueswhenrotatedaroundthe16-pointsamplingcircle,
zeroornegativeintensity(sincetheresponsequantityisde-
there being eight distinct values of M before duplication occurs
signedtoensureonlychess-boardintersectionshavepositive
duetosymmetry,andhalfofthoseeightsimplydifferinginsign
intensity).
depending on whether a given opposing pair of the four points
sampledareina“black”or“white”gridsquare.
• Nonmaximumsuppression–astandardtechniquetodis-
card non-maximal responses in a small area around each To achieve the first stage of the orientation binning, for each
pixeloftheresponseimage. Thismaybeusedtodetermine measure an average (AM) across those measures one orienta-
integerpixelco-ordinatesforasetofcandidatefeatures. tioneithersideofthecurrentmeasureisfound. Moreexplicitly,
3AM =M +M +M , n∈{0,1,2,3} (with care taken
n n−1 n n+1
• Responseconnectivity–truechess-boardvertexresponses when n−1 is −1 or n+1 is 4 to wrap modularly to 3 or 0 re-
typicallyspananumberofpixels;anytotallyisolatedposi- spectivelyandflipthesignoftheM inquestion). Theindexiof
tiveresponsepixelsmaybediscarded. the orientation which has the greatest absolute average measure
istaken,i.e. i=argmax |AM |. Tofindthefinalorientationbin
• Neighbourhood comparison – comparing the magnitudes n n
index, the sign of M is considered, with features with positive
i
ofmaximalresponsesoveralargearea,thoselessthansome
M beingconsignedtoadifferentfourorientationbinsthanthose
i
proportionofthegreatestresponses(whicharethoseoftrue
withanegativeM.
i
chess-board features) are viewed as false features and dis-
Becauseagridintersectionoftwoblackandtwowhitesquares
carded. In many respects this compensates for the lack of
has a rotational symmetry of order two, these eight bins corre-
intensity/contrastnormalizationinthedetector.
spondtoincrementsof22.5°.
Typicalresponsepatternsareshownforavarietyoffeaturero- Withregardtochess-boarddecoding, thislevelofgranularity
tationsinFigure8. Weobservethattheyaresymmetricalabout ensuresasatisfactorydistancebetweenalternatechess-boardver-
thefeaturecentreanditcanthereforebeseenthatforsub-pixello- tices, which can be viewed as 90° out of phase. With the 22.5°
calizationacentreofmasstechniquewillgivereasonableresults, distinction available here, two opposite sense features’ orienta-
a specific example being a 5 × 5 patch centred on the maximal tions can tolerate a distance-one (in bins) orientation labelling
pixel. This method is fast and in common use – de la Escalera errorwhileremainingdistinct.
andArmingol[2010]findthecentreofmassofeachofthepoints
resultingfromHarrisdetection,whileSunetal.[2008]alsofind
the centre of mass of their response clusters. More complex re- 7 Experimental results
finementtechniquessuchasthosetypicallyusedtopost-process
featuresdetectedwiththeHarrisandStephensdetectorcouldalso Insubstantiationoftheclaimsofrobustnessindetectionofchess-
beused,butarebeyondthescopeofthiswork’saimofdetecting board vertices, the ChESS detector must be compared against
featuresinitially. otherdetectorsinuseforthesameproblem.
6 Orientation labelling
7.1 Syntheticdata
Thesumresponsehasafurtheruse:byfindingtherotationaround Resiliencetonoiseandinvarianceofresponsemagnitudetorota-
the sampling ring at which the sum response is maximal, each tioncanbothbequantifiedbysimulatingafeaturepointatvary-
feature can be assigned one of eight orientations, relative to the ingrotationsandnoiselevels,performingfeaturedetectiononthe
6
generatedimage,subsequentlylocalizingthegreatestfeaturere-
sponseusinganumberofstrategies,andmeasuringthedistance
ofthispointfromtheco-ordinatesoftheoriginalsimulatedpoint.
Comparisons with the Harris and Stephens [1988] algorithm
and the SUSAN method (Smith and Brady 1997) are made be-
low, the SUSAN detector being another general purpose corner
detectorgivingaquantifiedfeatureresponsestrengthwithapub-
lished reference implementation. A further comparison is made (a)Capturedimage (b)Simulatedimage
againstthePTAMdetector,butthisisevaluatedseparatelydueto
thedetectoronlyprovidingabinaryresponse. Figure9: Realandsimulatedimagesofarotatedpoint
7.1.1 Simulationgeneration
It is desirable for the simulated images to bear a reasonable re-
semblancetorealdata,inorderfortheresultstobemeaningful.
Thesimulationimagesarethuscomposedoffourequalsizerect-
angles in two colours, arranged to define an intersecting point.
Sinceacommonimageformatofcameraoutputis1channelof8
bitsthetwocoloursaresetat64and191,approximatelyequidis-
(a)Capturedimage (b)Simulatedimage
tantfromsaturationandthemiddleoftheintensityrange.
The image is then rotated by some angle around the co- Figure 10: Real and simulated noisy images of a pixel grid-
ordinates of the intersection, using the ImageMagick library1 alignedpoint
withabi-linearinterpolationmethodspecified.Theimageisnext
cropped to VGA resolution, then a 3 × 3 Gaussian blur, using
two1Dpassesofa 1(cid:2)1 3 1(cid:3)filter,roughlycorrespondingtoa ting, and a noise variance of 5, and found to again be visually
5
0.675variance,isapplied. Thisproducesimagessimilartothose similar.
capturedbyawellfocussedVGAresolutioncamera.
Finally, noise, generatedfromrandomlysamplingaGaussian 7.1.2 Effectsofnoiseandrotationondetection
distributionwithaspecifiedvarianceisaddedtoeachpixelofthe
At a coarse level sub-pixel localization is unnecessary – simply
image,withsaturationoccurringatpixelintensitiesof0and255.
taking the integer pixel co-ordinates of the greatest response is
Avariantonthisapproachmayalsobesimulated. Themethod
sufficienttoprovideanoverallillustrationofbehaviour.Asimilar
describedabovemaybethoughtofasemulatingthecasewhere
connectivitymethodtothatdescribedinsection5isemployedto
the real-world edge is exactly incident on the edges of the cam-
provideaminimalfilter,discardingresponseswhicharenotcon-
era’s sensor elements (prior to rotation), but equally probable is
nectedtoanyoftheeightadjacentpixels(horizontally,vertically
thecaseoftheedgecoincidingwiththemiddleoftheelements.
anddiagonally).
Byinsertingatransitionrowandcolumnatmid-magnitude(128)
Figure 11 displays the performance of the three detectors un-
attherectanglebordersandrotatingaboutapointoffsetfromthe
der test. The “ChESS detector” is that detailed in the preceding
centre by half a pixel in x and y this case may also be tried. Of
sections;the“Harrisdetector”usesa5×5Sobelaperture,a3×
course,inrealitytheincidentedge’scentrewillfallbetweenthese
3blocksizeforthesubsequentboxfilter,andthefreeparameter
two cases, but together they ought to highlight any undesirable
k = 0.04; and the “SUSAN detector” uses Smith’s implementa-
pathologicalbehaviourpresentintheseextremecases.
tionofthisalgorithm2,withthebrightnessthresholdvalueat20.
In Figure 9b a portion of an image simulated with a rotation
TheHarrisparametershavebeenfoundempiricallytogivestrong
angle of 32.5° and a noise variance of 1 is shown. Figure 9a
responses on real data, and the SUSAN threshold is the default
showsasimilarportionofablackandwhiteintersectioncaptured
value. Thecolouringoftheplotscorrespondstothedistance(in
byarealcamera;thetwomaybeseentobevisuallysimilar.
pixels)ofthegreatestresponsedetectedfromwherethetruefea-
Similarly,Figure10a,animagecapturedwithashortexposure
turelies,i.e. themeasurederrorincreasesasthecolourchanges
(leading to higher noise), may be compared with Figure 10b, a
frombluetored.
portionofanimagesimulatedwithnorotation,half-pixeloffset-
2Available at http://users.fmrib.ox.ac.uk/~steve/susan/
1http://www.imagemagick.org/index.php susan2l.c
7
Distanceto ChESSdetector Harrisdetector, SUSANdetector, Distanceto
ChESSdetector Harrisdetector SUSANdetector centre(px) withpre-blur nosmoothing threshold=40 centre(px)
220 220
ntre40 40 40 200 ntre40 40 40 200
ce30 30 30 ce30 30 30
angle(°) Zerooffset120000 20 40 60 80 100120000 20 40 60 80 100120000 20 40 60 80 100 111124680000 angle(°) Zerooffset120000 20 40 60 80 100120000 20 40 60 80 100120000 20 40 60 80 100 111124680000
Rotation pixeloffsetcentre12340000 12340000 12340000 24681000000 Rotation pixeloffsetcentre12340000 12340000 12340000 24681000000
Half 00 20 40 60 80 100 00 20 40 60 80 100 00 20 40 60 80 100 0 Half 00 20 40 60 80 100 00 20 40 60 80 100 00 20 40 60 80 100 0
Noisevariance Noisevariance
Figure11: Basiccomparisonofdetectorperformanceatvarious Figure12:Furthercomparisonofvariantdetectorperformanceat
featureanglesandnoiselevels variousfeatureanglesandnoiselevels
It can immediately be seen that the new detector performs as default threshold, but still does not begin to compete with the
well or better than the Harris detector at all noise levels and ro- othertwodetectors,andinusewouldleadtoweakerfeaturesbe-
tations. Asexpectedthenewdetector’sresponsedisplaysperiod- ingmissed.WhiletheSUSANprincipleisintendedtonotrequire
icityabout22.5°,duetotheangularspacinginthesamplingring noise reduction, we note for completeness that from simulation
discussedinsection3. ItcanfurthermorebeseenthattheHarris not presented here, using the same pre-blur as employed with
detector’saccuracyvariesdependingonrotation–itisnoticeably the ChESS detector (and retaining the brightness threshold of
betteratzerorotationthanwhencloserto45°rotation. 40) merely improves the noise performance to being marginally
TheSUSANdetectorhasnotfarednearlyaswellastheother worsethanthesmoothedHarrisdetector;notadramaticimprove-
detectorsinthistestwithincreasingnoise;thelowdefaultbright- ment. For these reasons the SUSAN detector is not considered
ness threshold leads to noise features dominating at very low furtherinthisaccuracycomparison.
noiselevels. Itmayalsobeobservedthattherotationalresponse Looking in more detail at localization precision at low noise
isuneven. levels, Figure 13a and Figure 13b plot the error performance of
Consideringthedetectorsinmoredetail,somevariantsmustbe the remaining four detector schemes, using the 5 × 5 centre of
includedinthesimulationsforamoredirectcomparison.The5× mass sub-pixel localization method described previously, on the
5 Sobel operation applied in the Harris detector implementation same axes for varying noise (mean of all rotations), and rota-
hasaneffectofsmoothingtheinputimagebya5×5Gaussian tion (mean of low noise regions) respectively. Other localiza-
kernel. AstheChESSdetectorhasnosuchsmoothingstep,itis tionschemescouldbeemployed,butitisinformativetocompare
instructive to simulate two further variants: the ChESS detector the ability of the raw detection method without additional com-
processingimagessmoothedbya5×5Gaussiankernel(two1D plex refinement, not least to determine whether extensive post-
1 (cid:2)1 4 6 4 1(cid:3) (σ2 ≈ 1.04) passes), and a modified Har- processingisinfactnecessary.
16
ris detector with no initial smoothing. A further simulation of Figure13clearlyshowsthattheChESSdetectorvariantsper-
the SUSAN detector with a higher brightness threshold (40) is formbetterthantheHarrisdetectorsatallnoiselevels,andhave
alsowarrantedtoascertainitsperformancewhendetectingonly agoodandevenperformanceatallfeaturerotations. Theangular
strongfeatures. performanceofthetwonewdetectorvariantsiscomparable,but
The results of simulating these variants are plotted in Fig- thepre-blurredvariantismoreresilienttoimagenoise.
ure 12, again with the colour showing the error in terms of dis- ToformacomparisonagainstthePTAMdetector,whoseout-
tance. Blurringtheinputdatasignificantlyimprovesthenewde- putisaper-pixelBooleanresponseindicatingwhetheritisacor-
tector’s resilience to noise, allowing approximately double the nerfeature,theoutputoftheChESSdetectoristhresholded,with
noise variance before significant errors occur. Conversely, the thethresholdsetatapproximately1.5percentofthepositivere-
Harrisdetectorwithoutthepre-blurringstepbecomesevenmore sponse. Figure 14 presents the distance of the nearest detected
directional,andhasverypoornoiseperformance. corner feature from the true corner location, using a pixel grid
Setting the brightness threshold of the SUSAN detector to a alignedfeatureonly. Theplots’colourssaturateatadistanceof
largervalueyieldssomeimprovementinnoiseresilienceoverthe five pixels – any positive result detected a greater distance from
8
x) Distanceto
(p 2 ChESSdetector PTAMdetector centre(px)
centre 1.5 CCHhhaEErrSSisSSdddeeettteeeccctttooorrrwithpre-blur gle(°) 3400 3400 45
ue Harrisdetector,nosmoothing an 3
tr 1 on 20 20 2
om ati 10 10 1
efr 0.5 Rot 0 0 0
nc 0 20 40 60 80 100 0 20 40 60 80 100
Dista 00 5 10 15 20 25 30 35 40 45 50 55 60 Noisevariance
Noisevariance
Figure14: Basiccomparisonofbinaryresponsedetectorperfor-
(a)Performanceofthedetectorsforvaryingnoise(meanofallrotations)
manceatvariousfeatureanglesandnoiselevels
px) Pixel-gridalignedrotation Pixel-gridoffsetrotation Distanceto
e( 2 2 PTAMdetector,gate=20 PTAMdetector,gate=30 centre(px)
centr 1.5 1.5 e(°) 40 40 45
true 1 1 angl 30 30 3
from 0.5 0.5 ation 1200 1200 12
e ot
nc R 0 0 0
a 0 0 0 20 40 60 80 100 0 20 40 60 80 100
st 0 10 20 30 40 0 10 20 30 40
Di Rotationangle(°) Noisevariance
ChESSdetector ChESSdetectorwithpre-blur Figure 15: Further comparison of the variant PTAM detectors’
Harrisdetector Harrisdetector,nosmoothing
performancesatvariousfeatureanglesandnoiselevels
(b)Performanceofthedetectorsforvaryingrotation(meanoflownoise
regions)
be recorded. Hence while the simulated noise performance of
Figure13: Errorperformanceofthefourdetectorschemesusing the PTAM detector is seen to improve with a higher gate value,
5×5centreofmasssub-pixellocalizationmethodonthesame featureswouldhavetohaveblackandwhiteregionsdifferingin
axes intensity by more than forty (for gate=20) in order for the fea-
turetoberecorded,whereasthenewdetectorwillstillfindalow
contrastfeature,albeitwithasmallresponse.
the true feature is unlikely to be due to the true feature. By de-
faultthePTAMimplementationappliesaGaussianblurwithσ =
7.2 Realdata
1beforesamplingtheimage,soaGaussianblurwithsimilarσ is
appliedpriortoprocessingbyeitherdetectorinthecomparison.
Whileintheprevioussubsectionattentionwaspaidtotheaccu-
It can immediately be seen that in the simulation results the racyofthesimulation,itisneverthelesstruethatresultsobtained
PTAMdetectorfaresworsethantheChESSdetectorasthenoise throughsimulationdonotalwaysholdtrueinreality. Inthissub-
increases. A further poor performance region is visible under sectiontheaccuracyandrobustnessofthedetectorisvalidatedby
low noise conditions; this is due to the PTAM detector’s rejec- measuringtheerrorandconsistencyin3Dreconstructionofsur-
tionofcornerswhosecentralintensityissimilartothedetecting facesofknownshapeonwhichachess-boardpatternisprojected
region’smean, aninevitablesituationwiththesimulatedoptical andmultipleviewsofthesurfacerecorded(astandardStructured
bluracrosstheregionsoftwointensities. Lighttechnique). Theimportanceofaccuratelocalizationispar-
Considering only the comparatively weaker noise perfor- ticularly great as the extrinsic calibration of the cameras is per-
mance, like SUSAN the detector offers a parameter to recog- formedontheobserveddata,soanyerrorincalibrationresulting
nizeonlystrongerfeatures,thegatevalue. Plotsforgate=20and frompoorlocalizationwilltendtodegradethequalityofthere-
gate=30areshowninFigure15. constructionoverall.
Pixel intensities around the sampling ring must change by 2 Forthecameracalibrationandsurfacereconstructionthecom-
× gate in order for a white–black or black–white transition to bination of methods described in de Boer et al. [2010] (and in
9
more detail in de Boer [2010]) are used, drawing heavily on 100
LrealsiaebnlbeyanadndacSctuervaetnesoonna[2v0ar0i1e]ty. oTfhdeasteaihnavperevbieoeunsfsotuudnidest.o be sfully%) 80
ThereconstructiontestpermitsbothcomparisonoftheChESS cesd( 60
ce
feature detector against other detection schemes, and testing of suect 40
theChESSdetectorvariantsagainsteachother. Italsoconstructs ntsdet Cleandataset
theexperimentinsuchawayastotestthetwomostlikelyappli- Poi 20 Noisydataset
cations for this work, namely camera calibration and 3D recon- 0
Harrisdetector ChESSdetector ChESSdetector+blur
struction. Using a 5 × 5 centre of mass sub-pixel interpolation
methodineachcase,thevariantsundertestare: Figure16: Performanceofdetectorsontheflatplatedata
• Harrisdetector(parametersasusedinsimulation).
• ChESSdetectorwithoutpre-blur.
• ChESSdetectorwithpre-blur.
ThePTAMdetectorisnotconsideredhereduetoitsoutputnot
beingwellsuitedtosub-pixelfeaturelocalization.
Figure17:Illustrationofvariousfitstoastationaryflatplate,and
7.2.1 Flatplatecomparison
vectorsnormaltothefittedplanes
A flat plate is used to permit easy verification that the recon-
structed surface is planar. In the test a moving platform, whose
positionatanytimeispreciselyknown, movedtheplatetoward sinceweaimforadetectorthathasfewornooutliersitisfairto
andawayfromthecamerasoveratravellingdistanceofapprox- notmakeanyspecialefforttoexcludethemseparately.
imately95mm, whilethedistanceoftheplatefromthecameras The situation is illustrated in Figure 17 where a number of
wasaround1m. Followingthemotionperiodtheplatewasheld planes fitted to different frames captured during the rest period
at a constant displacement; the “rest” period. During the whole are depicted, along with their normal unit vectors. The dotted
recordedperiodover500projectedgridpointswereinviewand arrowshowsthemeannormalunitvector.
theseweresubsequentlyusedforcalibration. Foragoodstablefit,thedistanceofeachframe’svectorfrom
Therelativelylargemotionisintendedtoresultinanimproved thismeanvectorshouldbeverysmall,andinFigure18themean
calibration of the cameras’ extrinsic parameters, while the rest andvarianceofthesedistancesareplottedforallmethodsunder
period permits the plate’s reconstructed flatness to be compared test. The mean and variance over all frames of the SSE in each
overmanyframes,asnorealworldmotionispresent. frame’sfitarealsogiven. Thevaluesareplottedrelativetothose
Thefirstdatasetcontainsanoptimallylitandfocussedscene– oftheChESSdetectorwithoutpre-blurring,whichhasitsvalues
whatmightbeconsideredhighqualitydata. Figure16isaplotof normalized to one. To provide a sense of scale, the calculated
the percentages of grid points successfully found by each tested valuesforthisnormalizedcasehavethemeandistancefromthe
detectionmethod(irrespectiveoftheirpreciselocalization),with meannormalunitvector(expressedasanangleduetotheminute
theresultsfromthiswell-litdatasetgivenbythecross(×)mark- distancesinvolved)as0.131µrad,withavarianceof.0335µrad2,
ers. Itmayimmediatelybeseenthatforthis“clean”dataallthe and a mean fit SSE of 6.70mm2 with a variance of 0.357mm4
detectorsaresuccessful. overapatchof100points(perframe). Fromthisitcanbeseen
During the rest period a plane was fitted to the reconstructed thatinabsolutetermstheerrorisverylow–sub-millimetre.
surfaceofeachframe. Themethodemployedwasthatdescribed Compared to the detection success-rates, the fitting results
inMATLAB’sdocumentation(TheMathWorksInc.),wherealin- show greater variety in performance between the detectors, em-
ear regression that minimizes the perpendicular distances from phasizethepoorerlocalizationresultingfromtheHarrisdetector,
the data to the fitted model is found using Principle Component andreinforcetheconclusionfoundinsimulationthatuseofpre-
Analysis (PCA), forming a linear case of Total Least Squares. blurcanbebeneficialwhenusingthenewdetector.
Thisproducesaunitvectornormaltotheplane,andsummingthe Thecirclemarkers(◦)inFigure16andFigure19displaythe
squaredperpendiculardistancesfromthedatatothefittedplane sameinformationforaharderdataset, wherethelightlevelsare
asumofsquarederrors(SSE)iscalculated,indicatingthequality very low and hence the image noise level is much higher. The
of the fit. Any outliers will deteriorate the quality of the fit, but lighting difference between the datasets may be appreciated by
10