T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects

3.1 MB·

by  Tomas Hodan

T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects
Toma´sˇ Hodanˇ1,PavelHaluza1,Sˇteˇpa´nObdrzˇa´lek1,Jiˇr´ıMatas1,ManolisLourakis2,XenophonZabulis2
1CenterforMachinePerception,CzechTechnicalUniversityinPrague,CzechRepublic
2InstituteofComputerScience,FoundationforResearchandTechnology–Hellas,Heraklion,Greece
7
Abstract
1
0
We introduce T-LESS, a new public dataset for estimat-
2
ingthe6Dpose,i.e.translationandrotation,oftexture-less
n
rigid objects. The dataset features thirty industry-relevant
a
J objects with no significant texture and no discriminative
9 color or reflectance properties. The objects exhibit sym-
1 metriesandmutualsimilaritiesinshapeand/orsize. Com-
pared to other datasets, a unique property is that some of

]
theobjectsarepartsofothers.Thedatasetincludestraining
V
andtestimagesthatwerecapturedwiththreesynchronized
C
sensors, specifically a structured-light and a time-of-flight
.
s RGB-D sensor and a high-resolution RGB camera. There
c
are approximately 39K training and 10K test images from
[
eachsensor. Additionally,twotypesof3Dmodelsarepro-

1 vided for each object, i.e. a manually created CAD model
v
and a semi-automatically reconstructed one. Training im-
8
agesdepictindividualobjectsagainstablackbackground.
9
4 Testimagesoriginatefromtwentytestsceneshavingvary-
5 ing complexity, which increases from simple scenes with
0 severalisolatedobjectstoverychallengingoneswithmul-
.
1 tipleinstancesofseveralobjectsandwithahighamountof
0 clutterandocclusion.Theimageswerecapturedfromasys-
7 tematically sampled view sphere around the object/scene,
1 Figure 1. Examples of T-LESS test images (left) overlaid
andareannotatedwithaccurategroundtruth6Dposesof
: withcolored3Dobjectmodelsatthegroundtruth6Dposes
v allmodeledobjects. Initialevaluationresultsindicatethat
(right). Instances of the same object have the same color.
i
X thestateoftheartin6Dobjectposeestimationhasample
The goal is to find instances of the modeled objects and
roomforimprovement,especiallyindifficultcaseswithsig-
r estimatetheir6Dposes.
a nificant occlusion. The T-LESS dataset is available online
atcmp.felk.cvut.cz/t-less.
jectposecanbeusedtoenhanceone’sperceptionofreality
byaugmentingobjectswithextrainformationsuchashints
1.Introduction
forassemblyguidance.
Texture-lessrigidobjectsarecommoninhumanenviron- The visual appearance of a texture-less object is domi-
mentsandtheneedtolearn, detectandaccuratelylocalize natedbyitsglobalshape,color,reflectanceproperties,and
them from images arises in a variety of applications. The the configuration of light sources. The lack of texture im-
poseofarigidobjecthassixdegreesoffreedom,i.e. three pliesthattheobjectcannotbereliablyrecognizedwithtra-
in translation and three in rotation, and its full knowledge ditional techniques relying on photometric local patch de-
is often required. In robotics, for example, the 6D object tectors and descriptors [9, 31]. Instead, recent approaches
posefacilitatesspatialreasoningandallowsanend-effector that can deal with texture-less objects have focused on lo-
toactuponanobject. Inanaugmentedrealityscenario,ob- cal3Dfeaturedescription[33,51,19], andsemi-globalor
globaldescriptionrelyingprimarilyonintensityedgesand 1 2 3 4 5 6
depth cues [20, 24, 54, 5, 14, 21, 27]. Therefore, RGB-
D data consisting of aligned color and depth images, ob-
tainedwithwidelyavailableKinect-likesensors,havecome
7 8 9 10 11 12
toplayanimportantrole.
In this paper, we introduce a new public dataset for 6D
pose estimation of texture-less rigid objects. An overview
13 14 15 16 17 18
oftheincludedobjectsandtestscenesisprovidedinFig.2.
Thedatasetfeaturesthirtycommodityelectricalpartswhich
havenosignificanttexture, discriminativecolorordistinc-
19 20 21 22 23 24
tive reflectance properties, and often bear similarities in
shape and/or size. Furthermore, a unique characteristic of
theobjectsisthatsomeofthemarepartsofothers. Forex-
ample, objects7and8arebuiltupfromobject6, object9 25 26 27 28 29 30
is made of three copies of object 10 stacked next to each
other, whilst the center part of objects 17 and 18 is nearly
identicaltoobject13. Objectsexhibitingsimilarproperties
arecommoninindustrialenvironments. 1 2 3 4
The dataset includes training and test images captured
withatripletofsensors,i.e. astructuredlightRGB-Dsen-
sorPrimesenseCarmine1.09,atime-of-flightRGB-Dsen-
5 6 7 8
sorMicrosoftKinectv2,andanRGBcameraCanonIXUS
950 IS. The sensors were time-synchronized and had sim-
ilar perspectives. All images were obtained with an auto-
maticprocedurethatsystematicallysampledimagesfroma 9 10 11 12
view sphere, resulting in ~39K training and ~10K test im-
ages from each sensor. The training images depict objects
inisolationwithablackbackground,whilethetestimages
13 14 15 16
originate from twenty table-top scenes with arbitrarily ar-
ranged objects. Complexity of the test scenes varies from
thosewithseveralisolatedobjectsandacleanbackground
toverychallengingoneswithmultipleinstancesofseveral
17 18 19 20
objectsandwithahighamountofocclusionandclutter.Ad-
ditionally,thedatasetcontainstwotypesof3Dmeshmod-
elsforeachobject;onemanuallycreatedinCADsoftware
andonesemi-automaticallyreconstructedfromthetraining
Figure 2. T-LESS includes training images and 3D mod-
RGB-D images. All occurrences of the modeled objects
els of 30 objects (top) and test images of 20 scenes (bot-
in the training and test images are annotated with accurate
tom) – shown overlaid with colored 3D object models at
ground truth 6D poses; see Fig. 1 for their qualitative and
the ground truth poses. The images were captured from a
Sec.4.1fortheirquantitativeevaluation.
systematicallysampledviewspherearoundanobject/scene
Thedatasetisintendedforevaluatingvariousflavorsof
and are annotated with accurate ground truth 6D poses of
the 6D object pose estimation problem [23] and other re-
allmodeledobjects.
lated problems, such as 2D object detection [50, 22] and
objectsegmentation[49,17]. Sinceimagesfromthreesen-
sorsareavailable,onemayalsostudytheimportanceofdif-
ferentinputmodalitiesforagivenproblem. Anotheroption in complexity, so that it would provide different levels of
istousethetrainingimagesforevaluating3Dobjectrecon- difficultyandbereasonablyfuture-proof,i.e. solvable,but
struction methods [44], where the provided CAD models notsolvedbythecurrentstate-of-the-artmethods.Thediffi-
canserveasthegroundtruth. cultyofthedatasetfor6Dobjectposeestimationisdemon-
Our objectives in designing T-LESS were to provide a stratedbytherelativelylowperformanceofthemethodby
datasetofasubstantialbutmanageablesize,witharigorous Hodanˇ etal.[24]. Thismethodotherwiseachievesaperfor-
andcompletegroundtruthannotationthatisaccuratetothe mance close to the state of the art on the well-established
levelofsensorresolution,andwithasignificantvariability datasetofHinterstoisseretal.[20].
The remainder of the paper is organized as follows. models of another 6 textured objects and 170 test images
Sec. 2 reviews related datasets, Sec. 3 describes technical depictingtheobjectsplacedonakitchentable.
detailsoftheacquisitionandpost-processingoftheT-LESS The Challenge and Willow datasets [58], which were
dataset, Sec. 4 assesses the accuracy of the ground truth collectedforthe2011ICRASolutionsinPerceptionChal-
posesandprovidesinitialevaluationresults,andSec.5con- lenge, shareasetof35texturedhouseholdobjects. Train-
cludesthepaper. ingdataforeachobjectisgivenintheformof37RGB-D
training images that show the object from different views,
2.RelatedDatasets plus a color point cloud obtained by merging the training
images. The Challenge and Willow datasets respectively
First we review datasets for estimating the 6D pose of contain176and353testRGB-Dimagesofseveralobjects
specificrigidobjects, groupedbythetypeofprovidedim- insingleinstancesplacedontopofaturntable. TheWillow
ages, then we mention a few datasets designed for simi- datasetalsofeaturesdistractorobjectsandobjectocclusion.
larproblems. Ifnotstatedotherwise,thesedatasetssupply SimilaristheTUWdataset[1]thatpresents17texturedand
groundtruthannotationsintheformof6Dobjectposes. texture-less objects appearing in 224 test RGB-D images.
Insteadofaturntablesetup,imageswereobtainedbymov-
2.1.RGB-DDatasets
ingarobotaroundastaticclutteredenvironmentwithsome
objectsappearinginmultipleinstances.
Only a few public RGB-D datasets, from over one
hundred reported by Firman in [15], enable the evalua- The Rutgers dataset [37] is focused on perception for
tion of 6D object pose estimation methods. Most of the roboticmanipulationduringpick-and-placetasksandcom-
datasets reviewed in this section were captured with Mi- prises of images from a cluttered warehouse environment.
crosoft Kinectv1 or Primesense Carmine1.09, which rep- It includes color 3D mesh models for 24 mostly textured
resent the first generation of consumer-grade RGB-D sen- objectsfromtheAmazonPickingChallenge2015[11],that
sorsoperatingonthestructured-lightprinciple. Thedataset were captured in more than 10K test RGB-D images with
introduced in [17] was captured with Microsoft Kinectv2, variousamountsofocclusion.
whichisbasedonthetime-of-flightprinciple. Aldomaetal.[2]provide3Dmeshmodelswithoutcolor
For texture-less objects, the dataset of Hinterstoisser et information of 35 household objects that are both textured
al.[20]hasbecomeastandardbenchmarkusedinmostof andtexture-lessandareoftensymmetricandmutuallysim-
the recent work, e.g. [38, 4, 47, 24, 54]. It contains 15 ilarinshapeandsize. Thereare50testRGB-Dimagesof
texture-lessobjectsrepresentedbyacolor3Dmeshmodel. table-top scenes with multiple objects in single instances,
Eachobjectisassociatedwithatestsequenceconsistingof withnoclutterandvariouslevelsofocclusion.
~1200RGB-Dimages,eachofwhichincludesexactlyone The BigBIRD dataset [42] includes images of 125
instanceoftheobject.Thetestsequencesfeaturesignificant mostly textured objects that were captured in isolation on
2D and 3D clutter, but only mild occlusion, and since the aturntablewithmultiplecalibratedRGB-DandDSLRsen-
objects have discriminative color, shape and/or size, their sors.Foreachobject,thedatasetprovides600RGB-Dpoint
recognition is relatively easy. In the 6D localization prob- clouds, 600 high-resolution RGB images, and a color 3D
lem (where information about the number and identity of meshmodelreconstructedfromthepointclouds.SinceBig-
objectspresentintheimagesisprovidedbeforehand[23]), BIRD was acquired under very controlled conditions, it is
state-of-the-art methods achieve recognition rates that ex- not concerned with occlusions, clutter, lighting changes or
ceed95%formostoftheobjects. Brachmannetal.[4]pro- varying object-sensor distance. Georgakis et al. [17] pro-
videdadditionalgroundtruthposesforallmodeledobjects vide6735testRGB-Dimagesfromkitchenscenesinclud-
inoneofthetestsequencesfrom[20]. Thisextendedanno- ingasubsetoftheBigBIRDobjects. Groundtruthforob-
tation introduces challenging test cases with various levels jects in the test images is provided only in the form of 2D
ofocclusionandallowstheevaluationofmultipleobjectlo- boundingboxesand3Dpointlabeling.
calization,witheachobjectappearinginasingleinstance. Laietal.[29]createdanextensivedatasetwith300com-
Tejani et al. [47] presented a dataset with 2 texture-less mon household objects captured on a turntable from three
and 4 textured objects. For each object, a color 3D mesh elevations.Itcontains250KsegmentedRGB-Dimagesand
modelisprovidedtogetherwithatestsequenceofover700 22 annotated video sequences with a few hundred RGB-D
RGB-Dimages. Theimagesshowseveralobjectinstances framesineach.Groundtruthisprovidedonlyintheformof
with no to moderate occlusion, and with 2D and 3D clut- approximate rotation angles for training images and in the
ter. Doumanoglouetal.[14]provideadatasetwith183test formof3Dpointlabelingfortestimages.
imagesof2texturedobjectsfrom[47]that appearinmul- Schletteetal.[40]synthesizedRGB-Dimagesfromsim-
tiple instances in a challenging bin-picking scenario with ulated object manipulation scenarios involving 4 texture-
heavyocclusion. Furthermore,theyprovidecolor3Dmesh less objects from the Cranfield assembly benchmark [10].
Several small datasets that were used for evaluation of the
SHOT descriptor are provided by Salti et al. [39]. These
(cid:126)
datasetsincludesyntheticdataaswellasdataacquiredwith 3
aspacetime-stereomethodandanRGB-Dsensor.
2.2.Depth-onlyandRGB-onlyDatasets
(cid:126)
The depth-only dataset of Mian et al. [34] includes 3D 2
meshmodelsof5objectsand50testdepthimagesacquired
with an industrial range scanner. The test scenes contain
only the modeled objects that occlude each other. A sim-
ilar dataset is provided by Taati et al. [46]. The Desk3D (cid:126)
1
dataset [3] comprises of 3D mesh models for 6 objects
which are captured in over 850 test depth images with oc-
clusion,clutterandsimilarlylookingdistractorobjects.The
datasetwasobtainedwithanRGB-Dsensor,howeveronly
thedepthimagesarepubliclyavailable. Figure3. Acquisitionsetup: 1)turntablewithmarkerfield,
TheIKEAdatasetbyLimetal.[30]providesRGBim- 2)screenensuringablackbackgroundfortrainingimages,
ageswithobjectsbeingalignedwiththeirexactlymatched removed when capturing test images, 3) triplet of sensors
3D models. Crivellaro et al. [12] supply 3D CAD mod- attachedtoajigwithadjustabletilt.
els and annotated RGB sequences with 3 highly occluded
and texture-less objects. Mun˜oz et al. [36] provide RGB
sequences of 6 texture-less objects that are each imaged
inisolationagainstacleanbackgroundandwithoutocclu-
sion. Furthertotheabove,thereexistRGBdatasetssuchas
[13,50,38,25],forwhichthegroundtruthisprovidedonly
intheformof2Dboundingboxes.
2.3.DatasetsforSimilarProblems
The RGB-D dataset of Michel et al. [35] is focused on
articulated objects, where the goal is to estimate the 6D
pose of each object part, subject to the constraints intro-
ducedbytheirjoints. Therearealsodatasetsforcategorical
poseestimation.Forexample,the3DNet[55]andtheUoB-
Figure 4. Sample training (top) and test (bottom) im-
HOOC[53]containgeneric3DmodelsandRGB-Dimages
ages. Left: RGB-DimagesfromPrimesenseCarmine1.09.
annotated with 6D object poses. The UBC VRS [32], the
Middle: RGB-DimagesfromMicrosoftKinectv2. Right:
RMRC (a subset of NYU Depth v2 [41] with annotations
High-resolutionRGBimagesfromCanonIXUS950IS.For
derived from [18]), the B3DO [26], and the SUN RGB-
theRGB-Dimages,bottom-lefthalvesshowtheRGBcom-
D[43]provideno3Dmodelsandgroundtruthonlyinthe
ponentswhereasthetop-rightshowthedepthcomponents.
form of bounding boxes. The PASCAL3D+ [57] and the
ObjectNet3D [56] provide generic 3D models and ground
truth6Dposes,butonlyRGBimages.
The rest of the section describes the process of dataset
3.TheT-LESSDataset preparation,whichincludesimageacquisition,cameracal-
ibration,depthcorrection,3Dobjectmodelgenerationand
Comparedtotherevieweddatasets,T-LESSisuniquein
thegroundtruthposeannotation.
itscombinationofthefollowingcharacteristics. Itcontains
1) alarger numberof industry-relevant objects, 2) training
3.1.AcquisitionSetup
imagescapturedundercontrolledconditions,3)testimages
withlargeviewpointchanges,objectsinmultipleinstances, Thetrainingandtestimageswerecapturedwiththeaid
affected by clutter and occlusion; including test cases that ofthesetupshowninFig.3.Itconsistsofaturntable,where
arechallengingevenforthestate-of-the-artmethods,4)im- the imaged objects were placed, and a jig with adjustable
agescapturedwithasynchronizedandcalibratedtripletof tilt,towhichthesensorswereattached.Amarkerfieldused
sensors,5)accurategroundtruth6Dposesforallmodeled forcameraposeestimationwasaffixedtotheturntable.The
objects,and6)twotypesof3Dmodelsforeachobject. fieldwasextendedverticallytothesidesoftheturntableto
facilitate pose estimation at lower elevations. To capture images that are acquired with a dense sampling of view-
training images, the objects were placed in the middle of points, e.g. [13, 20, 38, 24]. To support such approaches,
the turntable and in the front of a black screen, which en- T-LESS offers training images of every object in isolation
suredauniformbackgroundatallelevations. Tointroduce fromafullviewsphere. Theseimageswereobtainedwith
anon-uniformbackgroundinthetestimages,asheetofply- a systematic acquisition procedure which uniformly sam-
woodwithmarkersatitsedgeswasplacedonthetopofthe pled elevation from 85◦ to −85◦ with a 10◦ step and the
turntable. In some scenes, the objects were placed on the completeazimuthrangewitha5◦ step. Viewsfromtheup-
topofotherobjects(e.g.books)togivethemdifferentele- per and lower hemispheres were captured separately, turn-
vationsandthusinvalidateagroundplaneassumptionthat ing the object upside down in between. In total, there are
might be made by an evaluated method. The depth of ob- 18×72 = 1296trainingimagesperobjectfromeachsen-
ject surfaces in the training and test images is in the range sor. Exceptionsareobjects19and20,forwhichonlyviews
0.53 − 0.92m, which is within the sensing ranges of the fromtheupperhemispherewerecaptured,specifically648
usedRGB-Dsensorsthatare0.35−1.4mforCarmineand images from elevation 85◦ to 5◦. These objects are hori-
0.5−4.5mforKinect. zontally symmetric at the pose in which they were placed
ontheturntable,thustheviewsfromtheupperhemisphere
3.2.CalibrationofSensors
aresufficienttocapturetheirappearance. Testsceneswere
capturedfromaviewhemispherewitha10◦ stepineleva-
Intrinsic and distortion parameters of the sensors were
tion(rangingfrom75◦ to15◦)anda5◦ stepinazimuth. A
estimated withthe standard checkerboard-based procedure
totalof7×72 = 504testimageswerecapturedperscene
using OpenCV [6]. The root mean square re-projection
byeachsensor.
error calculated at corners of the calibration checkerboard
squares is 0.51px for Carmine, 0.35px for Kinect, and Toremoveirrelevantpartsofthesceneintheimagespe-
0.43pxforCanon. FortheRGB-Dsensors,thecalibration riphery, the provided images are cropped versions of the
was performed with the RGB images. The depth images captured ones. Resolution of the provided images is as
were aligned to the RGB images using the factory depth- follows: 400 × 400px for training RGB-D images from
to-colorregistrationavailablethroughmanufacturer’sSDKs CarmineandKinect,1900×1900pxfortrainingRGBim-
(OpenNI2.2andKinectforWindowsSDK2.0). Thecolor agesfromCanon,720×540pxfortestRGB-Dimagesfrom
andaligneddepthimages,whichareincludedinthedataset, CarmineandKinectand2560×1920pxfortestRGBim-
are already processed to remove radial distortion. The in- agesfromCanon. SampleimagesareshowninFig.4.
trinsicparameterscanbefoundatthedatasetwebsite. Parts of the marker field were visible in some of the
All sensors were synchronized and extrinsically cali- trainingimages,especiallyatlowerelevations. Thesewere
brated with respect to the turntable, making it possible to masked to ensure a black background everywhere around
register any pair of images. Synchronization was essential the objects. To achieve this, we identified an object mask
since the images were taken while the turntable was spin- inanimagebyback-projectingitsCADmodelandgradu-
ning. The extrinsic calibration was achieved using fidu- ally darkened the image moving from the mask perimeter
cialBCH-codemarkersfromARToolKitPlus[52]. Specifi- towardstheimageborder.
cally,thedetectionofparticularmarkersinanimagecom-
3.4.DepthCorrection
binedwiththeknowledgeoftheirphysicallocationonthe
turntableprovidedasetof2D-3Dcorrespondences. These
Similarly to [16, 45], we observed that the depths mea-
were used to estimate the camera pose in the turntable co-
suredbytheRGB-Dsensorsexhibitasystematicerror. To
ordinate system by robustly solving the PnP problem and
remove it, we collected depth measurements d at projec-
then refining the estimated 6D pose by non-linearly min-
tions of the marker corners and computed their expected
imizing the cumulative re-projection error with the posest
depth values d from the known marker coordinates. The
e
library from [31]. The root mean square re-projection er-
measurements were collected from the depth range 0.53–
ror, which was calculated at marker corners in all test im-
0.92m in which the objects appear in the training and test
ages,is1.27pxforCarmine,1.37pxforKinect,and1.50px
images. We found the following linear correction models
forCanon. Thismeasurecombineserrorsinsensorcalibra-
byleastsquaresfitting:d =1.0247·d−5.19forCarmine,
c
tion,markerfielddetectionandsensorposeestimationand
and d = 1.0266·d−26.88 for Kinect (depth measured
c
is therefore larger than the aforementioned error in sensor
in mm). In [45], only scaling is usedfor the depth correc-
intrinsiccalibration.
tion. According to Foix et al. [16], a 3-degree polynomial
function suffices to correct depth in the 1–2m range. In
3.3.TrainingandTestImages
our case, a narrower range is used and we found a simple
Acommonstrategyfordealingwithpoorlytexturedob- linear polynomial to adequately account for the error: the
jectsistoadoptatemplate-basedapproachtrainedonobject correction reduced the mean absolute difference from the
expected depth d from 12.4mm to 2.8mm for Carmine
e
andfrom7.0mmto3.6mmforKinect. Theestimatedcor-
rectionwasappliedtoalldepthimages,requiringnofurther
actionfromthedatasetuser.
3.5.3DObjectModels
For each object, a manually created CAD model and
a semi-automatically reconstructed model are available
(Fig. 5). Both models are provided in the form of 3D
meshes with surface normals at model vertices. Surface
color is included only for the reconstructed models. The
normals were calculated using MeshLab [7] as the angle-
weightedsumoffacenormalsincidenttoavertex[48].
Thereconstructedmodelswerecreatedusingfastfusion,
Figure5. Examplesof3Dobjectmodels. Top: Manually
avolumetric3DmappingsystembySteinbru¨ckeretal.[44].
created CAD models. Bottom: Semi-automatically recon-
The input to fastfusion were the RGB-D training images
structed models which include also surface color. Surface
from Carmine and the associated camera poses estimated
normalsatmodelverticesareincludedinbothmodeltypes.
using the fiducial markers (see Sec. 3.2). For each object,
twopartialmodelswerefirstreconstructed,onefortheup-
perandanotherforthelowerviewhemisphere. Thepartial
3.6.GroundTruthPoses
models were then aligned using the iterative closest point
(ICP)algorithmappliedtotheirvertices.Thiswasfollowed
To obtain ground truth 6D object poses for images of a
by manual refinement that ensured correct registration of
test scene, a dense 3D model of the scene was first recon-
surface details that are visible only in color. The result-
structed with the system of Steinbru¨cker et al. [44]. This
ing alignment was applied to the camera poses to trans- wasaccomplishedusingall504RGB-Dimagesofthescene
form them into a common reference frame, and the up-
along with the sensor poses estimated using the turntable
dated poses were used to reconstruct the full object model
markers. The CAD object models were then manually
fromallimages. Thesemodelscontainedsomeminorarti-
alignedtothescenemodel. Toincreaseaccuracy,theobject
facts, e.g. small spikes, which were removed manually. It
modelswererenderedintoseveralselectedhigh-resolution
isnotedthatsomeoftheobjectscontainsmallshinymetal
scene images from Canon, misalignments were identified
parts whose depth is not reliably captured by the current
andtheposesweremanuallyrefinedaccordingly. Thispro-
depthsensors;ingeneral,anyglossyortranslucentsurface
cesswasrepeateduntilasatisfactoryalignmentoftheren-
isproblematic. Hence,someoftheseparts,suchastheplug
deringswiththesceneimageswasachieved.Thefinalposes
poles,werenotreconstructed.
weredistributedtoalltestimageswiththeaidoftheknown
ThereconstructedmodelswerealignedtotheCADmod- camera-to-turntablecoordinatetransformations. Thetrans-
els using the ICP algorithm and the alignment was fur- formed poses are provided as the ground truth poses with
ther refined manually. Models of both types are therefore eachtestimage.
defined in the same coordinate system and the provided
ground truth poses are valid for both of them. The origin 4.DesignValidationandExperiments
ofthemodelcoordinatesystemcoincideswiththecenterof
theboundingboxoftheCADmodel. This section presents an accuracy assessment of the
The geometrical similarity of the two model types was ground truth poses and examines the difficulty of T-LESS
assessed by calculating the average distance from vertices witharecent6Dlocalizationmethod.
of the reconstructed models to the closest surface points
4.1.AccuracyoftheGroundTruthPoses
of the corresponding CAD models. The average distance
over all object models was found to be 1.01mm, which is Aiming to evaluate the accuracy of the ground truth
very low compared to the size of objects that ranges from poses, we compared the captured depth images, after the
58.13mm for object 13 to 217.16mm for object 8. Dis- correctiondescribedinSec.3.4,withdepthimagesobtained
tancesintheoppositedirection,i.e. fromtheCADmodels bygraphicallyrenderingthe3Dobjectmodelsattheground
tothereconstructedmodels,arenotinformativesincesome truthposes. Ateachpixelwithavaliddepthvalueinboth
CADmodelscontaininnerpartsthatarenotrepresentedin images, we calculated the difference δ = d −d , where
c r
the reconstructed models. The Metro software by Cignoni d is the captured and d is the rendered depth. Table 1
c r
etal.[8]wasusedtomeasurethemodeldifferences. presents statistics of these differences, aggregated over all
trainingandtestdepthimages. Differencesexceeding5cm Sensor,modeltype µδ σδ µ|δ| med|δ|
and amounting to around 2.5% of the measurements were Carmine,CAD -0.60 8.12 4.53 2.57
considered to be outliers and were pruned before calculat- Carmine,reconst. -0.79 7.72 4.28 2.46
ing the statistics. The outlying differences may be caused Kinect,CAD 4.46 11.76 8.76 5.67
byerroneousdepthmeasurements,orbyocclusioninduced Kinect,reconst. 4.08 11.36 8.40 5.45
bydistractorobjectsinthecaseoftestimages.
Table1. Statisticsofdifferencesbetweenthedepthofob-
Therendereddepthsalignwellwiththedepthscaptured
jectmodelsatthegroundtruthposesandthecaptureddepth
by Carmine, as indicated by the mean difference µ being
δ
(inmm). µ andσ isthemeanandthestandarddeviation
close to zero. In the case of Kinect, we observed that the δ δ
ofthedifferences,µ andmed isthemeanandtheme-
RGBanddepthimagesareslightlymisregistered,whichis |δ| |δ|
dianoftheabsolutedifferences.
the cause of the positive bias in µ . The average absolute
δ
differenceµ islessthan5mmforCarmineand9mmfor
|δ|
Kinect,whichisneartheaccuracyofthesensors[28]andis
100
relatively small compared to the size of objects. The error
80
statistics are slightly favorable for the reconstructed mod- %]
els(asopposedtotheCADmodels),astheywereobtained all [ 60
c 40
e
fromthecaptureddepthimagesandthereforeexhibitsimi- R
20
larcharacteristicsandartifacts. Forexample,theplugpoles
0
1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930
are invisible to the RGB-D sensors and are missing in the Object ID
reconstructedmodels,butarepresentintheCADmodels.
100
4.2.6DLocalization %] 80
The recent template-based method of Hodanˇ et al. [24] call [ 4600
e
wasevaluatedonthe6Dlocalizationproblem. Theinputis R 20
comprisedofatestimagetogetherwiththeidentitiesofob- 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
jectinstancesthatarepresentintheimage,andthegoalisto Scene ID
estimatethe6Dposesoftheseinstances[23]. Themethod 100
wasevaluatedonalltestRGB-DimagesfromtheCarmine 80
steemnspolra.teTshweepraeragmeneeterarstewdefrreomsetthaes dtreasicnriinbgediminag[e2s4]f,rothme call [%] 4600
e
R
Carmine, andtheCADmodelswereemployedinthepose 20
refinement stage as detailed in [59]. Pose estimates were 0
0 20 40 60 80 100
evaluated as in [20], using the average distance error for Visible part of object surface [%]
objects with indistinguishable views. This error measures
Figure6. PerformanceofthemethodbyHodanˇ etal.[24]
the misalignment between the surface of model M at the
on the 6D localization problem. Shown are the recall per
ground truth pose (R¯,¯t) and at the estimated pose (Rˆ,ˆt),
object(top), recallperscene(middle), andrecallw.r.t. the
andisdefinedas:
percentageofthevisibleobjectsurface(bottom).
(cid:13) (cid:13)
e= avg min (cid:13)(cid:0)R¯x +¯t(cid:1)−(cid:0)Rˆx +ˆt(cid:1)(cid:13) .
(cid:13) 1 2 (cid:13)
x1∈Mx2∈M 2
Pose estimate (Rˆ,ˆt) is considered correct if e ≤ k·d, contains many similar objects and severe occlusions. Bot-
wherek =0.1anddisthelargestdistancebetweenanypair tomofFig.6plotstherecall,accumulatedoverallobjects,
ofmodelvertices,i.e. theobjectdiameter. Onlytheground as a function of the fraction of their image projection that
truth poses at which at least 10% of the object surface is isunoccluded. Therecallincreasesproportionallywiththis
visible were considered for the evaluation. The visibility fraction,illustratingthatocclusionisoneofthemainchal-
wasestimatedasin[23]. lengesinT-LESS.
Theperformanceismeasuredbyrecall,i.e.thepercent- The achieved mean recall over all objects is 67.2%,
ageofthegroundtruthposesforwhichacorrectposewas which suggests a significant margin for improvement. We
estimated. Fig. 6 presents achieved recall per object (top) notethatthesamemethodachievedameanrecallof95.4%
and recall per scene (middle). The objects with the lowest on the dataset of Hinterstoisser et al. [20], which is close
recallarethosethataresimilartootherobjects. Forexam- to the state of the art: [20] reports 96.6% and [5] reports
ple,object1isoftenconfusedwithobject2,asareobjects 99.0% on this dataset. The latter is not directly compara-
20,21and22. Likewise,testscenescontainingsimilarob- ble since it was calculated only over 13 out of 15 objects
jects are harder, with the hardest one being scene 20 that includedinthedataset.
5.Conclusion [9] A. Collet, M. Martinez, and S. S. Srinivasa. The MOPED
framework: Objectrecognitionandposeestimationforma-
ThispaperhaspresentedT-LESS,anewdatasetforeval-
nipulation. IJRR,2011.
uating 6D pose estimation of texture-less objects that can
[10] K. Collins, A. Palmer, and K. Rathmill. The development
facilitatesystematiccomparisonofpertinentmethods. The of a European benchmark for the comparison of assembly
dataset features industry-relevant objects and is character- robotprogrammingsystems.InRobottechnologyandappli-
ized by a large number of training and test images, accu- cations.1985.
rate6Dgroundtruthposes,multiplesensingmodalities,test [11] N.Correll,K.E.Bekris,D.Berenson,O.Brock,A.Causo,
scenes with multiple object instances and with increasing K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and
difficultyduetoocclusionandclutter. Initialevaluationre- P.R.Wurman.LessonsfromtheAmazonpickingchallenge.
sultsusingthedatasetindicatethatthestateoftheartin6D arXivpreprintarXiv:1601.05484,2016.
objectposeestimationhasampleroomforimprovement. [12] A. Crivellaro, M. Rad, Y. Verdie, K. M. Yi, P. Fua, and
V.Lepetit. Anovelrepresentationofpartsforaccurate3D
TheT-LESSdatasetisavailableonlineat: objectdetectionandtrackinginmonocularimages.InICCV,
cmp.felk.cvut.cz/t-less 2015. cvlab.epfl.ch/data/3d object tracking.
[13] D. Damen, P. Bunnun, A. Calway, and W. Mayol-Cuevas.
Acknowledgements Real-timelearninganddetectionof3Dtexture-lessobjects:
Ascalableapproach. InBMVC,2012.
ThisworkwassupportedbytheTechnologyAgencyof [14] A. Doumanoglou, R. Kouskouridas, S. Malassiotis,
theCzechRepublicresearchprogramTE01020415(V3C– and T.-K. Kim. Recovering 6D object pose and pre-
VisualComputingCompetenceCenter),CTUstudentgrant dicting next-best-view in the crowd. In CVPR, 2016.
www.iis.ee.ic.ac.uk/rkouskou/research/
SGS15/155/OHK3/2T/13, and the European Commission
6D NBV.html.
FP7 DARWIN Project, Grant No. 270138. The help of
[15] M. Firman. RGBD datasets: Past, present and future.
Jan Pola´sˇek and Avgousta Hatzidaki in creating the CAD
arXiv:1604.00999,2016.
modelsisgratefullyacknowledged.
[16] S. Foix, G. Alenya, and C. Torras. Lock-in time-of-flight
(ToF)cameras:asurvey. SensorsJournal,2011.
References
[17] G. Georgakis, M. A. Reza, A. Mousavian, P.-H. Le, and
[1] A. Aldoma, T. Fa¨ulhammer, and M. Vincze. Au- J. Kosecka. Multiview RGB-D dataset for object in-
tomation of “ground truth” annotation for multi-view stance detection. arXiv preprint arXiv:1609.07826, 2016.
RGB-D object instance recognition datasets. In IROS, cs.gmu.edu/˜robot/gmu-kitchens.html.
2014. repo.acin.tuwien.ac.at/tmp/permanent/ [18] R. Guo and D. Hoiem. Support surface prediction in in-
dataset index.php. door scenes. In ICCV, 2013. ttic.uchicago.edu/
[2] A. Aldoma, F. Tombari, L. Di Stefano, and M. Vincze. A ˜rurtasun/rmrc/indoor.php.
globalhypothesesverificationmethodfor3Dobjectrecog- [19] Y.Guo,M.Bennamoun,F.Sohel,M.Lu,J.Wan,andN.M.
nition. In ECCV, 2012. users.acin.tuwien.ac.at/ Kwok.Acomprehensiveperformanceevaluationof3Dlocal
aaldoma/datasets/ECCV.zip. featuredescriptors. IJCV,2016.
[3] U. Bonde, V. Badrinarayanan, and R. Cipolla. Robust [20] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski,
instance recognition in presence of occlusion and clut- K. Konolige, and N. Navab. Model based training, detec-
ter. In ECCV, 2014. sites.google.com/site/ tionandposeestimationoftexture-less3Dobjectsinheav-
ujwalbonde/publications/downloads. ilyclutteredscenes.InACCV,2012.campar.in.tum.de/
[4] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shot- Main/StefanHinterstoisser.
ton, and C. Rother. Learning 6D object pose estimation [21] S.Hinterstoisser,V.Lepetit,N.Rajkumar,andK.Konolige.
using 3D object coordinates. In ECCV, 2014. cvlab- Goingfurtherwithpointpairfeatures. InECCV,2016.
dresden.de/iccv2015-occlusion-challenge. [22] T.Hodanˇ,D.Damen,W.Mayol-Cuevas,andJ.Matas. Ef-
[5] E.Brachmann,F.Michel,A.Krull,M.Y.Yang,S.Gumhold, ficient texture-less object detection for augmented reality
and C. Rother. Uncertainty-driven 6D pose estimation of guidance. InISMARW,2015.
objectsandscenesfromasingleRGBimage. 2016. [23] T.Hodanˇ,J.Matas,andSˇ.Obdrzˇa´lek. Onevaluationof6D
[6] G. Bradski and A. Kaehler. Learning OpenCV: Computer objectposeestimation. InECCVWorkshoponRecovering
visionwiththeOpenCVlibrary.O’ReillyMedia,Inc.,2008. 6DObjectPose,2016.
[7] P.Cignoni,M.Callieri,M.Corsini,M.Dellepiane,F.Ganov- [24] T. Hodanˇ, X. Zabulis, M. Lourakis, Sˇ. Obdrzˇa´lek, and
elli,andG.Ranzuglia. MeshLab:anopen-sourcemeshpro- J.Matas. Detectionandfine3Dposeestimationoftexture-
cessingtool. InEurographicsItalianChapterConf.,2008. lessobjectsinRGB-Dimages. InIROS,2015.
[8] P.Cignoni,C.Rocchini,andR.Scopigno.Metro:measuring [25] E. Hsiao and M. Hebert. Occlusion reasoning for ob-
erroronsimplifiedsurfaces. InComputerGraphicsForum, ject detection under arbitrary viewpoint. TPAMI, 2014.
volume17,pages167–174.WileyOnlineLibrary,1998. www.cs.cmu.edu/˜./hebert/occarbview.html.
[26] A. Janoch, S. Karayev, Y. Jia, J. T. Barron, M. Fritz, [42] A.Singh, J.Sha, K.S.Narayan, T.Achim, andP.Abbeel.
K. Saenko, and T. Darrell. A category-level 3D object BigBIRD:Alarge-scale3Ddatabaseofobjectinstances. In
dataset: Putting the Kinect to work. In Consumer Depth ICRA,2014. rll.berkeley.edu/bigbird.
CamerasforComputerVision.2013. kinectdata.com. [43] S. Song, S. P. Lichtenberg, and J. Xiao. Sun RGB-D: A
[27] W.Kehl,F.Milletari,F.Tombari,S.Ilic,andN.Navab.Deep RGB-D scene understanding benchmark suite. In CVPR,
learningoflocalRGB-Dpatchesfor3Dobjectdetectionand 2015. rgbd.cs.princeton.edu.
6Dposeestimation. InECCV,2016. [44] F. Steinbru¨cker, J. Sturm, and D. Cremers. Volumetric
[28] K.KhoshelhamandS.Elberink. Accuracyandresolutionof 3D mapping in real-time on a CPU. In ICRA, 2014.
Kinectdepthdataforindoormappingapplications. Sensors, github.com/tum-vision/fastfusion.
2012. [45] J.Sturm,N.Engelhard,F.Endres,W.Burgard,andD.Cre-
[29] K.Lai,L.Bo,X.Ren,andD.Fox.Alarge-scalehierarchical mers. A benchmark for the evaluation of RGB-D SLAM
multi-viewRGB-Dobjectdataset. InICRA,2011. rgbd- systems. InIROS,2012.
dataset.cs.washington.edu.
[46] B. Taati, M. Bondy, P. Jasiobedzki, and M. Greenspan.
[30] J. J. Lim, H. Pirsiavash, and A. Torralba. Parsing Variable dimensional local shape descriptors for ob-
IKEA Objects: Fine Pose Estimation. In ICCV, 2013. ject recognition in range data. In ICCV, 2007.
ikea.csail.mit.edu. rcvlab.ece.queensu.ca/˜qridb/lsdPage.html.
[31] M. Lourakis and X. Zabulis. Model-based pose estima-
[47] A. Tejani, D. Tang, R. Kouskouridas, and T.-K. Kim.
tion for rigid objects. In Computer Vision Systems. 2013.
Latent-classhoughforestsfor3Dobjectdetectionandpose
www.ics.forth.gr/˜lourakis/posest. estimation. In ECCV, 2014. www.iis.ee.ic.ac.uk/
[32] D. Meger and J. J. Little. Mobile 3D object detection in rkouskou/research/LCHF.html.
clutter.InIROS,2011.www.cs.ubc.ca/labs/lci/vrs.
[48] G.Thu¨rrnerandC.A.Wu¨thrich. Computingvertexnormals
[33] A.Mian,M.Bennamoun,andR.Owens. Ontherepeatabil-
frompolygonalfacets. JournalofGraphicsTools,1998.
ityandqualityofkeypointsforlocalfeature-based3Dobject
[49] F.Tombari,L.DiStefano,andS.Giardino. Onlinelearning
retrievalfromclutteredscenes. IJCV,2010.
forautomaticsegmentationof3Ddata. In2011IEEE/RSJ
[34] A. S. Mian, M. Bennamoun, and R. Owens. Three-
InternationalConferenceonIntelligentRobotsandSystems,
dimensional model-based object recognition and
pages4857–4864.IEEE,2011.
segmentation in cluttered scenes. TPAMI, 2006.
[50] F.Tombari,A.Franchi,andL.DiStefano. BOLDfeatures
staffhome.ecm.uwa.edu.au/˜00053650/
todetecttexture-lessobjects. InICCV,2013.
recognition.html.
[51] F.Tombari,S.Salti,andL.DiStefano.Uniquesignaturesof
[35] F. Michel, A. Krull, E. Brachmann, M. Y. Yang,
histogramsforlocalsurfacedescription. InECCV,2010.
S. Gumhold, and C. Rother. Pose estimation of kine-
[52] D. Wagner and D. Schmalstieg. ARToolKitPlus for pose
matic chain instances via object coordinate regression.
trackingonmobiledevices. InCVWW,2007.
In BMVC, 2015. cvlab-dresden.de/iccv2015-
articulation-challenge. [53] K. Walas and A. Leonardis. UoB highly occluded object
[36] E.Mun˜oz,Y.Konishi,V.Murino,andA.D.Bue. Fast6D challenge II, 2016. www.cs.bham.ac.uk/research/
pose estimation for texture-less objects from a single RGB projects/uob-hooc.
image. In ICRA, 2016. www.iit.it/datasets/vgm- [54] P.WohlhartandV.Lepetit. Learningdescriptorsforobject
6d-pose-of-texture-less-objects-dataset. recognitionand3Dposeestimation. InCVPR,2015.
[37] C. Rennie, R. Shome, K. E. Bekris, and A. F. D. [55] W. Wohlkinger, A. Aldoma, R. B. Rusu, and M. Vincze.
Souza. A dataset for improved RGBD-based ob- 3DNet:Large-scaleobjectclassrecognitionfromCADmod-
ject detection and pose estimation for warehouse pick- els. In ICRA, 2012. repo.acin.tuwien.ac.at/tmp/
and-place. RA-L, 2016. www.pracsyslab.org/ permanent/3d-net.org.
rutgers apc rgbd dataset. [56] Y. Xiang et al. ObjectNet3D: A large scale
[38] R.Rios-CabreraandT.Tuytelaars. Discriminativelytrained database for 3D object recognition. In ECCV, 2016.
templatesfor3Dobjectdetection: Arealtimescalableap- cvgl.stanford.edu/projects/objectnet3d.
proach. InICCV,2013. [57] Y.Xiang,R.Mottaghi,andS.Savarese. BeyondPASCAL:
[39] S. Salti, F. Tombari, and L. D. Stefano. SHOT: Unique Abenchmarkfor3Dobjectdetectioninthewild. InWinter
signaturesofhistogramsforsurfaceandtexturedescription. ConferenceonApplicationsofComputerVision,2014.
CVIU. www.vision.deis.unibo.it/research/80- [58] Z. Xie, A. Singh, J. Uang, K. S. Narayan, and P. Abbeel.
shot. Multimodalblendingforhigh-accuracyinstancerecognition.
[40] C.Schletteetal. Anewbenchmarkforposeestimationwith InIROS,2013. rll.berkeley.edu/2013 IROS ODP.
ground truth from virtual reality. Production Engineering, [59] X.Zabulis,M.Lourakis,andP.Koutlemanis.3Dobjectpose
2014. www.mmi.rwth-aachen.de/exchange/data/ refinementinrangeimages. InICVS,2015.
pesi2014/benchmark.htm.
[41] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor
segmentationandsupportinferencefromRGBDimages. In
ECCV, 2012. cs.nyu.edu/˜silberman/datasets/
nyu depth v2.html.