Teaching Robots to Do Object Assembly using Multi-modal 3D Vision

2.7 MB·

by  Weiwei Wan

Teaching Robots to Do Object Assembly using Multi-modal 3D Vision
WeiweiWana,FengLub,ZepeiWua,KensukeHaradaa
aNationalInstituteofAdvancedIndustrialScienceandEngineering,Japan
bBeihangUniversity,China
Abstract
The motivation of this paper is to develop a smart system using multi-modal vision for next-generation mechanical assembly. It
includestwophaseswhereinthefirstphasehumanbeingsteachtheassemblystructuretoarobotandinthesecondphasetherobot
findsobjectsandgraspsandassemblesthemusingAIplanning.Thecrucialpartofthesystemistheprecisionof3Dvisualdetection
6andthepaperpresentsmulti-modalapproachestomeettherequirements: ARmarkersareusedintheteachingphasesincehuman
1beingscanactivelycontroltheprocess. Pointcloudmatchingandgeometricconstraintsareusedintherobotexecutionphaseto
0avoid unexpected noises. Experiments are performed to examine the precision and correctness of the approaches. The study is
2
practical: Thedevelopedapproachesareintegratedwithgraphmodel-basedmotionplanning,implementedonanindustrialrobots

nandapplicabletoreal-worldscenarios.
a
JKeywords:

63DVisualDetection,RobotManipulation,MotionPlanning
2

]
O1. Introduction humanbeingswhoareintelligentenoughtoactivelyavoidoc-
clusionsandshowgoodfeaturestovisionsystems. Themodal
R
employedinthisphaseisthemarkers(rgbimage)andthegeo-
. Themotivationofthispaperistodevelopasmartsystemus-
s metricrelationbetweenthemarkersandtheobjectmodels.The
ingmulti-modalvisionfornext-generationmechanicalassem-
c
tag“AR(RGB)”isusedforrepresentation.
[bly: Ahumanworkerassemblesmechanicalpartsinfrontofa

visionsystem; Thesystemdetectsthepositionandorientation
2
vof the assembled parts and learns how to do assembly follow- Second, we use depth cameras and match the object model
3ingthehumanworkerfsdemonstration;Anindustrialrobotper-
to the point cloud obtained using depth camera to roughly es-
7forms assembly tasks following the data learned from human
timatetheobjectpose, andusethegeometricconstraintsfrom
4
demonstration; It finds the mechanical parts in its workspace,
6 planar table surface to reinforce the precision. For one thing,
picks up them, and does assembly using motion planning and
0 therobotexecutionphaseisautomaticandisnotasflexibleas
1.assTehmebdlyiffiplcaunlntyinign.developing the smart system is precise vi- human teaching. Markerless approaches are used to avoid oc-
0 clusions. Fortheother,thepointcloudandextrinsicconstraints
sualdetection. Twoproblemsexistwherethefirstoneisinthe
6 arefusedtomakeupthepartiallossandnoises. Theassump-
1humanteachingphase,namelyhowtopreciselydetecttheposi-
tioniswhentheobjectisplacedonthesurfaceofatable,itsta-
:tionandorientationofthepartsinhumanhandsduringmanual
v bilizes at a limited number of poses inherent to the geometric
idemonstration;Thesecondoneisintherobotexecutionphase, constraints. TheposeshelptofreezesomeDegreeofFreedom
X
namely how to precisely detect the position and orientation of
andimproveprecision.Themodalemployedinthisphaseisthe
rthepartsintheworkspaceandperformassembly.
a cloudpointdataandthegeometricconstraintsfromtheplenary
Lots of detecting and tracking studies are available in con- surface. Thetag“Depth+Geom”isusedforrepresentation.
temporary publication, but none of them meets the require-
mentsofthetwoproblems. TheapproachesusedincludeRGB
images, markers, point cloud, extrinsic constraints, and multi- Moreover, we propose an improved graph model based on
modalsolutionswheretheRGBimagesandmarkersareshort our previous work to perform assembly planning and motion
inocclusions,thepointcloudandextrinsicconstraintsareshort planning. Experiments are performed to examine the preci-
inpartiallossandnoises,andthemulti-modalsolutionsarenot sion and correctness of our approaches. We quantitatively
clearlystatedandarestillbeingdeveloped. show the advantages of “AR(RGB)” and “Depth+Geom” in
This paper solves the two problems using multi-modal vi- next-generation mechanical assembly and concretely demon-
sion. First, we attach AR markers to the objects for assembly stratetheprocessofsearchingandplanningusingtheimprove
andtrackthembydetectingandtransformingthemarkerposi- graphmodel. Wealsointegratethedevelopedapproacheswith
tionsduringhumandemonstration. Wedon’tneedtoworryoc- KawadaNextageRobotsandshowtheapplicabilityofthemin
clusionssincetheteachingphaseismanualandisperformedby real-worldscenarios.
PreprintsubmittedtoNeurocomputing January27,2016
2. RelatedWork markersrequiresomemanualworkandtherearelimitationson
markersizes,viewdirections,etc.
Thispaperishighlyrelatedtostudiesin3Dobjectdetection
[31]usescircularconcentricringfiducialmarkerswhichare
forroboticmanipulationandassemblyandtheliteraturereview
placed at known locations on a computer case to overlay the
concentrates on the perception aspect. For general studies on
hidden innards of the case on the camerafs video feed. [32]
roboticgrasping,manipulation,andassembly,referto[1],[2],
usesARmarkerstolocatethedisplaylotsduringvirtualconfer-
and[3]. Theliteraturereviewemphasizesonmodel-basedap-
ence.Itexplainstheunderneathcomputationwell.[33]doesnft
proaches since the paper is motivated by next-generation me- directly use markers, but uses the offline matched keyframes,
chanicalassemblyandisabouttheindustrialapplicationswhere
which are essential the same thing, to correct online tracking.
precisioniscrucialandobjectmodelsareavailable. Formodel-
[34]usesARmarkerstorecognizeandcageobjects. [35]uses
lessstudies,referto[4]and[5]. Forappearance-basedstudies,
both balls (circles) and AR markers to estimate and track the
referto[6][7],and[8]. Moreover,learningapproachesareno
position of objects. More recently, [36] uses AR markers to
reviewed since they are not precise. Refer to [9] and [10] if
track the position of human hands and the operating tools and
interested.
uses the tracked motion to teach robots. The paper shares the
We archive and review the related work according to the
same assumption that human beings can actively avoid occlu-
modals used for 3D detection, including RGB images, mark-
sions.However,itdoesn’tneedanddidn’tanalyzetheprecision
ers,pointcloud,hapticsensing,extrinsicconstraints,andmulti-
sincethegoaloftrackingisintasklevel,insteadofthelowlevel
modalfusion.
trajectories.
2.1. RGBimages
2.3. Pointcloud
RGBimagesarethemostcommonlyusedmodalofrobotic
Point cloud can be acquired using a structured light camera
perception. Using RGB images to solve the model-based 3D
[37], stereovision camera [38], and LIDAR device [39], etc.
positionandorientationdetectionproblemiswidelyknownas
Therecentavailabilityoflowcostdepthsensorsandthehandy
the “model-to-image registration problem” [11] and is under
Point Cloud Library (PCL) [40] has widely disseminated 3D
theframeworkofPOSIT(PosefromOrthographyandScaling
sensing technology in research and commercial applications.
withIterations)[12]. Whenlookingforobjectsinimages, the
Thebasicflowofusingpointcloudfordetectionisthesameas
POSIT-basedapproachesmatchthefeaturedescriptorsbycom-
image-baseddetection: Thefirststepistoextractfeaturesfrom
paringthemostrepresentativefeaturesofanimagetothefea-
models and objects; The second step is to match the features
tures of the object for detection. The features could either be
anddetectthe3Dpose.
values computed at pixel points or histograms computed on a
[41] is one of the early studies that uses point cloud to de-
regionofpixels. Usingmorethanthreematches,the3Dposi-
tect the pose of an object. It is based on the Iterative Clos-
tionandorientationofanobjectcanbefoundbysolvingpoly-
estPoint(ICP)algorithm[42]whichiterativelyminimizesthe
nomialequations[13,14,15].Agoodmaterialthatexplainsthe
mean square distance of nearest neighbor points between the
matchingprocesscanbefoundin[16].Thepaperstudiesmulti-
model and the point cloud. Following work basically uses the
viewimagematching. Itisnotdirectlyrelatedto3Ddetection,
same technique, with improvements in feature extraction and
butexplainswellhowtomatchthefeaturedescriptors.
matching algorithms. The features used in point cloud detec-
Someofthemostcommonchoicesoffeaturesincludecorner
tionaremorevastthanthoseinimage-baseddetection,includ-
features [17] applied in [18] and [12], line features applied in
inglocaldescriptorslikesignatureofhistogramsoforientation
[19], [20] and [21], cylinder features applied in [21] and [22],
(SHOT)[43]andRadius-basedSurfaceDescriptor(RSD)[44],
and SIFT features [23] applied in [24] and [25]. Especially,
and global descriptors like Clustered Viewpoint Feature His-
[24]statedclearlythetwostagesofmodel-baseddetectionus-
togram(CVFH)[45]andEnsembleofShapeFunctions(ESF)
ing RGB images: (1) The modeling stage where the textured
[46]. Thematchingalgorithmsdidnftchangemuch,stickingto
3Dmodelisrecoveredfromasequenceofimages;(2)Thede-
RandomSampleConsensus(RANSAC)[13]andICP.Acom-
tectionstagewherefeaturesareextractedandmatchedagainst
pletereviewandusageofthefeaturesandmatchingalgorithms
those of the 3D models. The modeling stage is based on the
algorithmsin[16]. Thedetectionstageisopentodifferentfea- canbefoundin[47].
tures, different polynomial solving algorithms, and some opti-
mizationslikeLevenberg-Marquardt[26]andMean-shift[27], 2.4. Extrinsicconstraints
etc. [28]comparedthedifferentalgorithmsinthesecondstage. Likethemarkers,extrinsicconstraintsareusedtoassistpoint
cloud based detection. When the point cloud don’t provide
2.2. Markers enough features or when the point cloud are noisy and oc-
Incaseswheretheobjectsdon’thaveenoughfeatures,mark- cluded, it is helpful to take into account the extrinsic con-
ers are used to assist image-based detection. Possible marker straints. For example, the detection pipeline in [47] uses hy-
typesinclude: ARmarkers,coloredmarkers,andshapemark- pothesis,whichisoneexampleofextrinsicconstraints,tover-
ers, etc. The well known AR libraries [29, 30] provide robust ify the result of feature matching. [48] analyzes the functions
librariesandtheOptitrackdeviceprovideseasy-to-usesystems of connected object parts and uses them to refine grasps. The
for the different marker types. However, the applications with paper is not directly related to detection but is an example of
2
improving performance using extrinsic constraints caused by tioninthissection, anddescribeindetailthe“AR(RGB)”and
adjacentfunctionalunits. “Depth+Geom”approachesinthesectionsfollowingit.
Some other work uses geometric constraints to reduce am-
biguityofICP.[49]and[50]segment3Dclutterusingthege-
ometric constraints. They are not directly related to detection
butareusedwidelyastheinitialstepsofmanydetectionalgo-
rithms. [51]clusterspointcloudintopolygonpatchesanduses
RANSACwiththemultiplepolygonconstraintstoimprovethe
precisionofdetection.[52]usestableconstraintsforsegmenta-
tionandusesHoughvotingtodetectobjectposes.[53]usesthe
sliced2Dcontoursof3Dstableplacementstoreducethenoises
ofestimation. Itislikeourapproachbutiscontour-basedand
suffersfromambiguity.
2.5. Multi-modalfusion
Multi-modalapproachesaremostlythecombinationorrep-
etition of the previous five modals. For example, some work
fuses repeated modals to improve object detection. [54] fuses
colour, edge and texture cues predicted from a textured CAD
modelofthetrackedobjecttorecoverthe3Dpose,andisopen
Figure1: Theflowofthenext-generationmechanicalassembly. Theflowis
toadditionalcues.
composedofahumanteachingphaseandarobotexecutionphase.Inthehuman
Some other work uses visual tracking to correct the noises teachingphase,ahumanworkerdemonstratesassemblywithmarkedobjectsin
caused by fast motions and improve the precision of initial frontofanRGBcamera.Thecomputerdetectstherelationshipoftheassembly
parts.Intherobotexecutionphase,therobotdetectsthepartsintheworkspace
matches. ThefusedmodalsincludetheRGBimagemodaland
usingdepthcameraandgeometricconstraints, picksupthem, andperforms
themotionmodalwherethelateronecouldbeeitherestimated
assembly.
using image sequences or third-part sensors like Global Posi-
tioningSystem(GPS)orgyros. [55]isonerepresentativework
Fig.1 shows the flow of the next-generation mechanical as-
which fuses model motion (model-based tracking) and model
semblysystem. Itiscomposedofahumanteachingphaseand
detectioninRGBimagestorefineobjectposes. [20]fusesgyro
arobotexecutionphase. Inthehumanteachingphase,ahuman
dataandlinefeaturesofRGBimagestoreinforcetheposees-
workerdemonstrateshowtoassemblemechanicalpartsinfront
timationforhead-mounteddisplays. [56]usesgyrodata,point
ofa visionsystem. Thesystemrecordsthe relationshipofthe
descriptors,andlinedescriptorstogethertoimprovetheperfor-
twomechanicalpartsandsavesitasanintermediatevalue.
manceofposeestimationforoutdoorapplications.
In the robot execution phase, the robot uses another vision
[57]usespointcloudtoclusterthesceneandfindtheRegion-
systemtofindthemechanicalpartsintheworkspace,picksup
of-Interests(ROIs),andusesimagemodaltoestimatetheobject
them,andperformsassembly.Therelativepositionandorienta-
pose at respective ROIs. The fused modals are RGB images
tionoftheassemblypartsaretheintermediatevaluesperceived
andPointcloud. [58]alsocombinesimageanddepthmodals.
by the human teaching phase. The grasping configurations of
Itusestheimagegradientsfoundonthecontourofimagesand
the robotic hand and the motions to move the parts are com-
thesurfacenormalsfoundonthebodyofpointcloudtoestimate
putedonlineusingmotionplanningalgorithms.
objectposes.
The beneficial point of this system is it doesn’t need direct
To our best knowledge, the contemporary object detection
robot programming and is highly flexible. What the human
studiesdonotmeetourrequirementsaboutprecision.Themost
worker needs to do is to attach the markers and presave the
demanding case is robotic grasping and simple manipulation,
pose of the marker in the object’s local coordinate system so
whichisfarlessstrictthanregraspandassembly. Wedevelop
thatthevisionsystemcancomputetheposeoftheobjectfrom
ourownapproachesinthispaperbyfusingdifferentmodalsto
thedetectedmarkerposes.
dealwiththeproblemsintheteachingphaseandtherobotex-
The 3D detection in the two phases are denoted by the two
ecutionphaserespectively. Foronething, weuseARmarkers
“object detection” sub-boxes in Fig.1. The one in the human
todetectthe3Dobjectpositionsandorientationsduringhuman
teachingphaseusesARmarkerssincehumanbeingscaninten-
teaching. Fortheother,weusethecloudpointdataandthege-
tially avoid unexpected partial occlusions by human hands or
ometricconstraintsfromplanartablesurfaceduringrobotexe-
the other parts, and as well as ensure high precision. The one
cution. Thedetailsarepresentedinfollowingsections.
in the motion planning phase uses point cloud data to roughly
detecttheobjectpose,andusesthegeometricconstraintsfrom
3. SystemOverview planartablesurfacetocorrectthenoisesandimproveprecision.
Thedetailswillbeexplainedinfollowingsections.Beforethat,
We present an overview of the next-generation mechanical welistthesymbolstofacilitatereaders.
assemblysystemandmakeclearthepositionsofthe3Ddetec- p s ThepositionofobjectX onaplanerysurface. Weuse
X
3
A and B to denote the two objects and consequently R a). Duringrobotexecution,the(p a,R a)issetto:
B B B
use p sand p stodenotetheirpositions.
A B
RXs TheorientationofobjectXonaplanerysurface. Like pBa = pBa− pAa, RBp =RBa·(RAa)(cid:48) (1)
p s,XistobereplacedbyAorB.
p a TXhepositionofobjectXintheassembledstructure. and pAaandRAaaresettozeroandidentitymatrixrespectively.
X
R a TheorientationofobjectXintheassembledstructure. Onlytherelativeposesbetweentheassemblypartsareused.
X
p p The pre-assembly positions of the two objects. The
X
robotwillplanamotiontomovetheobjectsfromthe 5. 3DDetectionduringRoboticExecution
initialpositionstothepreassemblepositions.
gXf The force-closure grasps of object X. The letter f in- Theobjectposedetectionduringrobotexecutionisdoneus-
dicates the object is free, and is not in an assembled ingKinect,PointCloudLibrary,andgeometricconstraints.The
structureorlayingonsomething. detection cannot be done using markers since: (1) What the
gXs(cid:48) Theforce-closuregraspsofobjectXonaplanerysur- robotmanipulatesarethousandsofindustrialparts,itisimpos-
face. Itisassociatedwith pXsandRXs. sible to attach markers to all of them. (2) The initial configu-
gXs The collision-free and IK (Inverse Kinematics) feasi- ration of the object is unexpectable and the markers might be
blegraspsofobjectX onaplanerysurface. Itisalso occludedfromtimetotimeduringroboticpick-and-place. The
associatedwith pXsandRXs. detectionisalsonotapplicabletoimage-basedapproaches: In-
gXa(cid:48) Theforce-closuregraspsofobjectX inanassembled dustrial parts are usually mono colored and textureless, image
structure. Itisassociatedwith pXaandRXa. featuresarenotonlyhelplessbutevenharmful.
gXa The collision-free and IK (Inverse Kinematics) feasi- UsingKinectisnottrivialduetoitslowresolutionandpre-
blegraspsofobjectX intheassembledstructure. Itis cision. For example, the objects in industrial applications are
associatedwith pXaandRXa. usuallyinclutteranditisdifficulttosegmentoneobjectfrom
gXp(cid:48) The force-closure grasps of object X at the pre- anotherusingKinect’sresolution. Oursolutionistodividethe
assemblypositions. difficultyinclutterintotwosubproblems. First,therobotcon-
gXp Thecolllision-freeandIKfeasiblegraspsofobjectX sidershowtopickoutoneobjectfromtheclutter,andplacethe
atthepre-assemblypositions. objectonanopenplenarysurface. Second, therobotestimate
theposeofthesingleobjectontheopenplenarysurface. The
first subproblem doesn’t care what the object is or the pose of
4. 3DDetectionduringHumanTeaching
the object and its only goal is to pick something out. It is re-
ferred as a pick-and-place problem in contemporary literature
The object detection during human teaching is done using
andisillustratedintheleftpartofFig.3. Thefirstsubproblem
AR markers and a RGB camera. Fig.2 shows the flow of the
iswellstudiedandinterestedreadersmayreferto[59]forthe
detection and the poses of the markers in the object model’s
solutions.
localcoordinatesystem. Themarkersareattachedmanuallyby
Thesecondsubproblemistodetecttheposeofasingleob-
humanworkers.
ject on an open plenary surface and is shown in the right part
of Fig.3. It is much easier but still requires much proess to
meet the precision requirements of assembly. We concentrate
on precision and will discuss use point cloud algorithms and
geometricconstraintstosolvethesecondproblem.
Figure2: ObjectdetectionusingARmarkers. Beforehand,humanworkerat-
tachesthemarkersandpresavestheposeofthemarkerintheobject’slocal
coordinatesystem. Duringdetection,visionsystemcomputestheposeofthe
objectfromthedetectedmarkerposes(thethreesubfigures).Theoutputisthe
(pAa,RAa)and(pBa,RBa).
Duringdemonstration,theworkerholdsthetwoobjectsand
show the makers to the camera. We assume the workers have Figure3:Overcometheclutterproblembydividingtheposedetectionintotwo
subproblems.Thefirstoneispickingoutfromclutterwherethesystemdoesn’t
enough intelligence and can expose the markers to the vision
carewhattheobjectisortheposeoftheobjectanditsonlygoalistopick
system without occlusion. The detection process is conven- somethingout. Theproblemiswellstudied. Thesecondoneistodetectthe
tionalandcanbefoundinmanyARliterature: Giventheposi- poseofasingleobjectonanopenplenarysurface. Itisnottrivialsincethe
tionsofsomefeaturesinthemarkers’localcoordinatesystem, precisionofKinectisbad.
findthetransformmatrixwhichconvertsthethemintothepo-
sitionsonthecamerascreen.Inourimplementation,thehuman
5.1. Roughdetectionusingpointcloud
teachingpartisdevelopedusingUnityandtheARrecognition
isdoneusingtheVuforiaSDKforUnity. First, we roughly detect the pose of the object using CVFH
In the example shown in Fig.2, there are two objects where and CRH features. In a preprocessing process before start-
the detected results are represented by (p a, R a) and (p a, ing the detection, we put the camera to 42 positions on a unit
A A B
4
sphere,savetheviewofthecamera,andprecomputetheCVFH thecandidatesurfaceortooneartoitsboundary)areremoved.
andCRHfeaturesofeachview.Thisstepisvirtuallyperformed Anexampleofthestableplacementsforoneobjectisshownin
using PCL and is shown in the dashed blue frame of Fig.4. thestableplacementsframeboxofFig.5.
During the detection, we extract the plenary surface from the Given the raw detection result using CVFH and CRH fea-
pointcloud,segmenttheremainingpointscloud,andcompute tures, the robot computes its distance to the stable placements
the CVFH and CRH features of each segmentation. Then, we andcorrectitfollowingAlg.1.
matchtheprecomputedfeatureswiththefeaturesofeachseg-
ment and estimate the orientation of the segmentations. This Algorithm1:Noisecorrection
step is shown in the dashed red frame of Fig.4. The matched
Data: Rawresult: p,R ;
segments are further refined using ICP to ensure good match- Stableplacemrentsr: {R (i), i=1,2,...}
p
ing. ThesegmentationthathashighestICPmatchesandsmall-
Tableheight: h
t
est outlier points will be used as the output. An example is
Result: Correctedresult: p ,R
showninthe“Rawresult”frameboxofFig.4. 1 dmin ←+∞ c c
2 Rnear ←I
3 fori←1toRp.size()do
4 di ←||log(Rp(i)Rp(cid:48))||
5 if di <dminthen
6 dmin ←di
7 Rnear ←Rp(i)
8 end
9 end
10 Ryaw ←rotFromRpy(0,0,rpyFromRot(Rr).y)
Figure 4: Rawly detecting the pose of an object using model matching and 11 Rc ←Ryaw·Rnear
CwVeFpHrecaonmdpCutReHthfeeaCtVurFeHs. aInndaCpRrHeprfoeacteusrseinsgofpr4o2cedsisffebreefnotrevitehwesdaentedctsiaovne, 12 pc ←[pr.x,pr.y,ht](cid:48)
themasthetemplate. Thepreprocessingprocessisshowninthedashedblue
frame.Duringthedetection,wesegmenttheremainingpointscloud,compute
theCVFHandCRHfeaturesofeachsegmentation,andmatchthemtothepre- Inthispseudocode,Iindicatesa3×3identitymatrix. Func-
computedviewsusingICP.Thebestmatchisusedastheoutput. tions rotFromRpy and rpyFromRot converts roll, pitch, yaw
anglestorotationmatrixandviceverse. Thedistancebetween
tworotationmatricesiscomputedinline4.Thecorrectedresult
5.2. Noisecorrectionusinggeomtericconstraints
isupdatedatlines11and12.
6. GraspandMotionPlanning
After finding the poses of the parts on the plenary surface,
what the robot does next is to grasp the parts and assemble
them.Itincludestwosteps:Agraspplanningstepandamotion
planningstep.
6.1. Graspplanning
In the grasp planning step, we set the object at free space
Figure5: Correctingtherawresultusingthestableplacementsonaplenary
surface(geometricconstraints). Inapreprocessingprocessbeforethecorrec- andcomputetheforce-closuredandcollision-freegrasps. Each
tion,wecomputethestableplacementsoftheobjectonaplenarysurface. An graspisrepresentedusing g f={p ,p ,R}where p and p are
X 0 1 0 1
exampleisshowninthestableplacementsframebox.Duringthecorrection,we the contact positions of the finger tips, R is the orientation of
computethedistancebetweentherawresultsandeachofthestableplacements,
thepalm. Thewholesetisrepresentedbyg f,whichincludes
andcorrecttherawresultsusingthenearestpose. X
many g f. Namely,g f ={g f}.
X X X
Theresultofroughdetectionisfurtherrefinedusingthegeo- Giventheposeofapartontheplenarysurface,say pXs and
metric constraints. Since the object is on plenary surface, its RXs,theIK-feasibleandcollision-freegraspsthattherobotcan
stable poses are limited [60] and can be used to correct the usetopickuptheobjectiscomputedfollowing
noisesoftheroughlyestimatedresult. Theplacementplanning
g s =IK(g s(cid:48) )∩CD(g s(cid:48), planerysurface) (2)
includes two steps. First, we compute the convex hull of the X X X
objectmeshandperformsurfaceclusteringontheconvexhull.
where
Eachclusterisonecandidatestandingsurfacewheretheobject g s(cid:48) =R s·g f + p s (3)
maybeplacedon. Second,wecheckthestabilityoftheobjects X X X X
standingonthesecandidatesurfaces. Theunstableplacements R s·g f+p stransformsthegraspsatfreespacetothepose
X X X
(theplacementwheretheprojectionofcenterofmassisoutside oftheobject. g s(cid:48) denotesthetransformedgraspset. IK()finds
X
5
theIK-feasiblegraspsfromtheinputset.CD()checksthecolli- employ Transition-RRT to find the motion between the initial
sionbetweenthetwoinputelementsandfindsthecollision-free configurationandgoalconfiguration.
grasps. g s denotes the IK-feasible and collision-free grasps In practice, the flow in Fig.6 doesn’t work since the goal
X
thattherobotcanusetopickuptheobject. configuration is in the assembled structure and is in the nar-
Likewise,giventheposeofobjectAintheassembledstruc- row passages or on the boundaries of the configuration space.
ture,say p aandR a,theIK-feasibleandcollision-freegrasps Themotionplanningproblemisanarrow-passage[62]orpeg-
A A
thattherobotcanusetoassembleitiscomputedfollowing in-hole problem [63] which is not solvable. We overcome the
difficultybyaddingapre-assembleconfiguration: Forthetwo
gAa =IK(gAa(cid:48) )∩CD(gAa(cid:48),objB( pBa,RBa)) (4) objects A and B, we retract them from the structure following
theapproachingdirectionvaofthetwoobjectsandgetthepre-
where assembleposes p p,R p,and p p,R pwhere
g a(cid:48) =R a·g f + p a (5) A A B B
A A A A
p p = p a+0.5va, R p =R a (6)
R a · g f + p a transforms the grasps at free space to the A A A A
poseAofthAeobjecAtintheassembledstructure. g a(cid:48) denotesthe pBp = pBa−0.5va, RBp =RBa (7)
X
transformed grasp set. IK() and CD() are the same as Eqn(2).
Thegraspsassociatedwiththeretractedposesare
objB( p a,R a )indicatesthemeshmodelofobjectBatpose
B B
p a,R a. g a denotestheIK-feasibleandcollision-freegrasps
B B A
thattherobotcanusetoassembletheobject. g p =IK(g p(cid:48) ), whereg p(cid:48) =g a+0.5va (8)
A A A A
g p =IK(g p(cid:48) ), whereg p(cid:48) =g a−0.5va (9)
B B B B
6.2. Motionplanning
Note that the poses in Eqn(6-9) are in the local coordinate
In the motion planning step, we build a graph using the
of object A where p a is a zero vector and R a is an identity
elements in g s and g a, search the grasp to find high-level A A
X X matrix. GiventheposeofobjectAinworldcoordinate, p a(g)
keyframes, and perform Transition-based Rapidly-Exploring A
and R a(g), the grasps in the world coordinate are computed
RandomTree(Transition-RRT)[61]motionplanningbetween A
using
thekeyframestofindassemblemotions.
g a(g) =g p(g) (10)
A A
g a(g) = p a(g)+R a(g)·g a (11)
B A A B
g p(g) = p a(g)+R a(g)·g p (12)
A A A A
g p(g) = p a(g)+R a(g)·g p (13)
B A A B
Themotonplanningisthentofindamotionbetweenoneini-
tialconfigurationing stoagoalconfigurationing p(g)where
X X
X is either A or B. There is no motion between g p(g) and
A
g a(g)sincetheyareequaltoeachother. Themotionbetween
A
g p(g)andg a(g)ishardcodedalongva.
B B
Figure6: Theflowofthemotionplanning. Giveninitialandgoalposesofan
object(leftimagesintheupperleftandlowerleftframeboxes), wesearchits
availableinitialandgoalgraspsandusethecommongraspsandIKtogetthe
initialandgoalconfigurationsoftherobotarm. Then,wedomotionplanning
repeatedlybetweentheinitialandgoalconfigurationstofindasolutiontothe
desiredtask.
Fig.6showstheflow. TheobjectXinthisgraphisawooden
blockshownintheleftimageoftheupper-leftframebox. The
Figure7:ThegraspgraphbuiltusinggXsandgXp(g).Ithasthreelayerswhere
imagealsoshowstheposeofthisobjectontheplenarysurface,
thetoplayerencodesthegraspsassociatedwiththeinitialconfiguration,the
p s andR s. Whentheobjectisassembledinthestructure,its middle layer encodes the grasps associated with placements on planery su-
X X
pose p a andR a isshownintheleftimageofthebottom-left faces, andthebottomlayerencodesthegraspsassociatedwiththeassemble
framebXox. TheSgraspsassociatedwiththeposesareshownin pose. TheleftimageshowsonegXp(g)(thevirtualgraspillustratedincyan).
Itcorrespondstoanodeinthebottomlayer. Thesubimagesintheframebox
the right images of the two frame boxes. They are rendered illustratetheplacements(yellow)andtheirassociatedgrasps(green).
usingthecoloredhandmodel. Green,blue,andreddenoteIK-
feasible and collision free, IK-infeasible, and collided grasps Which initial and goal configuration to use is decided by
respectively. Webuildagraphtofindthecommongraspsand building and searching a grasp graph which is built using g s
X
6
andg p(g), andisshownintheframeboxofFig.7. Thegraph 7.2. Precisionoftheobjectdetectioninroboticexecution
X
isbasicallythesameas[60],buthasthreelayers. Thetoplayer
Then,weexaminethepecisionoftheobjectdetectioninthe
has only one circle and is mapped to the initial configuration.
robotexecutionphase. Threeobjectswitheightplacementsare
Thebottomlayeralsohasonlyonecircleandismappedtothe
used during the process. They are shown in the top row of
goalconfiguration. Themiddlelayersarecomposedofseveral
Fig.9. The plenary surface is set in front of the robot and is
circleswhereeachofthemmapsapossibleplacementonaple-
dividedintofourquarters. Weplaceeachplacementintoeach
narysurface. Eachnodeofthecirclesrepresentsagrasp: The
quoter to get the average values. There are five data rows di-
onesintheupperlayersarefromg s,andtheonesinthebot-
X vided by dashed or solid lines in Fig.9 where the first four of
tomlayersarefromg p(g)).Theonesfromthemiddlelayersare
X themshowtheindividualdetectionprecisionateachquoterand
thegraspsassociatedwiththeplacements. Theorientationsof
thelastoneshowstheaveragedetectionprecision. Thedetec-
theplacementsareevenlysampledonline. Theirpositionsare tionprecisionisthedifferencebetweenthedetectedvalueand
fixedtotheinitialpositionp s. Ifthecirclessharesomegrasps
A thegroundtruth. Sinceweknowtheexactmodeloftheobject
(graspswiththesame p , p ,Rvaluesintheobject’slocalco-
0 1 andtheheightofthetable,thegroundtruthisknowbeforehand.
ordinatesystem),weconnectthematthecorrespondentnodes.
Theaveragedetectionprecisioninthelastrowarethemeanof
We search the graph to find the initial and goal configurations theabsolutedifference.
and a sequence of high-level keyframes, and perform motion
Inside each data grid there are four triples where the upper
planning between the keyframes to perform desired tasks. An
two are the roughly detected position and orientation and the
exemplaryresultwillbeshownintheexperimentsection.
lower two are the corrected values. The roughly detected re-
sultsaremarkedwithredshadowsandthecorrectedresultsare
markedingreen. Thethreevaluesofthepositiontriplesarethe
7. ExperimentsandAnalysis x,y,zcoordinates,andtheirmetricsaremillimeters(mm). The
three values of the orientation triples are the roll, pitch, yaw
anglesandtheirmetricsaredegree(◦).
We performed experiments to examine the precision of the
Theresultsshowthatthemaximumerrorsofroughposition
developedapproaches,analyzetheprocessofgraspandmotion
detectionare-1.1mm,3.3mm,and2.3mmalongx,y,andzaxis.
planning,anddemonstratetheapplicabilityofthestudyusinga
TheyaremarkedwithredboxesinFig.9. Aftercorrection,the
KawadaNextageRobot. Thecamerausedinthehumanteach-
maximumpositionerrorschangeto-1.1mm,3.6mm,0.0mmre-
ingphaseisalogicoolHDWebcamC615. Thecomputersys-
spectively. Theyaremarkedwithgreenboxes. Themaximum
tem is a Lenovo Thinkpad E550 laptop (Processor: Intel Core
errors of rough orientation detection are -26.3◦, 26.1◦, and -
i5-52002.20GHzClock,Memory: 4G1600MHzDDR3). The
19.5◦ inroll, pitch,andyawangles. Theyaremarkedwithred
depthsensorusedintheroboticexecutionphaseisKinect. The
boxes. Theerrorschangeto0.0,0.0,and-21.4aftercorrection.
computer system used to compute the grasps and motions is
Thecorrectionusinggeometricconstraintscompletelycorrects
a Dell T7910 workstation (Processor: Intel Xeon E5-2630 v3
the erros along z axis and in roll and pitch angles. It might
with 8CHT, 20MB Cache, and 2.4GHz Clock, Memory: 32G
slightlyincreasetheerrorsalongxandyandinyawbuttheval-
2133MHzDDR4).
uesareignorable. Theaverageperformancecanbefoundfrom
the data under thee dual solid line. The performance is good
enoughforroboticassembly.
7.1. Precisionoftheobjectdetectioninhumanteaching
Inaddition,thesecondandthirdsubfigurerowsofFig.9plot
someexamplesoftheroughlydetectedposesandthecorrected
First,weexaminetheprecisionofobjectdetectionduringhu-
poses. Readersmaycomparethemwiththedatarows.
manteaching. Weusetwosetsofassemblypartsandexamine
the precision of five assembly structures for each set. More-
7.3. Simulationandreal-worldresults
over,foreachassemblystructure,weexaminethevaluesofat
fivedifferentorientationstomaketheresultsconfident. Fig.10showsthecorrespondencebetweenthepathsfoundby
The two sets and ten structures (five for each) are shown in thesearchingalgorithmsandtheconfigurationsoftherobotand
the first row of Fig.8. Each structure is posed at five different theobject. Thestructuretobeassembledinthistaskisthefirst
orientations to examine the precision. The five data rows un- one (upper-left one) shown in Fig.8. We do motion planning
derthesubimagerowinFig.8aretheresultattheorientations. alongthepathsrepeatedlytoperformthedesiredtasks.
Eachgridofthedatarowsisshownintheform∆d(∆r,∆p,∆y) Theassemblyprocessisdividedintotwostepwitheachstep
where ∆d is the difference in the measured |p a − p a| and corresponds to one assembly part. In the first step, the robot
A B
the actual value. (∆r,∆p,∆y) are the difference in the mea- finds object A on the table and moves it to a goal pose using
sured roll, pitch, and yaw angles. The last row of the figure the three-layer graph. The subfigures (1)-(4) of Fig.10 shows
showstheaveragedetectionerrorofeachstructureintheform this step. In Fig.10(1), the robot computes the grasps associ-
|∆d|(|∆r|,|∆p|,|∆y|)where|·|indicatestheabsolutevalue. The atedwiththeinitialposeandgoalposeofobjectA.Theasso-
metricsaremillimeter(mm)fordistanceanddegree(◦)forori- ciated grasps are rendered in green, blue, and red colors like
entation. The precision in position is less than 1mm and the Fig.1. They corresponds to the top and bottom layer of the
precisioninorientationislessthan2◦onaverage. graspshowninFig.10(1’). InFig.10(2),therobotchoosesone
7
Figure8: Resultsoftheobjectdetectedduringhumanteaching. Theimagerowshowsthestructuretobeassembled. Eachstructureisposedatfivedifferent
orientationstoexaminetheprecisionandthedetectederrorindistanceandorientationareshowninthefivedatarowsbelow.Eachgridofthedatarowsisshownin
theform∆d(∆r,∆p,∆y)where∆disthedifferenceinthemeasured|p a−p a|andtheactualvalue.(∆r,∆p,∆y)arethedifferenceinthemeasuredroll,pitch,and
A B
yawangles.Thelastrowistheaveragedetectionerror.Themetricsaremillimeterfordistanceanddegreefororientation.
Figure9: Resultsoftheobjectdetectedduringroboticexecution. Thefigureincludesthreesubfigurerowsandfivedatarowswherethefirstdatarowshowthe
targetobjectpose,thesecondandthirdsubfigurerowsplotsomeexamplesoftheroughlydetectedposesandthecorrectedposes. Thefivedatarowsaredivided
bydashedorsolidlineswherethefirstfourofthemshowtheindividualdetectionprecisionatfourdifferentpositionsandthelastoneshowstheaveragedetection
precision. Eachdatagridofthedatarowsincludefourtripleswheretheuppertwo(underredshadow)aretheroughlydetectedpositionandorientationandthe
lowertwo(undergreenshadow)arethecorrectedvalues.Thethreevaluesofthepositiontriplesarethex,y,zcoordinates,andtheirmetricsaremillimeters(mm).
Thethreevaluesoftheorientationtriplesaretheroll, pitch,yawanglesandtheirmetricsaredegree(◦). Themaximumvaluesofeachdataelementaremarked
withcoloredframeboxes.
8
Figure10: ThesnapshotsofassemblingthestructureshowninFig.8. Itisdividedintotwostepwiththefirststepshownin(1)-(4)andthesecondstepshownin
(5)-(8). Inthefirststep,therobotpicksupobjectAandtransfersittothegoalpose. Inthesecondstep,therobotfindsobjectBonthetableandassemblesitto
objectA.Thesubfigures(1’)-(8’)showscorrespondentpathedgesandnodesonthethree-layergraph.
feasible(IK-feasibleandcollision-free)graspfromtheassoci- matchingtofindarawposeanduseextrinsicconstraintstocor-
ated grasps and does motion planning to pick up the object. rectthenoises. Weexaminetheprecisionoftheapproachesin
The selected grasp corresponds to one node in the top layer the experiment part and demonstrate that the precision fulfills
ofthegraph, whichismarkedwithredcolorinFig.10(2’). In assemblytasksusingagraphmodelandanindustrialrobot.
Fig.10(3),therobotpicksupobjectAandtransfersittothegoal The future work will be on the manipulation and assembly
pose using a second motion planning. This corresponds to an aspect. The current result is position-based assembly. It will
edgeinFig.10(3’)whichconnectsthenodeinonecircletothe beextendedtoforce-basedassemblytaskslikeinserting,snap-
nodeinanother. Theedgedirectlyconnectstothegoalinthis ping,etc.,inthefuture.
example and there is no intermediate placements. After that,
the robot moves its arm back at Fig.10(4), which corresponds
References
toanodeinthebottomlayerofthegraphshowninFig.10(4’).
Inthesecondstep, therobotfindsobjectBonthetableand [1] T.Lozano-Perez,J.L.Jones,E.Mazer,P.A.O’Donnell,HANDEY:A
RobotTaskPlanner,TheMITPress,1992.
assemblesittoobjectA.Thesubfigures(5)-(8)Fig.10(b)show
[2] M.T.Mason,MechanicsofRoboticManipulation,TheMITPress,2001.
it. InFig.10(5),therobotcomputesthegraspsassociatedwith
[3] M.Dogar,A.Spielberg,S.Baker,D.Rus,Multi-robotgraspplanningfor
theinitialposeandgoalposeofobjectB.Theyarerenderedin sequentialassemblyoperations,in:ProceedingsofInternationalConfer-
green,blue,andredcolorslikeFig.10(1)andarecorrespondent eneonRoboticsandAutomation(ICRA),2015.
[4] C.Goldfeder,Data-DrivenGrasping,Ph.D.thesis,ColumbiaUniversity
tothetopandbottomlayerofthegraspshowninFig.10(5’). In
(2002).
Fig.10(6),therobotchoosesonefeasiblegraspanddoesmotion [5] I.Lenz,H.Lee,A.Saxena,DeepLearningforDetectingRoboticGrasps,
planningtopickuptheobject. Theselectedgraspcorresponds InternationalJournalofRoboticsResearch(IJRR).
tothemarkednodeinFig.10(6’). InFig.10(7),therobotpicks [6] H.Murase,S.K.Nayar,VisualLearningandRecognitionof3DObjects
fromAppearance,InternationalJournalofComputerVision(IJCV).
up object B and assembles it to the goal pose using a second
[7] P.Mittrapiyanuruk,G.N.Desouza,A.C.Kak,Calculatingthe3D-Pose
motion planning which corresponds to an edge in Fig.10(7’). ofRigid-ObjectsusingActiveAppearanceModels, in: Proceedingsof
Finally,therobotmovesitsarmbackatFig.10(8)and(8’). InternationalConferenceonRoboticsandAutomation(ICRA),2004.
[8] S.Zickler, M.Veloso, DetectionandLocalizationofMultipleObjects,
Thesubfigures(1”)-(8”)inthethirdrowshowhowtherobot
in: Proceedings of International Conference on Humand Robots (Hu-
executes the planned motion. They correspond to (1)-(8) and
manoids),2006.
(1’)-(8’)inthefirsttworows. [9] M.Stark,M.Goesele,B.Schiele,BacktotheFuture: LearningShape
Modelsfrom3DCADData,in: ProceedingsofBritishMachineVision
Conference,2011.
[10] J.Liebelt,C.Schmid,Multi-viewObjectClassDetectionwitha3DGeo-
8. ConclusionsandFutureWork
metricModel,in:ProceedingsofComputerVisionandPatternRecogni-
tion(CVPR),2010.
Wepresentedprecise3Dvisualdetectionapproachesinthis [11] P.Wunsch,G.Hirzinger,RegistrationofCADModelstoImagesbyIter-
ativeInversePerspectiveMatching,in:ProceedingsofInternationalCon-
paper to meet the requirements of a smart mechanical assem-
ferenceonPatternRecognition,1996.
blysystem. In thehumanteaching phaseofthe systemwhere [12] P.David,D.DeMenthon,R.Duraiswami,H.Samet,SoftPOSIT:Simul-
humanbeingscontroltheoperationandcanactivelyavoidoc- taneousPoseandCorrespondenceDetermination,in:ProceedingsofEu-
clusion,weuseARmarkersandcomputetheposeoftheobject ropeanConferenceonComputerVision,2002.
[13] M.A.Fischler,R.C.Bolles,RandomSampleConsensus: AParadigm
by detecting the markers’ poses. In the robot execution phase
for Model Fitting and Applications to Image Analysis and Automated
where occlusions happen unexpectedly, we use point cloud Cartography,AssociationforComputingMachinery.
9
[14] D.DeMenthon, L. S. Davis, Model-Based Object Pose in 25 Lines of (ICRA),2011.
Code,InternationalJournalofComputerVision. [41] C.Schutz,H.Hugli,AugmentedRealityusingRangeImages,in: SPIE
[15] C.-P.Lu,G.D.Hager,E.Mjolsness,FastandGloballyConvergentPose PhotonicsWest,TheEngineeringRealityofVirtualReality,1997.
EstimationfromVideoImages,TransactiononPatternAnalysisandMa- [42] P.J.Besl,N.D.McKay,AMethodforRegistrationof3-DShapes,Trans-
chineIntelligence(PAMI). actionsonPatternAnalysisandMachineIntelligence(PAMI).
[16] F.Schaffalitzky,A.Zisserman,Multi-viewMatchingforUnorderedIm- [43] F.Tombari,S.Salti,L.D.Stefano,UniqueSignaturesofHistogramsfor
ageSets,or“HowdoIOrganizeMyHolidaySnaps?”,in:Proceedingsof LocalSurfaceDescription,in: ProceedingsofEuropeanConferenceon
EuropeanConferenceonComputerVision,2002. ComputerVision(ECCV),2010.
[17] C.J.Harris,ACombinedCornerandEdgeDetector,in: Proceedingsof [44] Z.C.Marton,D.Pangercic,N.Blodow,M.Beetz,Combined2D-3DCat-
AlveyVisionConference,1988. egorizationandClassificationforMultimodalPerceptionSystems,Inter-
[18] K.W.Chia,A.D.Cheok,S.J.Prince,Online6-DOFAugmentedReality nationalJournalofRoboticResearch(IJRR).
RegistrationfromNaturalFeatures,in:ProceedingsofEuropeanConfer- [45] A.Aldoma, etal., CAD-modelrecognitionand6DOFposeestimation
enceonComputerVision,2002. using3Dcues,in:ICCVWorkshops,2011.
[19] P.David,D.DeMenthon,R.Duraiswami,H.Samet,SimultaneousPose [46] W.Wohlkinger,M.Vincze,EnsembleofShapeFunctionsfor3DObject
andCorrespondenceDeterminationusingLineFeatures,in:Proceedings Classification,in: ProceedingsofInternationalConferenceonRobotics
ofComputerVisionandPatternRecognition,2003. andBiomimetics(ROBIO),2011.
[20] G.Klein,T.Drummond,RobustVisualTrackingforNon-instrumented [47] A.Aldoma,Z.C.Maron,F.Tombari,M.Vincze,Tutorial: PointCloud
Augmented Reality, in: Proceedings of International Symposium on Library: Three-DimensionalObjectRecognitionand6DOFPoseEsti-
MixedandAugmentedReality,2003. mation,IEEERoboticsandAutomationMagzine.
[21] E.Marchand, F.Chaumette, Virtualvisualservoing: Aframeworkfor [48] Y.Shiraki,K.Nagata,N.Yamanobe,A.Nakamura,K.Harada,D.Sato,
real-timeaugmentedreality,in:ProceedingsofEurographic,2002. D.N.Nenchev,ModelingofEverydayObjectsforSemanticGrasp,in:
[22] K.Harada,K.Nagata,T.Tsuji,N.Yamanobe,A.Nakamura,Y.Kawai, ProceedingsofInternationalSymposiumonRobotandHumanInteractive
ProbabilisticApproachforObjectBinPickingApproximatedbyCylin- Communication,2014.
ders,in: ProceedingsofInternationalConferenceonRoboticsandAu- [49] M.J.Schuster,J.Okerman,H.Nguyen,J.M.Rehg,C.C.Kemp,Per-
tomation(ICRA),2013. ceivingClutterandSurfaceforObjectPlacementinIndoorEnvironment,
[23] D.G.Lowe,DistinctiveImageFeaturesfromScale-invariantKey-points, in: ProceedingsofInternationalConferenceonHumanoidRobots(Hu-
InternationalJournalofComputerVision(IJCV). manoids),2010.
[24] I.Gordon,D.G.Lowe,WhatandWhere: 3DObjectRecognitionwith [50] G.Smonath,M.Rohith,D.Metaxas,C.Kambhamettu,D-Clutter:Build-
AccuratePose,LectureNotesinComputerScience. ingObjectModelLibraryfromUnsupervisedSegmentationofCluttered
[25] A.Collet,D.Berenson,S.S.Srinivasa,D.Ferguson,ObjectRecognition Scenes, in: ProceedingsofComputerVisionandPatternRecognition),
andFullPoseRegistrationfromaSingleImageforRoboticManipulation, 2010.
in:ProceedingsofInternationalConferenceonRoboticsandAutomation [51] M.Saval-Calvo,J.Azorin-Lopez,A.Fuster-Guillo,J.Garcia-Rodriguez,
(ICRA),2009. Three-dimensional Planar Model Estimation using Multi-constraint
[26] W.Press,S.Teukolsky,W.Vetterling,NumericalRecipesinC:TheArt KnowledgebasedonK-meansandRANSAC,AppliedSoftComputing.
ofScientificComputing,TheCambridgePress,1992. [52] L.Goron,Z.-C.Marton,G.Lazea,M.Beetz,RobustlySegmentingCylin-
[27] Y.Cheng,Meanshift,Modeseeking,andClustering,TransactiononPat- dricalandBox-likeObjectsinClutteredScenesusingDepthCamera,in:
ternAnalysisandMachineIntelligence(PAMI). ProceedingsofGermanConferenceonRobotik,2012.
[28] A. Ramisa, D. Aldavert, S. Vasudevan, R. Toledo, R. L. de Mantaras, [53] E.C.Cheung,C.Cao,J.Pan,Multi-contourInitialPoseEstimationfor
EvaluationofThreeVisionbasedObjectPerceptionMethodsforaMobile 3DRegistration,in: ProceedingsofInternationalConferenceonIntelli-
Robot,JournalofIntelligentRoboticSystems(JIRS). gentRobotsandSystems(IROS),2015.
[29] M.Fiala,ARTag: AFiducialMarkerSystemusingDigitalTechniques, [54] G.Taylor, L.Kleeman, FusionofMultimodalVisualCuesforModel-
in: ProceedingsofComputerVisionandPatternRecognition(CVPR), basedObjectTracking, in: ProceedingsofAustralasianConferenceon
2005. RoboticsandAutomation(ACRA),2003.
[30] D.Wagner,D.Schmalstieg,ARToolKitPlusforPoseTrackingonMobile [55] T. Kempter, A. Wendel, H. Bischof, Online Model-Based Multi-Scale
Devices,in:ProceedingsofComputerVisionWinterWorkshop,2007. PoseEstimation,in:ProceedingsofComputerVisionWinterWorkshop,
[31] V.Sundareswaran,R.Behringer,VisualServoing-basedAugmentedRe- 2012.
ality,in: ProceedingsofInternationalWorkshoponAugmentedReality, [56] G.Reitmayr,T.Drummond,GoingOut: RobustModel-BasedTracking
1998. forOutdoorAugmentedReality,in:ProceedingsofInternationalSympo-
[32] H.Kato,M.Billinghurst,MarkerTrackingandHMDCalibrationfora siumonMixedandAugmentedReality,2006.
Video-basedAugmentedRealityConferencingSystem,in: Proceedings [57] D.Pangercic,V.Haltakov,M.Beetz,FastandRobustObjectDetectionin
ofInternationalWorkshoponAugmentedReality,1999. HouseholdEnvironmentsusingVocabularyTreeswithSiftDescriptors,
[33] L.Vacchetti,V.Lepetit,P.Fua,StableReal-time3DTrackingusingOn- in: InternationalConferenceonIntelligentRobotsandSystems(IROS),
lineandOfflineInformation, TransactiononPatternAnalysisandMa- WorkshoponActiveSemanticPerceptionandObjectSearchintheReal
chineIntelligence(PAMI). World,2011.
[34] S.Makita, K.Okita, Y.Maeda, MotionPlanningfor3DMultifingered [58] S.Hinterstoisser,S.Holzer,C.Cagniart,S.Ilic,K.Konolige,N.Navab,
CagingwithObjectRecognitionusingARPictureMarkers,in:Proceed- V.Lepetit, MultimodalTemplatesforReal-TimeDetectionofTexture-
ingsofInternationalConferenceonMechatronicsandAutomation,2015. lessObjectsinHeavilyClutteredScenes,in:ProceedingsofInternational
[35] F.Suligoj,B.Sekoranja,M.Svaco,bojanJerbic,ObjectTrackingwith ConferenceonComputerVision(ICCV),2011.
aMultiagentRobotSystemandaStereoVisionCamera,ProcediaEngi- [59] Y.Domae, D.Okuda, Y.Taguchi, K.Sumi, T.Hirai, FastGraspability
neering. EvaluationonSingleDepthMapsforBinPickingwithGeneralGrippers,
[36] K.Ramirez-Amaro,M.Beetz,G.Cheng,TransferringSkillstoHuman in:ProceedingsofInternationalConferenceonRoboticsandAutomation
Robots by Extracting Semantic Representations from Observations of (ICRA),2014.
HumanActivities,ArtificialIntelligence. [60] W.Wan,K.Harada,DevelopingandComparingSingle-armandDual-
[37] B.Freedman,A.Shpunt,M.Machline,Y.Arieli,DepthMappingusing armRegrasp,IEEERoboticsandAutomationLetters(RA-L).
ProjectedPatterns(2012). [61] L.Jaillet,J.Cortes,T.Simeon,Transition-basedRRTforpathplanning
[38] S.M.Choi,E.G.Lim,J.I.Cho,D.H.Hwang,StereoVisionSystemand incontinuouscostspaces,in:Proc.ofIROS,2008.
StereoVisionProcessingMethod(2012). [62] H.Liu,D.Ding,W.Wan,PredictiveModelforPathPlanningusingK-
[39] P.E.Bauhahn, B.S.Fritz, B.C.Krafthefer, SystemsandMethodsfor nearDynamicBrdigeBuilderandInnerParzenWindow,in:Proceedings
SafeLaserImaging(2009). ofInternationalConferenceonIntelligentRobotsandSystems(IROS),
[40] R. B. Rusu, S. Cousins, 3D is Here: Point Cloud Library (PCL), in: 2008.
Proceedings of International Conference on Robotics and Automation [63] S.kookYun,CompliantManipulationforPeg-in-Hole: isPassiveCom-
10