Visual SLAM and Surface Reconstruction for Abdominal Minimally Invasive Surgery

134 Pages·2015·8.86 MB·English

UUnniivveerrssiittyy ooff SSoouutthh FFlloorriiddaa
DDiiggiittaall CCoommmmoonnss @@ UUnniivveerrssiittyy ooff
SSoouutthh FFlloorriiddaa
USF Tampa Graduate Theses and Dissertations USF Graduate Theses and Dissertations
1-1-2015
VViissuuaall SSLLAAMM aanndd SSuurrffaaccee RReeccoonnssttrruuccttiioonn ffoorr AAbbddoommiinnaall
MMiinniimmaallllyy IInnvvaassiivvee SSuurrggeerryy
Bingxiong Lin
University of South Florida, [email protected]
Follow this and additional works at: https://digitalcommons.usf.edu/etd
Part of the Robotics Commons
SScchhoollaarr CCoommmmoonnss CCiittaattiioonn
Lin, Bingxiong, "Visual SLAM and Surface Reconstruction for Abdominal Minimally Invasive Surgery"
(2015). USF Tampa Graduate Theses and Dissertations.
https://digitalcommons.usf.edu/etd/5849
This Dissertation is brought to you for free and open access by the USF Graduate Theses and Dissertations at
Digital Commons @ University of South Florida. It has been accepted for inclusion in USF Tampa Graduate Theses
and Dissertations by an authorized administrator of Digital Commons @ University of South Florida. For more
information, please contact [email protected].
VisualSLAMandSurfaceReconstructionforAbdominalMinimallyInvasiveSurgery
by
BingxiongLin
Adissertationsubmittedinpartialfulfillment
oftherequirementsforthedegreeof
DoctorofPhilosophy
DepartmentofComputerScienceandEngineering
CollegeofEngineering
UniversityofSouthFlorida
MajorProfessor:YuSun,Ph.D.
XiaoningQian,Ph.D.
DmitryGoldgof,Ph.D.
RichardGitlin,Sc.D.
YunchengYou,Ph.D.
DateofApproval:
March20,2015
Keywords:3Dreconstruction,Laparoscopelocalization,Featuredetection,Vesselfeature,Tissue
deformation
Copyright(cid:13)c 2015,BingxiongLin
DEDICATION
Tomyparents.
ACKNOWLEDGMENTS
First and foremost, I would like to thank my advisor, Yu Sun. Over the past five years, he has taught
mehowtochooseresearchtopics,howtomeasuretheresearchprogress,howtoaddressresearchproblems.
He led me to the fascinating 3D Computer Vision when I started my study here, and has helped me grow
and explore in this area since then. His trust in my ability has built up my confidence in my research. He
has shown a great support on my job hunting and his suggestions are invaluable to me. He has also been a
goodfriendinlifeandthe“coffeetime”onFridayafternoonhasbroughtmealotoffun.
I would also like to thank Xiaoning Qian, who has always been a great teacher and friend to me. His
patienceandkindnesssometimesmakemefeelthatheismypeerfriend. Hisdoorisalwaysopenwhenever
I have a question. He has given me massive suggestions and comments on how to approach a research
problemandhowtowriteatechnicalpaper,andhasalwaysbeenpatientwithmystupidgrammarerrors. It
ismygreathonorthathetookaflighttoTampaandattendedmydissertationdefenseinperson.
Of course, I am also very grateful to other committee members and professors who have helped me in
my research. Dr. Dmitry Goldgof has provided many insightful suggestions on my research, especially
duringtheComputerVisionseminars. Dr. RichardGitlinandDr. YunchengYouhavealsoprovidedmany
useful comments. Dr. Jaime Sanchez has brought me to the surgery room, explained to me the surgical
procedures,andhelpedmecollectdatathatiscriticalinthisdissertation.
Iownmythankstomylabmatesandfriends,whohavehelpedmeduringmyresearchandmademylife
heremorecolorful. TheyareAdrianJohnson,YunLin,FillipeSouza,RajmadhanEkambaram,Yongqiang
Huang,JieZhang,JunyiTu,HaifangWu,ShaogangRen,MengLv,YingyingZhang,MuZhou,XiangZuo,
Huixing Zhai, David Paulius, Roger Milton, Baishali Chaudhury, Hao Li, Ran Rui, Tao Wang, Wen Zhao,
LukaiGuo,RuiGuo,WeiYuan,GangLiu,DongpingDu,andZunjunChen.
IwouldnothavepursuedaPhDwithoutthesupportsfrommyfamily. Mymom,DaibiYang,hastaught
medeterminationandhardwork,whichledmethroughmyPhD.Mydad,MinlianLin,hasencouragedme
tofollowmyheart. Mybrothers,BingqiangLinandBingcaiLin,havesupportedmyideasanddecisions.
TABLEOFCONTENTS
LISTOFTABLES iv
LISTOFFIGURES v
ABSTRACT viii
CHAPTER1 MOTIVATIONANDCHALLENGES 1
1.1 BackgroundandMotivation 1
1.2 Challenges 2
1.3 Summary 3
CHAPTER2 LITERATUREREVIEW 4
2.1 NotetoReader 4
2.2 MotivationandMISDatasets 4
2.3 FeatureDetectionandFeatureTracking 6
2.3.1 FeatureDetection 7
2.3.2 FeatureTracking 8
2.3.3 Discussion 10
2.4 ReconstructionwithoutCameraMotion 11
2.4.1 StereoCue 11
2.4.2 ActiveMethods 12
2.4.3 ShadingandShadowCue 13
2.4.4 Discussion 14
2.5 RigidMIS-VSLAM 14
2.5.1 MonocularCamera 16
2.5.2 StereoCameras 19
2.5.3 Discussion 20
2.6 DynamicMIS-VSLAM 20
2.6.1 State-of-the-artDSSFM 21
2.6.1.1 DSSFMwithMonocularCameras 21
2.6.1.2 DSSFMwithStereoCameras 23
2.6.2 DSSFMinMISEnvironment 24
2.6.3 MovingInstrumentTracking 26
2.6.4 Discussion 27
i
CHAPTER3 VESSEL-BASEDIMAGEFEATUREDETECTION 28
3.1 NotetoReader 28
3.2 VesselFeatureIntroduction 28
3.3 VesselFeatureDetection 29
3.3.1 MethodOverview 29
3.3.2 DetectingCandidateBranchingPoints 30
3.3.3 BloodVesselEnhancement: Vesselness 32
3.3.4 BloodVesselEnhancement: Ridgeness 33
3.3.5 BranchingPointDetection(RBCT) 35
3.3.6 ConnectedComponentLabelingandNon-maximalSuppression 37
3.3.7 BranchingSegmentDetection(RBSD) 38
3.3.8 ComputationalAnalysisandRunTimeResults 40
3.4 ExperimentsandResults 41
3.4.1 InVivoDatasetsandGroundTruths 42
3.4.2 RepeatabilityandNumberofPoints 45
3.4.3 PatchMatchingCorrectness 47
CHAPTER4 VESSEL-FEATURE-BASED3DRECONSTRUCTION 50
4.1 TraditionalStereoMatching 50
4.2 VesselFeatureStereoMatching 51
4.2.1 BranchingPointMatching 52
4.2.2 BranchingSegmentMatching 54
4.3 Large-areaDense3DReconstruction 57
4.4 ExperimentsandResults 58
4.4.1 InVivoDatasets 59
4.4.2 VesselFeatureStereoMatchingAccuracy 59
4.4.3 Large-area3DReconstructionResults 60
CHAPTER5 SHADOW-CASTING-BASED3DRECONSTRUCTION 62
5.1 NotetoReader 62
5.2 Introduction 62
5.3 Shadow-Scanning-Based3DReconstruction 63
5.3.1 SystemOverview 63
5.3.2 ExtractingShadowCurves 64
5.3.3 IntersectingShadowCurveswithEpipolarLines 67
5.3.4 FieldSurfaceInterpolation 68
5.3.5 3DReconstruction 71
5.4 ExperimentsandResults 71
5.4.1 PhantomandexvivoImages 72
5.4.2 DisparityMaps 74
5.4.3 3DReconstructionResults 75
5.4.4 NumericalComparison 77
5.4.5 RobustnessAnalysis 80
ii
CHAPTER6 VISUALSLAMINADEFORMINGENVIRONMENT 83
6.1 NotetoReader 83
6.2 Introduction 83
6.2.1 PTAM 84
6.3 StereoscopePTAM 84
6.3.1 DeformingPointDetection 85
6.3.2 Non-linearStereoPoseRefinement 87
6.3.3 MapInitialization 88
6.3.4 StereoBundleAdjustment 88
6.4 ExperimentalandResults 88
6.4.1 TrackingAccuracy 88
6.4.2 EvaluationwithInVivoData 91
CHAPTER7 PERIPHERYAUGMENTATIONSYSTEMANDEVALUATION 94
7.1 PeripheryAugmentationSystemDesign 94
7.2 InVivoEvaluation 95
7.2.1 InterfaceandPointEstimationProcedure 96
7.2.2 EnvironmentAwarenessMeasures 98
7.2.3 ResultsandAnalysis 98
CHAPTER8 CONCLUSIONANDFUTUREDIRECTIONS 101
8.1 SummaryofContributions 101
8.2 FutureResearchDirections 102
REFERENCES 104
APPENDICES 118
AppendixA PermissiontoReproduceMaterial 119
iii
LISTOFTABLES
Table2.1 SummaryofpubliclyavailableMISdatasets. 6
Table2.2 SummaryofdifferentapproachesinDSSFM. 22
Table2.3 DynamicvisualSLAMinMIS. 24
Table2.4 Summaryofthestate-of-the-artmethodsinMIS-VSLAM. 25
Table3.1 Eigenvaluesanalysistowardsvesselandbranchingpoints(0 < λ1 < λ2). 31
Table3.2 RuntimeofdifferentstepsinRBSD. 41
Table3.3 Parametersofstate-of-the-artfeaturedetectorsusedinthischapter. 42
Table3.4 Summaryoftheadopteddatasets. 43
Table3.5 ThetotalnumberofpointsinallbranchingsegmentsdetectedbyRBSD. 46
Table4.1 Summaryofdatasetsusedhere. 59
Table4.2 Thedisparityerrorsand3Dpositionerrorsoftheproposevesselfeature
matchingmethodacrossthesixscenes. 60
Table5.1 Disparityerrorofthefourmethodsoverthefiveexperiments. 80
Table5.2 3Dpositionerrorofthefourmethodsoverthefiveexperiments. 80
Table5.3 Thepercentageofpixelsinimagewithvaliddisparityvalue. 81
Table6.1 Meanerrorandvarianceofthetrackingresults. 89
Table7.1 Theoverallmeanandstandarddeviationof theangleerrorsanditsStu-
dent’st-testresults. 99
Table7.2 The overall mean and standard deviation of the pixel errors and its Stu-
dent’st-testresults. 99
iv
LISTOFFIGURES
Figure1.1 ThediagramofatypicalabdominalMISsetup. 2
Figure2.1 ThedichotomyofMIS-VSLAMmethodsbasedoncameramotionsand
scenetypes. 5
Figure2.2 IllustrationofMIS-VSLAMmethodsinrigidandstaticscenes. 15
Figure2.3 IllustrationofMIS-VSLAMmethodsindynamicordeformingscenes. 21
Figure3.1 Illustration of vessel features: branching point (detected by RBCT) and
branchingsegment(detectedbyRBSD). 30
Figure3.2 Overviewofthebranchingpointandbranchingsegmentdetection. 30
Figure3.3 IllustrationofeigenvaluesofHessianmatrix(0 < λ1 < λ2). 32
Figure3.4 ExampleofvesselenhancementusingFrangivesselness. 33
Figure3.5 Comparisonofa)binaryridgeimageandb)ourridgenessimage. 35
Figure3.6 Typicalexampleofacircletestatabranchingpointona)rawimage,b)
vesselnessimage,c)binaryridgeimage,andd)ridgenessimage. 36
Figure3.7 Ridgenessvalueofpixelsalongthecirclearoundabranchingpoint. 37
Figure3.8 Candidatebranchingpointsa)beforeandb)afterRBCT. 37
Figure3.9 Illustrationofthetwo-passtestvesseltracingprocess. 39
Figure3.10 Illustrationofbranchingsegmentdetection. 40
Figure3.11 Sampleimagesoftheseveninvivovideoclips. 43
Figure3.12 Illustrationofselectedgroundtruthpointcorrespondences(green)across
differentviewsinscene3. 44
Figure3.13 Repeatabilityscoresofdifferentmethodsinthesevenscenes. 46
Figure3.14 Thenumberoffeaturepointsdetectedbydifferentfeaturepointdetectors
inthesevenscenes. 47
v
Figure3.15 Patchmatchingcorrectnessofdifferentmethodsinthesevenscenes. 49
Figure3.16 Oneexampleofcorrelation-basedpatchmatchingusing“fixedrangeim-
agesearch”withfeaturedetectorRBCT. 49
Figure4.1 Oneexampleofstereoimagesbefore(top)andafterrectification(down). 51
Figure4.2 Thedetectedbranchingpointsareshownascyandots. 53
Figure4.3 Diagramofvesselvectors 53
Figure4.4 Illustrationoflocaladjustingforcandidatebranchingpoints. 54
Figure4.5 Illustrationofbranchingpointmatchingresults. 54
Figure4.6 Illustrationofvesselpixelsearchrange(epipolarband)intherightimage. 56
Figure4.7 Oneexampleofthevessel-pixelmatchingresults. 56
Figure4.8 Illustrationofthevesselpixelmatchingwithonemis-matchedpoint(red
solidrectangle). 57
Figure4.9 Illustrationofvesselpixelmatchingresultsontheridgemask. 57
Figure4.10 Oneexampleofrecovered3Dvesselsfromapairofstereoimages. 58
Figure4.11 Exemplarimages(leftchannel)ofthefivedatasets. 59
Figure4.12 Therecovered3Dvesselnetworkbyintegrating3Dvesselsfromdiffer-
entviews. 60
Figure4.13 The untextured and textured 3D models that are recovered based on the
obtained3Dvesselnetwork. 61
Figure5.1 Outlineoftheproposedmethodwithfourmajorsteps. 64
Figure5.2 One example of a) the difference image and b) the shadow-mask image
withshadowareashownaswhite. 66
Figure5.3 Illustrationofaccumulated-shadow-maskimage. 67
Figure5.4 ShadowcurvesbeforeandafterLWR. 68
Figure5.5 Illustrationofintersectionofshadowcurveandepipolarlines. 69
Figure5.6 Mappingdefinedbyapairofcurvesandepipolarlines. 70
Figure5.7 Mappingdefinedbytwopairsofcurvesandepipolarlines. 71
Figure5.8 Imagesoftheexprimentsetup. 73
vi