monocular pose estimation and shape reconstruction of quasi-articulated objects with consumer

170 Pages·2017·16.04 MB·English

UUnniivveerrssiittyy ooff KKeennttuucckkyy
UUKKnnoowwlleeddggee
Theses and Dissertations--Computer Science Computer Science
2014
MMOONNOOCCUULLAARR PPOOSSEE EESSTTIIMMAATTIIOONN AANNDD SSHHAAPPEE
RREECCOONNSSTTRRUUCCTTIIOONN OOFF QQUUAASSII--AARRTTIICCUULLAATTEEDD OOBBJJEECCTTSS WWIITTHH
CCOONNSSUUMMEERR DDEEPPTTHH CCAAMMEERRAA
Mao Ye
University of Kentucky, [email protected]
RRiigghhtt cclliicckk ttoo ooppeenn aa ffeeeeddbbaacckk ffoorrmm iinn aa nneeww ttaabb ttoo lleett uuss kknnooww hhooww tthhiiss ddooccuummeenntt bbeenneefifittss yyoouu..
RReeccoommmmeennddeedd CCiittaattiioonn
Ye, Mao, "MONOCULAR POSE ESTIMATION AND SHAPE RECONSTRUCTION OF QUASI-ARTICULATED
OBJECTS WITH CONSUMER DEPTH CAMERA" (2014). Theses and Dissertations--Computer Science. 25.
https://uknowledge.uky.edu/cs_etds/25
This Doctoral Dissertation is brought to you for free and open access by the Computer Science at UKnowledge. It
has been accepted for inclusion in Theses and Dissertations--Computer Science by an authorized administrator of
UKnowledge. For more information, please contact [email protected].
SSTTUUDDEENNTT AAGGRREEEEMMEENNTT::
I represent that my thesis or dissertation and abstract are my original work. Proper attribution
has been given to all outside sources. I understand that I am solely responsible for obtaining
any needed copyright permissions. I have obtained needed written permission statement(s)
from the owner(s) of each third-party copyrighted matter to be included in my work, allowing
electronic distribution (if such use is not permitted by the fair use doctrine) which will be
submitted to UKnowledge as Additional File.
I hereby grant to The University of Kentucky and its agents the irrevocable, non-exclusive, and
royalty-free license to archive and make accessible my work in whole or in part in all forms of
media, now or hereafter known. I agree that the document mentioned above may be made
available immediately for worldwide access unless an embargo applies.
I retain all other ownership rights to the copyright of my work. I also retain the right to use in
future works (such as articles or books) all or part of my work. I understand that I am free to
register the copyright to my work.
RREEVVIIEEWW,, AAPPPPRROOVVAALL AANNDD AACCCCEEPPTTAANNCCEE
The document mentioned above has been reviewed and accepted by the student’s advisor, on
behalf of the advisory committee, and by the Director of Graduate Studies (DGS), on behalf of
the program; we verify that this is the final, approved version of the student’s thesis including all
changes required by the advisory committee. The undersigned agree to abide by the statements
above.
Mao Ye, Student
Dr. Ruigang Yang, Major Professor
Dr. Miroslaw Truszczynski, Director of Graduate Studies
MONOCULARPOSEESTIMATIONANDSHAPERECONSTRUCTIONOF
QUASI-ARTICULATEDOBJECTSWITHCONSUMERDEPTHCAMERA
DISSERTATION
Adissertationsubmittedinpartialfulfillmentof
therequirementsforthedegreeof
DoctorofPhilosophy
intheCollegeofEngineering
attheUniversityofKentucky
By
MaoYe
Lexington,Kentucky
Director: Dr. RuigangYang,ProfessorofComputerScience
Lexington,Kentucky2014
Copyright©MaoYe2014
ABSTRACTOFDISSERTATION
MONOCULARPOSEESTIMATIONANDSHAPERECONSTRUCTIONOF
QUASI-ARTICULATEDOBJECTSWITHCONSUMERDEPTHCAMERA
Quasi-articulated objects, such as human beings, are among the most commonly seen ob-
jectsinourdailylives. Extensiveresearchhavebeendedicatedto3Dshapereconstruction
and motion analysis for this type of objects for decades. A major motivation is their wide
applications, such as in entertainment, surveillance and health care. Most of existing stud-
iesreliedononeormoreregularvideocameras. Inrecentyears,commoditydepthsensors
have become more and more widely available. The geometric measurements delivered by
thedepthsensorsprovidesignificantlyvaluableinformationforthesetasks. Inthisdisserta-
tion, we propose three algorithms for monocular pose estimation and shape reconstruction
ofquasi-articulatedobjectsusingasinglecommoditydepthsensor. Thesethreealgorithms
achieveshapereconstructionwithincreasinglevelsofgranularityandpersonalization. We
then further develop a method for highly detailed shape reconstruction based on our pose
estimationtechniques.
Ourfirstalgorithmtakesadvantageofamotiondatabaseacquiredwithanactivemarker-
basedmotioncapturesystem. Thismethodcombinesposedetectionthroughnearestneigh-
bor search with pose refinement via non-rigid point cloud registration. It is capable of ac-
commodatingdifferentbodysizesandachievesmorethantwicehigheraccuracycompared
toapreviousstateoftheartonapubliclyavailabledataset.
The above algorithm performs frame by frame estimation and therefore is less prone
to tracking failure. Nonetheless, it does not guarantee temporal consistent of the both the
skeletalstructureandtheshapeandcouldbeproblematicforsomeapplications. Toaddress
this problem, we develop a real-time model-based approach for quasi-articulated pose and
3D shape estimation based on Iterative Closest Point (ICP) principal with several novel
constraintsthatare criticalformonocularscenario. Inthisalgorithm, wefurtherproposea
novelmethodforautomaticbodysizeestimationthatenablesitscapabilitytoaccommodate
differentsubjects.
Duetothelocalsearchnature,theICP-basedmethodcouldbetrappedtolocalminima
inthecaseofsomecomplexandfastmotions. Toaddressthisissue,weexplorethepoten-
tialofusingstatisticalmodelforsoftpointcorrespondencesassociation. Towardsthisend,
weproposeaunifiedframeworkbasedonGaussianMixtureModelforjointposeandshape
estimation of quasi-articulated objects. This method achieves state-of-the-art performance
onvariouspubliclyavailabledatasets.
Based on our pose estimation techniques, we then develop a novel framework that
achieves highly detailed shape reconstruction by only requiring the user to move naturally
in front of a single depth sensor. Our experiments demonstrate reconstructed shapes with
richgeometricdetailsforvarioussubjectswithdifferentapparels.
Lastbutnottheleast,weexploretheapplicabilityofourmethodontworeal-worldap-
plications. First of all, we combine our ICP-base method with cloth simulation techniques
for Virtual Try-on. Our system delivers the first promising 3D-based virtual clothing sys-
tem. Secondly,weexplorethepossibilitytoextendourposeestimationalgorithmstoassist
physical therapist to identify their patients movement dysfunctions that are related to in-
juries. Our preliminary experiments have demonstrated promising results by comparison
with the gold standard active marker-based commercial system. Throughout the disserta-
tion, we develop various state-of-the-art algorithms for pose estimation and shape recon-
struction of quasi-articulated objects by leveraging the geometric information from depth
sensors. Wealsodemonstratetheirgreatpotentialsfordifferentreal-worldapplications.
KEYWORDS: Pose Estimation, Shape Reconstruction, Articulated Objects, Depth Sen-
sor,Non-rigidRegistration
Author’ssignature: MaoYe
Date: December17,2014
MONOCULARPOSEESTIMATIONANDSHAPERECONSTRUCTIONOF
QUASI-ARTICULATEDOBJECTSWITHCONSUMERDEPTHCAMERA
By
MaoYe
DirectorofDissertation: RuigangYang
DirectorofGraduateStudies: MiroslawTruszczynski
Date: December17,2014
TOMYBELOVEDFAMILIES
MygrandparentsZaishengYeandYuhuaHuang
MyparentsYongmingYeandHuipingHou
MybrotherLiangYe
MywifeZiyuJia
ACKNOWLEDGMENTS
Fortheseyears,Ihavereceivedatremendousamountofhelpandsupportfrommyadvisor,
my colleagues, my friends and my families. Without them, I will not have been able to
finishthisdissertationandearnmyPh.D.degree.
I would like to express my first sincere appreciation to my advisor, Dr. Ruigang Yang,
for introducing me into the exciting area of computer vision and for years of support and
guidance throughout my Ph.D. journey. I am so grateful that I have been able to conduct
research on topicsthat I am really interestedin under Dr. Yang’s supervision. Theinspira-
tions and encouragement I have received from Dr. Yang have had significantly impacts on
my ways of thinking, exploring and doing. I am more than fortunate to have been able to
walkthroughmyPh.D.journeywithhim.
I also want to thank all my committee members, Dr. Fuhua Cheng, Dr. Sen-ching
Cheung, Dr. Nathan Jacobs and Dr. Qiang Ye. I really appreciate their efforts on guiding
mystudyandonmydissertations.
IhavealsobeensoluckytosharemyPh.D.lifewithagroupoflovelypeople. Theyare
allsocreative,inspiringtoworkwithandsonicetogetalongwith. Weworkedandplayed
together for all these memorable years. They have all contributed to my work and my
life. Iwanttoexpressmythankfulnesstoallofthem,inparticularLiangWang,Xianwang
Wang,XinyuHuang,MiaoLiao,QingZhang,JizhouGao,ChenxiZhang,BoFu,HuiLin,
ChangpengTi,YajieZhao,ChaoDu,YongwookSong,YinHu,YanHuang,XinanLiuand
YigongZhang.
Mostimportantly,mymostsinceregratitudesareforallmyfamilymemberswhohave
offered me unconditional love throughout my life, in particular my grandparents, my par-
ents, my brother and my wife. They have always been the most significant part of my life,
iii
mygreatestandstrongestsupportwheneverIwasdown. Everythinggoodinmylife,Iowe
tothem.
iv
TABLEOFCONTENTS
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
TableofContents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
ListofTables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
ListofFigures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Chapter1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 HistoricReview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Chapter2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1 Marker-lessMotionCapture . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1.1 Color-basedMethods . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.2 Depth-basedMethods . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.3 TemplateAdaptation . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 ShapeReconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.1 Model-basedApproaches . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.2 Model-freeApproaches . . . . . . . . . . . . . . . . . . . . . . . . 16
Chapter3 Preliminaries: ShapeandPoseRepresentations . . . . . . . . . . . . . 18
3.1 ArticulatedDeformationModel . . . . . . . . . . . . . . . . . . . . . . . . 18
3.2 LinearBlendSkinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Twist-BasedRepresentation . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4 IncrementalPoseUpdate . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Chapter4 DataDrivenHumanPoseEstimation . . . . . . . . . . . . . . . . . . . 26
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 PointCloudSegmentationandDenoising . . . . . . . . . . . . . . . . . . 28
4.3 ModelBasedMotionEstimation . . . . . . . . . . . . . . . . . . . . . . . 31
4.3.1 PointCloudAlignment . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3.2 Nearest-NeighborSearchinLow-DimensionalSubspace . . . . . . 33
4.4 ExperimentalResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.4.1 TheEffectivenessofDenoising . . . . . . . . . . . . . . . . . . . . 36
4.4.2 TheEffectivenessofPoseRefinement . . . . . . . . . . . . . . . . 36
4.4.3 ViewpointIndependency . . . . . . . . . . . . . . . . . . . . . . . 37
4.4.4 FailureDetection . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.4.5 DatabaseDependency . . . . . . . . . . . . . . . . . . . . . . . . 39
v