Real-Time Reconstruction of Static and Dynamic Scenes Der Technischen Fakultät der Friedrich-Alexander-Universität Erlangen-Nürnberg zur Erlangung des Grades Dr.-Ing. vorgelegtvon Michael Zollhöfer ausHerzogenaurach AlsDissertationgenehmigt vonderTechnischenFakultät derFriedrich-Alexander-UniversitätErlangen-Nürnberg TagdermündlichenPrüfung: 22.12.2014 VorsitzendedesPromotionsorgans: Prof.Dr.-Ing.habil.MarionMerklein Gutachter: Prof.Dr.GüntherGreiner Prof.Dr.Christianeobalt Revision1.00 ©2014,CopyrightMichaelZollhöfer [email protected] AllRightsReserved AlleRechtevorbehalten Abstract WiththereleaseoftheMicrosoXbox360Kinect,anaffordablereal-time RGB-Dsensorisnowavailableonthemassmarket. ismakesnewtech- niques and algorithms, which have previously been only available to re- searchersandenthusiasts, accessibleforaneverydayusebyabroadaudi- ence. Applicationsrangefromtheacquisitionofdetailedhigh-qualityre- constructionsofeverydayobjectstotrackingthecomplexmotionsofpeo- ple.Inaddition,thecaptureddatacanbedirectlyexploitedtobuildvirtual reality applications, i.e. virtual mirrors, and can be used for gesture con- trolofdevicesandmotionanalysis.Tomaketheseapplicationseasy-to-use inoureverydaylife, theyshouldbeintuitivetocontrolandprovidefeed- backatreal-timerates.Inthisdissertation,wepresentnewtechniquesand algorithmsforbuildingthree-dimensionalrepresentationsofarbitraryob- jectsusingonlyasinglecommodityRGB-Dsensor, manuallyeditingthe acquiredreconstructionsandtrackingthenon-rigidmotionofphysically deforming objects at real-time rates. We start by proposing the use of a statisticalpriortoobtainhigh-qualityreconstructionsofthehumanhead usingonlyasinglelow-qualitydepthframeofacommoditysensor.Weex- tendthisapproachandobtainevenhigherqualityreconstructionsatreal- timeratesbyexploitingallinformationofacontiguousRGB-Dstreamand jointly optimizing for shape, albedo and illumination parameters. ere- aer,weshowthatamovingsensorcanbeusedtoobtainsuper-resolution reconstructionsofarbitraryobjectsatsensorratebyfusingalldepthobser- vations. Wepresentstrategiesthatallowustohandleavirtuallyunlimited reconstructionvolumebyexploitinganewsparsescenerepresentationin combinationwithanefficientstreamingapproach. Inaddition,wepresent ahandlebaseddeformationparadigmthatallowstheusertoeditthecap- turedgeometry,whichmightconsistofmillionsofpolygons,usinganinter- activeandintuitivemodelingmetaphor. Finally,wedemonstratethatthe motionofarbitrarynon-rigidlydeformingphysicalobjectscanbetracked atreal-timeratesusingacustomhigh-qualityRGB-Dsensor. i Acknowledgements is dissertation captures the compressed technical contributions and al- gorithmicachievementsoffoursolidyearsofwork. Istartedmyworkin January2011withastronginterestingeometryprocessing, optimization problems, rendering and GPGPU programming, but quite clueless about a suitable field of research. Little did I know that all my rather different interestswouldmagicallylineup... afewyearslater. isallwouldnothavebeenpossiblewithouttheconstantsupport,friendly encouragements and selfless contributions of my colleagues and cowork- ers.MysupervisorGüntherGreineralwayssupportedmeandgavemethe freedom to fully pursue my own research interests. Jochen Süßmuth in- troducedmetotheworldofnon-rigidregistrationandsupervisedmedur- ingmystudiesandinthefirstyearofmyPhD.MatthiasNießnerhelped me to rediscover my will to conduct research and fueled my curiosity in online 3D reconstruction methods. Marc Stamminger helped during all deadlineswithhisadviceandcraedalotofamazingillustrations. Frank Bauer, our Blender guru, shared his tricks and helped with the produc- tion of videos. I am thankful for all this support and feel in great debt. Whatimpressedmemostisthatallofmycolleaguesmanagedtoendure myconstantrantsaboutthealgorithmsthatbreakunderrealworldcondi- tions:ChristianSiegl,KaiSelgrad,MagdalenaPrus,MichaelMartinek,Jan Kretschmer, Quirin Meyer, Roberto Grosso, Matteo Colaianni, Matthias Innmann, Henry Schäfer, Benjamin Keinert, Franziska Bertelshofer and ChristophWeber. I feelprivileged for theopportunity towork in such a greatandcreativeenvironmentwithpeoplethathavebecomemorethan justcolleaguestomeoverthelastfouryears. Iwouldalsoliketothankallofmystudentsthatworkedwithmeduring theirBachelorandMasterprojects. Togetherwelearnedalotandgotex- posedtonewinterestingtopicsandideas. EzgiSerthelpeddevelopingthe presentedlatticebaseddeformationapproach. Justusieshelpedmesub- stantially with my first DFG project and the presented interactive model iii basedreconstructionmethod.Iamhappythathewillcontinuemyresearch atthechair. ankstoShahramIzadiforgivingmetheopportunitytoconductresearch in his group at Microso Research Cambridge. I met a lot of cool peo- pleduringmystay:AndrewFitzgibbon,ChristophRhemann,Christopher Zach,DavidKim,CemKeskinandSeanFanello.Itwasathrillingtimefull ofhardwork,excellentfoodandgreatparties. ankstoMatthiasNießner,MatthewFisher,ChengleiWuandChristian eobalt for helping with my real-time non-rigid reconstruction project andAngelaDaiforthenicevoiceover. IamalsogratefultotheGermanResearchFoundation(DFG)forfunding myworkoverthelastfouryearsundergrantSTA-662/3--1andGRK-1773. Lastbutnotleast,thankstomyparentsKerstinandFranzforsupporting meonallofmyendeavors<3. October 2014 MZ iv Contents 1 MotivationandFundamentals 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Real-TimeRGB-DSensors. . . . . . . . . . . . . . . . . . 2 1.3 GeneralPurposeGPUProgramming . . . . . . . . . . . . 7 2 OptimizationTheory 11 2.1 ModelFitting . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 LeastSquaresOptimization . . . . . . . . . . . . . . . . . 13 2.3 LinearLeastSquaresOptimization . . . . . . . . . . . . . 15 2.4 Non-linearLeastSquaresOptimization . . . . . . . . . . . 16 3 ContributionandOutline 19 I ReconstructionofPersonalizedAvatars 23 4 Introduction 25 5 RelatedWork 27 6 Method 29 6.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 6.2 DataAcquisition . . . . . . . . . . . . . . . . . . . . . . . 29 6.2.1 DataPreparation . . . . . . . . . . . . . . . . . . 30 6.2.2 FaceandFeatureDetection . . . . . . . . . . . . . 31 6.2.3 FaceSegmentation. . . . . . . . . . . . . . . . . . 33 6.3 FittingaGenericFaceModel . . . . . . . . . . . . . . . . 33 6.3.1 FittingEnergyTerm . . . . . . . . . . . . . . . . . 35 6.3.2 RegularizationEnergyTerm . . . . . . . . . . . . 36 6.3.3 Optimization . . . . . . . . . . . . . . . . . . . . 36 6.3.4 Fittingthemorphablefacemodel . . . . . . . . . . 37 v Contents 7 Results 39 8 Conclusion 41 II ModelbasedReconstructionoftheHumanHead 43 9 Introduction 45 10 RelatedWork 47 10.1 Model-free3D-Reconstruction . . . . . . . . . . . . . . . 47 10.2 Model-based3D-Reconstruction . . . . . . . . . . . . . . 48 11 Method 51 11.1 PipelineOverview . . . . . . . . . . . . . . . . . . . . . . 51 11.2 HeadPoseEstimation . . . . . . . . . . . . . . . . . . . . 52 11.3 DataFusion . . . . . . . . . . . . . . . . . . . . . . . . . . 53 11.4 EstimatingModelParameters . . . . . . . . . . . . . . . . 54 11.4.1 StatisticalShapeModel . . . . . . . . . . . . . . . 54 11.4.2 ObjectiveFunction . . . . . . . . . . . . . . . . . 55 11.4.3 ParameterInitialization . . . . . . . . . . . . . . . 56 11.4.4 JointNon-LinearGPUOptimizer . . . . . . . . . 57 12 Results 59 12.1 RuntimeEvaluation . . . . . . . . . . . . . . . . . . . . . 59 12.2 ReconstructionQuality . . . . . . . . . . . . . . . . . . . 60 12.3 Applications . . . . . . . . . . . . . . . . . . . . . . . . . 60 13 Conclusion 65 III Online3DReconstructionatScale 67 14 Introduction 69 15 Relatedwork 71 vi
Description: