Cage-based Performance Capture

148 Pages·2014·5.347 MB·English

by  Yann Savoye (auth.)

Studies in Computational Intelligence 509
Yann Savoye
Cage-based
Performance
Capture
Studies in Computational Intelligence
Volume 509
Series Editor
J. Kacprzyk, Warsaw, Poland
For furthervolumes:
http://www.springer.com/series/7092

Yann Savoye
Cage-based Performance
Capture
123
Dr. YannSavoye
Lyon
France
ISSN 1860-949X ISSN 1860-9503 (electronic)
ISBN 978-3-319-01537-8 ISBN 978-3-319-01538-5 (eBook)
DOI 10.1007/978-3-319-01538-5
SpringerChamHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2013949253
(cid:2)SpringerInternationalPublishingSwitzerland2014
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor
informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodology now known or hereafter developed. Exempted from this legal reservation are brief
excerpts in connection with reviews or scholarly analysis or material supplied specifically for the
purposeofbeingenteredandexecutedonacomputersystem,forexclusiveusebythepurchaserofthe
work. Duplication of this publication or parts thereof is permitted only under the provisions of
theCopyright Law of the Publisher’s location, in its current version, and permission for use must
always be obtained from Springer. Permissions for use may be obtained through RightsLink at the
CopyrightClearanceCenter.ViolationsareliabletoprosecutionundertherespectiveCopyrightLaw.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Our deepest fear is not that we are
inadequate. Our deepest fear is that we are
powerful beyond measure. It is our light, not
our darkness that most frightens us. [...]
Your playing small does not serve the world.
There is nothing enlightened about shrinking
so that other people won’t feel insecure
around you. We are all meant to shine, [...].
It’s not just in some of us; it’s in everyone.
And as we let our own light shine, we
unconsciously give other people permission
to do the same. As we are liberated from our
own fear, our presence automatically
liberates others.
Marianne Williamson
A Return to Love: Reflections on the
Principles of a Course in Miracles
Acknowledgments
The armadillo and horse models are courtesy of Stanford University.
The crane, handstand, and samba mesh sequences datasets are courtesy of MIT
Adobe. Cited figures are copyrighted by corresponding authors and publishers
(please see associated references as copyright notice).
vii
Contents
1 General Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Research Context. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 From Real-World to Vision-based Animation. . . . . . . . . 3
1.1.2 High-Level Research Directions . . . . . . . . . . . . . . . . . . 4
1.1.3 Domains of Application. . . . . . . . . . . . . . . . . . . . . . . . 7
1.2 The Problem: Non-rigid Parametrization
for Video-based Animation. . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3 Motivation, Objectives and Novelty. . . . . . . . . . . . . . . . . . . . . 10
1.4 Major Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.5 Our Original Contributions. . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 Thesis Organization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 Sparse Constraints Over Animatable Subspaces . . . . . . . . . . . . . . 17
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2 Related Works on Interactive Deformable Surface. . . . . . . . . . . 19
2.2.1 Deformable Models. . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.2 Surface-based Parameterization. . . . . . . . . . . . . . . . . . . 21
2.2.3 Skeleton-based Parameterization. . . . . . . . . . . . . . . . . . 22
2.2.4 Space-based Parameterization. . . . . . . . . . . . . . . . . . . . 23
2.2.5 Cage-based Parameterization . . . . . . . . . . . . . . . . . . . . 24
2.3 Our Approach: Cage-based Shape Inverse kinematics . . . . . . . . 28
2.3.1 Laplacian-based Mean-Value Cage . . . . . . . . . . . . . . . . 30
2.3.2 Dual-Laplacian Surface Regularization . . . . . . . . . . . . . 33
2.3.3 Screen-space Surface Constraints . . . . . . . . . . . . . . . . . 34
2.3.4 Indirect Dual-Laplacian Cage-based Fitting . . . . . . . . . . 36
2.3.5 Experimental Results and Evaluation. . . . . . . . . . . . . . . 38
2.3.6 A Mathematical Study. . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4 Beyond the Cage: A Generalization to Skeletal Structures . . . . . 42
2.4.1 Differential Skeletal Editing. . . . . . . . . . . . . . . . . . . . . 43
2.4.2 Rigidity-Breaking Skeletal Optimization . . . . . . . . . . . . 43
2.4.3 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.5 Discussion and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 47
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
ix
x Contents
3 Reusing Performance Capture Data . . . . . . . . . . . . . . . . . . . . . . . 53
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.2 Related Works from Capture to Re-Use of Dynamic Surfaces. . . 55
3.2.1 Acquiring Surface in Motion . . . . . . . . . . . . . . . . . . . . 55
3.2.2 Performance Capture Reuse . . . . . . . . . . . . . . . . . . . . . 61
3.3 Our Approach: Cage-based Animation Fitting. . . . . . . . . . . . . . 65
3.3.1 Non-rigid Dynamic Shape Analysis. . . . . . . . . . . . . . . . 65
3.3.2 Cage-based Conversion from Performance
Mesh Animation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.3.3 Estimation of Space-Time Cages
for Non-rigid Surfaces. . . . . . . . . . . . . . . . . . . . . . . . . 69
3.3.4 Experimental Results and Video-based Evaluation . . . . . 71
3.4 Beyond the Cage: Animation Cartoonization. . . . . . . . . . . . . . . 77
3.4.1 Foundation of Animation Cartoonization . . . . . . . . . . . . 77
3.4.2 Cartoonization of Multi-View Data. . . . . . . . . . . . . . . . 78
3.4.3 Space-Time Exaggerating of Life-Like Surfaces. . . . . . . 80
3.4.4 Depicting Video-Infused Appearance. . . . . . . . . . . . . . . 84
3.4.5 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4 Toward Non-rigid Dynamic Cage Capture . . . . . . . . . . . . . . . . . . 93
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.2 Related Works from Reconstruction to Registration. . . . . . . . . . 96
4.2.1 Image-based Shape Reconstruction . . . . . . . . . . . . . . . . 96
4.2.2 Non-rigid Shape Registration . . . . . . . . . . . . . . . . . . . . 101
4.3 Our Approach: Handle-Aware Detached Registration. . . . . . . . . 108
4.3.1 Non-rigid Registration Setup . . . . . . . . . . . . . . . . . . . . 108
4.3.2 Target Point-Clouds Reconstruction . . . . . . . . . . . . . . . 109
4.3.3 Normal-Guided Pairwise Correspondences. . . . . . . . . . . 113
4.3.4 Iterative Elasto-Plastic Optimization . . . . . . . . . . . . . . . 115
4.3.5 Weight-Control Update Rules. . . . . . . . . . . . . . . . . . . . 119
4.3.6 Experimental Results. . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.4 Beyond The Cage: Toward Captured RGB-Images . . . . . . . . . . 123
4.5 Discussions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
5 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
5.2 Overall Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.3 Perspectives and Future Directions . . . . . . . . . . . . . . . . . . . . . 141
Chapter 1
General Introduction
A
s an introduction to this book, we provide a general overview concerning the
reusabledigitalizationofcapturedanimation:fromcapturingthereal-worldto
editing of video-based animation. In particular, we present and discuss a concise
statementconcerningtheproblemofnon-rigidparametrizationforvideo-basedcap-
tured animation. To enforce the importance of this problem, we demonstrate the
impactofthisresearchareabythevariousdomainsofapplication.Inaddition,we
clearly exhibit the motivations, objectives, and difficulties of the tackled class of
problems.Hence,weyetuncoveraresearchroadmapofourapproachandoriginal
contributionstoalleviatethisproblem.Finally,wedrawthechapters’organization
forthisbook.
1.1 ResearchContext
FortyYears.OverfortyyearshavebeenrequiredinthefieldofComputerGraphics
andComputerVisiontobematureenoughfromthefirstuseofcomputer-generated
threedimensionalgraphicalmodel(asseeninthemovieFutureWorldin1976),tothe
firstaccomplishedphotorealistic3Dmovie(asseeninthemovieAvatar in2009).
There is no doubt that Computer-based Imagery Technologies have significantly
influencedandchangedmodernsociety,andhaveasignificantimpactonoureveryday
lifeoverthelasttwodecades.Inthepast,technologyhasmadeenormousleapsin
termofpowerandavailability.Moreprecisely,tremendoustechnologicalprogress
ledtoaffordableCPUcomputationpowerinrecentyears.Memoryandgraphiccards
haveboostedthegenerationofsynthesisimages.Intheeraofcomputers,theadvent
oftechnologieshasenablednewpossibilitiesinrendering,animation,modelingand
real-timewhole-bodyinteraction(seecontrastbetweenFigs.1.1and1.2).
Strong Duality. On the one hand, Computer Graphics techniques deal with the
designofcomputermodelsorrepresentationinordertoproducehigh-qualityimages.
On the other hand, Computer Vision tools are able to extract information from
Y.Savoye,Cage-basedPerformanceCapture, 1
StudiesinComputationalIntelligence509,DOI:10.1007/978-3-319-01538-5_1,
©SpringerInternationalPublishingSwitzerland2014