3D Video and Its Applications PDF
Preview 3D Video and Its Applications
3D Video and Its Applications Takashi Matsuyama (cid:2) Shohei Nobuhara (cid:2) Takeshi Takai (cid:2) Tony Tung 3D Video and Its Applications TakashiMatsuyama TakeshiTakai GraduateSchoolofInformatics GraduateSchoolofInformatics KyotoUniversity KyotoUniversity Sakyo,Kyoto,Japan Sakyo,Kyoto,Japan ShoheiNobuhara TonyTung GraduateSchoolofInformatics GraduateSchoolofInformatics KyotoUniversity KyotoUniversity Sakyo,Kyoto,Japan Sakyo,Kyoto,Japan ISBN978-1-4471-4119-8 ISBN978-1-4471-4120-4(eBook) DOI10.1007/978-1-4471-4120-4 SpringerLondonHeidelbergNewYorkDordrecht LibraryofCongressControlNumber:2012940250 ©Springer-VerlagLondon2012 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation, broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection with reviews or scholarly analysis or material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer. PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations areliabletoprosecutionundertherespectiveCopyrightLaw. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Whiletheadviceandinformationinthisbookarebelievedtobetrueandaccurateatthedateofpub- lication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityforany errorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,withrespect tothematerialcontainedherein. Printedonacid-freepaper SpringerispartofSpringerScience+BusinessMedia(www.springer.com) To ourcolleagues,students, andfamilies Preface Thisbookaddresses3Dvideoproductiontechnologiesandapplicationsdeveloped by our laboratory in Kyoto University, Japan, over the past ten years and more. In 1996, we started the Cooperative Distributed Vision project, where a group of networkconnectedactivecamerasmonitora3Drealworldscenetocooperatively detect and track people in real time. At the last stage of the project in 1999, we appliedthesystemtosynchronizedmulti-viewvideodatacapturetomeasurefull3D humanshapeandmotion,whichwasthenfollowedbythedevelopmentoftexture mappingmethodstogeneratefull3Dvideoaround2000. Since then, we have been conducting successively work to improve multi-view video capture systems in both image resolution and object movable space, imple- mentparallelprocessingmethodstoreconstruct3Dshapeandmotioninrealtime using a PC cluster system, develop accurate 3D shape and motion reconstruction algorithms as well as high fidelity texture mapping and lighting environment esti- mationmethods.Withthese3Dvideoproductiontechnologies,in2002,westarted toexploreapplicationsof3Dvideoincludinginteractive3Dvisualization,3Dcon- tentediting,anddatacompressionmethodstocultivatetheworldof3Dvideo. Thisbookgivesacomprehensiveviewofthestate-of-the-artof3Dvideoproduc- tion technologies and applications we developed, as well as related contemporary visualinformationmediatechnologieswhichwillhelpgraduatestudentsandyoung researcherstounderstandtheworldof3Dvideo.Sincetheemployedtechnologies includeaverywiderangeoftechnicaldisciplinescoveringrealtimesynchronized multi-viewvideocapture,objecttrackingwithagroupofactivecameras,geometric andphotometriccameracalibration,parallelprocessingbyaPCclustersystem,2D imageandvideoprocessing,3Dshapeandmotionreconstruction,texturemapping and image rendering, lighting environment estimation, attractive 3D visualization, visual contents analysis and editing, 3D body action analysis, and data compres- sion,weputasreferencesbooksandtechnicalsurveypapersonthesefundamental technicalareasforreaderstounderstandbackgroundknowledgeof3Dvideo. Althoughwe haveestablished technicalskills and know-howfor implementing multi-viewvideocapturesystems,andthequalityofgenerated3Dvideohasbeen significantlyimprovedwithadvancedtechnologies,ahighfidelity3Dvideoproduc- vii viii Preface tionsystemanditscasualusagesineverydaylifeenvironmentsarestillourfuture researchtargets.Wehopethisbookwillpromotefurtherexplorationsoftheworld of3Dvideo. TakashiMatsuyama ShoheiNobuhara TakeshiTakai TonyTung Acknowledgements Our research activities on 3D video over the past decade have been supported by theJapanesegovernmentunderseveraldifferentprograms:ResearchfortheFuture ProgrambytheJapanSocietyforthePromotionofScience(1996–2000),Grant-in- Aid for Scientific Research (2001–2003) and National Project on Development of HighFidelityDigitizationSoftwareforLarge-ScaleandIntangibleCulturalAssets (2004–2008) by the Ministry of Education, Culture, Sports, Science & Technol- ogy, and Strategic Information and Communications R&D Promotion Programme bytheMinistryofInternalAffairsandCommunications(2006–2008).Asof2011, we have got another Grant-in-Aid for Scientific Research (2011–2012) and a col- laborative research with Nippon Telegraph and Telephone Corporation to explore furtheradvanced3Dvideotechnologiesandapplications.Weareverygratefulfor allthesesupports. Since the 3D video project has long been one of the major research topics in our laboratory, many undergraduate, master and Ph.D. students, as well as faculty membersandpostdoctoralresearchershavebeenengagedintheproject.Somehave stayed in our laboratory and contributed to this book, and others are working in universities,researchinstitutions,andcompaniestodevelopnewinformationtech- nologies. Among others, Prof. Toshikazu Wada and Dr. Xiaojun Wu established foundationsof3Dvideoproductionsystems,Profs.NorimichiUkitaandShinsaku Hiuraimplementedarealtimeactivemulti-targettrackingsystemwithagroupof network connected pan-tilt-zoom cameras, Dr. Atsuto Maki, Mr. Hiromasa Yoshi- moto, and Mr. Tatsuhisa Yamaguchi developed a 3D video capture system of an object moving in a wide area with a group of active cameras, and Prof. Hitoshi Habe and Dr. Lyndon Hill implemented a sophisticated 3D video coding method. Mr.QunShidevelopedagazeestimationmethodfromcaptured3Dvideo.Without theireffortsandenthusiasm,wecouldnothaveexploredsuchawiderangeof3D videoproductiontechnologiesandapplicationsascoveredinthisbook. Needless to say, our everyday research activities are supported by the staffs of ourlaboratoryanduniversity.Wewouldliketothankallofourformerandcurrent secretariesfortheirdevotedworkonprojectproposalwriting,budgetmanagement, travelscheduleplanning,conferenceandworkshoporganization,paperandhome- ix x Acknowledgements pagepreparation,andsoon.Theircharmingsmilesencouragedustoattackdifficult problemsinresearchaswellasmanagement. Last,butofcoursenotleast,wewouldliketoexpressoursincereappreciations to our families for their everlasting support and encouragement, especially to our wives,Mrs.AkemiMatsuyama,YoshikoNobuhara,ShihoTakai,andBiddaCamilla SolvangPoulsen. Contents 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 VisualInformationMediaTechnologies. . . . . . . . . . . . . . . 1 1.2 WhatIsandIsNot3DVideo? . . . . . . . . . . . . . . . . . . . . 3 1.3 ProcessingSchemeof3DVideoProductionandApplications . . . 5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 PartI Multi-viewVideoCapture 2 Multi-cameraSystemsfor3DVideoProduction . . . . . . . . . . . . 17 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.1 Single-CameraRequirements . . . . . . . . . . . . . . . . 18 2.1.2 Multi-cameraRequirements . . . . . . . . . . . . . . . . . 19 2.2 StudioDesign . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2.1 CameraArrangement . . . . . . . . . . . . . . . . . . . . 20 2.2.2 Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.2.3 Lens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.4 Shutter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.2.5 Lighting . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.2.6 Background . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.2.7 StudioImplementations . . . . . . . . . . . . . . . . . . . 27 2.3 CameraCalibration . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3.1 GeometricCalibration . . . . . . . . . . . . . . . . . . . . 29 2.3.2 PhotometricCalibration . . . . . . . . . . . . . . . . . . . 36 2.4 PerformanceEvaluationof3DVideoStudios . . . . . . . . . . . . 41 2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3 Active Camera System for Object Tracking and Multi-view Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.1.1 Fundamental Requirements for Multi-view Object Observationfor3DVideoProduction . . . . . . . . . . . . 45 xi xii Contents 3.1.2 Multi-viewVideoCaptureforaWideArea . . . . . . . . . 46 3.2 Cell-BasedObjectTrackingandMulti-viewObservation. . . . . . 50 3.2.1 ProblemSpecificationsandAssumptions . . . . . . . . . . 50 3.2.2 BasicSchemeoftheCell-BasedObjectTrackingand Multi-viewObservation . . . . . . . . . . . . . . . . . . . 51 3.2.3 DesignFactorsforImplementation . . . . . . . . . . . . . 52 3.2.4 Cell-BasedCameraControlScheme . . . . . . . . . . . . 54 3.3 AlgorithmImplementation . . . . . . . . . . . . . . . . . . . . . 55 3.3.1 ConstraintsbetweenDesignFactorsandSpecifications . . 56 3.3.2 StudioDesignProcess . . . . . . . . . . . . . . . . . . . . 62 3.3.3 Cell-BasedCameraCalibration . . . . . . . . . . . . . . . 63 3.3.4 Real-TimeObjectTrackingAlgorithm . . . . . . . . . . . 65 3.4 PerformanceEvaluations . . . . . . . . . . . . . . . . . . . . . . 65 3.4.1 QuantitativePerformanceEvaluationswithSynthesized Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.4.2 QuantitativePerformanceEvaluationwithRealActive Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.5 DesigningaSystemforLargeScaleSportScenes . . . . . . . . . 77 3.5.1 ProblemSpecifications . . . . . . . . . . . . . . . . . . . 78 3.5.2 CameraandCellArrangements . . . . . . . . . . . . . . . 80 3.6 ConclusionandFutureWorks . . . . . . . . . . . . . . . . . . . . 83 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 PartII 3DVideoProduction 4 3DShapeReconstructionfromMulti-viewVideoData . . . . . . . . 91 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 4.2 Categorizationof 3D Shape ReconstructionMethods for 3D VideoProduction . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.2.1 VisualCues for Computing3DInformationfrom2D Image(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.2.2 Full3DShapeReconstruction . . . . . . . . . . . . . . . . 96 4.2.3 DynamicFull3DShapeReconstructionfor3DVideo Production . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.3 DesignFactorsof3DShapeReconstructionAlgorithms . . . . . . 106 4.3.1 Photo-ConsistencyEvaluation . . . . . . . . . . . . . . . . 106 4.3.2 VisibilityandOcclusionHandling. . . . . . . . . . . . . . 109 4.3.3 ShapeRepresentationandOptimization. . . . . . . . . . . 113 4.4 Implementations and Performance Evaluations of 3D Shape ReconstructionAlgorithms . . . . . . . . . . . . . . . . . . . . . 118 4.4.1 3DShapefromMulti-viewImages . . . . . . . . . . . . . 118 4.4.2 Simultaneous3DShapeandMotionEstimationfrom Multi-viewVideoDatabyaHeterogeneousInter-frame MeshDeformation . . . . . . . . . . . . . . . . . . . . . . 131 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
The list of books you might like
