image-based transformation of viewpoint and scene appearance

111 Pages·1998·1.62 MB·English

IMAGE-BASED TRANSFORMATION OF
VIEWPOINT AND SCENE APPEARANCE
By
StevenMaxwellSeitz
A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE
REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCES)
atthe
UNIVERSITYOFWISCONSIN–MADISON
1997
IMAGE-BASED TRANSFORMATION OF
VIEWPOINT AND SCENE APPEARANCE
By
StevenMaxwellSeitz
A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE
REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
(COMPUTER SCIENCES)
atthe
UNIVERSITYOFWISCONSIN–MADISON
1997
i
Abstract
Thisthesisaddressestheproblemofsynthesizingimagesofrealscenesunderthree-dimensionaltrans-
formations in viewpoint and appearance. Solving this problem enables interactive viewing of remote
scenes on a computer, inwhich a user can move avirtual camera through the environment and virtu-
ally paint or sculpt objects in the scene. It is demonstrated that a variety of three-dimensional scene
transformations canberenderedonavideodisplaydevicebyapplyingsimpletransformations toaset
of basis images of the scene. The virtue of these transformations is that they operate directly on im-
agesandrecoveronlythesceneinformation thatisrequired inordertoaccomplish thedesired effect.
Consequently, they are applicable in situations where accurate three-dimensional models aredifficult
orimpossible toobtain.
Acentraltopicistheproblemofviewsynthesis,i.e.,renderingimagesofarealscenefromdifferent
cameraviewpointsbyprocessingasetofbasisimages. Towardsthisend,twoalgorithmsaredescribed
thatwarpandresamplepixelsinasetofbasisimagestoproducenewimagesthatarephysically-valid,
i.e., they correspond to what a real camera would see from the specified viewpoints. Techniques for
synthesizing other types of transformations, e.g., non-rigid shape and color transformations, are also
discussed. Thetechniques arefoundtoperform wellonawidevarietyofrealandsynthetic images.
A basic question is uniqueness, i.e., for which views is the appearance of the scene uniquely de-
termined from the information present in the basis views. An important contribution is a uniqueness
result for the no-occlusion case, which proves that all views on the line segment between the two
camera centers areuniquely determined from twouncalibrated viewsofascene. Importantly, neither
densepixelcorrespondencenorcamerainformationisneeded. Fromthisresult,aviewmorphingalgo-
rithmisderivedthatproduceshighqualityviewpointandshapetransformationsfromtwouncalibrated
images.
To treat the general case of many views, a novel voxel coloring framework is introduced that
facilitates the analysis of ambiguities in correspondence and scene reconstruction. Using this frame-
work, anewtype of scene invariant, called color invariant, isderived, whichprovides intrinsic scene
information useful for correspondence and view synthesis. Based on this result, an efficient voxel-
based algorithm is introduced to compute reconstructions and dense correspondence from a set of
basis views. This algorithm has several advantages, most notably its ability to easily handle occlu-
sion and views that are arbitrarily far apart, and its usefulness for panoramic visualization of scenes.
These factors also make the voxel coloring approach attractive as a means for obtaining high-quality
three-dimensional reconstructions fromphotographs.
ii
Acknowledgements
Thisthesisisdedicatedtomyparents,GaryandSheilaSeitz,whotaughtmethateducationisalifelong
pursuitandinspired metopursueaPh.D.
I wish to thank Chuck Dyer for his guidance and support throughout my graduate studies. His
advice and infallible judgement on all matters academic and otherwise have been a tremendous in-
fluence and an invaluable resource for me. I am very fortunate to have had the opportunity to work
withhimduringmytimeatU.W.Madison. Iwouldalsoliketothanktheothermembersofmythesis
committee, Nicola Ferrier, Vladimir Lumelsky, Vadim Shapiro, and Jude Shavlik for their comments
andinsights onthiswork.
I am especially grateful to Larry Rowefor introducing meto scientific research as an undergrad-
uate atU.C.Berkeley and forencouraging metopursue graduate studies incomputer science. Kyros
Kutulakos contributed keyideastothisworkandprovided astronginfluence bothaspeerandcollab-
orator.
In the course of graduate studies, I have benefited from interactions with many people. I would
especiallyliketothankBrianRoddy,AaronSeitz,KathySuchenski,SanjayTiwari,andJohnWatrous
formanyilluminating discussions. Adviceandinsightsonthisworkwereprovidedbyseveralcurrent
andformermembersofthevisiongroupatU.W.Madison, including RusselManning,BrianMorgan,
Andrew Prock, Dan Reznik, Brent Seales, and Steve Walden. I also thank the following people who
graciously permitted their imagestobeused inthisthesis, inorder ofappearance: John Watrous, Su-
huiChiang,ShubuMukherjee,DionisiosPnevmatikatos,AlainKagi,EltonGlaser,BabakFalsafi,and
DougBurger.
Most of all, I am deeply grateful to Sandhya Subramanian for her endless encouragement and
enthusiasm, andformakingtheselastseveralyearstrulyenjoyable.
iii
Contents
Abstract i
Acknowledgements ii
1 Introduction 1
1.1 ExamplesofImage-Based SceneTransformations . . . . . . . . . . . . . . . . . . . 3
1.2 Transforming RealScenes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 ThesisContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 ThesisOutline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Notation 8
2.1 Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Color . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3 TheTwo-ViewCase: ViewMorphing 9
3.1 2Dvs. 3D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 RelatedWorkonViewSynthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.4 Uniqueness andMeasurability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.5 Monotonicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.5.1 ImpactforViewSynthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5.2 Identifying Monotonic Scenes . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.5.3 Orthographic Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.6 ImageMorphing forViewSynthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.7 Shape-Preserving Morphs: ParallelViews . . . . . . . . . . . . . . . . . . . . . . . . 20
3.8 Occlusions andVisibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.8.1 ConstantVisibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.8.2 ChangesinVisibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.9 Shape-Preserving Morphs: Non-ParallelViews . . . . . . . . . . . . . . . . . . . . . 24
3.9.1 ImageReprojection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.9.2 AThreeStepAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.9.3 SingularViewConfigurations . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.9.4 Producing theMorph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.10 Uncalibrated ViewMorphing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.10.1 ProjectiveTransformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
iv
3.10.2 ComputingPrewarps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.10.3 Specifying Postwarps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.11 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.11.1 Correspondence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.11.2 ImageResampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.11.3 PrewarpStability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.12 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.13 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4 Single-ViewMorphing 39
5 N-ViewMorphing 41
5.1 StrongMonotonicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6 AMulti-ViewApproach: VoxelColoring 44
6.1 PixelCorrespondence forViewSynthesis . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1.1 OpticalFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.1.2 Stereo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6.1.3 FeatureTracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.1.4 ContourTrackingandVolumeIntersection . . . . . . . . . . . . . . . . . . . 47
6.1.5 EPIAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.1.6 SceneSpaceMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.1.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2 TheVoxelColoringProblem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.3 ColorInvariants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.3.1 TheOrdinalVisibilityConstraint . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.3.2 Properties ofColorInvariants . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.4 AVoxelColoringAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.4.1 LayeredSceneDecomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.4.2 VoxelConsistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.4.3 ASingle-PassAlgorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6.4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
6.4.5 Optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.5.1 ResultsonRealImages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
6.5.2 ResultsonSynthetic Images . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
v
7 Editing 73
7.1 TheUserView: EditingbyExample . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
7.1.1 Plenoptic Painting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
7.1.2 Plenoptic Scissoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
7.1.3 Plenoptic Morphing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
7.1.4 NewViewGeneration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
7.2 EditPropagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7.2.1 Painting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
7.2.2 Scissoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
7.2.3 Morphing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8 Conclusions 79
8.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.2 LimitationsandFutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
Bibliography 83
A ComputingPrewarps 92
A.1 TheFundamentalMatrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
A.2 Choosing theHomographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
A.2.1 AligningtheImagePlanes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
A.2.2 AligningtheScanlines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
A.2.3 InfiniteEpipoles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
A.2.4 Selecting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
d0
A.2.5 Orthographic Prewarps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
B OrthographicViewMorphing 99
B.1 ImageInterpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
B.2 TheNeedforRectification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
vi
List of Figures
1.1 Image-based SceneTransformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 ImagetransformationofPhotographs,Drawings,andPaintingsEnables3DVisualiza-
tionofHistoricObjectsandScenes. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 An Image Morph Combines 2D Interpolations of Shape (via Image Warping) and
Color(viaCross-Dissolve)toCreateTransitionsthatAppearStrikinglyThree-Dimensional.
4
3.1 ViewMorphing BetweenTwoImages ofanObject TakenFrom TwoDifferent View-
pointsProducestheIllusion ofPhysicallyMovingaVirtualCamera. . . . . . . . . . . 10
3.2 VisualAmbiguity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 TheMonotonicity Constraint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.4 Correspondence UnderMonotonicity. . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.5 AShape-Distorting Morph. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.6 Morphing ParallelViews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.7 ViewMorphing inThreeSteps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.8 SingularViews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.9 ViewMorphing Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.10 Viewmorphsofthreeimagepairswithautomatic stereomatching. . . . . . . . . . . . 34
3.11 FacialViewMorphs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.12 ImageMorphing VersusViewMorphing: HelicopterToy. . . . . . . . . . . . . . . . 37
3.13 ImageMorphing VersusViewMorphing: MonaLisa. . . . . . . . . . . . . . . . . . . 37
4.1 Photocorrection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.1 N-ViewMorphing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.1 VoxelColoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.2 SpatialAmbiguity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.3 ColorAmbiguity. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
6.4 ColorInvariants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.5 CompatibleCameraConfigurations. . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.6 LayeredSceneTraversal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
6.7 Selectedbasisimagesforadinosaur toyandarose. . . . . . . . . . . . . . . . . . . . 65
6.8 ImageAcquisition Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.9 VoxelColoringofaDinosaurToy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
6.10 VoxelColoringofFlower. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
vii
6.11 Placementofinputcameraviewpoints forapanoramic synthetic roomscene. . . . . . 68
6.12 PanoramicRoomSceneReconstruction. . . . . . . . . . . . . . . . . . . . . . . . . 69
6.13 EffectsofTextureDensityonVoxelReconstruction. . . . . . . . . . . . . . . . . . . 70
6.14 EffectsofImageNoiseandVoxelSizeonReconstruction. . . . . . . . . . . . . . . . 71
7.1 Examples of Plenoptic Image Editing Operations Applied to Photographs of a Di-
nosaur Toy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
A.1 APairofViewscanbeMadeCoincidentWithaPlane byRotatingtheImagePlanes
E
AboutTheirLinesofIntersection With . . . . . . . . . . . . . . . . . . . . . 94
i di E
I
B.1 Monotonic ViewingGeometry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
B.2 ViewsGeneratedbyImageInterpolation. . . . . . . . . . . . . . . . . . . . . . . . . 100
viii
List of Tables
6.1 Comparison ofCorrespondence Approaches. . . . . . . . . . . . . . . . . . . . . . . 46
6.2 VoxelResolution Effects. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64