THE ART OF SEEING: VISUAL PERCEPTION IN - Research Areas

128 Pages·2005·3.24 MB·English

THE ART OF SEEING: VISUAL PERCEPTION IN
DESIGN AND EVALUATION OF
NON-PHOTOREALISTIC RENDERING
BYANTHONYSANTELLA
ADissertation submittedtothe
GraduateSchool—NewBrunswick
Rutgers,TheStateUniversityofNewJersey
inpartialfulfillmentoftherequirements
forthedegreeof
DoctorofPhilosophy
GraduatePrograminComputerScience
Writtenunderthedirectionof
DougDeCarlo
andapprovedby
NewBrunswick,NewJersey
May,2005

ABSTRACT OF THE DISSERTATION
The Art of Seeing: Visual Perception in Design and
Evaluation of Non-Photorealistic Rendering
by Anthony Santella
Dissertation Director: Doug DeCarlo
Visual displays such as art and illustration benefit from concise presentation of in-
formation. We present several approaches for simplifying photographs to create such
concise, artistically abstracted images. The difficulty of abstraction lies in selecting
what is important. These approaches apply models of human vision, models of image
structure, and new methods of interaction to select important content. Important loca-
tionsareidentifiedfromeyemovementrecordings. Usingaperceptualmodel,features
are then preserved where the viewer looked, and removed elsewhere. Several visual
stylesusingthismethodarepresented. Theperceptualmotivationforthesetechniques
makes predictions about how they should effect viewers. In this context, we validate
our approach using experiments that measure eye movements over these images. Re-
sults also provide some interesting insights into artistic abstraction and human visual
perception.
ii
Acknowledgements
Thanks go to the many people whose help and support was essential in making this
work possible. None of this would have happened without my advisor Doug DeCarlo.
Thanks go also to my other committe members: Adam Finkelstein, Eileen Kowler,
Casimir Kulikowski and Peter Meer for their advice and encouragement at various (in
somecasesmany)stagesofthisprocess.
Thanks go also to the many friends and family members who have supported and
keptmesanethroughthislongprocess. Iwouldn’thavesurviveditwithoutmyparents
and brothers Nick and Dennis. Special thanks go to Bethany Weber. Thanks also to
JimHousell,alltheoldNYUcrowd,thegradgroupatSt. Petersandallthesupportive
soulsintheCSDepartment,RuCCSandtheVILLAGE.
Finally, thanks go to Phillip Greenspun for photos used in several renderings that
appear in chapters 7 and 9, as well as models Marybeth Thomas, Adeline Yeo and
FrancoFigliozzi. SpecialthankstoGeorgioDellachiesaforlookingequallythoughtful
incountlessillustrativeexamples.
iii
Table of Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
ListofFigures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1. Inspirations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.1. ArtisticPractice . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2. Psychology . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.1.3. ComputerGraphics . . . . . . . . . . . . . . . . . . . . . . . 7
1.2. OurGoal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2. AbstractioninComputerGraphics . . . . . . . . . . . . . . . . . . . . 11
2.1. ManualAnnotation . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2. AutomaticMethods . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3. LevelOfDetail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3. HumanVision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1. EyeMovements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.1. EyeMovementControl . . . . . . . . . . . . . . . . . . . . . 19
3.1.2. SalienceModels . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2. EyeTracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.3. LimitsofVision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3.1. ModelsofSensitivity . . . . . . . . . . . . . . . . . . . . . . 24
3.3.2. SensitivityAwayfromtheVisualCenter . . . . . . . . . . . . 26
3.3.3. ApplicabilitytoNaturalImagery . . . . . . . . . . . . . . . . 26
iv
4. VisionandImageProcessing . . . . . . . . . . . . . . . . . . . . . . . 30
4.1. ImageStructureFeaturesandRepresentation . . . . . . . . . . . . . 30
4.2. Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.3. EdgeDetection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
5. OurApproach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.1. EyetrackingasInteraction . . . . . . . . . . . . . . . . . . . . . . . 38
5.2. UsingVisibilityforAbstraction . . . . . . . . . . . . . . . . . . . . . 40
6. PainterlyRendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.1. ImageStructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.2. ApplyingtheLimitsofVision . . . . . . . . . . . . . . . . . . . . . 43
6.3. Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
7. ColoredDrawings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.1. FeatureRepresentation . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.1.1. Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.1.2. Edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
7.2. PerceptualModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
7.3. Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
7.4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
8. PhotorealisticAbstraction . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.1. ImageStructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8.2. MeasuringImportance . . . . . . . . . . . . . . . . . . . . . . . . . 65
8.3. ResultsandDiscussion . . . . . . . . . . . . . . . . . . . . . . . . . 67
9. Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
v
9.1. EvaluationofNPR . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
9.1.1. AnalysisofEyeMovementData . . . . . . . . . . . . . . . . 75
9.2. Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
9.2.1. Stimuli . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
9.2.2. Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
9.2.3. PhysicalSetup . . . . . . . . . . . . . . . . . . . . . . . . . 78
9.2.4. CalibrationandPresentation . . . . . . . . . . . . . . . . . . 79
9.3. Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
9.3.1. DataAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 80
9.3.2. StatisticalAnalysis . . . . . . . . . . . . . . . . . . . . . . . 82
9.4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
9.4.1. QuantitativeResults . . . . . . . . . . . . . . . . . . . . . . 86
9.4.2. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
9.5. EvaluationConclusion . . . . . . . . . . . . . . . . . . . . . . . . . 92
10. FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
10.1. ImageModeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
10.1.1. Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . 95
10.1.2. Edges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
10.2. PerceptualModels . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
10.3. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
11.Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
CurriculumVita . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
vi
List of Figures
1.1. (a) Henri de Toulouse-Lautrec’s “Moulin Rouge—La Goulue” (Litho-
graphic print in four colors, 1891). (b) Odd Nerdrum’s “Self-portrait
as Baby” (Oil, 2000). Artists control detail as well as other features
such as color and texture to focus a viewer on important features and
create a mood. La Goulue’s swirling under-dress is a highly detailed
focal point of the image, and contributes to the picture’s air of reck-
less excitement. Artists have a fair amount of latitude in how they
allocate detail to create an effect. Nerdrum renders his eyes (usually
one of the most prominent features in a portrait) in a sfumato style
that makes them almost nonexistent. Detail is instead allocated to the
child’s prophetic gesture. These choices change a common baby pic-
tureintosomethingmysteriousandunsettling. . . . . . . . . . . . . 4
1.2. Judith Schaechter’s, “Corona Borealis” (Stained glass, 2001). Skill-
ful artists use the formal properties and constraints of a medium for
expressive purposes. The high dynamic range provided by transmit-
ted light and the heavy black outlines of the lead caming that holds
the glass together are used to set the figure off from the background
creatingapowerfulimageofjoyinisolation. . . . . . . . . . . . . . 5
2.1. Direct placement of strokes. Complete control of abstraction is pos-
sible when a user provides actual strokes that are rendered in a given
style. Reproducedfrom[Durandetal,2001]. . . . . . . . . . . . . . 11
2.2. Manual annotation for textural indication. Important edges on a 3D
model are marked and have texture rendered near them, while it is
omitted in the interior. Reproduced from [Winkenbach and Salesin,
1994]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
vii
2.3. Manual local importance images. Hand painted images can indicate
importantareastoberenderedingreaterdetailorfidelity. Reproduced
from[Hertzmann,2001] . . . . . . . . . . . . . . . . . . . . . . . . 12
2.4. (a)originalimage. (b)correspondingsaliencemap[Ittietal,1998]. (c)
corresponding salience map [Itti and Koch, 2000]. Salience methods
picks out potentially important areas on the basis of contrast in some
space(notlimitedtointensity). Thetwomethodspicturedheredifferin
the method of normalization used to enhance contrast between salient
andnonsalientregions. . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1. Patterns of eye movements of a single subject over an image when
given different instructions. Note (1) free observation which shows
fixations that are relatively dispersed yet still focused on relevant ar-
eas. Contrast it with (3) where the viewer is instructed to estimate the
figures’ages. ReproducedfromYarbus1967. . . . . . . . . . . . . . 18
3.2. Similareffectsto[Yarbus,1967]areeasily(evenunintentionally)achieved
when using eye tracking for interaction. Circles are fixations, their di-
ameter is proportional to duration. The first viewer was instructed to
findtheimportantsubjectmatterintheimage. Thesecondviewerwas
told to ’just look at the image’. The viewer assumed, from prior expe-
rience in perceptual experiments, that he was going to be later asked
detailed questions about the contents of the scene. This resulted in a
muchmorediffusepatternofviewing. . . . . . . . . . . . . . . . . . 19
3.3. Log-log plot of contrast sensitivity from equation (3.2) This function
isusedtodefineathresholdbetweenvisibleandinvisiblefeatures. . 25
3.4. CorticalMagnificationdescribesthedrop-offofvisualsensitivitywith
angulardistancefromthevisualcenter. . . . . . . . . . . . . . . . . 27
viii
4.1. (a) Scale space of one dimensional signal. Features disappear through
scale space but no new features appear. (b) Plot of inflection points of
anotheronedimensionalsignalthroughscalespace. Reproducedfrom
[Witkin1983] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.2. Interval tree for 1D signal illustrating decomposition of the signal into
ahierarchy. Reproducedfrom[Witkin1983]. . . . . . . . . . . . . . 33
5.1. (a) Computing eccentricities with respect to a particular fixation at p.
(b)Asimpleattentionmodeldefinedasapiecewise-linearfunctionfor
determining the scaling factor a for fixation f based on its duration
i i
t. Very brief fixations (below t ) are ignored, with a ramping up (at
i min
t )toamaximumlevelofa . . . . . . . . . . . . . . . . . . . . 40
max max
6.1. Painterly rendering results. The first column shows the fixations made
by a viewer. Circles are fixations, size is proportional to duration, the
baratthelowerleftisthediameterthatcorrespondstoonesecond. The
second column illustrates the painterly renderings built based on that
fixationdata. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.2. Detail in background adjacent to important features can be inappro-
priately emphasized. The main subject has a halo of detailed shutter
slats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.3. Sampling strokes from an anisotropic scale space avoids giving the
imageanoverallblurredlook,butproducesasomewhatjaggedlookin
backgroundareas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.4. Color and contrast manipulation. Side by side comparison or render-
ing with and without color and contrast manipulation (precise stroke
placementvariesbetweenthetwoimagesduetorandomness). . . . . 48
7.1. Slices through several successive levels of a hierarchical segmentation
treegeneratedusingourmethod. . . . . . . . . . . . . . . . . . . . . 51
7.2. Linedrawingstyleresults. . . . . . . . . . . . . . . . . . . . . . . . 60
ix