Toward a Perceptually-relevant Theory of Appearance

109 Pages·2017·29.09 MB·English

by  Pascal Barla

Toward a Perceptually-relevant Theory of Appearance
Pascal Barla
To cite this version:
Pascal Barla. Toward a Perceptually-relevant Theory of Appearance. Graphics [cs.GR]. Université
Bordeaux, 2017. ￿tel-01759272￿
HAL Id: tel-01759272
https://hal.inria.fr/tel-01759272
Submitted on 10 Apr 2018
HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est
archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
Toward a Perceptually-relevant
Theory of Appearance
By
PASCAL BARLA
A dissertation submitted to Bordeaux University in accor-
dancewiththerequirementsoftheHABILITATION A DIRIGER
LES RECHERCHES intheFacultyofComputerSciences.
OCTOBER 9TH, 2017
Jurymembers:
PierrePoulin(UniversitédeMontréal):rapporteur
FabriceNeyret(LaboratoireJeanKuntzmann):rapporteur
TamyBoubekeur(TelecomParisTech):rapporteur
DanielMeneveaux(UniversitédePoitiers):examinateur
MathiasPaulin(UniversitéPaulSabatier):examinateur
XavierGranier(Institutd’Optique):directeur

ACKNOWLEDGEMENTS / REMERCIEMENTS
I’dliketothankalotofpeopleforprovidingmewithpersonalandscientificsupport,butsincea
gooddealofthemdonoteasilyreadEnglish,I’llquicklyswitchtoFrenchforalittle“aparté”.
J’aimerais tout d’abord remercier Deborá pour avoir su subtilement me rappeler qu’il y a
d’autreschosesdanslavieaumoinsaussiimportantesquelarecherche,desortequejedevrais
parfoisarrêtertemporairementdemeprendrelatêteetm’occuperdelavaisselleparexemple.
Jeseraiéternellementreconnaissantenversmonpèreetmamèrepourm’avoirfaitcroire—à
tord—qu’ilétaittoutàfaitnormaldesuivremapassionprécocepourl’informatiquegraphique;
je n’ai réalisé que sur le tard que cela n’était pas chose courante, naif et chanceux comme je
suis.Jesuiségalementreconnaissantdelaclairvoyancedemesgrand-parentsqui,bienquene
parlantpresquepasunmotd’Anglais,ontparfaitementcomprislesensdumot“deadline”,comme
pouvaiententémoignerleurssouriresempreintsdecompassion.J’aimeraisaussisaluerletalent
d’actricesdematanteetdemacousine,quiontvraimentsûdonneruneimpressiondeprofond
intérêtlorsquejeleurexposaitcequipourmoidevaitconstituerunethéorieperceptuellement
pertinentedel’apparence;vousm’avezpresqueberné!Jevaisbienêtreobligédesaluericimes
amisd’enfance,quiontpourleurpartuniquementretenuletoutpremierdemesrésultatsde
rechercheoùuncubejauneétaitutiliséàdesfinsdetests;mêmesideleurpointdevue,mon
boulotconsistetoujoursàm’amuseravecdescubesjaunes,etàêtrepayépourça!
Jevaismaintenantpasseràmafamillescientifique,encommençantparJoëlleThollot,qui
durantmathèseasucontenirleMarseillaisenmoi,enbonneencadrante/psychologuequ’elleest.
Elleaeudenombreux“petits”depuis,etonformeunegrandefamilleheureuse.Jemedoisicide
remercierPascalGuitton,pasuniquementcarilestchauvecommemoi(8-),maisaussiparcequ’il
asuffisamentinsistépourquejetented’obtenirunpostedechercheurenuntempslointainoùil
n’étaitpasnécessaired’avoirunprixNobel.Denombreusesexcuses/remerciementsvontsans
surpriseauxmembresdeséquipesIparlapuisManao,pouravoirpatiemmentetpéniblement
écouté au moins la moitié du flot de paroles que je peux produire sans discontinuer lors de
réunions d’équipes, de brainstormings, de déjeuners, voire de pauses café ou même WC. Ma
famillescientifiques’estagrandielorsquej’aipuparticiperauréseauPRISMdès2013,etjesuis
extrêmementreconnaissantenversRolandFlemingdem’avoirenrôlédansceprojetpassionant;
jesuisparcontremoinsreconnaissantenversEdwardSnowdenpouravoirrévélél’existanced’un
autreprojetdumêmenomenjuin2013,cequej’aitrouvéuntantinetégoïstedesapartauvu
descomplicationsquecelaapuentraînerdenotrecôté.
Enfin,j’aimeraisremerciernaturellementlesmembresduJuryd’HDR,pourtoutletempsque
vousaurezinvestidanslalecturedecedocument,danslestransportsenvuedelasoutenance,
et dans les conséquences qui vont s’en suivre sous la forme de longs emails de discussions
interminablesdontvousserezlesinnocentesvictimes.
i

TABLE OF CONTENTS
Page
1 Introduction 1
1.1 Acloserlookatdepiction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Thegraphicspipeline(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Fourstagesofappearance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 StretchingtowardOpticsandVision . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Image-spacetechniques 11
2.1 Imagecreation&manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 Diffusioncurves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.2 Imageflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2 Expressiveshading&compositing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.1 Surfaceflows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.2 MatCapmanipulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3 Object-spacetechniques 43
3.1 Radiancetweaking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.1.1 Localwarpingandscaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.1.2 Multi-scaleenhancement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.2 Materialdesign . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2.1 Glossyappearance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.2.2 Anisotropicreflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
4 Conclusionsandperspectives 79
4.1 Blueprintforatheoryofappearance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.1.1 Belowtheobjectsurface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.1.2 Betweentheobjectandtheimage . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.1.3 Abovetheimageplane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.2 Beyondappearance:EcologicalOptics . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
Author’sbibliography 91
Bibliography 93
iii

R 1
E
T
P
A
H
C
INTRODUCTION
ComputerGraphicstechniqueshavehistoricallybeendrivenbyavarietyofactiveapplica-
tiondomains,fromtheproductionofanimations,movies,visualeffectsandvideogames,to
scientificvisualizationorproductdesignforinstance.Asaresult,thefieldhasevolvedcon-
jointlywithnewtechnologies(suchasGraphicsProcessingUnits,orGPUs),bringinginnovations
that often mix theoretical with practical considerations. One may argue that the first virtues
of Computer Graphics are automation and interactivity. Automation tools have for instance
lead to a progressive move from 2D hand-drawn to 3D animated characters in feature-length
movies1, as well as the near disappearance of animatronics in special effects, being replaced
byvirtualcreatures.Interactivetechniqueshavepermittedvideogamestobecometheleading
entertainment industry, from casual games on mobile devices, to independent or mainstream
games,andmorerecentlyseriousgamesandthegeneralizedgamificationofdifferenttypesof
activities.Differentartisticpracticeshaveappearedalongtheway,correspondingtodifferent
typesoftoolsusedforspecificproductionstages.Forinstance,whilesomeartistsareexpertat
modeling3Dshapes(e.g.,withsoftwaresuchasZBrushorMudbox),otherswillfocusonmaterial
ortextures(e.g.,usingSubstanceorQuixeltools),whereasartistsworkinginpostproductionwill
dealwithintegrationwithlivefootage(e.g.,usingNukeorAfterEffects).Differentapplications
alsoplacedifferentconstraints,andartistsmustincorporatethemtoproduceeffectiveimages,
suchasinscientificillustrationsforMedicineorArcheologywherelegibilityiskey.
Asidefrombeingaversatilecreationmedium,ComputerGraphicsalsoconstituteanunprece-
dentedtoolboxforprobinghowwe(humans)perceivetheworldaroundus.Visualperceptionhas
beenoneofthemainmotivationsfortheresearchprojectsIhaveparticipatedto,anditwillthus
bearecurringthemeinthefollowingpages.Inparticular,Iwillfocusonappearance,whichwe
mayatthisstagelooselydefineas“theresultoftheinteractionbetweenlight,shapeandmatter”.
Besidesautomationandinteraction,onemayask:Whatcannowbedoneintermsofappearance
manipulation that was not possible before the advent of Computer Graphics? In principle, by
offeringcontrolovereachandeveryingredientthatmakeupanimage,physically-impossible
tweaking and wild appearance variations should become possible. However, all of this is not
1Fortunately,frame-by-frame2Danimationremainsverymuchaliveinotherapplicationdomains,fromshort
movies,tomusicvideos,advertisementandevendocumentaries.
1
CHAPTER 1. INTRODUCTION
really new, as testified by the work of traditional artists as discussed in Section 1.1, and we
shouldattheveryleasttakeinspirationfromthem.
WhatisnewisthatComputerGraphicsprovideasystematicmeanstotinkerwiththeimage
formationprocess,whichnotonlyopensthewaytoautomationandinteractivity,butalsogives
ustheopportunitytobetterunderstandthevisualingredientsthatconstituteobjectappearance.
But where should we start from? After a quick presentation of the classical image production
pipelinesinSection1.2,IlaydownaplanintowhichIdivemoreprogressivelyinSection1.3,
introducingthetechniquesthatwillbedescribedindetailsinthefollowingchapters.Sincethe
long-termgoalistoassemblethesedisparateelementsintoacoherenttheoryofappearance,I
willfocusonprovidinginsights,notproofsorimplementationdetails.HumanVisionandPhysical
Opticswillnaturallyplayafundamentalroleinthisendeavor,butasdiscussedinSection1.4,
thesedomainsstillremainfarapartfromeachother,somuchthatitwillbenecessarytoweave
linksbetweenthemifonewantstounderstandappearancefully.
Ihavebeenmoreandmorefascinatedbyboththeentertaining/creativeaspectsofapplied
computergraphicstechniques,andtheeye-openinginsightsgainedfromdissectingtheimage
formationprocessinamorefundamentalmanner.Ihavetriedtosharethisdualpassionwith
thecolleaguesandstudents(manyofthemnowcolleagues)Ihavehadthechancetoworkwith,
andIhopetoshareitwithreadersofthisdissertationaswell.
1.1 A closer look at depiction
Themanipulationofappearanceinimagesisanoldhumanactivity,andevenaquicklookatthe
historyofArtrevealstheseskillshavebeenaroundforaverylongtime.Forinstance,paintings
from 100 BC found among the ruins of Pompeï exhibit effective “trompe-l’oeil” that cleverly
manipulate composition and perspective. The artwork of Jerome Bosch, in the 16th century,
showsatremendousvarietyofmaterialappearances,rangingfromminerals,tometals,skins,
differenttypesofwoods,ofglass,etc.Theratherrecentadventofphotographyinthe19thcentury
hasaddedanewwaytocreateimageswithitsowncodes,practices,and(relative)easeofuse.
However,photographyshouldstillbeconsideredasimageproduction,notasasimplerecordof
reality(seeFigure1.1);indeed,thechoiceofviewpoint,time,parametersallparticipateinthe
qualityofthefinalpicture(photographyliterallymeans“drawingwithlight”!).
(a)“Lesmains” (b)“TheTetonsandtheSnakeRiver”
byBenoitCourti byAnsellAdams
Figure1.1: Skillfully-executedphotographyismorethanamererecordofreality.
2
1.1. A CLOSER LOOK AT DEPICTION
Painting, drawing and photography have the power to elicit an infinite range of impres-
sions: they depict the shape of objects, plants or animals, their materials and textures, their
relativedepthsinthescene,thetimeofdayatwhichtheimagehasbeentakenorisimagined
tocorrespond,theenvironmentinwhichthescenetakesplacesuchasopenorcloseareas,the
size of the different elements that may be observed, etc. Tactile or olfactory emulations may
even be elicited by properly depicting familiar visual patterns that recall touch or smell by
association. The artist often does not stop at producing or capturing a scene that can merely
be recognized, but also works at conveying a mood, maybe tension or emotion between char-
acters, atmospheres in landscapes, even functions or purposes when a mechanism is depicted
or an action is taking place. This is already getting us a bit far from the topic of appearance,
although not too far since cognitive aspects of a painting or photograph rely at least in part
on simpler aspects of appearance. For instance, tension can be conveyed by wrinkles on body
parts(Figure1.1a),grandeurisenhancedbycontrastsinluminosityandtexturepatterns(Fig-
ure1.1b),andanatmosphereissetbyaproperchoiceoftones,shadowsandcontrasts(Figure1.2).
“Meules,effetdeneige,lematin” “Meules,findel’été”
byClaudeMonet byClaudeMonet
Figure1.2: Non-visualimpressionsmaybeconveyedbyaproperchoiceofcolorsandshades.
Letmecomebacktotheconceptofappearance,whicharisesfromthe(imagined,observedor
captured)interactionsbetweenlighting,andtheshapeandmaterialsofobjectspresentinthe
depictedscene.Sinceweonlyhaveanindirect,remotevisualaccesstotheseobjects,appearance
must then manifest itself as a variety of image patterns that are diagnostic of specific shapes,
materials or lighting configurations. Even if we somehow make use of these patterns to give
sensetoanimage,wedosowithoutevengivingitathought.Itisthusdifficulttopinpointwhat
effectivecombinationofimagepatternsmakesaplausibledepictionofascene,asanystudentof
artquicklyrealizes:whatiseffortlesslyperceivedrevealstobechallengingtoreproduce.
Paintersmustlearnhowtocreatesuchpatternswithvarioustoolsandmedia,anddepending
onthelattertheappropriatetechniquewillvary.Asasimpleexample,considersurfacehighlights.
They may be directly painted in white on top of existing dried paint layers when using oil or
acrylic;whereaswithwatercolor,theyareratherconveyedbyleavingthewhitesurfaceofthe
paperuntouched.Studentsofphotographyarefacedwiththechallengeoffindingandcapturing
therightpatternsattherighttimeandviewpoint.Theymayadditionallymakeuseofadequate
lenses,focallengthsorevencolorfilterswhenworkinginblackandwhite.Studiophotography
goesonestepfurtherbyprovidingcontroloverthelightingenvironment.
Ihopetoshowthatdespitethewidevarietyofartisticstylesorphotographicpractices,the
sought-for patterns are essentially the same. The work of many artists suggest that this is
indeedthecasebyblurringthedistinctionbetweenphotographyandpainting.Inphotography,
3