Real-time Specularity Detection and Recovery

129 Pages·2013·3.35 MB·English

Real-time Specularity Detection
and Recovery
Qing Tian
Master of Engineering
Department of Electrical and Computer Engineering
McGill University
Montreal, Quebec
2013-04-15
A thesis submitted to McGill University
in partial fulfillment of the requirements of the degree of
Master of Engineering
(cid:2)c Qing Tian 2013
ACKNOWLEDGEMENTS
Firstandforemost, Iwouldliketoexpress mysinceregratitudetomysupervisor,
Prof. James J. Clark, one of the kindest professors I have ever met. His knowledge,
enthusiasm, patience, andlogicalwayofthinkinghaveinspiredmeinmyresearchand
have laid the foundation for this thesis. It was a happy and unforgettable experience
working with him.
Also, I would like to thank my parents for their understanding, support, and respect
for my decisions, and the great deal of freedom they give me to explore the world
myself.
In addition, I would like to thank all the instructors of the courses I have taken at
McGill such as Prof. Martin D. Levine, Prof. Paul Kry, Prof. Tal Arbel, Prof.
Kaleem Siddiqi, Prof. Frank P. Ferrie, and Prof. Doina Precup, from whom I have
learnednotonlythetechnical knowledgebutalsoapositiveattitudetowardsresearch
and life.
I am also thankful to my lab mates Jonathan Bouchard, Victor Ng, Amin Haji
Abolhassani, Mehdi Rezagholizadeh, Meltem Demirkus. By discussing with them
many pitfalls have been avoided. I would also like to thank the staff at the Centre
for Intelligent Machines (CIM): Ms. Marlene Gray, Mr. Jan Binder, Mr. Nick
Wilson. They help me get used to many software and hardware resources at CIM.
Moreover, thanks are also due to the good friends I made here at McGill, with whom
ii
I have had the pleasure of studying and working during my master career. These
friends include Lin Su, Matthew Balazsi, Ivan Aslamov, and so on.
Last but not least, I would like to acknowledge the financial support from McGill
University and my supervisor (through the funding from the Natural Sciences and
Engineering Research Council of Canada (NSERC), the Consortium en Innovation
Numrique du Quebec (CINQ), and Mokko Inc.) during my master’s career.
iii
ABSTRACT
Specularityisaverycommonphenomenonintherealworldandconfoundsmany
computer vision tasks such as stereo. The first purpose of this thesis is to design a
real-time algorithm of specularity detection. After that, with the knowledge of where
the specularities are, a stereo correspondence approach robust to specularity is pro-
posed. Finally, a specularity recovery method is presented to recover the underlying
diffuse color using the stereo correspondence information.
For real-time specularity detection, a new concept of unnormalized Wiener entropy
(UW Entropy) is first proposed in this thesis, which has the desirably simple final
form and requires no information about the lighting condition, surface structure,
imaging process, pre-segmentation, polarization state, and so forth. However, like
other specularity detection methods based on color alone, some false positives may
be detected. To distinguish between genuine specularities and false positives, a Sup-
port Vector Machine is learned in the proposed SpecLBP space as well as three other
spaces as comparisons. An alternative version is also presented for the beam-splitter
based stereo pairs in the 3D movie industry, where the curse of side-effect of the
beamsplitter is turned into a blessing for identifying problematic specularities. After
the genuine specularities are spotted, a new specularity-invariant stereo correspon-
dence method is proposed. By constructing an UW Entropy based matching energy
and minimizing it in the MAP-MRF framework using graph cuts, a disparity map
robust to specularities can be gained, which offers a precious piece of information for
iv
specularity recovery in the ending part of this thesis.
Experiment results show our methodology’s efficacy in real-time specularity detec-
tion, specularity-invariant stereo correspondence, as well as specularity recovery and
demonstrate our methodolodgy’s great potential for the 3D movie industry. By com-
paring theperformance ofthe proposedSpecLBP codeandthree other LBPvariants,
the SpecLBP code’s better performance justifies our claim that the best texture code
is task specific, not the one that captures the most information.
v
ABRE´GE´
La r´eflexion sp´eculaire est un ph´enom`ene fr´equemment observ´e dans la nature.
Et pourtant, ce type de r´eflexion pose encore probl`eme `a plusieurs algorithmes
de vision artificielle telle que l’interpr´etation de l’imagerie st´er´eoscopique. Cette
th`ese a pour premier objectif de concevoir un algorithme temps r´eel capable de
d´etecter les r´eflexions sp´eculaires. Par la suite, connaissant l’endroit ou` apparait ce
ph´enom`ene dans l’image, une approche pour la correspondance st´er´eoscopique ro-
buste aux r´eflexions sp´eculaires est propos´ee. Et enfin, une m´ethode de r´ecup´eration
de la couleur diffuse sous-jacente aux r´eflexions est pr´esent´ee, en tirant profit de
l’information acquise par la correspondance st´er´eo.
Pour effectuer la d´etection des r´eflexions sp´eculaires en temps r´eel, un nouveau
concept d’entropie de Weiner non normalis´ee (entropie UW) est d’abord propos´e par
cette th`ese. L’entropie UW est caract´eris´ee par une formulation analytique simple
qui ne requi`ere aucune information suppl´ementaire sur les conditions d’´eclairage, la
structure de la surface, la prise d’image, l’´etat de polarisation de la lumi`ere, aucune
pr´esegmentation et ainsi de suite. Cependant, comme d’autres m´ethodes de d´etection
des r´eflexions sp´eculaires bas´ees seulement sur la couleur, de faux positifs peuvent
ˆetre obtenus. Pour faire la distinction entre les r´eflexions sp´eculaires v´eritables et les
faux positifs, un s´eparateur `a vaste marge (SVM) est entrain´e dans l’espace (cid:2) Spe-
cLBP (cid:3) propos´e, et ´egalement dans trois autres espaces de la litt´erature, `a titre
de comparaison. Une adaptation du syst`eme est ´egalement pr´esent´ee pour traiter
vi
les paires d’images st´er´eo obtenues `a l’aide d’un miroir semi-argent´e, tel qu’utilis´e
dans l’industrie du film 3D, ou` les effets ind´esirables du miroir deviennent plutˆot
d’une aide pr´ecieuse pour localiser les r´eflexions probl´ematiques. Pour faire suite `a
la d´etection des r´eflexions authentiques, une nouvelle m´ethode de correspondance
st´er´eo robuste aux r´eflexions est propos´ee. En formulant l’entropie UW sous forme
d’´energie, et en minimisant cette ´energie dans le cadre d’un PAM-MRF (r´esolu en
utilisant des coupes de graphes), une carte de disparit´e st´er´eoscopique robuste aux
r´eflexionspeutˆetreacquise.Cette´evaluationdeladisparit´eenpr´esence desr´eflexions
est une information pr´ecieuse pour la r´ecup´eration de la couleur qui est pr´esent´ee
dans la derni`ere partie de cette th`ese.
Les r´esultats exp´erimentaux obtenus d´emontrent l’efficacit´e des m´ethodes propos´ees
pour la d´etection des r´eflexions sp´eculaires en temps r´eel, pour la correspondance
st´er´eo en pr´esence de r´eflexions, ainsi que pour la r´ecup´eration de la couleur sous-
jacente aux r´eflexions. Les exp´erimentations permettent ´egalement de d´emontrer le
potentiel de cette m´ethode pour l’industrie du cin´ema 3D. En comparant la perfor-
mance de la repr´esentation (cid:2) SpecLBP (cid:3) propos´ee et les trois autres variantes de
LBP, la performance sup´erieure du code (cid:2) SpecLBP (cid:3) valide notre hypoth`ese selon
laquelle une repr´esentation de la texture est meilleure, lorsqu’adapt´ee `a une taˆche
sp´ecifique, et non lorsqu’elle capture un maximum d’informations.
vii
TABLE OF CONTENTS
ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
ABRE´GE´ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi
LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xii
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Target Problem in the 3D Movie Industry . . . . . . . . . . . . . 2
1.3 Overview and Outline of the Thesis . . . . . . . . . . . . . . . . . 6
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Specularity Detection . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 Color Space Analysis Based Approaches . . . . . . . . . . . 11
2.1.2 Approaches Based on Physical Cues Other Than Color . . 14
2.2 Stereo Correspondence in the Presence of Specularity . . . . . . . 19
2.3 Specular Regions Recovery . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3 Real-time Unnormalized Wiener Entropy Based Specularity Detection . . 25
3.1 What is Specularity? . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 Theory of Unnormalized Wiener Entropy and Its Use in Specular-
ity Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2.1 Specularity Detection Based on Unnormalized Wiener
Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
viii
3.2.2 Unnormalized Wiener Entropy’s Relationship to Biological
Reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.3 The Developed Specularity Detector and Its Results . . . . . . . . 44
3.3.1 UW Entropy Based Real-time Specularity Detector . . . . 44
3.3.2 Experiment Results . . . . . . . . . . . . . . . . . . . . . . 45
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4 Genuine Specularity Selection Using One Image . . . . . . . . . . . . . . 52
4.1 Methodology of Genuine Specularity Selection . . . . . . . . . . . 52
4.2 Feature Extraction and Encoding . . . . . . . . . . . . . . . . . . 53
4.2.1 LBP - Local Binary Patterns Feature . . . . . . . . . . . . 53
4.2.2 Omni LBP - An Extension to the Naive LBP . . . . . . . . 54
4.2.3 A Selected Subset of Omni LBP for Specularity Detection-
SpecLBP . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.2.4 SpecLBP-Based Histogram Feature of Specularity . . . . . 60
4.3 Classifier Training and Testing . . . . . . . . . . . . . . . . . . . . 60
4.4 Experiment and Results . . . . . . . . . . . . . . . . . . . . . . . 61
4.4.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . 61
4.4.2 Results and Analysis . . . . . . . . . . . . . . . . . . . . . 61
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5 Problematic Specularity Detection in the 3D Movie Industry . . . . . . . 64
5.1 Polarization by Surface Reflection . . . . . . . . . . . . . . . . . . 65
5.1.1 Physical Essence of Polarization by Reflection . . . . . . . 65
5.1.2 Conditions for Polarization by Reflection . . . . . . . . . . 67
5.2 Polarizing Effect of Beamsplitter . . . . . . . . . . . . . . . . . . . 71
5.3 False Positives Removal and Problematic Specularity Detection . . 72
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6 Stereo Correspondence in Presence of Specularities . . . . . . . . . . . . 80
6.1 Stereo Correspondence as a Labelling Problem . . . . . . . . . . . 80
6.2 Specularity-Aware Energy Function for Stereo Correspondence . . 81
6.2.1 Mutual Information as a Similarity Measure . . . . . . . . 83
6.2.2 UW Entropy Based Specularity Aware Data Term . . . . . 85
6.2.3 Specularity Unaware Smoothness Term . . . . . . . . . . . 87
6.3 MRF Optimization Using Graph Cuts . . . . . . . . . . . . . . . . 88
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
ix
7 Specular Region Recovery Using Stereo Information . . . . . . . . . . . . 92
7.1 Specular Direction Calculation Using Correspondence Information 92
7.2 Specular Amount Calculation . . . . . . . . . . . . . . . . . . . . 94
7.2.1 Identification of the Regions with Uniform Diffuse Intensity 95
7.2.2 Intensity Propagation Over the Contours . . . . . . . . . . 97
7.2.3 Specular Magnitude by Intensity Subtraction . . . . . . . . 99
7.3 Specularity Recovery and Special Processing of Saturated Pixels . 99
7.4 An Alternative Approach for the 3D Movie Industry . . . . . . . . 100
7.5 Experiment and Specularity Removal Results . . . . . . . . . . . . 101
7.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
KEY TO ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
x