Real-time image based lighting with streaming HDR-light

45 Pages·2013·2.28 MB·English

LiU-ITN-TEK-A--12/071--SE
Real-time image based lighting
with streaming HDR-light probe
sequences
Saghi Hajisharif
2012-10-26
Department of Science and Technology Institutionen för teknik och naturvetenskap
Linköping University Linköpings universitet
SE-601 74 Norrköping, Sweden 601 74 Norrköping
LiU-ITN-TEK-A--12/071--SE
Real-time image based lighting
with streaming HDR-light probe
sequences
Examensarbete utfört i Medieteknik
vid Tekniska högskolan vid
Linköpings universitet
Saghi Hajisharif
Handledare Joel Kronander
Examinator Jonas Unger
Norrköping 2012-10-26
Upphovsrätt
Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare –
under en längre tid från publiceringsdatum under förutsättning att inga extra-
ordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner,
skriva ut enstaka kopior för enskilt bruk och att använda det oförändrat för
ickekommersiell forskning och för undervisning. Överföring av upphovsrätten
vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan användning av
dokumentet kräver upphovsmannens medgivande. För att garantera äktheten,
säkerheten och tillgängligheten finns det lösningar av teknisk och administrativ
art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i
den omfattning som god sed kräver vid användning av dokumentet på ovan
beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan
form eller i sådant sammanhang som är kränkande för upphovsmannens litterära
eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se
förlagets hemsida http://www.ep.liu.se/
Copyright
The publishers will keep this document online on the Internet - or its possible
replacement - for a considerable time from the date of publication barring
exceptional circumstances.
The online availability of the document implies a permanent permission for
anyone to read, to download, to print out single copies for your own use and to
use it unchanged for any non-commercial research and educational purpose.
Subsequent transfers of copyright cannot revoke this permission. All other uses
of the document are conditional on the consent of the copyright owner. The
publisher has taken technical and administrative measures to assure authenticity,
security and accessibility.
According to intellectual property law the author has the right to be
mentioned when his/her work is accessed as described above and to be protected
against infringement.
For additional information about the Linköping University Electronic Press
and its procedures for publication and for assurance of document integrity,
please refer to its WWW home page: http://www.ep.liu.se/
© Saghi Hajisharif
Abstract
This work presents a framework for shading of virtual objects using high dynamic
range (HDR) light probe sequences in real-time. The method is based on using
HDR environment map of the scene which is captured in an on-line process by HDR
video camera as light probes [32]. In each frame of the HDR video, an optimized
CUDA kernel is used to project incident lighting into spherical harmonics in real
time. Transfer coefficients are calculated in an offline process. Using precomputed
radiance transfer the radiance calculation reduces to a low order dot product
between lighting and transfer coefficients. We exploit temporal coherence between
frames to further smooth lighting variation over time. Our results show that
the framework can achieve the effects of consistent illumination in real-time with
flexibility to respond to dynamic changes in the real environment. We are using
low-order spherical harmonics for representing both lighting and transfer functions
to avoid aliasing.
v
Acknowledgments
Iwould like to thank my examiner, Jonas Unger and my supervisor Joel Kronander
for providing a great atmosphere for working and helping me during my thesis.
I would like to thank my dear parents for all their love and supports specially
during my studies in Sweden. And lots of thanks to my dear Ehsan, for all his
patience.
vii
Contents
1 Introduction 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Radiometry . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 BRDF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3 Global Illumination . . . . . . . . . . . . . . . . . . . . . . 4
1.1.4 Image-based lighting . . . . . . . . . . . . . . . . . . . . . . 5
1.1.5 Pre-computed radiance transfer . . . . . . . . . . . . . . . . 5
1.1.6 CUDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 Previous Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 Method 17
2.0.1 Algorithm overview . . . . . . . . . . . . . . . . . . . . . . 17
2.1 Off-line pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Light probe image processing . . . . . . . . . . . . . . . . . . . . . 19
2.3 Lighting SH projection . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4 Temporal filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Lighting reconstruction and rendering . . . . . . . . . . . . . . . . 22
3 Implementation 25
4 Results 27
5 Conclusions and Future Work 33
Bibliography 35
ix
Chapter 1
Introduction
Compositing and merging synthetic objects in the real world scene containing
real objects is one of the challenging tasks for many areas of computer graphics
like visual effects and augmented reality. As an example in visual effects area, the
producers are interested in combining their videos with computer generated objects
such as digital actors and props. This process requires consistent local and distant
lighting between synthetic and real components. Similarly, augmented reality(AR)
is based on adding synthetic objects to the real world environment and enriching
it in a way that virtual objects are perceived as coexisting with physical world.
Visual quality is one of the important aspects and even with the objects correctly
placed in the environment, if their visual appearance do not match the scene it
instantly gives away their artificiality.
Rendering realistic objects with correct lighting of the real environment is a
demanding task and requires the knowledge of the surroundings. Image Based
Lighting (IBL), [5], is a widely used technique for photo-realistic rendering of
virtual objects so that they can be seamlessly composited into still or video footage
captured in real scenes. The key idea of IBL is to capture the lighting present in
the real scene and use this information to illuminate the virtual objects. The
scene lighting in traditional IBL is measured by capturing an omni-directional
High Dynamic Range (HDR) image, or HDRi, at a single point in space. Such a
panoramic HDRi is generally called a light probe. Since the HDRi captures the full
dynamic range in the scene (from the direct light sources to the parts of the scene
that are in shadow), the light probe can be thought of as a measurement of the
scene radiance incident at the point in space where the panorama was captured,
and can be used as an approximation of the lighting in the scene during rendering.
The ease of use and level of realism attainable have now made IBL a standard
tool in most production pipelines, and even in real-time applications (based on
approximations).
Traditional IBL has been limited to only static lighting environments. This is
due to the fact that there are no cameras available on the market, that can capture
true HDR images in a single shot. HDR images are commonly captured using
exposure bracketing, a series of differently exposed images covering the dynamic
1
2 Introduction
⊥
Figure 1.1. Definition of radiance L : flux per unit projected area dA and per unit
solid angle dω
range of the scene that are combined into a final HDR image. This limits the
capture to static scenes and still images. However, recent developments in sensor
and imaging hardware, [31, 30, 15], have now made it possible to capture HDR-
video (HDRv). This in turn also enables the capture of light probe video sequences,
and thus IBL with dynamic real world lighting environments and moving light
probes.
In this report we are presenting a technique and system overview for real-time
IBL using HDRv light probe sequences. Our method is based on the precomputed
radiance transfer method which was introduced by Sloan et al.[28]. Real world
scene lighting is recorded using a high quality and high resolution 4Mpixel HDRv
camera running at 25 or 30 frames per second (fps). Using a real-time CUDA
kernel, the input HDR images are processed and the spherical radiance distribu-
tion described by each frame in the video sequence is projected onto a low order
spherical harmonics basis and used for image based rendering of synthetic objects.
This report is structured as : in section 1.1 we are providing the reader the ba-
sic information that is required for understanding the concept of this work. This
sections contains a brief summery of radiance, global illumination, image based
lighting, precomputed radiance transfer. In section 2 we will discuss in deep the
method we are using in our framework. Section 3 and 4 show the results of our
implementation and we can see the performance and problems of this method. In
section 5 we will have a conclusion and future works.
1.1 Background
In this section we will have a brief review over the related works in real-time image
based lighting and a short summary of the necessary theoretical background in
computer graphics and global illumination. Firstly, we will begin with an overview
on radiometry and photometry.
1.1.1 Radiometry
The energy that is received from the environment by the eye is called light which
enables human brain to get information about the surroundings. Light interacts
1.1 Background 3
Figure 1.2. Irradiance E coming from direction Li.
with near-by objects due to their material properties. Some objects absorb or
reflect the light while the others refract it and among all these interactions some
part of light finally reaches our eyes. All global illumination methods are trying to
simulate the properties of the real light in the modelled scene. Therefore under-
standing the nature of the light and its propagation in the environment is a very
important to have an acceptable illumination.
The word radiometry is referred to the science of measuring light energy. As
light travels through the air, we are interested in the measurements of the energy
that passes through a surface. This quantity is called flux φ which is defined as a
ratio of total radiant energy flow Q per unit of time (watts). The radiant flux that
is coming away from any direction per unit of area over a surface is called irradi-
dΦ 2
ance and is defined as: E = . The unit of irradiance is watts/m . The radiant
dA
intensity, I is the change of radiant power with respect to a solid angle. Based on
these definitions, radiosity is defined as the density of the radiant flux leaving a
dφ 2
surface: M = and is also expressed with units in watts/m . Radiance is also
dA
a radiometric measure which describes the amount of flux that passes through or
2
is emitted from a unit projected area per unit solid angle(watts/(steradians.m )):
2 2
d Φ d Φ
L = = , (1.1)
⊥
dωdA dωdAcosθ
2
where, L is the radiance(watts/(steradians.m )), φ is the radiant flux(watts),
θ is the angle between the surface normal and the specified direction, A is the
2
area of the surface(m ) and ω is the solid angle(steradians). Figure 1.1 displays
the concept of radiance. Generally, Radiance L is the power that a ray which is
leaving or arriving a point on a surface contains.
The radiance leaving a point x along the direction Θ is denoted as L(x → Θ).
Similarly, L(x ← Θ) represents radiance from direction Θ towards point x.
1.1.2 BRDF
To understand how light propagates through the scene, we need to know its inter-
action with the objects in the environment. Different materials interact with light
in different ways such as reflectance, refraction and absorption. When light hits a
surface on point P with incident direction Ψ, it can leave the surface at the same
Light source
Camera
4 Introduction
Figure 1.3. BRDF(photo taken from wikipedia)
point or other point such as q with the exitant direction θ. Since in this report we
are not considering subsurface scattering we assume that the light leaves the sur-
face at the same point. Figure 1.3 shows the visual definition of this phenomena.
Thus we can describe the reflectance properties of a surface by a function that is
called bidirectional reflectance distribution function or BRDF. BRDF at point x
is defined as follow:
dL(x → θ)
fr(x, ψ → θ) = (1.2)
L(x ← ψ)cos(Nx, ψ)dωψ
where fr(x, ψ → θ) is the BRDF and cos(Nx, ψ) is the cosine of the angle
between the incident direction vector ψ and the normal vector at the point x, Nx.
The BRDF is denoted over the entire sphere of directions around a surface point
x (4π steradians).
1.1.3 Global Illumination
Global illumination is a term that refers to a group of algorithms in computer
graphics which are meant to add realism to the 3D scenes. These algorithms con-
sider not only the direct illumination where the light is coming directly from the
light source but also the indirect illumination where the light rays come bouncing
from other surfaces in the scene.Theoretically light refraction, reflection and shad-
ows are global illumination effects. However in practice these are considered as
local effects and global illumination is referred to the simulation of diffuse inter-
reflections and caustics. Algorithms such as Radiosity [26], ray tracing [8], path
tracing [16], photon mapping [14] and image based lighting are some examples
of global illumination. All these algorithm are an attempt to solve the rendering
equation which was firstly introduced by Kajia [12]:
∫
L(x → Θ) = Le(x → Θ) + fr(x,Ψ → Θ)L(x ← Ψ)cos(Nx,Ψ)dωΨ (1.3)
Ωx