Thesis--- Identification of Billboards in a Live Handball Game

161 Pages·2006·3.02 MB·English

Table of Contents

ABSTRACT ............................................................................................................................................3
I. INTRODUCTION...............................................................................................................................4
1.1. BACKGROUND ...............................................................................................................................4
1.2. IMAGE AND VIDEO ISSUES .............................................................................................................7
1.3. VARIOUS APPROACHES..................................................................................................................9
II. IMAGE FEATURES FOR TRACKING ......................................................................................12
2.1. EDGES..........................................................................................................................................12
2.2. COLOR .........................................................................................................................................14
2.3. HISTOGRAM.................................................................................................................................16
III. DEFORMABLE TEMPLATE MATCHING..............................................................................18
3.1. BASIC THEORY ............................................................................................................................19
3.1.1. Bayes Theorem....................................................................................................................19
3.1.2. Bayesian Formulation of the Deformation..........................................................................19
3.2. DEFORMATION MODELS ..............................................................................................................21
3.3. ALGORITHM.................................................................................................................................24
IV. CONDITIONAL DENSITY PROPAGATION (CONDENSATION) .......................................27
4.1. BASIC THEORY ............................................................................................................................28
4.1.1. Modelling Shape and Motion ..............................................................................................28
4.1.2. Discrete Time Propagation of State Density.......................................................................28
4.1.3. Temporal Propagation of Conditional Densities ................................................................29
4.1.4. Dynamic Models .................................................................................................................30
4.1.5. Measurement.......................................................................................................................30
4.1.6. Propagation ........................................................................................................................31
4.1.7. Factored Sampling..............................................................................................................32
4.2. THE CONDENSATION ALGORITHM ...............................................................................................33
4.3. TEMPLATE REPRESENTATION ......................................................................................................35
4.3.1. B-spline Curves...................................................................................................................35
4.3.2. Template Curves .................................................................................................................40
4.3.3. Affine Representation of B-spline Curves ...........................................................................47
4.4. CONDENSATION TRACKER...........................................................................................................53
4.4.1. Dynamic Model...................................................................................................................53
4.4.2. Observation Model..............................................................................................................56
4.4.3. Initialization........................................................................................................................58
4.4.4. Detailed Algorithm..............................................................................................................59
V. EXPERIMENT AND FINDINGS ..................................................................................................63
5.1. EXPERIMENT PURPOSE.................................................................................................................63
5.2. DATA SET ....................................................................................................................................64
5.3. ERROR MEASUREMENT................................................................................................................66
5.3.1. Mean Square Error .............................................................................................................66
5.3.2. Confidence Interval.............................................................................................................68
5.4. EXPERIMENTS ON DEFORMABLE TEMPLATE MATCHING .............................................................69
5.4.1. Experimental Results...........................................................................................................69
5.4.2. Findings and Discussions ...................................................................................................76
5.5. EXPERIMENTS ON CONDENSATION ALGORITHM..........................................................................78
5.5.1. Define an Optimal Template ...............................................................................................78
5.5.2. Coefficients A , A , and B ..............................................................................................79
0 1
5.5.3. Result and Findings ............................................................................................................85
5.6. FURTHER DISCUSSIONS..............................................................................................................104
5.6.1. Optimal Definition of Final Q at Measurement Step ........................................................104
5.6.2. Optimal Number of Particles ............................................................................................105
1
5.6.3. Initialization and the Stability of the Filter.......................................................................106
VI. COMPARISON OF TEMPLATE MATCHING AND CONDENSATION ...........................107
VII. CONCLUSION AND FUTURE IMPROVEMENTS..............................................................109
APPENDIX .........................................................................................................................................112
A. GRADIENT MAGNITUDE FOR PIXEL REPRESENTATION.................................................................112
B. COLOR MODEL CONVERSION FROM RGB TO HSI........................................................................112
C. ESTIMATION OF A0, A1, AND B BY MAXIMUM LIKELIHOOD ESTIMATION METHOD. ....................113
D. IMAGES FROM THE EXPERIMENT..................................................................................................115
D.1. The Results of Section 5.4 Experiments on Deformable Template Matching......................115
D.2. A0 s, A1 s and Bs Calculated from the Training Data in Section 5.5.2..............................124
D.3. Results of Section 5.5 Experiments on Condensation .........................................................125
E. MATLAB CODE.............................................................................................................................131
REFERENCES ...................................................................................................................................159

2
Abstract

Replacing commercial billboards around the ballgame field in a live TV transmission
has huge commercial potential. The fewer parties are involved in this process, the
more profitable it is. This thesis addresses part of this process in a handball game: the
way to track a billboard in a live handball game in real-time without knowledge about
camera conditions. The information provided beforehand is the approximate location
of cameras and the designs of the billboards. We present two methods to detect non-
rigid objects, namely deformable template matching and condensation algorithm, and
evaluate their accuracy and speed.

The template matching method seeks an object that best matches a deformable
template using the edge direction and the gradient magnitude of the edge. It can detect
the right object quite accurately. However, it is too slow to achieve real-time tracking.

The condensation algorithm predicts the location of the target object using a non
linear dynamic model. Following this the observation model determines the new
probability distribution for the next step by comparing the edge feature between
samples and the B-spline template. B-spline curves are flexible for representing
various shapes. The condensation algorithm is flexible and fast enough to be able to
achieve real-time tracking. However, it is difficult to create an appropriate dynamic
model suitable for many different settings.

Last but not least, we wish to thank our supervisor, Kim Streenstrup Pederson, for his
advice and encouragement. We also thank Maz Spork at Bopa Vision for materials as
well as advice from a practical point of view.

3
I. Introduction

1.1. Background

The analysis and manipulation of live TV transmissions has huge commercial
potential. For example, while a TV station in Denmark is live broadcasting a football
match in England, the TV station would be able to replace the advertisement
billboards in the stadium with Danish advertisements without the viewers in Denmark
being able to notice the underlying process. It is even more flexible and profitable if
the TV station in Denmark can achieve this replacement without any information
from other parties, such as the location and properties of the involved cameras. To
implement such a system several problems need to be solved.

This project has been done in collaboration with the Danish company Bopa Vision,
which would like to create a product for real-time commercial replacement for live
TV of, for instance, sports events. Bopa Vision provided us with parts of a video
taped handball game, several pictures with entire billboards, and a map of the
approximate locations of the billboards. The given video sequence consists of many
1
clips taken by different cameras.

Replacing billboards in a live handball game includes a lot of work as depicted in
Figure 1: identifying clips, identifying the location of billboards with advertisement in
each frame, taking care of the change in the appearance of billboards by motion blur,
zooming, shadows, and obstacles in front of the billboards, and placing another
commercial in the right place by adjusting for changes in global illumination without
losing picture quality or introducing delays in the transmission.

1
By a clip we mean a sequence of frames taken by one camera
4
Video sequence

Design of existing
Identifying clips

billboards

Clips

• Motion blur

• Zooming
Identifying billboards in real-time

• Shadow

• Occlusion
Design of new Position of billboards
billboards or

Transformation

Replacing the design of billboards in real-time
• Illumination

Clips with new

billboards

Connecting clips into a sequence

Figure 1 Diagram of the billboard replacement process

Given the limited scope of our project we will mainly focus on the second part of such
a system. Given a clip, we will detect the relevant billboard and provide either the
transformation parameters or the location of the billboard corresponding to each
frame in the video sequence. Even though Bopa Vision’s request is detecting all
billboards in the scene, we focus on the primary stage of detecting one billboard at a
time. A robust tracker should be able to deal with any occasions happening in the
scene, such as shadows, lighting changes, and situations by overlapping objects and
objects coming into the scene or moving out of it (Stauffer and Grimson, 2000).

In the identification process, we will deal with the following issues:
1. Should the process be done frame by frame? Can it be done on an entire clip?
2. How should the billboard be defined? We are supposed to have only limited
information about the scene before hand, such as the picture of a billboard. Is
5
it possible to detect billboards only from their shape? And if it is the case, will
connected billboards be considered as one?
3. What should the output be? What kind of coordinate descriptions should be
given to describe the locations of the billboards? Since the output should be
ready for the replacement process, it should also identify different kind of
advertisements.
4. How should distorted billboards be handled? Even though every input is a clip
taken by one camera, the camera zooms and pans all the time, which can result
in motion blur. Furthermore, the lighting conditions in the stadium may
change. These factors make the identification rather difficult.
5. How should a billboard, which is not totally visible, be identified? Sometimes
one object or a few objects will be in front of a billboard, which makes part of
the billboard invisible. So we might see only part of the billboard or a
billboard which is disconnected by obstacles.
6. How should we identify obstacles and generate masks for them. As mentioned
in Issue 5, it is often the case that a few obstacles will be in front of the
billboards. We need to identify these obstacles and make masks for them, so
that the frame can be ready for the replacement process. These masks are very
important as we need to put the object back to the original places after
replacing the billboard.

Regarding Issue 1, we start by detecting a template frame by frame, even though the
motion between two consecutive frames is in general not very large. We need to find
the accurate location of the template in order to maintain the sequence’s continuality.
On the other hand, it is impossible to know beforehand when the camera makes a
sudden big panning. If we skip a frame where the camera pans a lot, it is very likely
we will lose the track of the template.

Regarding issue 2, among many image features, we will discuss edges, colors,
histograms and shapes. Especially we will look at edges and mainly use this feature in
our algorithms.

Regarding issue 3, the output could be coordinates of each billboard. However, all
billboards move in the same way based on some transformation functions. Therefore
6
it is also possible to predict the locations of all billboards in a frame if we concentrate
on detecting this common transformation function. We will consider this issue in
Chapter IV.

Regarding issue 5, we will perform a few experiments of the case where a few players
running in front of a billboard. However, we leave the more thorough discussion of
this issue together with issue 4 and issue 6 for future research.

1.2. Image and Video Issues

In order to successfully locate and track objects in a video sequence, we need to
understand the features of the targeted objects and some issues concerning the video
sequence.

Useful features of an image are normally the ones that are detectable to the human
eyes, such as color, texture, edges and etc. These features are often used in image
segmentation and object recognition. In our project, each billboard has its unique
feature. Most of the billboards are rectangular containing text of a certain color and
font. But color itself might not be efficient in identifying billboards in our project,
because some billboards have the same colors and the background scene may also
have the same color as the billboards. So it is natural that we will start the tracking by
representing each billboard by a unique feature. This could be the shape of the logos
represented on the billboard and some additional color information. In order to
identify features of each billboard, we will use the color and edge details of each
frame. We will discuss color and image edges in chapter II.

Another source of concern is related to the motion in the video sequence. In video
sequences objects are blurred and the view frequently changes due to the zooming and
panning of the camera. This makes tracking more difficult. Moreover most target
objects do not have exactly the same features as the template image because of not
only motion blur, but also occlusions and lighting changes etc. Even the same object’s
features will change from frame to frame due to the above mentioned factors and the
object will be deformed by the zooming and panning of the camera as well as the
change of the 3 dimensional (3D) orientation of the billboard. Moving objects in a
7
video sequence have inherent motion blur, especially fast moving objects. Very often,
an object is partly occluded in the scene, for example, in our project, the billboards are
very often behind the handball players and parts of the billboard will be outside the
screen. This will cause a failure in locating and tracking the billboards. Figure 2
shows an example of an EL GIGANTEN billboard being occluded by a player.

One player is

in front of

ELGIGANTEN

Figure 2 A frame from the video sequence
Figure 3 is an example of motion blur taken from one of the clips. The images blur a
lot when the camera turns quickly to trace the ball. On top of that, players who are
running fast blur a lot as well.

Figure 3 Images with motion blur
(taken from the 120 th frame in ’10-2.avi’)
A player on the left and the billboard behind her are heavily blurred because she is running faster than
the other players.

In our video sequence, the camera moves constantly, sometimes very fast, in order to
follow the handball and the players. In this way, almost all the billboards we want to
track are motion blurred.

The task at this stage, detecting billboards, in the whole project of commercial
replacement is to provide the accurate location of the billboard in each frame where
another billboard can be inserted. But when the replaced billboard is put into the
scene, the same motion blur need to be generated in order to make the scene look
8
natural. It looks odd, for example, if a blurred edge of the original billboard is visible
at the border of the new, replaced billboard.

Due to the limitation in time, we will only cover motion blur and occlusion briefly
later in the thesis.

Our thesis will be based on the following assumptions:
1) A sequence of frames in the same clip is taken by the same camera.
2) The following information is given: (1) the designs of all billboards, (2) layout
of the billboards in the sports arena (relative positioning of billboards and
cameras)
3) Billboards have to be detected even though they are occluded or part of them
is invisible in the screen.
4) All kinds of billboards can be detected by the system we implement.
5) Real time performance can be achieved by the use of programming languages
such as C++. We have, however, decided to use Matlab and lower the speed
requirement reflecting the expected gains in speed from using a faster
programming language.
6) Clips are already made, which means that separating a video sequence into
clips is out of the scope of this project. The provided video sequence is an
MPEG2 file, which matlab cannot read. Therefore the clips are converted into
avi files, which is readable to matlab. We will use these converted images.
Somewhat surprisingly, the amount of frames is reduced when the MPEG2 file
is converted to an avi file and thereby, the data set becomes smaller than the
original one.

1.3. Various Approaches

There are several real-time tracking approaches using various image features for
different purposes. For video surveillance systems, where a camera neither moves nor
zooms, background subtraction works well because it detects only the pixels which
have changed their colors or intensities depending on whether it uses color images or
9
2
gray scale images (Stauffer and Grimson, 2000) (Lipton et. al., 1998). Lipton et. al.
mention, however, that background subtraction is not robust to changes in object size,
orientation and lighting conditions, which happen often in a handball game.
Combining more features into background subtraction for improving accuracy and/or
speed, Berriss, W. P. et. al. (2003) investigate a color-based approach for MPEG-7
standard and Koller et. al. (1994) use contour tracker and an affine motion model for a
robust real-time traffic scene surveillance. Background subtraction is effective only
when the objects change neither their shape nor size.

Another application of real-time tracking of deformable objects is area tracking, for
example, tracking a speaker’s head in a video conference. Fieguth and Terzopoulos
(1997) develop a very fast color-based model for tracking a speaker’s head. For this
purpose, we do not need the information on precise positions as long as the object is
seen in the screen.

On the contrary, accuracy is one of the important requirements for our project. In
addition, billboards not only change their size but also may deform by the projection
from the three-dimensional (3D) viewing frame to the two-dimensional (2D) viewing
plane. Therefore, we need more precise information on the positions or
transformations of deformable objects for every frame. Deformable template
matching addressed by Jain et. al. (1996, 1998) may be able to achieve this goal by
detecting the precise positions or transformation of an object in non-linear systems. A
disadvantage of this approach is that it is time consuming (Kervrann and Heitz, 1994).
The length of processing time also depends on the capacity of the computer used for
the analysis.

If deformable template matching is too slow to be able to achieve on-lien tracking,
another approach desirable for on-line tracking is to predict the motion as Yang and
Waibel (1996) suggest. Kalman filtering is a popular approach for linear tracking
when clutters do not occur in the image (Isard and Blake, 1996). In the handball game
we need to deal with clutters because players often overlap in front of billboards.

2
Explanation of color and intensity follows in Chapter 2.
10