Automatic Detection of Commercial Blocks in Broadcast TV Content Alexandre Ferreira Gomes

96 Pages·2016·3.26 MB·English

Automatic Detection of Commercial Blocks
in Broadcast TV Content

Alexandre Ferreira Gomes

Thesis to obtain the Master of Science Degree in
Electrical and Computer Engineering

Supervisors
Prof. Maria Paula dos Santos Queluz Rodrigues
Prof. Fernando Manuel Bernardo Pereira

Examination Committee
Chairperson: Prof. José Eduardo Charters Ribeiro da Cunha Sanguino
Supervisor: Prof. Maria Paula dos Santos Queluz Rodrigues
Members of the Committee: Prof. João Magalhães

November 2016

Acknowledgements
In first place, I would like to thank to my father Mário, my mother Carmen and my sisters Filipa
and Inês for supporting me in every moments of this journey and for always do everything to help
me. Anything I say is not enough to express my sense of debt to you. At the same level, a special
word to Sara Mendes, whose grace and love made everything easier and makes me to improve
every single day.
A special thanks to professors Paula Queluz and Fernando Pereira, for all the availability,
guidance and for helping me to better understand the importance of having a critical view about
every situation – valuable lessons that will be useful in all my life; and to André Alexandre and
Luis Nunes for the company and for sharing some tears with me!
I would also thank to my uncle Rodrigo and my grandmother Isabel for the precious late meals
and all the breakfasts; my grandparents Silvino and Deolinda for helping me to grow up as a
better person; my uncles Pedro and Tânia and my beautiful cousins Afonso and Xavier for the
amazing and relaxing Saturday afternoons in their home; to my cousin Ricardo for helping me out
in a very sensitive moment; to Lurdes and her good mood.
Finally, but not least at all, a heartfelt thanks to all my good friends that have been with me in
past few years and will for sure remain in the next few decades: Alexandre Gabriel, Ricardo
Sousa, Tiago Sebastião, Miguel Ramos, Pedro Gama, Guilherme Gil, João Melo, João Silva,
David Oliveira, Ricardo Joaquinito and Carlos Silva. Also, a special word to João Brogueira. To
my favourite Civil Engineering guys: Beatriz Loura, Filipe Vale, Mariana Antunes, João Rafael
and also Ana Santos for her importance in some crucial moments. Finally, a warm hug to André
Antunes, Gonçalo Vieira, Joana Freitas and Bárbara Santos.

i
Abstract
As the global economy evolves, companies need to improve their marketing solutions in order
to get some advantage over competitors; TV advertising commercials have emerged as a major
tool for achieving this goal. From the video content point of view, TV commercials have some
specific characteristics as they all target to capture the viewers´ attention. Naturally, it is also
these characteristics that make it possible to automatically detect advertising content and
eventually skip it. Commercials are always packed and broadcasted together in the so-called
commercial blocks, containing a given amount of individual commercials. Moreover, their
structure depend not only on the country and its relevant legislation, but also on the specific
broadcaster, according to their advertising strategy and style.
Motivated by the solutions proposed along the last few years for TV commercials detection,
this Thesis presents an overview of the available state-of-the-art - notably to understand the
current weaknesses - and proposes a new and effective solution. The proposed method for TV
commercials detection is based on the presence or absence, in the screen, of a TV channel logo,
which is a specific type of Digital on-Screen Graphic (DoG), as this logo is never present in
commercial blocks. After segmenting the video, using a shot change detector, the resulting video
shots are analyzed in terms of color and shape, to conclude on the existence or not of DoGs on
the video content. A DoGs Database system containing the DoGs acquired over time is built and
continuously updated. A systematic control of the DoGs Database is performed to conclude about
the nature of each DoG and to classify each video segment as Regular Program or Commercial
Block. For the used video dataset, that resulted from recordings of three different Portuguese TV
channels, a minimum accuracy of 93,9% on commercials detection was achieved; furthermore,
the measured and reported processing time suggests that the proposed solution could enable
real time (i.e., while recording) detection of commercial blocks.

Keywords: TV advertising; commercial blocks; shot detection; Digital on-Screen Graphics; logos
detection; video processing.

ii
Resumo
À medida que a economia global se desenvolve, as empresas têm a necessidade de
melhorar as suas soluções de marketing de modo a obter alguma vantagem sobre a
concorrência; neste âmbito, os anúncios televisivos têm emergido como uma ferramenta
essencial para atingir este objetivo. Do ponto de vista do conteúdo, os anúncios publicitários têm
algumas características específicas para que possam captar a atenção dos telespectadores.
Naturalmente, são também essas características que permitem detetar automaticamente o
conteúdo comercial. Os anúncios publicitários são habitualmente combinados e transmitidos
pelos operadores televisivos em blocos comerciais que contêm um conjunto de anúncios
sucessivos a diferentes marcas e entidades. A estrutura e o modo como a publicidade é
transmitida em televisão dependem não apenas do país e da legislação em vigor, mas também
do operador específico e da sua estratégia e abordagem à questão da publicidade.
Motivado pelas soluções propostas ao longo dos anos, nesta Tese apresenta-se o estado-
da-arte na área da deteção de blocos publicitários, analisando-se as debilidades dos métodos
existentes, e propõe-se uma solução nova e eficaz. A solução proposta é baseada na presença
(ou ausência), no ecrã, do logo de um canal televisivo, já que este nunca está presente em blocos
publicitários; este logo é um caso particular de DoG – Digital on-Screen Graphic. Após segmentar
o vídeo a analisar, utilizando um detetor de mudança de shots, os segmentos vídeo resultantes
são analisados em termos de forma e cor, de modo a concluir-se sobre a existência, ou não, de
DoGs no conteúdo vídeo. Neste contexto, é construída uma base de dados de DoGs cujo objetivo
é armazenar os DoGs adquiridos ao longo do tempo e que é continuamente atualizada à medida
que a análise do vídeo avança. É também realizado um controlo sistemático da informação que
está na base de dados de DoGs de modo a que se conclua sobre a natureza de cada DoG.
Finalmente, classifica-se cada segmento de vídeo previamente fragmentado como Programa ou
Bloco Comercial, tendo em conta a classificação atribuída aos DoGs. Para o conjunto vídeos de
teste utilizado, e que resultou de gravações de três canais de televisão portugueses, obteve-se
uma exatidão mínima de 93,9% na deteção de tramas pertencentes a blocos comerciais;
adicionalmente, o tempo de processamento medido sugere que a solução proposta permitirá a
deteção de segmentos comerciais em tempo real (isto é, durante a gravação).

Palavras-chave: Publicidade em TV; blocos comerciais; deteção de shots; Digital on-Screen
Graphics; deteção de logos; processamento de vídeo.

iii
Table of Contents
Acknowledgements .................................................................................................................. i
Abstract .................................................................................................................................... ii
Resumo ................................................................................................................................... iii
Table of Contents .................................................................................................................... iv
Index of Figures ..................................................................................................................... vii
Index of Tables ....................................................................................................................... ix
List of Acronyms ..................................................................................................................... x
Chapter 1 - Context and Objectives ........................................................................................ 1
1.1 Motivation .................................................................................................................. 1
1.2 Objectives .................................................................................................................. 2
1.3 Main Contributions ..................................................................................................... 3
1.4 Thesis Outline ............................................................................................................ 3
Chapter 2 - TV Commercials: Legal Framework and Characterization .................................. 4
2.1 Legal Framework ....................................................................................................... 4
2.1.1 Advertising Legal Framework in the European Union ....................................... 4
2.1.2 Legal Framework for Advertising in Portugal .................................................... 4
2.2 Typical Structure of a Commercial Block .................................................................. 5
2.3 Intrinsic Characteristics ............................................................................................. 6
2.3.1 High Scene Cut Rates ....................................................................................... 6
2.3.2 Text Presence ................................................................................................... 6
2.3.3 Audio Jingles ..................................................................................................... 7
2.3.4 Audio Level ........................................................................................................ 7
2.4 Extrinsic Characteristics ............................................................................................ 7
2.4.1 Commercial Block Separator ............................................................................. 7
2.4.2 TV Channel Logo ............................................................................................... 8
2.4.3 Black Frames ..................................................................................................... 8
2.4.4 Time Duration .................................................................................................... 9
2.4.5 Commercials Repetition .................................................................................... 9
Chapter 3 - Overview of TV Commercials Detection Schemes ............................................ 10
3.1 Knowledge-based Detection ................................................................................... 10
3.1.1 The First Steps - Black Frames and Silence ................................................... 10
3.1.2 Going Deeper – Cut Rates .............................................................................. 12
3.1.3 Motion Analysis ............................................................................................... 14
3.1.4 Logo Detection ................................................................................................ 14
3.1.5 Audio Analysis ................................................................................................. 16
iv
3.1.6 Text Detection ................................................................................................. 17
3.1.7 Still images detection ....................................................................................... 19
3.2. Repetition-based Detection ................................................................................. 19
3.2.1. Lienhart et al. (1997) ....................................................................................... 20
3.2.2 J. M. Gauch and A. Shivadas (2005) .............................................................. 21
3.2.3 Li et al. (2008) .................................................................................................. 22
Chapter 4 - Proposed Solution: Architecture and Algorithms ............................................... 24
4.1 Learning about Commercials and Logos with Real TV Content ............................. 24
4.2 Characterizing TV Channel Logos .......................................................................... 27
4.3 Proposed System Architecture ................................................................................ 29
4.3.1 Designing the System...................................................................................... 29
4.3.2 Architecture walkthrough ................................................................................. 30
4.4 Shot Change Detection and Segmentation ............................................................. 32
4.4.1 Luminance Histogram Operations ................................................................... 33
• Luminance Frame Histogram Computation............................................................. 33
• Luminance Histogram Distance Computation ......................................................... 33
4.4.2 Adaptive Threshold Computation .................................................................... 33
4.4.3 Hard Cut Detection Decision ........................................................................... 34
4.4.4 Forced Segmentation ...................................................................................... 34
4.5 DoG Acquisition Algorithm ...................................................................................... 34
4.5.1 Video Segment Edges & Color Analysis ......................................................... 34
4.5.2 DoG Detection ................................................................................................. 42
4.6 DoGs Database Updating & DoG Type Decision .................................................... 46
4.6.1 DoGs Matching ................................................................................................ 48
4.6.2 DoGs Insertion in DoGs Database .................................................................. 49
4.6.3 Database Update & Management ................................................................... 50
4.6.3.1 Basic Solution Rationale .............................................................................. 50
4.6.3.2 Advanced Solution Rationale ...................................................................... 51
4.7 Video Segment Classification .................................................................................. 53
Chapter 5 – Performance Evaluation .................................................................................... 55
5.1 Test Material ............................................................................................................ 55
5.1.1 Shot Change Detection Assessment Dataset ................................................. 55
5.1.2 DoG Acquisition Assessment Dataset ............................................................. 56
5.1.3 Global Solution for Detecting Commercials Assessment Dataset .................. 58
5.2 Performance Assessment Methodology and Metrics .............................................. 59
5.2.1 Shot Change Detection Assessment ............................................................... 60
5.2.2 DoG Acquisition Assessment .......................................................................... 60
v
5.2.3 Global Solution for Detecting Commercials Assessment ................................ 61
5.3 Results and Analysis ............................................................................................... 62
5.3.1 Shot Change Detection Assessment Experiment ........................................... 62
5.3.2 DoG Acquisition Algorithm Assessement ........................................................ 64
5.3.3 Global Solution Assessment ........................................................................... 68
Chapter 6 - Summary and Future Work ................................................................................ 72
6.1 Summary ................................................................................................................. 72
6.2 Future Work ............................................................................................................. 74
Appendix A - DoGs Detection – Example of the complete process ..................................... 75
1. Video test sequence characterization ........................................................................... 75
2. Screenshots extracted from each shot .......................................................................... 75
3. Key Frames Edge Fusion step ...................................................................................... 77
4. SPMs Intersection step ................................................................................................. 78
5. Color Map of the detected DoG .................................................................................... 78
6. Heat Map ....................................................................................................................... 78
Appendix B - SPMs Intersection step results........................................................................ 79
Bibliography .......................................................................................................................... 80

vi
Index of Figures
Figure 1.1 - Some well-known commercials produced for Super Bowl………. ............................................. 1
Figure 2.1 - Typical structure of commercial block in the Portuguese TV channels………..……….…………5
Figure 2.2 - Examples of (A) fading, fade-out first, then fade-in [15]; (B) Dissolving [16]. ............................. 6
Figure 2.3 - Example of a TV commercial with text in different places and with different fonts. .................... 7
Figure 2.4 - Example of RTP (Portuguese public television) initial commercial block separator. .................. 8
Figure 2.5 - TV Channel Logo present in the top left corner. ........................................................................ 8
Figure 3.1 - Conditions imposed by Sadlier et al. to detect commercial blocks by using BF/SF series (image
from [23]). ................................................................................................................................................... 11
Figure 3.2 – Comparison of the Cuts per Minute metric for a commercial block and a movie [13]. ............ 12
Figure 3.3 - Comparison of the Cuts per Minute metric for a commercial block and a newscast [11]. ........ 12
Figure 3.4 – The time averaged gradient (in the first row); Binary mask obtained after morphological
processing (in the second row). .................................................................................................................. 15
𝑆𝑆𝑆𝑆 𝐿𝐿𝐿𝐿
Figure 3.5 – Binary image obtained after step 2.c. (top right corner); edge detection result (bottom right
corner). ....................................................................................................................................................... 18
Figure 3.6 – The result after step 2.e. (see the text boxes in the left image). ............................................. 18
Figure 3.7 – Examples of FPMI, each line representing a different type [12]. ............................................. 19
Figure 3.8 – Discount Tire Co.’s “Thank you!” commercial. ........................................................................ 20
Figure 3.9 – Confidence Level Assignment. ............................................................................................... 23
Figure 4.1 - (a) Lamborghini’s TV Commercial screenshot, where a flash of the brand logo can been seen
in the lower left corner (extracted from https://www.youtube.com/watch?v=Xd0Ok-MkqoE); (b) Screenshot
extracted from a Portuguese TV commercial: the commercial brand logo appears in the upper right corner
during the whole commercial…………………………………………………………………………….…….......25
Figure 4.2 - Screenshot from a Portuguese TV series: in the upper left and upper right corners, the TV
channel logo and the TV series logo, respectively. ..................................................................................... 26
Figure 4.3 - Screenshot from a Portuguese TV news program: in the upper left corner, the TV channel logo;
in the lower right corner, the news program logo (“BomDia Portugal”), the current time (“06:38”) and live
traffic information. ....................................................................................................................................... 26
Figure 4.4 - Screenshot from a Portuguese broadcaster self-promotion commercial: the program logo is
placed in the upper right corner during the whole self-promotion................................................................ 26
Figure 4.5 - Difficult logo examples: (a) The TV channel logo in the upper left corner is over a highly textured
zone, making it hard to detect; (b) The TV channel logo in the upper left corner is over the sky, making it
almost impossible to detect as its color is quite similar to the background color. ........................................ 28
Figure 4.6 - Difficult logo example: in the upper left corner, the TV channel logo contains a dark shadow
surrounding a colored graphical object. ...................................................................................................... 28
Figure 4.7 - Example of texture variations in a logo along time, notably in the central letter “I” and in the red
and orange regions. .................................................................................................................................... 28
Figure 4.8 - Example of a dynamic logo in terms of shape with snowflakes constantly falling on the logo. 29
Figure 4.9 - RTP1 logos: the old and the new. ............................................................................................ 29
Figure 4.10 - Global System architecture. ................................................................................................... 31
Figure 4.11 - Shot Change Detection and Segmentation module flowchart. ............................................... 32
Figure 4.12 - Schematic representation of the Luminance Histogram Distance Computation between
consecutive frames. .................................................................................................................................... 33
Figure 4.13 - Video Segment Edges& Color Analysis module flowchart. .................................................... 35
vii
Figure 4.14 - (a) SL = 53; ; : 1, 11, 21, 31, 41, 51 (b) SL = 76;
; KFs indexes: 1, 11, 21, 31, 41, 51, 61, 71 (c) SL = 110; ; : 1, 12,
NKF= ceil0.1 ×SL=6 KFs indexes NKF=
23, 34, 45, 56, 67, 78, 89, 100. ................................................................................................................... 36
ceil0.1 ×SL= 8 NKF= 10 KFs indexes
Figure 4.15 - DoG Areas Definition: the white boxes signalized as ULC, URC, LLC and LRC. .................. 37
Figure 4.16 - Canny Edge Detector output for the video frame in Figure 4.15. ........................................... 38
Figure 4.17 - Screenshot extracted from the video segment “rtp1_demo1” [ref:
https://www.dropbox.com/s/9rlzlcsajgad9dz/rtp1_demo1.mp4?dl=0], with a TV channel logo in the upper left
corner. ......................................................................................................................................................... 38
Figure 4.18 - KFs edges maps obtained for the video sequence “rtp1_demo1”. ........................................ 39
Figure 4.19 - Key Frames Edges Fusion output for the ULC region (video sequence “rtp1_demo1”)......... 40
Figure 4.20 - Key Frames Edges Fusion output for the ULC region after dilation (video sequence
“rtp1_demo1”). ............................................................................................................................................ 40
Figure 4.21 - Heat map representing the mean chrominances (Cr and Cb) variance. ................................. 41
Figure 4.22 - Static Pixels Map (SPM) for the video sequence used as example. ...................................... 42
Figure 4.23 - DoG Detection module flowchart. .......................................................................................... 43
Figure 4.24 - Result of SPMs Intersection for various values of Nseg: (a) Nseg = 4; (b) Nseg= 5; (c) Nseg = 6. . 43
Figure 4.25 - DoG Presence Verification: (a) DoG in DDB; (b) SPMs Intersection map under verification; (c)
Final result after DoG Presence Verification step, which contains the pixels classified as Similar in this stage.
.................................................................................................................................................................... 45
Figure 4.26 - DoGs Database Updating & Logo Type Decision module flowchart. ..................................... 47
Figure 4.27 - Video Segment Classification module flowchart. ................................................................... 54
Figure 4.28 - Output structure that should to be corrected. ......................................................................... 54
Figure 5.1 - Structure of the “sicNotGS” test sequence. ............................................................................. 58
Figure 5.2 - Structure of "tviGS" test sequence. .......................................................................................... 58
Figure 5.3 - Structure of the “rtp1GS” test sequence. ................................................................................. 59
Figure 5.4 - Strong brightness change in consecutive frames, visible in the upper right corner. ................ 63
Figure 5.5 - Examples of situations where the DoGA algorithm failed with false negatives. ....................... 66
Figure 5.6 - Sequence of frames (in different shots) showing failures, notably false negatives on the lower
left corner and the lower right corner); also a “RTP1” logo shape transition (in the stripes part) can be
observed on the upper left corner. .............................................................................................................. 67
Figure 5.7 - Example of a situation where the algorithm fails by detecting false positives in the lower corners,
due to highly textured background. ............................................................................................................. 68
Figure A.1 - Screenshot extracted from the first shot of “rtp1_example” video test sequence. ................... 75
Figure A.2 - Screenshot extracted from the second shot of “rtp1_example” video test sequence. ............. 76
Figure A.3 - Screenshot extracted from the third shot of “rtp1_example” video test sequence. .................. 76
Figure A.4 - Screenshot extracted from the fourth shot of “rtp1_example” video test sequence. ................ 76
Figure A.5 - Screenshot extracted from the fifth shot of “rtp1_example” video test sequence. ................... 76
Figure A.6 - Key Frames Edges Fusion step output obtained for each shot of “rtp1_example” video test
sequence . .................................................................................................................................................. 77
Figure A.7 - Key Frames edges maps that represent the fifth shot of the sequence................................... 77
Figure A.8 - SPMs intersection output for the sequence "rtp1_example". ................................................... 78
Figure A.9 - Color map of the detected DoG. .............................................................................................. 78
Figure A.10 - Heat map corresponding to the Key Frames Edge Fusion map of shot 5. ............................ 78
Figure B.1 - Some results from the SPMs Intersection step for each TV channel tested in DoG Acquisition
Algorithm Assessement. ............................................................................................................................. 79

viii