Data-Driven Facial Expression Analysis from Live Video

154 Pages·2017·2.82 MB·English

Data-Driven Facial
Expression Analysis from Live Video

by

Wee Kiat, Tay

A thesis
submitted to the Victoria University of Wellington
in partial fulfilment of the
requirements for the degree of
Master of Science
In Computer Graphics.
Victoria University of Wellington
2017
Abstract
Emotion analytics is the study of human behavior by analyzing the
responses when humans experience different emotions. In this thesis, we
research into emotion analytics solutions using computer vision to detect
emotions from facial expressions automatically using live video.
Considering anxiety is an emotion that can lead to more serious
conditions like anxiety disorders and depression, we propose 2 hypotheses
to detect anxiety from facial expressions. One hypothesis is that the
complex emotion “anxiety” is a subset of the basic emotion “fear”. The
other hypothesis is that anxiety can be distinguished from fear by
differences in head and eye motion.
We test the first hypothesis by implementing a basic emotions
detector based on facial action coding system (FACS) to detect fear from
videos of anxious faces. When we discover that this is not as accurate as
we would like, an alternative solution based on Gabor filters is
implemented. A comparison is done between the solutions and the Gabor-
based solution is found to be inferior.
The second hypothesis is tested by using scatter graphs and
statistical analysis of the head and eye motions of videos for fear and
anxiety expressions. It is found that head pitch has signific ant differences
between fear and anxiety.
As a conclusion to the thesis, we implement a systems software
using the basic emotions detector based on FACS and evaluate the
software by comparing commercials using emotions detected from facial
expressions of viewers.
i
Acknowledgements
I would like to thank my supervisor Dr. Taehyun Rhee and my co-
supervisors Dr. Harvey Ho and Prof. Neil Dodgson. It’s with their
guidance, support and valuable advice that I can complete my master’s
research and thesis.
I would also like to thank my fellow postgraduate students from the
computer graphics group for sharing their research and providing
valuable feedback during our weekly group meetings.
In addition, I would like to give special thanks to Auckland
Bioengineering Institute for sponsoring the research grant to support my
thesis.
Finally, I would like to take this opportunity to express my
gratitude to my brother and parents for their support and encouragement
given to me throughout my life. Without them, it would have been
possible for me to pursue my interests in computer graphics and
undertake this master’s degree.

iii
Contents
1. Introduction ............................................................................................. 11
1.1 Motivation of thesis ...................................................................... 11
1.2 Objectives of thesis ........................................................................ 12
1.3 Research Methodology ................................................................. 13
1.3.1 Literature survey on solutions for detecting anxiety ..... 13
1.3.2 Propose hypothesis on facial expression of anxiety ....... 13
1.3.3 FACS-based solution......................................................... .14
1.3.4 Gabor-based solution ......................................................... 16
1.3.5 Detecting head and eye movement ................................. 18
1.3.6 System application for detecting emotions ..................... 19
1.4 Structure of thesis .......................................................................... 19
2. Background and Related Works ............................................................ 21
2.1 Basic Emotions ............................................................................... 21
2.2 Facial Action Coding System (FACS) .......................................... 22
2.3 Facial Expression of Anxiety ........................................................ 26
2.4 Detecting emotions from facial images ....................................... 28
2.4.1 FACS-based detection methods ....................................... 28
2.4.2 Gabor-based detection methods ...................................... 31
2.5 Beyond Basic Emotions ................................................................ 34
2.6 Affective Applications .................................................................. 36
2.6.1 Detecting emotional stress from facial expressions for
driving safety ................................................................................. 36
2.6.2 Video Classification and Recommendation using
Emotions ........................................................................................ .37
2.6.3 Predicting Movie Ratings from Audience Behavior .s.... 38
2.6.4 Intelligent Advertising Billboards ................................... 39
v
vi CONTENTS
2.6.5 Predicting Online Media Effectiveness using Smile
Response.......................................................................................... 40
3. Databases and Tools ................................................................................ 43
3.1 Facial Expression Databases ........................................................ .43
3.1.1 Japanese Female Facial Expression (JAFFE) ................... 43
3.1.2 Extended Cohn-Kanade Database (CK+) ........................ 44
3.1.3 Mind Reading DVD .......................................................... .46
3.1.4 Affectiva-MIT Facial Expression Dataset (AM-FED)..... 49
3.2 Tools Used ..................................................................................... 50
3.2.1 OpenFace ............................................................................ 51
3.2.2 Weka ................................................................................... 51
3.2.3 Matlab ................................................................................. 52
3.2.4 LibSVM ............................................................................... 53
3.2.5 WebRTC ............................................................................. 55
3.2.6 Kurento ............................................................................... 57
4. Detecting Emotions using Facial Action Units ..................................... 59
4.1 Existing solution and its limitations ............................................ 59
4.2 Proposed Solution and Implementation .................................... 60
4.3 Testing Methodology ................................................................... 63
4.3.1 How to select the test candidates .................................... 63
4.3.2 How to interpret the results ............................................. 66
4.4 Results ........................................................................................... .67
4.4.1 Classifier performance using CK+ Database .................. 67
4.4.2 Classifier performance using JAFFE Database ............... 70
4.4.3 Detecting fear from anxiety videos ................................. 71
4.5 Analysis and Discussion .............................................................. .71
5. Detecting Emotions using Gabor Filter ................................................. 75
5.1 Proposed Solution ......................................................................... 75
CONTENTS vii
5.2 Implementation ............................................................................... 77
5.3 Results ............................................................................................... 82
5.3.1 CK+ Database ...................................................................... 82
5.3.2 JAFFE Database ................................................................... 83
5.4 Analysis and discussion ................................................................. 85
6. Detecting Head and Eye Motion............................................................ 89
6.1 Method ........................................................................................... .89
6.2 Results ............................................................................................ .92
6.3 Analysis and discussion ............................................................... 97
7. Systems Implementation .......................................................................101
7.1 Introduction ................................................................................. 101
7.2 System Architecture .................................................................... 101
7.2.1 Overview .......................................................................... 101
7.2.2 Kurento Client ................................................................. 104
7.2.3 Kurento Media Server ...................................................... 106
7.2.4 Applications Server ......................................................... .107
7.3 Prototype ...................................................................................... 107
7.3.1 Specifications.................................................................... 107
7.3.2 Implementation ............................................................... 108
7.4 Evaluation of System .................................................................. 109
7.4.1 Procedure .......................................................................... 110
7.4.2 Evaluation methodology ................................................. 110
7.4.3 Test Results ....................................................................... 112
7.5 Analysis and discussion ............................................................. .115
8. Conclusion ..............................................................................................117
8.1 Summary and Findings .............................................................. .117
8.2 Limitations and Future Work ..................................................... 118