Deep Physiological Arousal Detection in a Driving Simulator

88 Pages·2017·2.77 MB·English

Master Thesis

Deep Physiological Arousal
Detection in a Driving
Simulator

Aaqib Saeed
August 2017
Deep Physiological Arousal Detection in a
Driving Simulator
Master Thesis
by
Aaqib Saeed
1651854
MSc Computer Science
Data Science and Smart Services
Enschede, August 2017
Graduation Committee:
Dr.Ir. Maurice van Keulen, University of Twente
Dr. Stojan Trajanovski, Philips Research
Prof. Dr. Jan B.F. van Erp, University of Twente
Dr. A.K. Ramakrishnan, University of Twente

Acknowledgments
I am thankful to many people for their support, contribution and trust which made two
years of master studies very exciting, and an experience in itself. I would like to thank
my advisers Stojan Trajanovski, Maurice van Keulen and Jan van Erp for their guidance,
critical feedback, thought-provoking discussions and above all giving me freedom to pursue
my own ideas. Their unfailing help not just enabled me to get a better understanding of
applied Machine Learning but helped me develop a research mindset. I also want to thank
my internship supervisors (in Fit2Perform project) Adrienne Heinrich and Victor Kallen for
sparking my interest in biometrics data analytics.
I owe a special thanks to my friends back home in Pakistan, Nouman Farooq, Inam Akbar
and Mudasir Ali for all their support and motivation. I am also very grateful to my teachers
from undergraduate studies especially, M.Qasim Pasta, Faraz Zaidi and Husain Parvez for
their encouragement and being an inspiration.
But most importantly, I am very thankful to my parents for all their love, sacrifices and
countless efforts in providing new opportunities to me and my siblings for a better future.
A special word of gratitude to my fiance, Zaharah A.Bukhsh, for always being there for me
and being my emotional support in making this journey wonderful.
3

Abstract
Driving is an activity that requires considerable alertness. Insufficient attention, imperfect
perception,inadequateinformationprocessing,andsub-optimalarousalarepossiblecausesof
poor human performance. Understanding of these causes and the implementation of effective
remedies is of key importance to increase traffic safety and improve driver’s well-being.
For this purpose, we used deep learning algorithms to detect arousal level, namely, under-
aroused, normal and over-aroused for professional truck drivers in a simulated environment.
The physiological signals are collected from 11 participants by wrist wearable devices. We
presented a cost effective ground-truth generation scheme for arousal based on a subjective
measure of sleepiness and score of stress stimuli. On this dataset, we evaluated a range of
deep neural network models for representation learning as an alternative to handcrafted
feature extraction. Our results show that a 7-layers convolutional neural network trained
on raw physiological signals (such as heart rate, skin conductance and skin temperature)
outperforms a baseline neural network and denoising autoencoder models with weighted F-
score of 0.82 vs. 0.75 and Kappa of 0.64 vs. 0.53, respectively. The proposed convolutional
model not only improves the overall results but also enhances the detection rate for every
driver in the dataset as determined by leave-one-subject-out cross-validation.
5

Contents
Acknowledgments 3
Abstract 5
Contents 7
List of Figures 11
List of Tables 13
List of Acronyms 15
1 Introduction 1
2 Background Study and Related Work 7
2.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Recognizing Physiological Arousal . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.1 Fatigue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.2 Stress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Deep Learning for Sequence Classification . . . . . . . . . . . . . . . . . . . . 12
2.4 Analysis of Existing Approaches . . . . . . . . . . . . . . . . . . . . . . . . . 14
3 Data and Methodology 15
3.1 Experimental Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 Data Collection and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
7
Contents
3.3 Arousal Ground Truth Annotation . . . . . . . . . . . . . . . . . . . . . . . . 17
3.4 Pre-processing and Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . 18
4 Deep Neural Networks for Arousal Classification 23
4.1 Problem Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2 Neural Network and Denoising Autoencoder . . . . . . . . . . . . . . . . . . . 24
4.3 Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 Recurrent Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.5 Hybrid Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.6 Training Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.7 Tackling Data Imbalance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.7.1 Threshold-Moving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.7.2 Over-Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.7.3 Weighted Categorical Cross Entropy . . . . . . . . . . . . . . . . . . . 37
5 Experiments and Discussion 39
5.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1.1 Dataset Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1.2 Synthetic Data Generation . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1.3 Evaluation Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.1.4 Performance Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.1.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.1.6 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.2.1 Validation of Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.2.2 Convolutional and Recurrent Neural Networks . . . . . . . . . . . . . 45
5.2.3 Evaluation of Hybrid Models . . . . . . . . . . . . . . . . . . . . . . . 51
5.2.4 Effect of techniques to solve Data Imbalance . . . . . . . . . . . . . . 52
5.2.5 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
8
5.2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6 Conclusion and Future Work 61
Appendix 63
A1 Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
A2 Oversmapled classes per driver using SMOTE . . . . . . . . . . . . . . . . . . 63
A3 Cost matrices for Threshold-Moving method . . . . . . . . . . . . . . . . . . . 64
A4 Illustration of ground truth unavailability . . . . . . . . . . . . . . . . . . . . 65
Bibliography 67