Talis Vertriest Urban Data Mining applied to sound sensor networks

147 Pages·2016·5.07 MB·English

Urban Data Mining applied to sound sensor networks
Talis Vertriest
Supervisor: Prof. dr. ir. Dick Botteldooren
Counsellor: Prof. dr. ir. Bert De Coensel
Master's dissertation submitted in order to obtain the academic degree of
Master of Science in Industrial Engineering and Operations Research
Department of Information Technology
Chair: Prof. dr. ir. Daniël De Zutter
Faculty of Engineering and Architecture
Academic year 2015-2016
"The author gives permission to make this master dissertation available for
consultation and to copy parts of this master dissertation for personal use. In all
cases of other use, the copyright terms have to be respected, in particular with regard
to the obligation to state explicitly the source when quoting results from this master
dissertation."
(June 1, 2016)

I
Preface
While my name is on the front cover of this thesis, I am by no mean its sole
contributor. There are a number of people behind this project who deserve to be both
acknowledged and thanked: committed supervisors, generou s friends and a warm and
supportive family.
I would like to thank my thesis committee members, Professor Dick B otteldooren and
Professor Bert De Coensel for their guidance and unrelenting support through this
process. Both have routinely shared their pass ion and knowledge, which are to the
great benefit of this thesis.
I thank all my friends and in particular my friend and Post Doc Scientist in
Neurosciences Dr. Ken Veys for his valuable contribution and advice. This research
looks very different because o f his expertise, technical help, and the additional
computation force of his MacBook p ro.
I would like to express my deepest appreciation for my parents who did everything in
their power and beyond to fire fight my worries, concerns and anxieties, and have
worked to instill great confidence in both myself and my work.
Most importantly of all, I feel blessed that I am able to accomplish this thesis and
show intensive gratitude towards Mother Nature and all the people that contributed to
who I am today. In the same vein, I would like to extend great thanks to the
University of Ghent, to all professors and to everybody involved in the education of
our society.
II
Abstract
Title: URBAN DATA MINING APPLIED TO SOUND SENSOR
NETWORKS
Name: TALIS VERTRIEST
Supervisor: PROF. DR. IR. DICK BOTTELDOOREN
Counsellor: PROF. DR. IR. BERT DE COENSEL
Degree: MASTER OF SCIENCE
Major Field: INDUSTRIAL ENGINEERING AND OPERATIONS
RESEARCH
Department: INFORMATION TECHNOLOGY
Faculty: ENGINEERING AND ARCHITECTURE
University: UNIVERSITY OF GHENT
Academic year: 2015-2016
Almost every activity or event produces sound patterns, making sound a valuable
source of information in the analysis of environments. As one of our senses, sound
directly contributes to the human perception of places. Solely from sound information,
people are able to distinguish danger from safe situations, unusual events from normal
activity. This thesis designs a program in the attempt to detect those (point) anomalies
that people would define as abnormal as well as contextual anomalies, which are less
obvious for human perception.
Raw audio signals are not suitable as a direct input to a classifier. As a consequence,
the data is transformed into a representation that lends itself to successful learning,
known as feature extraction. This thesis focuses on unsupervised learning because of
its multi applicable character. More specifically it applies data exploration rather than
field knowledge for feature extraction. Spectrograms are treated as series of
meaningless numbers instead of audio representative digits. Gaussian Mixture Models
describe the data per minute and its parameters define the features.
Whether or not supplemented with time features, those Gaussian features are the key
ingredient for the feature vectors. For classification, feature vectors are again being
clustered, using different techniques and dimensionalities, depending on the type of
anomaly that is searched for (point, contextual or conceptual). Conceptual anomalies
are beyond the scope of this thesis, but for point anomalies as well as contextual
anomalies, Gaussian Mixture Models outperform intelligent Kmeans and form the
basic clustering technique for this research.
In parallel, a more known but rather classical method is applied, based on spectral
features. Both techniques are compared based on their computational intensity and
results, revealing the qualities of the newly designed technique based on data
exploration for feature extraction and unsupervised learning for classification.
Key words: Sound sensors - Anomaly detection - Data exploration - Unsupervised
Learning - Gaussian Mixture Models
III
Extended Abstract
Urban Data Mining applied to sound sensor networks
Talis Vertriest
Supervisor(s): Dick Botteldooren, Bert De Coensel
 Environmental Anomaly Detection is a much broader
Abstract This thesis develops a system that detects unusual research and its abilities go beyond those of the previous
situations on time, inspired by the respective property that bespoken recognition based researches. While speech
humans exhibit in their everyday life quite effortless. It therefore recognition and sound event recognition only search for
uses audio information from sound sensors as its sole input. The certain real-time events, they reject all other unclassified data
technique used for Feature Extraction is Data Exploration. More as useless, while in in anomaly detection the whole dataset is
specifically, a Gaussian Mixture Model describes the timeframe now of interest, whether including a certain event or not. The
of one minute without overlap, thereby capturing temporal as
well as spectral relations into one single model. The parameters reason for this is the input data. Urban soundscapes, in
of these GMM's form the key ingredients for feature vectors. contrary to speech or specific audio events, capture an almost
These can be added up with additional time features, depending unlimited variety of sounds with a very high level of noise.
on the type of anomaly that is sought after. Three types of Furthermore, many sources interfere simultaneously. The
anomalies can be distinguished; Unusual Events, Unusual audio input can thus not be compared anymore with a
Minutes and Contextual Anomalies, with the focus on the former taxonomy of possible contents, or in other words, no labelled
type. Classification is based on Unsupervised Machine Learning: data is available anymore. Every signal is part of the system
a GMM classifier clusters the feature vectors that are, once and helps defining whether new incoming data are normal or
suspected anomalous, subject to human supervision for labelling. abnormal, accumulating to Big Data.
The characteristics of the labelled samples are again learned by
the system, reducing human supervision over time and
converging to a zero total error rate. A Linear Program defines II. APPROACHES FOR ANOMALY DETECTION IN BIG SOUND
the optimal mutual error ratio. DATA
Keywords Sound sensors, Data Exploration, Gaussian
Mixture Model, Anomaly Detection, Unsupervised Machine A. Anomaly Types
Learning, Error Ratio Three types of anomalies are distinguished, based on what a
human supervisor would notify as anomalous. Unusual events
are single unusual events, capturing only a certain frequency
I. INTRODUCTION range in a certain time frame, e.g. a gunshot, a thunderstorm.
Today, more than half of the world’s population lives in Unusual timespans (one minute) contain a strange
urban areas, highlighting the need to improve urban combination of possibly normal events, e.g. someone playing
environments. Among human senses, hearing is second only music during a heavy storm. Contextual anomalies describe an
to vision and has additional advantages according to unusual moment rather than content. Children playing on the
complexity and versatility, making it it optimal manner for street are for example frequently occurring, but not during the
understanding urban and conceptual settings. middle of the night. The focus of this research is on unusual
The main focus of audio classification systems is Speech events, however also the search engines for the other
Recognition (SR), because of the many evident application anomalies are also set up.
fields and its well-defined area of content. Speech
Recognition has traditionally relied on field knowledge for B. Methodology
feature extraction, a popular choice being the Mel Frequency With the focus of audio classification systems on
Cepstral Coefficients (MFCC). recognition, where a form of labelled data is available, the
In the past years, with the rapid growth of technology, the traditional path for feature extraction is by data knowledge in
awareness of the many useful applications of audio the form of Band filters, spectral and temporal features. For
classification grew and Environmental Sound Recognition classification, the provision of labelled data allows Supervised
(ESR) gained attention. However, the hand-crafted features Machine Learning, or Semi-supervised Machine Learning
that are successful for Speech Recognition perform poorly for when labelled data is replaced by a well-informed supervisor.
noisy environments, and the urge for new techniques rose. For this research, no labelled data is available and because
The advantage of sound recognition is that only certain events the audio data is irreversibly transformed to spectrograms by a
are sought after and those are usually provided as labelled Discrete Fourrier Transform (DFT), direct and accurate
samples. The features to be extracted can thus be studied in supervision is inconceivable. Feature extraction is therefore
relation to their content. This is called semi-supervised based on data exploration and the classification process
learning. happens through Unsupervised Machine learning. In parallel,
a classical approach based on spectral features is run for
T. Vertriest is with the Information Technology Department, Ghent
University (UGent), Gent, Belgium. E-mail:
comparison and to obtain a better insights for decision are pictured. A Gaussian Mixture Model of five components
making. describes 480 subsequent spectrograms (1-minute), capturing
spectral and temporal features in one model. It makes use of
the known fact that data points in vicinity of time or
frequency, do not differ a lot from each other.

Figure 2 & 3: GMM per 480 spectrograms
Five parameters describe each of the five Gaussian
components: two are allocated to the mean and three to the
Figure 1: Classification Methodology covariance. Each minute is thus described by a feature vector
of 25 digits, a significant data reduction of 600 times.
C. Related work
B. Classification
Research depends either on the extraction of classically
known low-level features, or there is labelled data available so The classification technique depends on the type of
features can be derived by data exploration. The major part of anomaly. At first, unusual events are in fact unusual Gaussian
research even combines both field knowledge for feature components. When plotting the mean values of all Gaussian
extraction and supervised learning for classification. It lays in components, their histogram suggests a GMM classifier.
men's nature to apply knowledge rather than to dive into the Different dimensions have been tested; 2D clustering only the
unknown and furthermore, it performs quite accurately for the mean values, 5D clustering the mean values and covariances
fields that are most attractive and thus received most attention: and 5D standardized, whereby mean and covariances are
speech and music. Ntalampiras et al [1] for example rely on rescaled. The not rescaled 5D method performs the best. The
MFCC's and have labelled data available. Their evident feature vectors are thus clustered by a 5D GMM classifier and
conclusion is that it only works accurately for speech involved Gaussian components with a low probability density
samples. Radhakrishnan et al [2] also rely on MFCC's for according to the built model, are alerted anomalous.
feature extraction and have samples of 'normal' audio samples,
in the search for anomalies. This research already belongs to
the semi-supervised category because it is now unknown what
is looked after. Data exploration for feature extraction, as well
as unsupervised classification, is still in its infancy. A
combination of both seems even inexistent and this is why this
thesis is unique, because it combines data exploration with
unsupervised classification. The reason is very simple,
standard low-level features have been proven to be ineffective
for environmental audio signals, there is little to no additional
information about what types of features could be significant,
there is no labelled data available and there is no possibility to
create them by human supervising. Interesting work for
feature extraction is the idea of scattering, done by Salomon
and Bello, applied in two of their papers [3] [4]. Although
they start from the Mel spectrum, it is an interesting start point

to discover features. Also Cai, Lu et al [5] scatter the feature
Figure 3: Histogram mean values of all Gaussian components
vectors and apply statistical parameters. For this thesis,
instead of scattering low-level features, the spectrograms
Unusual minutes are detected by statistical approach. Each
could be scattered and described by a more complex statistical
minute is described by a feature vector of five digits, the
model instead of basic parameters. Classification inspiration
cluster numbers of its containing Gaussian components. Two
comes from the same papers [4] [5] [6] for their use of K-
different measures can be calculated per feature vector; the
means, but especially the use of GMM's is attractive for both
joint probability and the joint correlation, depending on a non-
feature extraction as well as classification, inspired by
scientifically proven point of view. Joint probability assumes
Ntalampiras et al [1].
the Gaussian components or underlying audio events
independent from each other while Joint correlation assumes
III. PROPOSED MODEL: GMM
them dependent. In case one of the two values is lower than a
prescribed threshold, the minute is alerted as anomalous.
A. Feature Selection Contextual anomalous count an additional continuous time
By scattering different spectrograms in one single graph, feature, representing the hour of the day. The feature vectors
relations between data points in frequency as well as in time per minute consist out of 12 digits, five times the cluster
V
centroid mean values of the containing Gaussian components minutes ordered in descending number of their anomalous
and two clock coordinates for the time feature. Clustering is content.
done with a 12D GMM.
V. RESULTS NEW APPROACH
C. Anomaly Threshold Definition
Unusual events cannot visually be divided into categories,
In order to define one or more thresholds for anomaly they seem to be a misfit concerning the number of Gaussian
assignment, one must consider the types of errors occurring, components applied. All of these anomalies are reprocessed,
their impact and interaction. A linear program defines the and assigned with their preferred number of components,
optimal threshold: using the Akaike Information Criterion. The newly created
feature vectors are classified with the same threshold as
𝐹𝑁
𝑚𝑖𝑛 ∗ 𝐶𝑓𝑛 + 𝐹𝑃 ∗ 𝐶𝑓𝑝 + 𝑇𝑃 ∗ 𝑅𝑡𝑝
before. Roughly 1/6th of the anomalies is still considered
𝐻
anomalous, but after supervision, the newly assigned Gaussian
components are not representative for the underlying
or
anomalous scatter points, for which the reprocessing is
𝑓𝐹𝑁(𝑡) eliminated. The supervision of the anomalies defined by five
𝑚𝑖𝑛 ∗ 𝐶𝑓𝑛 + 𝑓𝐹𝑃(𝑡) ∗ 𝐶𝑓𝑝 + 𝑓𝑇𝑃(𝑡) ∗ 𝑅𝑡𝑝
𝐻 Gaussian components results in 52% of true positives, 48% of
false positives, a ratio that remains similar also amongst the
least anomalous of that threshold. This suggests many false
where:
negatives. The threshold must be broadened and the
𝐹𝑁: number of False Negatives
supervision is of crucial importance, no hard coded threshold
𝐹𝑃: number of False Positives
can be set. In addition Machine Learning of false positives
𝑇𝑃: number of True positives
𝐻: factor for moral damage to human beings will help convergence to a zero error rate.
𝐶𝑓𝑛: Cost per false negative
Unusual minutes are still due to supervision. Only 35,7% of
𝐶𝑓𝑝: Cost per false negative
anomalies based on joint probability are the same as those
𝑅𝑡𝑝: Revenue per True positive
defined by unusual events, and none of those based on joint
t : threshold
correlation.
In order to solve this LP, the functions of threshold, as well
Contextual anomalies are hard to examine, because a deep
as the precise cost per error must be known. As this is never
knowledge of the environment is necessary, as all the alerted
the case in reality, different thresholds must be set to learn
anomalies visually appear to be normal.
those parameters.
The equation above defines the optimal threshold for the
VI. CONCLUSION AND OUTLOOK
optimal ratio of errors, in a steady state. However, it does not
reduce the total error rate. Therefore, Machine Learning is
A. Conclusion
applied after the supervision of alerted anomalies by an
authorized person. Assuming that the opinion of the Acoustic information is a highly valuable source of
supervisor is always correct, they are grouped into true information for environmental context awareness. One of the
positives and false positives. The characteristics of the false main difficulties of this thesis is that the transformed data is
positives are learned by another GMM classifier, that will be not reversible to its original audio waves, which makes
applied onto newly incoming alerted anomalies, converging to acoustic supervision impossible. Another difficulty is that the
a zero total error rate. data is of significant size, calling for computational efficient
techniques and creative thinking. The unsupervised approach
IV. CLASSICAL APPROACH: SPECTRAL FEATURES has the advantage that all results are directly originating from
the input data; no other knowledge can be mistakenly applied.
The approach of Gaussian Mixture Models does not only
A. Feature Extraction
allow significant data reduction, it also captures both spectral
Each spectrogram is described by nine characteristics, low-
relations and temporal relations in a single model.
level spectral features; Spectral Energy, Spectral Centroid,
When looking at the results of the unusual events, the
Spectral Spread, Spectral Roll-off Point, Spectral Entropy,
created model fits the data very accurately, and where it does
Spectral Kurtosis or flatness, Spectral Skewness, Spectral
not, a supervisor helps classifying true positives and false
Slope and Noisiness. Down sampling to 1/1000 was necessary
positives. The latter ones are input for another GMM classifier
because of the excessive size and computation forces needed.
that is gradually updated and not only replaces the human
After standardizing the data, Principle Component Analysis
supervisor over time but also reduces the total error rate.
(PCA) decorrelates and recombines these features into pseudo
Instead of applying a hard threshold on the nomination of
independent features. Kaiser's stopping rule or otherwise
anomalies, a more intuitive and morally correct technique is
called the eigenvalue one criterion is applied. With the extra
applied. The rate of false positives is initially taken too high
margin of one, three remaining features are considered
and human supervisor assigns each anomaly with a label:
optimal.
'false positive' or 'true positive'. The false positives are stored
and their characteristics are learned by the system. This self-
B. Classification
enhancement, also called machine learning, gradually
A 3D GMM classifier is applied onto the feature vectors,
decreases the rate of false positives and increases the accuracy
defining seven clusters. For a valuable comparison with the
of the system.
newly proposed model, the feature vectors are divided into
Besides the significant data reduction, the speed of the
timeframes of 480 spectrograms without overlap. Per minute,
program and the advantages of unsupervised learning, another
the number of anomalous feature vectors are counted and the
advantage of this research is that the developed technique can
VI
be applied on any environment. The technique will learn the
location's specific features and increase accuracy levels with
time.
B. Outlook
The duration of a thesis project allows only a certain
deepness of research, so evidentially, there is room for
improvement.
Conceptual anomalies are not addressed in this research.
The GMM's only encounter small-scale temporal relations, in
between one minute and one day. Although, the evolution of
the environment over time is also very important and could
reveal trends, seasonality, ...
Another interesting topic for future work is to build
taxonomies for different types of environments. Instead of
using a huge training set every time this program is applied
onto a new environment, the knowledge of likewise locations
could be used to converge faster and improve the level of
anomaly accuracy.
ACKNOWLEDGEMENTS
I would like to thank my thesis committee members,
Professor Dick Botteldooren and Professor Bert De Coensel
for their guidance and unrelenting support through this
process. Both have routinely shared their passion and
knowledge, which are to the great benefit of this thesis.
REFERENCES
[1] S. Ntalampiras, I. Potamitis, N. Fakotakis. (s.a.). On acoustic
surveillance of hazardous situations. University of Patras, Greece:
department of Electrical and Computer Engineering
[2] R. Radhakrishnan, A. Divakaran, P. Smaragdis. (2005). Audio Analysis
for Surveillance Applications. Cambridge: Mitsubishi Electric
Research Labs.
[3] R. Radhakrishnan, A. Divakaran, P. Smaragdis. (2005). Audio analysis
for surveillance applications. in IEEE WASPAA’05. pp. 158-161.
[4] J. Salamon, J.P. Bello. (s.a.). Feature learning with deep scattering for
urban sound analysis. Center for urban science and progress. New
York University.
[5] R. Cai, L. Lu, A. Hanjalic. (s.a). Unsupervised Content Discovery in
Composite Audio. Delft University of Technology: Department of
Mediamatics, Tshinghua University: Department of Computer Science.
[6] J. Salamon, J.P. Bello. (s.a). Unsupervised Feature Learning for Urban
Sound Classification. New York University: Center for Urban Science
and Progress, Music and Audio Research Laboratory.
VII
Table of contents
Preface II
Abstract III
Extended Abstract IV
Table of contents VIII
List of figures XI
List of Matlab Graphs XII
1. Introduction 1
1.1. Motivation 2
1.2. Challenges of Environmental Anomaly Detection 5
1.2.1. Big Data 5
1.2.2. Taxonomy 6
2. Approaches for Anomaly Detection in Big Sound Data 9
2.1. Concept Introduction 9
2.1.1. Input Data types 9
2.1.2. Anomaly types 10
2.1.3. Methodology 12
2.2. Feature extraction 13
2.2.1. Field Knowledge 13
Low-level spectral features 14
Low-level harmonic features 15
Low-level perceptual features 16
Mid-level Temporal Features 17
2.2.2. Data exploration 19
2.3. Classification 19
2.3.1. Supervised 20
2.3.2. Unsupervised 22
2.4. Related work 26
2.4.1. Supervised 26
2.4.2. Unsupervised 28
2.4.3. Conclusions 30
3. Proposed Model New Approach: GMM. 32
3.1. Concept 32
3.2. Data Preparation 32
3.2.1. Missing Data 32
3.2.2. Reorganization 32
3.3. Programming Language 33
3.3.1. Efficiency 33
3.4. Feature Selection 34
VIII