Behaviour analysis within Drive-by-downloads Mohammad ALLY REHAZ Puttaroo University of ...

382 Pages·2017·4.9 MB·English

by  Mohammad Puttaroo

A BEHAVIOURAL STUDY IN RUN-
TIME ANALYSIS
ENVIRONMENTS AND DRIVE-BY-
DOWNLOAD ATTACKS

MOHAMMAD ALLY REHAZ PUTTAROO

A thesis submitted in partial fulfilment of the
requirements of The University of West London
for the degree of Doctor of Philosophy
in
Cyber Security and Operating Systems

July 2017
Acknowledgements
The process of undertaking this PhD has provided a life changing process of
learning and development for me. It would not have been possible to undertake the
PhD without the guidance, support and experience of many people.
Firstly, I would like to thank my supervisor and mentor, Professor Peter
Komisarczuk. Peter was exactly what every student who is crazy enough to start a
PhD needed. With his invaluable expertise within this subject, there was never a time
where Peter did not thoroughly support and provide me with direction. This direction
was greatly appreciated during the difficult periods during my PhD. I am truly lucky
to have had met and been able to work under the guidance of such an exemplary
gentleman and Professor. To put into perspective, he kept me as his PhD student
years after having left the original institution and never failed to uphold but the
highest standards of supervisory and support. Thank you, Peter, there are no words
to describe your kindness and dedication for what has been one hell of a journey.
My parents have been a constant source of strength, motivation and support during
my studies: from mild encouragement to listening to me worry about my progress
throughout the journey. They were there when experiments failed, and environments
broke, providing me with a constant stream of positivity and perseverance. Without
them, it would have proved impossible to successfully complete this thesis. Thank
you for being supportive and providing me the means to take part of this (insane)
journey. I would like to thank my lovely sister Madina for all the joys she brings to me
along with Mazmoon and Munira.
1
My beloved wife Imrana deserves an award for the endless emotional support and
for the countless hours proof-reading my thesis and learning about honeypots just to
accommodate my endless monologues, debates, doubts and reviews.
I would also like to extend my sincerest gratitude towards the supervisory team: Dr.
Renato Amorim who has one of the sharpest eyes when it comes to reviewing
transcripts whilst also being an amazing co-author who has read through this thesis
countlessly and helped tremendously in the betterment. Professor Thomas Roth-
Berghofer has also been a constant source of feedback and review. These were
key individuals in the refinement of the studies.
The feedback from Anastasia, Christian and Junaid at the internal school board was
very helpful and finding thesis reviewers that are willing to go through an entire
thesis is a rare commodity. I would like to thank them for the time and effort put into
the betterment of the work.
I extend my thanks towards the University of West London for the scholarship and
sponsorship that it provided over the years and the team at the graduate school. This
is extended to the IT department for accommodating my needs and experiments. It
must have been terrifying for an IT department to be told that malicious website
analysis needed to take place on a network and without the accommodation
required, this research would not have been possible.
The Honeynet project has been an indirect contributor for this work by inspiring
motivation for research focus in this field. In particular, I would like to thank the
Polish CERT (CERT Polska) for their various security projects, which led to this
research.

2
For my parents, who have firmly believed in education from the very first few years of
my life and everyone who have supported me get through the struggles PhD journey
and grow as a person.

3
Abstract
In the information age, the growth in availability of both technology and exploit kits
have continuously contributed in a large volume of websites being compromised or
set up with malicious intent. The issue of drive-by-download attacks formulate a high
percentage (77%) of the known attacks against client systems. These attacks
originate from malicious web-servers or compromised web-servers and attack client
systems by pushing malware upon interaction. Within the detection and intelligence
gathering area of research, high-interaction honeypot approaches have been a long-
standing and well-established technology. These are however not without
challenges: analysing the entirety of the world wide web using these approaches is
unviable due to time and resource intensiveness. Furthermore, the volume of data
that is generated as a result of a run-time analysis of the interaction between website
and an analysis environment is huge, varied and not well understood. The volume of
malicious servers in addition to the large datasets created as a result of run-time
analysis are contributing factors in the difficulty of analysing and verifying actual
malicious behaviour. The work in this thesis attempts to overcome the difficulties in
the analysis process of log files to optimise malicious and anomaly behaviour
detection.
The main contribution of this work is focused on reducing the volume of data
generated from run-time analysis to reduce the impact of noise within behavioural
log file datasets. This thesis proposes an alternate approach that uses an expert
lead approach to filtering benign behaviour from potentially malicious and unknown
behaviour. Expert lead filtering is designed in a risk-averse method that takes into
account known benign and expected behaviours before filtering the log file.
Moreover, the approach relies upon behavioural investigation as well as potential for
4
system compromisation before filtering out behaviour within dynamic analysis log
files. Consequently, this results in a significantly lower volume of data that can be
analysed in greater detail. The proposed filtering approach has been implemented
and tested in real-world context using a prudent experimental framework. An
average of 96.96% reduction in log file size has been achieved which is transferable
to behaviour analysis environments.
The other contributions of this work include the understanding of observable
operating system interactions. Within the study of behaviour analysis environments,
it was concluded that run-time analysis environments are sensitive to application and
operating system versions. Understanding key changes in operating systems
behaviours within Windows is an unexplored area of research yet Windows is
currently one of the most popular client operating system. As part of understanding
system behaviours for the creation of behavioural filters, this study undertakes a
number of experiments to identify the key behaviour differences between operating
systems. The results show that there are significant changes in core processes and
interactions which can be taken into account in the development of filters for updated
systems.
Finally, from the analysis of 110,000 potentially malicious websites, typical attacks
are explored. These attacks actively exploited the honeypot and offer knowledge on
a section of the active web-based attacks faced in the world wide web. Trends and
attack vectors are identified and evaluated.

5
Contents
Acknowledgements .................................................................................................... 1
Abstract ...................................................................................................................... 4
Chapter 1 – Introduction ........................................................................................... 21
1.1 Motivation ....................................................................................................... 21
1.2 Research goals ............................................................................................... 23
1.3 Contributions................................................................................................... 24
1.3.1 Methodology in developing the expert driven behavioural filters .............. 24
1.3.2 Identification of differences in behavioural manifestations in different
versions operating systems. .............................................................................. 25
1.3.3 Observed active exploits and malicious behaviours attacking honeypots 26
1.4 Thesis organisation ......................................................................................... 26
1.5 Publications .................................................................................................... 29
Chapter 2 – Literature review ................................................................................... 31
2.1 Introduction to drive-by-download and research area ..................................... 31
2.1.1 Drive-by-download attacks ....................................................................... 31
2.1.2 Detection, intelligence gathering parties and technologies ...................... 37
2.1.3 Collecting drive-by-download samples ..................................................... 40
2.2 Methods of analysing drive-by-downloads ...................................................... 41
2.2.1 Low-interaction ......................................................................................... 42
2.2.2 High-interaction ........................................................................................ 43
2.2.3 Hybrid-interaction ..................................................................................... 45
6
2.2.4 Chosen approach and rationale ............................................................... 47
2.3 Malware .......................................................................................................... 48
2.3.1 Malware analysis ...................................................................................... 48
2.3.2 Research around malware analysis ......................................................... 51
2.3.3 Malware packers ...................................................................................... 53
2.3.4 Differentiating drive-by-download and malware analysis ........................ 55
2.5 Instruments for drive-by-download collection and analysis ............................. 56
2.5.1 Capture-HPC ............................................................................................ 56
2.5.2 Cuckoo sandbox ...................................................................................... 58
2.6 Related work ................................................................................................... 59
2.6.1 Malicious Domain name servers .............................................................. 59
2.6.2 Systems monitoring state change ............................................................ 59
2.6.3 JavaScript research ................................................................................. 62
2.6.4 Current trends in client honeypot research .............................................. 63
2.6.5 Applicability of machine learning .............................................................. 64
2.6.6 Clustering approaches ............................................................................. 65
2.6.7 Classification approaches ........................................................................ 66
2.6.8 Closely related work ................................................................................. 68
2.7 Gap identification and research questions ...................................................... 70
2.7.1 Behaviour filtering in behaviour analysis environments ............................ 71
2.7.2 Expert driven filtering and exclusion lists ................................................. 72
7
2.7.3 Windows 7 applicability and operating system behaviours ...................... 76
2.7.4 Research scope and summary ................................................................. 77
Chapter 3 – Methodology and experiment design .................................................... 82
3.1 Safe, prudent and valid malware experimentation .......................................... 86
3.1.1 Correctness in malware experimentation ................................................. 88
3.1.2 Realism in malware experimentation ....................................................... 90
3.1.3 Transparency in malware experimentation............................................... 92
3.1.4 Safety in malware experimentation .......................................................... 93
3.1.5 Conclusion ............................................................................................... 95
3.2 Drive-by-download resource gathering ........................................................... 97
3.2.1 Creation of the data corpus ...................................................................... 99
3.2.2 Justification for the creation of the malware data corpus ....................... 101
3.2.3 Sample size evaluation .......................................................................... 102
3.3 Experimental setup ....................................................................................... 104
3.3.1 Experimental methodology justification .................................................. 105
3.3.2 Capture-BAT .......................................................................................... 106
3.3.3 Duration of malicious webpage analysis ................................................ 107
3.4 Testing the validity of experimental setup ..................................................... 111
3.5 Alternate resource gathering evaluations ..................................................... 111
3.6 Roundup ....................................................................................................... 111
Chapter 4 – Expert driven behaviour filter development ......................................... 115
8
4.1 Background .................................................................................................. 116
4.2 Research requirements ................................................................................ 117
4.2.1 Limitations of Capture-BAT within the Windows 7 environment ............. 118
4.2.2 Identification of behavioural vectors within Windows 7 .......................... 119
4.2.3 Exclusion list creation rationale .............................................................. 120
4.2.4 Aims and goals of exclusion lists ............................................................ 123
4.3 Methodology used in developing expert driven behavioural filters ................ 124
4.3.1 Decision making process behind classifications of behaviour ................ 127
4.3.2 Processing behavioural log files to synthesise expert driven behavioural
filters ............................................................................................................... 128
4.4 Inside an exclusion list .................................................................................. 130
4.4.1 Typical Windows 7 native system calls .................................................. 133
4.4.2 System behaviours: The Windows 7 file system behaviours .................. 136
4.4.3 System behaviours: The Windows 7 processes’ behaviours.................. 142
4.4.4 System behaviours: The Windows 7 registry behaviours ....................... 148
4.4.5 Differences between Windows 7 with Windows XP system calls ........... 157
4.5 The development cycle of exclusion lists ...................................................... 161
4.5.1 Environment set up ................................................................................ 162
4.5.3 Created exclusion lists within a Windows 7 application profile ............... 166
4.5.4 Limitations of behavioural filters ............................................................. 167
4.7 Contributions to knowledge and key findings ............................................ 168
Chapter 5 – Behaviour analysis environment experiments .................................... 171
9