Intelligent Mobile Malware Detection

191 Pages·2022·5.847 MB·English

by  Tony Thomas, Teenu John, Mamoun Alazab, Roopak Surendran

Intelligent Mobile
Malware Detection
The popularity of Android mobile phones has caused more cybercriminals to
create malware applications that carry out various malicious activities. The
attacks, which escalated after the COVID-19 pandemic, proved there is great
importance in protecting Android mobile devices from malware attacks.
Intelligent Mobile Malware Detection will teach users how to develop
intelligent Android malware detection mechanisms by using various graphs
and stochastic models. The book begins with an introduction to the Android
operating system accompanied by the limitations of the state-of-the-art static
malware detection mechanisms as well as a detailed presentation of a hybrid
malware detection mechanism. The text then presents four different system call-
based dynamic Android malware detection mechanisms using graph centrality
measures, graph signal processing and graph convolutional networks. Further,
the text shows how most of Android malware can be detected by checking the
presence of a unique subsequence of system calls in its system call sequence.
All the malware detection mechanisms presented in the book are based on
the authors’ recent research. The experiments are conducted with the latest
Android malware samples, and the malware samples are collected from public
repositories. The source codes are also provided for easy implementation of the
mechanisms. This book will be highly useful to Android malware researchers,
developers, students and cyber security professionals to explore and build
defense mechanisms against the ever-evolving Android malware.
Security, Privacy, and Trust
in Mobile Communications
Series Editor: Brij B. Gupta
This series will present emerging aspects of the mobile communication
landscape, and focuses on the security, privacy, and trust issues in mobile
communication-based applications. It brings state-of-the-art subject
matter for dealing with the issues associated with mobile and wireless
networks. This series is targeted for researchers, students, academicians,
and business professions in the field.
Nanoelectric Devices for Hardware and Software Security
Balwinder Raj & Arun Kumar Singh
Cross-Site Scripting Attacks: Classification, Attack, and
Countermeasures
B. B. Gupta & Pooja Chaudhary
Smart Card Security: Applications, Attacks, and Countermeasures
B. B. Gupta & Megha Quamara
Computer and Cyber Security: Principles, Algorithm,
Applications, and Perspectives
Brij B. Gupta
Intelligent Mobile Malware Detection
Tony Thomas, Roopak Surendran, Teenu S. John & Mamoun Alazab
Intelligent Mobile
Malware Detection
Tony Thomas, Roopak Surendran
and Teenu S. John
Kerala University of Digital Sciences, Innovation and Technology
(Digital University of Kerala), Kerala, India
Mamoun Alazab
College of Engineering, IT and Environment at
Charles Darwin University, Australia
First edition published 2023
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742
and by CRC Press
4 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN
CRC Press is an imprint of Taylor & Francis Group, LLC
© 2023 Tony Thomas, Roopak Surendran, Teenu S. John, and Mamoun Alazab
Reasonable efforts have been made to publish reliable data and information, but the author and
publisher cannot assume responsibility for the validity of all materials or the consequences of their use.
The authors and publishers have attempted to trace the copyright holders of all material reproduced
in this publication and apologize to copyright holders if permission to publish in this form has not
been obtained. If any copyright material has not been acknowledged please write and let us know so
we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information
storage or retrieval system, without written permission from the publishers.
For permission to photocopy or use material electronically from this work, access www.copyright.com
or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. For works that are not available on CCC please contact mpkbookspermissions@tandf.
co.uk
Trademark notice:Product or corporate names may be trademarks or registered trademarks and are
used only for identification and explanation without intent to infringe.
ISBN: 978-0-367-63871-9 (hbk)
ISBN: 978-1-032-42109-4 (pbk)
ISBN: 978-1-003-12151-0 (ebk)
DOI: 10.1201/9781003121510
Typeset in Erewhon font
by KnowledgeWorks Global Ltd.
Contents
Preface ix
Acknowledgements xi
AbouttheAuthors xiii
Symbols xv
1 InternetandAndroidOS 1
1.1 AndroidOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 Linuxkernel . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Nativelibraries . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.3 Androidruntime . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.4 Applicationframework . . . . . . . . . . . . . . . . . . . . 3
1.1.5 Applicationlayer . . . . . . . . . . . . . . . . . . . . . . . 4
1.2 AndroidApplicationDevelopment . . . . . . . . . . . . . . . . . 4
1.3 GooglePlaystore . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 IntentsandIntentFilters . . . . . . . . . . . . . . . . . . . . . . 6
1.5 AndroidSecurity . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5.1 Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5.2 Applicationsandbox . . . . . . . . . . . . . . . . . . . . . 7
1.5.3 Applicationsignature . . . . . . . . . . . . . . . . . . . . . 7
1.5.4 Dataencryption . . . . . . . . . . . . . . . . . . . . . . . . 7
1.6 InternetofThings . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6.1 ArchitectureofIoT . . . . . . . . . . . . . . . . . . . . . . 8
1.6.1.1 Sensorlayer . . . . . . . . . . . . . . . . . . . . 9
1.6.1.2 Gatewaysandnetworks . . . . . . . . . . . . . . 9
1.6.1.3 Managementservicelayer . . . . . . . . . . . . . 9
1.6.1.4 Applicationlayer . . . . . . . . . . . . . . . . . 9
1.7 AndroidThings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.8 IoTSecurity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.8.1 MalwareThreatsinIoT . . . . . . . . . . . . . . . . . . . . 11
1.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
v
vi Contents
2 AndroidMalware 13
2.1 PCMalwarevs.AndroidMalware . . . . . . . . . . . . . . . . . . 13
2.2 TrendsinMalware . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 TrendsinWindowsmalware . . . . . . . . . . . . . . . . . 15
2.2.2 TrendsinAndroidmalware . . . . . . . . . . . . . . . . . 15
2.3 TypesofMalwareDetectionMechanisms . . . . . . . . . . . . . . 16
2.4 MalwareTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.5 MalwareAttacksinAndroid . . . . . . . . . . . . . . . . . . . . . 19
2.5.1 Drivebydownloadattack . . . . . . . . . . . . . . . . . . 19
2.5.2 Updateattack . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.5.3 Repackingattack . . . . . . . . . . . . . . . . . . . . . . . 19
2.6 HistoryofMalwareAttacksinAndroid . . . . . . . . . . . . . . . 19
2.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3 StaticMalwareDetection 23
3.1 ReverseEngineeringandStaticAnalysis . . . . . . . . . . . . . . . 23
3.1.1 ReverseengineeringusingApktoolandDex2jar. . . . . . . 23
3.1.2 Staticmalwareanalysistools . . . . . . . . . . . . . . . . . 24
3.2 ComponentsofAndroidApplication . . . . . . . . . . . . . . . . . 25
3.3 APICallAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3.1 API’susedbymalwareapplications . . . . . . . . . . . . . 27
3.4 APICall-BasedStaticDetection . . . . . . . . . . . . . . . . . . . 29
3.4.1 MechanismsusingtheindependentoccurrenceofAPI . . . 29
3.4.2 MechanismsUsingAPICallGraphs . . . . . . . . . . . . . 30
3.5 PermissionandIntent-BasedStaticDetection . . . . . . . . . . . . 31
3.5.1 Permissionanalysis . . . . . . . . . . . . . . . . . . . . . . 31
3.5.1.1 Permissionsusedbythemalwareapplications . . 32
3.5.1.2 Component-basedpermissionescalationattack . . 34
3.5.2 Intent-basedanalysis . . . . . . . . . . . . . . . . . . . . . 36
3.5.2.1 Intentsusedformalwareattacks. . . . . . . . . . 36
3.5.2.2 Intent-basedvulnerabilities . . . . . . . . . . . . 38
3.5.3 Malwaredetectionusingpermissionsandintents . . . . . . 39
3.6 Opcode-BasedStaticDetection . . . . . . . . . . . . . . . . . . . 40
3.6.1 Malwaredetectionusingopcodes . . . . . . . . . . . . . . 40
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4 DynamicandHybridMalwareDetection 43
4.1 Emulator-BasedDynamicAnalysis . . . . . . . . . . . . . . . . . 43
4.2 DynamicMalwareDetectionMechanisms . . . . . . . . . . . . . . 44
4.2.1 Systemmetricandtrafficanalysis(Category1) . . . . . . . 44
4.2.2 Networkpacketanalysis(Category2) . . . . . . . . . . . . 46
4.2.3 SensitiveAPIcallanalysis(Category3) . . . . . . . . . . . 46
4.2.4 Systemcallanalysis(Category4) . . . . . . . . . . . . . . 47
4.2.4.1 SystemcallfrequencyorTF-IDF-basedmethods . 47
Contents vii
4.2.4.2 Systemcalldependencygraphormarkov
chain-basedmethods . . . . . . . . . . . . . . . 48
4.2.4.3 Systemcallphylogeny-basedmethods . . . . . . 49
4.2.4.4 System call behavior or sequence analysis-based
methods . . . . . . . . . . . . . . . . . . . . . . 49
4.3 HybridAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.3.1 Hybriddetectionbasedonasingleclassifier(Category1) . 50
4.3.2 Hybriddetectionbasedonensembleclassifiers
(Category2) . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.4 CorrelationAmongStaticandDynamicFeatures . . . . . . . . . . 52
4.4.1 TreeaugmentedNaiveBayes(TAN)model . . . . . . . . . 52
4.5 HybridAnalysiswithTANClassifier . . . . . . . . . . . . . . . . 53
4.5.1 DependenciesamongAPIcalls,permissionandsystemcalls 54
4.5.2 Ridgeregularizedlogisticregression(RRLR) . . . . . . . . 54
4.5.3 Probabilityestimation . . . . . . . . . . . . . . . . . . . . 56
4.5.4 Anomalydetection . . . . . . . . . . . . . . . . . . . . . . 56
4.5.4.1 Apppermissionanalysis . . . . . . . . . . . . . . 57
4.5.4.2 StaticAPIfunctioncallanalysis . . . . . . . . . . 57
4.5.4.3 Systemcallanalysis . . . . . . . . . . . . . . . . 58
4.5.5 MalwaredetectionusingTAN-basedmodel . . . . . . . . . 58
4.6 ExperimentsandAnalysis . . . . . . . . . . . . . . . . . . . . . . 59
4.6.1 Trainingphase . . . . . . . . . . . . . . . . . . . . . . . . 60
4.6.1.1 EstimationofthresholdforL ,L ,L . . . . . . . 62
1 2 3
4.6.1.2 Conditionalprobabilityestimation . . . . . . . . 62
4.6.2 Evaluationphase . . . . . . . . . . . . . . . . . . . . . . . 64
4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5 DetectionUsingGraphCentralityMeasures 69
5.1 DigraphfromSystemCallSequence . . . . . . . . . . . . . . . . . 70
5.2 CentralityMeasuresfromSystemCallDigraph . . . . . . . . . . . 71
5.3 MalwareDetectionPhase . . . . . . . . . . . . . . . . . . . . . . 74
5.4 ExperimentsandAnalysis . . . . . . . . . . . . . . . . . . . . . . 75
5.4.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
5.4.2 Performanceresults . . . . . . . . . . . . . . . . . . . . . . 76
5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6 GraphConvolutionalNetworkforDetection 79
6.1 IntroductiontoGCN . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.2 GCN-BasedMalwareDetection . . . . . . . . . . . . . . . . . . . 80
6.2.1 Systemcallgraphconstruction . . . . . . . . . . . . . . . . 80
6.2.2 GCNforlowdimensionalfeaturerepresentation. . . . . . . 83
6.2.3 TrainingofGCN . . . . . . . . . . . . . . . . . . . . . . . 84
6.2.4 SystemcallgraphclassificationusingGCN . . . . . . . . . 85
6.3 ExperimentsandAnalysis . . . . . . . . . . . . . . . . . . . . . . 85
6.3.1 Implementationdetails . . . . . . . . . . . . . . . . . . . . 86
viii Contents
6.4 DetectionofEmergingMalware . . . . . . . . . . . . . . . . . . . 88
6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7 GraphSignalProcessing-BasedDetection 91
7.1 GraphSignalProcessingandItsApplications . . . . . . . . . . . . 91
7.2 GraphSignalsfromSystemCallSequence . . . . . . . . . . . . . 92
7.3 MachineLearningClassificationforMalwareDetection . . . . . . 95
7.3.1 Constructionoflow-dimensionalfeaturevectors . . . . . . 96
7.4 ExperimentsandAnalysis . . . . . . . . . . . . . . . . . . . . . . 97
7.4.1 Experimentalsetup . . . . . . . . . . . . . . . . . . . . . . 97
7.4.2 PerformanceanalysiswithvariousMLclassifiers . . . . . . 99
7.5 MiscellaneousOperationsonGraphSignals . . . . . . . . . . . . . 100
7.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
8 SystemCallPattern-BasedDetection 103
8.1 ExtractionofPatternsFromSystemCallSequences . . . . . . . . . 103
8.1.1 RepresentingsystemcallsequenceasergodicMarkovchain 104
8.1.2 Computationofinformationinsystemcallsequence . . . . 104
8.1.3 Identificationofsystemcallpatterns . . . . . . . . . . . . . 105
8.2 SystemcallpatternsinWalkinwattrojan . . . . . . . . . . . . . . . 106
8.3 MalwareDetectionandClassificationBasedonSystemCall
Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
8.4 ExperimentsandAnalysis . . . . . . . . . . . . . . . . . . . . . . 110
8.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
9 ConclusionsandFutureDirections 115
9.1 RecentMalwareAttacks . . . . . . . . . . . . . . . . . . . . . . . 115
9.2 IdentifyingExploitationAttacks . . . . . . . . . . . . . . . . . . . 116
9.3 MitigatingEmulatorEvasionandCodeCoverageProblem . . . . . 117
9.4 ResiliencetotheChangeinSystemCallSequence . . . . . . . . . 118
9.5 CollusionAttack . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Appendix 119
Bibliography 153
Index 173
Preface
Nowadays, smart phones are widely used for making phone calls, sending mes-
sages, storing personal data, browsing the Internet, online banking and more.
Becauseofthis,smartphoneshavebecometargetsforcyber-attacksinvolvingmal-
ware.Cybercriminalsaretargetingsmartphonestospreadmalwareinordertosteal
money and confidential data stored in those phones. Malware applications such as
trojanSMSandtrojanbankercancausegreatfinanciallosstousers.TrojanSMScan
sendSMSmessagestopremiumratenumbersinthebackgroundandtrojanbanker
canstealtheonlinebankingdetailsofauserwithouttheuser’sknowledge.Therefore,
ithasbecomeveryessentialtosecuresmartphonesagainstmalwareattacks.
WiththewidespreadusageoftheAndroidoperatingsystem,thenumberofmal-
waretargetingAndroidsmartphoneshasrisenseveralfolds.Almost98%ofsmart-
phonemalwarearedesignedforAndroiddevices.Mostoftheexistinganti-malware
products are still relying on static and signature-based malware detection mecha-
nisms.Staticanalysisisamethodofdetectingmalwareapplicationbyanalyzingthe
sourcecodeoftheapplicationwithoutexecutingit.Insignature-basedanalysis,the
hashvalueofanapplicationiscomparedwithalistofhashvaluesofknownmalicious
applicationsforidentifyingwhethertheapplicationisoneamongthelistedmalware.
These detection mechanisms can be easily evaded by code transformation attacks.
Hence,itisessentialtodevelopnovelmalwaredetectionmechanismsbasedondy-
namicanalysisforaccuratemalwaredetection.Dynamicanalysismechanismscon-
siderruntimeinformationsuchassystemmetrics,networklevelinformation,system
callsandmorefordetectingthemaliciousbehavioroftheapplication.Amalicious
applicationtypicallyinvokessensitiveAPIsinanautomatedmannertoperformpriv-
ilegedoperations.ThisautomatedinvocationofAPIcallsgetsreflectedinthesystem
callsequenceoftheapplication.Hence,systemcallsareconsideredasoneofmost
effectivefeaturesforcapturingthemaliciousbehaviorofanapplication.
Most of the existing system call-based malware detection mechanisms consider
the system call frequencies or co-occurrences in the system call sequence for mal-
waredetection.Thesystemcallfrequency-basedmechanismsusemachinelearning
classifierstodetectmalwarebasedontheindependentoccurrencesofeachindividual
system call in the entire sequence. These mechanisms do not consider the relation-
shipsamongthesystemcallsinasystemcallsequence.Insystemcallco-occurrence-
based mechanisms, the mutual relationships between system calls in the sequence
areconsideredformalwaredetection.However,theseapproachesdonotconsiderthe
complex relationships among the system calls crucial for identifying the malicious
behaviorofanapplication.
ix