DEPARTMENT OF THE AIR FORCE AIR UNIVERSITY

104 Pages·2014·0.5 MB·English

by  Justin R. Ball

DETECTIONANDPREVENTIONOFANDROIDMALWAREATTEMPTINGTO
ROOTTHEDEVICE
THESIS
JustinR.Ball,Captain,USAF
AFIT-ENG-14-M-08
DEPARTMENT OF THE AIR FORCE
AIR UNIVERSITY
AIR FORCE INSTITUTE OF TECHNOLOGY
Wright-Patterson Air Force Base, Ohio
DISTRIBUTIONSTATEMENTA:
APPROVEDFORPUBLICRELEASE;DISTRIBUTIONUNLIMITED
The views expressed in this thesis are those of the author and do not reflect the official
policyorpositionoftheUnitedStatesAirForce,theDepartmentofDefense,ortheUnited
StatesGovernment.
This material is declared a work of the U.S. Government and is not subject to copyright
protectionintheUnitedStates.
AFIT-ENG-14-M-08
DETECTIONANDPREVENTIONOFANDROIDMALWAREATTEMPTINGTO
ROOTTHEDEVICE
THESIS
PresentedtotheFaculty
DepartmentofElectricalandComputerEngineering
GraduateSchoolofEngineeringandManagement
AirForceInstituteofTechnology
AirUniversity
AirEducationandTrainingCommand
inPartialFulfillmentoftheRequirementsforthe
DegreeofMasterofScienceinCyberspaceOperations
JustinR.Ball,B.S.C.S.
Captain,USAF
March2014
DISTRIBUTIONSTATEMENTA:
APPROVEDFORPUBLICRELEASE;DISTRIBUTIONUNLIMITED
AFIT-ENG-14-M-08
DETECTIONANDPREVENTIONOFANDROIDMALWAREATTEMPTINGTO
ROOTTHEDEVICE
JustinR.Ball,B.S.C.S.
Captain,USAF
Approved:
//signed// 14Feb2014
MajThomasE.Dube,PhD(Chairman) Date
//signed// 14Feb2014
BarryE.Mullins,PhD(Member) Date
//signed// 14Feb2014
COLGrantA.Jacoby,PhD(Member) Date
AFIT-ENG-14-M-08
Abstract
Every year, malefactors continue to target the Android operating system. Malware
which root the device pose the greatest threat to users. The attacker could steal stored
passwords and contact lists or gain remote control of the phone. Android users require a
systemtodetecttheoperationofmalwaretryingtorootthephone.
This research aims to detect the Exploid, RageAgainstTheCage, and Gingerbreak
exploits on Android operating systems. Reverse-engineering 21 malware samples lead
to the discovery of two critical paths in the Android Linux kernel, wherein attackers can
use malware to root the system. By placing sensors inside the critical paths, the research
detected all 379 malware samples trying the root the system. Moreover, the experiment
tested16,577benignapplicationsfromtheOfficialAndroidMarketandthirdpartyChinese
marketswhichtriggeredzerofalsepositiveresults.
Unlike static signature detection at the application level, this research provides
dynamic detection at the kernel level. The sensors reside in-line with the kernel’s
source code, monitoring network sockets and process creation. Additionally, the research
demonstrates the steps required to reverse engineer Android malware in order to discover
future critical paths. Using the kernel resources, the two sensors demonstrate efficient
asymptotictimeandspacereal-worldmonitoring. Furthermore,thesensorsareimmuneto
obfuscationtechniquessuchasrepackaging.
iv
TableofContents
Page
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
TableofContents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
ListofFigures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii
ListofTables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 ResearchContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Assumptions/Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Preview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
II. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 OffensiveTechniques(Rootkits) . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.1 Enhancing Stealthiness And Efficiency of Android Trojans and
DefensePossibilities(EnSEAD) . . . . . . . . . . . . . . . . . . . 4
2.2.2 AndroidplatformBasedLinuxKernelRootkit . . . . . . . . . . . 5
2.3 DefensiveTechniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3.1 StaticAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3.1.1 Extending Android Security Enforcement with a Secu-
rityDistanceModel . . . . . . . . . . . . . . . . . . . . 7
2.3.1.2 TrustDroid(TM) . . . . . . . . . . . . . . . . . . . . . . 8
2.3.1.3 SemanticallyRichApplication-centricSecurityinAndroid 9
2.3.2 DynamicAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.3.2.1 YAASE:YetAnotherAndroidSecurityExtension . . . . 11
2.3.2.2 SecuritycontrolsforAndroid . . . . . . . . . . . . . . . 12
2.3.2.3 Kernel-based Behavior Analysis for Android Malware
Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2.4 DetectingcovertcommunicationonAndroid . . . . . . . 14
2.3.2.5 ACloud-BasedIntrusionDetectionSystemForAndroid
Smartphones . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.2.6 A Cloud-Based Intrusion Detection And Response Sys-
temForMobilePhones . . . . . . . . . . . . . . . . . . 16
v
Page
2.3.2.7 Android Malware Detection via a Latent Network Be-
haviorAnalysis . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Surveys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.4.1 GoogleAndroid: AComprehensiveSecurityAssessment . . . . . . 17
2.4.2 DissectingAndroidMalware: CharacterizationandEvolution . . . 18
2.4.3 Mysmartphoneisasafe! Theuser’spointofviewregardingnovel
authenticationmethodsandgradualsecuritylevelsonsmartphones 20
2.4.4 SmartphoneSecurityChallenges . . . . . . . . . . . . . . . . . . . 22
2.4.5 Androidbotnetsontherise: Trendsandcharacteristics . . . . . . . 23
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
III. Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3.1 DevelopKernelSensorsInsideCriticalPaths . . . . . . . . . . . . 26
3.3.2 AddSensorsInsideKernelSourceCode . . . . . . . . . . . . . . . 27
3.3.3 AndroidOperatingSystem . . . . . . . . . . . . . . . . . . . . . . 28
3.4 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.5 PerformanceMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.6 Boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.7 Workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
IV. Reverse-EngineeringAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.2 ResearchContribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
4.3 ReverseEngineeringMalware . . . . . . . . . . . . . . . . . . . . . . . . 34
4.3.1 AndroidMalwareGenomeProject . . . . . . . . . . . . . . . . . . 35
4.3.2 ReversingApproach . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.4 GingermasterAndroidMalwareFamily . . . . . . . . . . . . . . . . . . 35
4.4.1 BehavioralAnalysisApproachandFindings . . . . . . . . . . . . 36
4.4.2 StaticAnalysisApproachandFindings . . . . . . . . . . . . . . . 37
4.5 zHashAndroidMalwareFamily . . . . . . . . . . . . . . . . . . . . . . . 40
4.6 DroidKungFu3AndroidMalwareFamily . . . . . . . . . . . . . . . . . . 43
4.7 SummaryofFindings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.8 MalwareDetectionSet-Up . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.8.1 CreateAndroidEmulatorUsingAndroidVirtualDeviceManager . 48
4.8.2 InstallingtheSensors . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.8.3 SpeedoftheSensors . . . . . . . . . . . . . . . . . . . . . . . . . 49
vi
Page
4.8.4 TestingProcedures . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
V. DetectionAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.2 Gingerbreak . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.3 zHash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
5.4 DroidKungFu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.5 FalsePositives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.6 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.7 AdvantageandDisadvantageoftheProposedDetectionAlgorithm . . . . . 60
5.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
VI.Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6.2 SummaryofResearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6.3 RecommendationsforFutureResearch . . . . . . . . . . . . . . . . . . . . 63
6.4 ResearchContributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
Appendix: ToolsNeeded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Appendix: ScriptsUsed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Appendix: Sensorcodeinsidethekernel . . . . . . . . . . . . . . . . . . . . . . . . 75
Appendix: Originalmalwaresignaturepaths . . . . . . . . . . . . . . . . . . . . . . 80
Appendix: SequenceDiagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
vii
ListofFigures
Figure Page
2.1 PolicyTreeforSaint[33] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.1 DifferentLayersoftheAndroidOperatingSystem[2] . . . . . . . . . . . . . . 27
3.2 TestingFlowchart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.1 Evolutionof DroidKungFuMalware[22] . . . . . . . . . . . . . . . . . . . . 43
4.2 Summaryofthethreemalwarefamilies . . . . . . . . . . . . . . . . . . . . . 46
5.1 Gingerbreakbinarydetection . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.2 Gingermasterdetectionresultsagainstthedifferentpayloads . . . . . . . . . 54
5.3 2011Gingermasterdetectionresultsfromfourantivirussoftware[20] . . . . 54
5.4 zHashbinarydetection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
5.5 2011zHashDetectionResultsFromFourAntivirusSoftware[20] . . . . . . . 55
5.6 zHashdetectionresultsagainstthedifferentpayloads . . . . . . . . . . . . . . 56
5.7 DroidKungFubinarydetection . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.8 Sourcecodechangeinside/system/core/adb.c:adb main() . . . . . . . . 57
5.9 DroidKungFudetectionresultsagainstthedifferentpayloads . . . . . . . . . . 58
5.10 2011DroidKungFudetectionresultsfromfourantivirussoftware[20] . . . . . 59
E.1 SequenceDiagramof Exploidexploit . . . . . . . . . . . . . . . . . . . . . . 86
E.2 SequenceDiagramof Gingerbreakexploit . . . . . . . . . . . . . . . . . . . 87
E.3 SequenceDiagramof RATCexploit . . . . . . . . . . . . . . . . . . . . . . . . 88
viii
ListofTables
Table Page
2.1 SummaryofResearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.1 Accuracyoftheexperiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
ix