Capturing and Analyzing Internet Worms - Computer Science @ UC

156 Pages·2007·1.81 MB·English

JedidiahRichardCrandall
June2007
ComputerScience
CapturingandAnalyzingInternetWorms
Abstract
Thisdocumentisaboutmalwareanalysis,withaparticularfocusonexploit-basedInternet
worms that spread from one host to another over the network by exploiting a software
vulnerability in the new host being attacked. Based on our experiences analyzing real
worms that use this method of worm propagation we develop a model that divides this
attack into three stages: the exploit vector ((cid:15)) where the machine being attacked is still
running its vulnerable code, the bogus control data (γ) that is the part of the attack that is
directly involved in control flow hijacking, and the payload (π) where the worm code is
beingexecutedinsteadofthe codeoftheattackedsystem.
The Epsilon-Gamma-Pi model will be defined more formally in Chapter 3. In
this document the particular focus will be on control data attacks, but the model general-
izes to hijacking of control flow at any level of abstraction. What we will show in this
dissertation is that malware analysis put into the context of the Epsilon-Gamma-Pi model
can take advantage of various limitations placed on the worm at each of the stages. Re-
searchers and malware analysis professionals can benefit greatly from an understanding
of the differences between the stages in terms of the adversarial model, the polymorphic
and metamorphic techniques to evade signature detection, and the amount of information
about the threat that can be discovered in a particular stage. Three specific examples are
describedindetail: Minos,anarchitecturalmechanismtocatchcontroldataattacksintheγ
stage;DACODA,atooltoanalyzeattackinvariantsthatlimitpolymorphisminthe(cid:15) stage;
and Temporal Search, a method to analyze the π stage anddiscovertimebomb attacks in a
worm’spayload.
CapturingandAnalyzingInternet Worms
By
JEDIDIAHRICHARDCRANDALL
B.S.(Embry-Riddle AeronauticalUniversity)2002
DISSERTATION
Submittedinpartial satisfactionofthe requirementsforthe degreeof
DOCTOROF PHILOSOPHY
in
Computer Science
inthe
OFFICEOF GRADUATESTUDIES
ofthe
UNIVERSITYOFCALIFORNIA
DAVIS
Approved:
Committeeincharge
2007
–i–
CapturingandAnalyzingInternet Worms
Copyright2007
by
JedidiahRichardCrandall
Tomydad–forcountlessconversations onthepatio
abouthistory, art,creativity, politics,whatsciencemeans,
andallofthethingsthatmakeascientistcomplete,
and,
tomymom–sincemynaturaltechnicalabilitiescouldn’t
havepossiblycomefrommydad.
–ii–
Contents
ListofFigures vi
ListofTables vii
1 Introduction 1
2 Minos(γ) 3
2.1 AboutMinos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.4.1 DiscussionofPolicyTradeoffs . . . . . . . . . . . . . . . . . . . . . . . . 12
2.5 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.6 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6.1 HardwareEmulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6.2 OperatingSystemChanges . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.6.3 VirtualMemorySwapping . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6.4 WindowsandBSDImplementations . . . . . . . . . . . . . . . . . . . . . 23
2.7 ExperimentalMethodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.7.1 FalsePositiveRate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.7.2 EffectivenessatStoppingAttacks . . . . . . . . . . . . . . . . . . . . . . 24
2.7.3 VirtualMemorySwappingOverhead . . . . . . . . . . . . . . . . . . . . 27
2.8 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.8.1 Falsepositives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.8.2 ExploitTests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.8.3 Virtualmemoryswappingoverhead . . . . . . . . . . . . . . . . . . . . . 34
2.9 SecurityAssessmentforMoreAdvancedAttacks . . . . . . . . . . . . . . . . . . 34
2.9.1 Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.9.2 BestPractices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.9.3 IntegrityTracking: AFundamentalTradeoff . . . . . . . . . . . . . . . . . 36
2.9.4 Non-ControlDataAttacks . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.10 TheHannibalExploit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.11 Follow-onResearchfromMinos . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.12 ConclusionsfromtheMinosWork . . . . . . . . . . . . . . . . . . . . . . . . . . 41
–iii–
3 ExperienceswithMinosHoneypots 42
3.1 AbouttheMinosHoneypots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.2 Exploits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.2.1 Control Flow is Usually Diverted Directly to the Attacker’s Executable
CodeviaaNOPSled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.2 NOPSledsareaNecessaryTechniqueforDealingwithUncertaintyAbout
theLocationofthePayloadCode . . . . . . . . . . . . . . . . . . . . . . 46
3.2.3 HackersHaveNotYetDemonstratedtheNeededTechniquestoWritePoly-
morphicWormCode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3 TheEpsilon-Gamma-PiModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.3.1 Epsilon((cid:15))=Exploit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.3.2 Gamma(γ)=BogusControlData . . . . . . . . . . . . . . . . . . . . . . 56
3.3.3 Pi(π)=Payload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3.4 OnRowSpacesandRanges . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.3.5 PolymorphismintheEpsilon-Gamma-PiModel. . . . . . . . . . . . . . . 58
3.4 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4 DACODA((cid:15)) 61
4.1 AboutDACODA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.2.1 TheNeedtoBeVulnerability-Specific . . . . . . . . . . . . . . . . . . . . 62
4.2.2 DACODA:TheDavisMalcodeAnalyzer . . . . . . . . . . . . . . . . . . 63
4.2.3 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.2.4 StructureoftheChapter . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.3 TheEpsilon-Gamma-PiModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.3.1 PolymorphismandMetamorphism . . . . . . . . . . . . . . . . . . . . . . 69
4.3.2 MotivationfortheModel . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.3.3 TheNeedforanOracle . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.4 HowDACODAWorks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.5 ExploitsAnalyzedbyDACODA . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.5.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.5.2 CodeRedIIasaConcreteExample . . . . . . . . . . . . . . . . . . . . . 77
4.5.3 ComplexitiesandChallenges . . . . . . . . . . . . . . . . . . . . . . . . . 78
4.6 Poly/Metamorphism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.6.1 WhatarePoly-andMetamorphism? . . . . . . . . . . . . . . . . . . . . . 85
4.6.2 Whatisavulnerability? . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.6.3 ThePD-Requires-ProvidesModel . . . . . . . . . . . . . . . . . . . . . . 88
4.7 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.8 ConclusionoftheChapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5 TemporalSearch(π) 92
5.1 AboutTemporalSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.2 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
5.2.1 ProposedApproachandContributionsofthisChapter . . . . . . . . . . . 94
5.2.2 StructureoftheChapter . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.3 Automated,Behavior-Based Analysis . . . . . . . . . . . . . . . . . . . . . . . . 96
5.4 TemporalSearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
5.4.1 HowTimeisMeasuredbyaSystem . . . . . . . . . . . . . . . . . . . . . 98
–iv–
5.4.2 SymbolicExecution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
5.4.3 TheBasicIdea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.4.4 LinuxExample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
5.4.5 TimePerturbationinWindows . . . . . . . . . . . . . . . . . . . . . . . . 106
5.4.6 ComparingHowTimersareUsed . . . . . . . . . . . . . . . . . . . . . . 106
5.4.7 WhyMustPerceivedTimebePerturbed? . . . . . . . . . . . . . . . . . . 108
5.5 DiscoveringPredicates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
5.5.1 Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
5.5.2 CodeRedv1(noCME[173]assigned) . . . . . . . . . . . . . . . . . . . 110
5.5.3 Blaster.E(noCMEassigned) . . . . . . . . . . . . . . . . . . . . . . . . . 110
5.5.4 Klez.A(noCMEassigned) . . . . . . . . . . . . . . . . . . . . . . . . . . 111
5.5.5 MyParty.A(noCMEassigned) . . . . . . . . . . . . . . . . . . . . . . . . 111
5.5.6 KamaSutra(CME-24)andSober.X(CME-681) . . . . . . . . . . . . . . 112
5.5.7 SummaryofResultsonDiscoveringPredicates . . . . . . . . . . . . . . . 113
5.6 RecoveringtheTimetable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.6.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5.6.2 DiscoveringTimetableEntries . . . . . . . . . . . . . . . . . . . . . . . . 116
5.6.3 AnIllustratingExample . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.6.4 CompletingtheTimetable . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.7 ChallengesforFutureWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.7.1 RegularMalware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.7.2 EvasiveMalware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.8 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.8.1 VirtualMachines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.8.2 TimePerturbation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.8.3 IntrusionDetection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.8.4 MalwareAnalysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
5.9 ConclusionsaboutTemporalSearch . . . . . . . . . . . . . . . . . . . . . . . . . 124
6 ConcludingRemarksandFutureWork 125
6.1 QuestionsforFutureResearch . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
6.1.1 Whatisaninstanceofmalware? . . . . . . . . . . . . . . . . . . . . . . . 125
6.1.2 Whatisthecontextofmalwareanalysis? . . . . . . . . . . . . . . . . . . 129
6.1.3 Howtokeephumansintheloop? . . . . . . . . . . . . . . . . . . . . . . 131
6.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
A TokensDiscoveredbyDACODAforSelectedExploits 133
Bibliography 134
–v–
List of Figures
2.1 DefinitionsoftrustfordifferentMinosimplementations. . . . . . . . . . . . . 5
2.2 Minos in an out-of-order execution microprocessor core. *Based on size and
compatibilitysettings. **Ignoredfor32-bitloadsandstores. . . . . . . . . . . 16
2.3 Thegccstresstest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.4 Linuxwebserveroveronemonth . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.5 gccVirtualMemorySwappingPerformance . . . . . . . . . . . . . . . . . . . 32
2.6 vprVirtualMemorySwappingPerformance . . . . . . . . . . . . . . . . . . . 32
2.7 mcfVirtualMemorySwappingPerformance . . . . . . . . . . . . . . . . . . . 33
2.8 bzip2VirtualMemorySwappingPerformance . . . . . . . . . . . . . . . . . . 33
3.1 AnOverly-SimpleModelofBufferOverflowExploits . . . . . . . . . . . . . . 43
3.2 TheEpsilon-Gamma-PiModelforControlDataExploits . . . . . . . . . . . . 54
4.1 TheEpsilon-Gamma-PiModel. . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.1 Howtimersareused. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
5.2 Grammarforpredicatesandexpressions,whereeip ∈ EIP,n ∈N. . . . . . . 115
5.3 Excerptfromctime()’ssourcecode. . . . . . . . . . . . . . . . . . . . . . . . 117
5.4 Annotated trace with weakest preconditions (shaded). The post-assertion is
shownonthelastline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
–vi–
List of Tables
2.1 TheexploitsthatweattackedMinoswith. . . . . . . . . . . . . . . . . . . . . . 28
2.2 TheexploitsthatothersactuallyattackedMinoswith. . . . . . . . . . . . . . . 28
3.1 ActualExploitsMinoshasStopped. . . . . . . . . . . . . . . . . . . . . . . . . 43
3.2 CharacteristicsoftheExploits . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3 RegisterSpringsPresentinPhysicalMemoryfortheDCOMexploit . . . . . . 44
3.4 CharacteristicsoftheProjections . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.1 HowQuadExpressionsareHandled. . . . . . . . . . . . . . . . . . . . . . . . . 71
4.2 SpecialRulesandExampleInstructions. . . . . . . . . . . . . . . . . . . . . . 72
4.3 ExploitsAnalyzedbyDACODA. . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.4 Whereexploitsarediscovered. . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4.5 SignatureTokens. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
–vii–
Acknowledgments
I would first like to thank my three advisors, who are also my committee members: Fred Chong,
for teaching me all of the things I didn’t know about being an effective researcher and member
of a research community–especially all of the things I didn’t know that I didn’t know; Felix Wu,
who imparted to me a sense of what’s really important in academia–namely the people you work
with;andZhendongSu–ifIcancarrywithmeevenhalfoftheenthusiasmthatProfessorSuhasfor
workingwithstudentsanddiscussingtheirideasIknowI’lldoverywellasajuniorfacultymember.
AlmosttheentirerestoftheComputerSciencedepartmentfaculty, andafewE.C.E.pro-
fessorsas well, helped me out along theway in one way or another (coming to practice talks, dis-
cussingresearchproblems,etc.). Justtonameafew: MattBishop,HaoChen,PremDevanbu,Matt
Farrens, Matt Franklin, Karl Levitt, Chip Martel, Norm Matloff, Ron Olsson, Phil Rogaway, and
VenkateshAkella. Helen Wangat MSRhelped me out in a lotof different ways, Mark Oskin, Di-
anaFranklin,andTimSherwoodneverhesitatedtosharetheirexperiencewiththeiracademiclittle
brother,andNickWeaverwasalwaysveryencouraging. I’vehadthepleasureofworkingalongside
some wonderful fellow students, including Daniela Alvim Seabra de Oliveira, Gary Wassermann,
everyone in PL reading group, and my labmates: Deen Copsey, Ravishankar Rao, John Oliver,
Paul Sultana, Darshan Thaker, Erik Czernikowski, Susmit Biswas and the others at UCSB, and
SetsoMetodi. TheworkinSection5.6ismore-soGaryWassermann’s workthanmyown,butwas
includedinChapter5forcompleteness.
I thank the department’s wonderful administrative staff, especially the graduate coordi-
nators Kim Reinking and Mary Reid. Also, none of the work I’ve done would have been possible
withoutopensourcesoftwaresoIthanktheopensourcedevelopers.
It has been very important to me to have my family (my parents, sisters, grandparents,
aunts, uncles, and nieces and nephews) with me during the past 22 years of education. My dad
especially has shared his graduate school experience, and also his father’s, with me–a gratuitous
quotefromDante’sInfernoisnotquiteaslunaticallyoutlandishasapictureoftwofliesprocreating
orapotterywheelmountedupsidedownonagarageceiling,buthopefullytheworkpresentedhere
hastheCrandallmarkonit.
–viii–