Improving End-to-End Availability Using Overlay Networks David Godbe Andersen

150 Pages·2012·2.31 MB·English

Improving End-to-End Availability Using Overlay Networks
by
David Godbe Andersen
S.M.ComputerScience,MassachusettsInstituteofTechnology(2001)
B.S.,ComputerScience,UniversityofUtah(1998)
B.S., Biology,UniversityofUtah(1998)
Submittedtothe DepartmentofElectricalEngineeringandComputerScience
inpartial fulfillmentofthe requirementsforthedegreeof
DoctorofPhilosophyinComputerScienceandEngineering
atthe
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
February2005
c MassachusettsInstituteofTechnology2004. Allrightsreserved.
(cid:13)
Author...........................................................................
Department ofElectricalEngineeringandComputerScience
December22, 2004
Certified by.......................................................................
HariBalakrishnan
AssociateProfessorofComputerScienceandEngineering
ThesisSupervisor
Acceptedby......................................................................
Arthur C.Smith
Chairman,DepartmentCommitteeon GraduateStudents
2
Improving End-to-EndAvailabilityUsing OverlayNetworks
by
DavidGodbeAndersen
SubmittedtotheDepartmentofElectricalEngineeringandComputerScience
onDecember22,2004,inpartialfulfillment ofthe
requirementsforthedegreeof
DoctorofPhilosophyinComputerScienceandEngineering
Abstract
Theend-to-endavailabilityofInternetservicesisbetweentwoandthreeordersofmagnitudeworse
than other important engineered systems, including the US airline system, the 911 emergency re-
sponse system, and the US public telephone system. This dissertation explores three systems de-
signed to mask Internet failures, and, through a study of three years of data collected on a 31-site
testbed,whythesefailureshappenandhoweffectivelytheycanbemasked.
A core aspect of many of the failures that interrupt end-to-end communication is that they fall
outside the expected domain of well-behaved network failures. Many traditional techniques cope
with link and router failures; as a result, the remaining failures are those caused by software and
hardware bugs, misconfiguration, malice, or the inability of current routing systems to cope with
persistentcongestion. TheeffectsofthesefailuresareexacerbatedbecauseInternetservicesdepend
upon the proper functioning of many components—wide-area routing, access links, the domain
namesystem, andtheservers themselves—and afailurein anyofthemcanprovedisastroustothe
properfunctioningoftheservice.
ThisdissertationdescribesthreecomplementarysystemstoincreaseInternetavailabilityinthe
face of such failures. Each system builds upon the idea of an overlay network, a network created
dynamicallybetweenagroupofcooperatingInternethosts. Thefirst twosystems,ResilientOverlay
Networks (RON) and Multi-homed Overlay Networks (MONET) determine whether the Internet
path between two hosts is working on an end-to-end basis. Both systems exploit the considerable
redundancy available in the underlying Internet to find failure-disjoint paths between nodes, and
forwardtraffic alongaworkingpath. RONisabletoavoid50%oftheInternetoutagesthatinterrupt
communication between a small group of communicating nodes. MONET is more aggressive,
combining an overlay network of Web proxies with explicitly engineered redundant links to the
Internetto also mask clientaccess link failures. Eighteen months of measurementsfroma six-site
deployment of MONET show that it increases a client’s ability to access working Web sites by
nearlyanorderofmagnitude.
WhereRONandMONETcombataccidentalfailures,theMaydaysystemguardsagainstdenial-
of-serviceattacksbysurroundingavulnerableInternetserver witharingoffiltering routers. May-
daythenusesasetofoverlaynodestoactasmediatorsbetweentheserviceanditsclients,permitting
onlyproperlyauthenticatedtraffic toreachtheserver.
ThesisSupervisor: HariBalakrishnan
Title: AssociateProfessorofComputerScienceandEngineering
3
4
Tomyparents,MaryLouGodbeandJerryRichardAndersen,
andtomygrandfatherHamptonClawsonGodbe,
whoalwaysencouraged metodiscoverthings,
instillinginmethecuriositytobecomeascientist,
andtheimpatiencetobecomeacomputerscientist.
5
6
Acknowledgments
I am deeply indebted to my advisor, Hari Balakrishnan, for making five years of graduate school
one of the best periods of my life. For each effort I put into my research and this dissertation, I
think Hari put two. I arrived at MIT quite without a clue, and rather worried that the admissions
committeewouldsoonrealizetheirdrasticerrorininvitingme. Fortunately, itwasdifficult tostray
toofarafield whenstrivingtofollowHari’sexampleandexemplaryadvice. Hewasabetteradvisor
thanIimaginedpossible.
Frans Kaashoek let me steal an enormous amount of his time and wisdom as I walked a line
betweennetworksandsystems. Frans’svigorousapproachbothtoresearchandtolifehelpedshow
methefunthatcanbehadinacademia. Ibenefited enormouslyfromthetimeIspentvisitingPDOS
groupmeetingsandgettinganinfusionofhard-coresystems.
In addition to so generously serving on my committee, and on my typically short notice, John
Guttagtaughtmemanyoftherightquestionstoaskaboutresearch. Iknowofnobodybettertoturn
towhenwondering,“What’sthereallyimportantthingaboutwhatI’mdoinghere?”
IhavetothankRobertMorrisfortwothings: First,asoneofthepeoplebehindtheRONproject,
forhisearlyguidanceandfrequentadvice;andsecond,forsharinghispassionforallthingstiming-
related and letting me frequently barge in and borrow a rubidium oscillator or frequency counter
whenIwasavoidingmyrealwork.
Alex Snoeren, my officemate for my first three years at MIT, showed me how to swim, both
in research and socially, in a very different pool than the one from which I came. He was a great
mentor, office-mate, and a damn smart blackboard off which to bounce my crazy idea or dumb
question of the week. He’s a great friend to have. Thank you, Alex. On the topic of swimming, I
oweChristineAlvaradoforsimilarhelp,bothasanoccasionaltrainingpartnerandteacher, andfor
successfully juggling her own Ph.D. research while encouraging those around her to occasionally
fleetheconfines oftheircubiclestomeetotherpeople.
Alex,AllenMiu,StanRost,andJonSalzmadeouroffice inroom512anexcellent—andloud—
placetowork. Stan,thankyouforthemusic,andalltherest. Allen,thankyouformakingmenotice
thebadmintoncourts. I’llmissourdailyinteraction.
Nick Feamstertolerated me as both a friend, an officemate and an apartment-mate, and did an
excellentjob at all of the above. I may have gotten moreresearch donewhile runningand cycling
with Nick than I did in the office. Where my research touched BGP, Nick was right there. We
builtacrazymeasurementinfrastructure,andhopefully, we’ll keeponbuildingit. Onanunrelated
note, I dedicatethe eradicationof the word“methodology”in this dissertationbothto Nick andto
Michael Walfish. If I ever manage to write half as well as Walfish, I’ll count myself happy. My
future sushi parties, however, must be dedicated to Jaeyeon Jung. Mythili Vutukuru and Sachin
Kattialsodisplayedimpressivenoise-toleranceduringourshorttimeofsharinganoffice.
I spent my time at MIT in the Networks and Mobile Systems group, whose members’ diverse
researchinterests always let melearn somethingnew, whetherabout wirelessnetworks, streaming
databases, BGP routing, or whatnot. Brett Hull, Kyle Jamieson, Bodhi Priyantha, Eugene Shih,
andGodfreyTanalldeserveblamefordoingcoolthingsandbeingwillingtoteachmeaboutthem.
RohitRaohelpedwritethesecondandmuchimproved versionsoftheMONETproxyandparallel
DNS server, thereby greatly improving my chances of graduating. Magda and Michel, I raise a
Costco membership card in your honor. Dina Katabi provided much good feedback about nearly
everythingIwonderedabout.
EddieKohlerandDavidMazie`resshowedmethetruePDOSspirit,carriedonbyJohnJannotti,
BryanFord,KevinFu,MichaelKaminsky,AthichaMuthitacharoen,EmilSit,BenjieChen,Jinyang
LiMax Krohn, andJacob Strauss. Sameerand Mandi, thankyou for many a Christmasparty, and
7
many a good conversation. Rodrigo and Chuck, don’t stop debunking peer-to-peer. Thank you to
ChrisLaasforneverlettingmeBS,andtoDan,John,FrankandStriblingforHalo. DougDecouto,
JohnBicketandSanjitBiswas,DanielAguayo,andBenChambers’sRoofnetshowedmethatgood
workcan(andshould)beinspirationalandextremelycool. Doug,goodwindforyoursails. Thomer
GilandAlexYipprovidedinspirationinresearchand cycling.
DorothyCurtisandMichelGoraczkoputupwithmyconstantrequestsformorediskspace,my
occasionallybreakingthenetwork,andthemanyoutrageousrequestsImadeatridiculoushours. I
placed a similar burden on the LCS infrastructure group; I give particular thanks to Garrett Woll-
mannandMaryAnnLadd,whocarvedthroughtheadministrativebureaucracytoletmeinstallDSL
lines,collectroutingdata,andaccessdedicatedInternetconnections. JayLepreau,MikeHibler,and
LeighStollerandtherestoftheEmulabcrewprovidedinvaluableassistancewithmanagingandrun-
ningtheRONtestbed. ThanksalsotothemanyanonymoususersatMITwhousedandabusedthe
proxytoprovideuswithdatatoanalyzeMONET.
MIT has an amazing collection from whom to learn, and a group of professors and research
scientists who shared generously with their much-demanded time. David Karger did his best to
teachmesometheory,evenifthatwasliketryingtoteachapigtofly. DavidGiffordintroducedme
tothejoysofmachinelearningthroughcomputationalbiology. JohnWroclawskiandKarenSollins
somehowalwaysmadetimetotalktomeaboutthingsnetworkarchitectureandotherwise.
MyresearchwassupportedbyaMicrosoftResearchfellowship,DARPA,theNationalScience
Foundation,andMIT.Manythankstoallformakingthisresearchpossible.
Mymother,MaryLouGodbe,bravelyandpatientlyproof-readmanyofthepapersIwrote,and
proof-read a draft version of this dissertation, steeping herself deeper in computer science jargon
thanI’msuresheeverimagined.
While I’m deliberately leaving out many good friends and family from this list (the list of
academic people is too long already—and I fear it’s drastically incomplete!), I do owe a heart-
felt thank you to Naomi Kenner for supporting me during my job search and related trauma that
temporarilyturnedmeintoastressed-outbasketcase. Thankyou,Ny. Yourock. Manythankstomy
friendsfromtheouting,cycling,triathlon,andmastersswimmingclubs. Youkeptmecomparatively
sane for fiveyears. Rob Jagnow was an excellent friend and adventure partner, who also provided
a sterling example of how many responsibilities one can successfully juggle if one really tries.
Between Rob, Hector Briceno, Luke Sosnowski, and Robert Zeithammer, I was never lacking for
friendsorinspirationtodocrazythingsinvolvingthemountains.
My friends and family, named and unnamed, form the richest tapestry in which anyone could
hopetolive. Youknowwhoyouare,andpleaseknowhowmuchIappreciateyouall.
8
Contents
1 Introduction 15
1.1 End-to-EndServiceAvailability . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.2 ChallengestoAvailability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.3 CopingWithFailuresandDegradation . . . . . . . . . . . . . . . . . . . . . . . . 23
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2 RelatedWork 29
2.1 AvailabilityandPerformanceofInternetSystems . . . . . . . . . . . . . . . . . . 29
2.2 InternetInfrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.3 ImprovingEnd-to-EndAvailability . . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.4 DenialofService . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.5 OverlayNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3 Method 39
3.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2 MeasurementandValidation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.3 FailuresandPerformanceMetrics . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4 TheRONTestbed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.5 TestbedOverview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.6 CloselyRelatedTestbeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4 ResilientOverlayNetworks 51
4.1 DesignGoals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.2 ExploratoryStudyResults . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3 SoftwareArchitecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4 RoutingandPathSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5 PolicyRouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.6 DataForwarding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.7 BootstrapandMembershipManagement . . . . . . . . . . . . . . . . . . . . . . . 63
4.8 ImplementationandtheRONIPTunnel . . . . . . . . . . . . . . . . . . . . . . . 63
4.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5 RONEvaluation 71
5.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.2 OvercomingPathOutages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.3 OvercomingPerformanceFailures . . . . . . . . . . . . . . . . . . . . . . . . . . 79
5.4 RONRoutingBehavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
9
5.5 SecurityDiscussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
6 Multi-homedOverlayNetworks 87
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.2 SystemArchitecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.3 ObtainingMultiplePaths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
6.4 WaypointSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
6.5 ReducingServerOverhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
6.6 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
6.7 ExplicitMulti-homing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.8 ICP+withConnectionSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
6.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7 MONETEvaluation 103
7.1 MONETTestbedandDataCollection . . . . . . . . . . . . . . . . . . . . . . . . 103
7.2 CharacterizingFailures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.3 HowWelldoesMONETWork? . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.4 ServerFailuresandMulti-homedServers . . . . . . . . . . . . . . . . . . . . . . 114
7.5 DiscussionandLimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
7.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
8 PreventingDenial-of-Service AttacksusingOverlayNetworks 119
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
8.2 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
8.3 AttacksandDefenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
8.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
8.5 PracticalDeploymentIssues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
9 Conclusion 135
9.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
9.2 OpenIssuesandFutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
10