Topological Trends of Internet Content Providers

0.28 MB·English

by  Yuval Shavitt

Topological Trends of Internet Content Providers
Yuval Shavitt Udi Weinsberg
School of Electrical Engineering School of Electrical Engineering
Tel-Aviv University, Israel Tel-Aviv University, Israel
Email: [email protected] Email: [email protected]
Abstract—The Internet is constantly changing, and its hier- usingobservationsoftrafficflows.Inthispaperwetakeafirst
archy was recently shown to become flatter. Recent studies of look at the changing connectivity of large content provider
2 inter-domain traffic showed that large content providers drive networks, from a topological point of view. Unlike previous
1 this change by bypassing tier-1 networks and reaching closer
workthatstudiedvarioustrafficcharacteristics[14],[26],[21],
0 to their users, enabling them to save transit costs and reduce
2 reliance of transit networks as new services are being deployed, we consider the trends observed in the connectivity of the
and traffic shaping is becoming increasingly popular. networks in the Autonomous Systems (AS) graph.
n
In this paper we take a first look at the evolving connectivity We achieve this using a 5-year longitudinal study of the
a
of large content provider networks, from a topological point
J AS graph, focusing on 5 major content providers: Google,
of view of the autonomous systems (AS) graph. We perform
Yahoo!,MSN,Amazon,andFacebook.Thefirstthreearewell
4 a 5-year longitudinal study of the topological trends of large
content providers, by analyzing several large content providers established, large content providers, that have been around

] and comparing these trends to those observed for large tier-1 before the beginning of our study, in 2006. Amazon provides
I
N networks. We study trends in the connectivity of the networks, a uniqueopportunityto study a contentproviderthatchanged
neighbor diversity and geographical spread, their hierarchy, the
its scope (from an online store to a cloud service host)
.
s adoption of IXPs as a convenient method for peering, and
and Facebook reveals the high-paced growth of an extremely
c theircentrality.Ourobservationsindicatethatcontentproviders
[ graduallyincreaseanddiversifytheirconnectivity,enablingthem popular content provider. Using a comparative approach, we

to improve their centrality in the graph, and as a result, tier-1 examine 5 major transit providers, namely AT&T, Qwest,
1
networks lose dominance over time. Level3, Sprint, and Global Crossing (Glbx). All of these are
v
4 large, tier-1 transit networks [33], that have been used by
I. INTRODUCTION
3 content providers for transit services over the years.
8 TheInternetisaconstantlyevolvingnetwork,quicklyadapt- In this paper we create a snapshot of the AS-level graph
0
ing to customer needs and financial forces. Up until recently every3months,usingamonthofactivetraceroutes,fromlate
.
1 it was common to picture an hierarchical Internet [16], [12], 2006 till early 2011. We then study the connectivity trends,
0 [7], in which networks are either tier-1, large networks that meaning,howcontentandtransitprovidersare connectedand
2
provide global transit functionalities, tier-2, smaller Internet evolveovertime.WelookatthenumberofneighboringASes,
1
Service Providers (ISPs) that provide Internet connectivity to types of networks they connect with and spatial spread for
:
v their customers, or stub networks that produce and consume understanding these trends. We then look at the adoption of
i
X content [39]. IXPs which are a convenient and cost-effective method for
r However, in recent years the Internet is changing. The peering between ASes. We then study the changes in the
a appearance and rapid growth of large content providers, such hierarchal position of content providers and conclude with
as Google, Yahoo! and others, is gradually changing the studying their centrality.
roles of key Internet players to accommodate their needs. Understanding the evolving trends of AS connectivity has
First, largecontentprovidersproducehugeamountof content implications on different aspects of the Internet ecosystem.
that is consumed by users around the globe, inducing heavy The decreasing dominance of large transit providers we ob-
traffic on tranist networks. Although wholesale transit prices serveindicatesa changeinthe waytraffic flowsandnetworks
are decreasing by roughly 30% each year [32], and transit interconnect. These in turn have direct implications on the
providers offer various wholesale pricing plans to accommo- operational decisions that drive ASes, their connectivity and
date these needs, such as tiered prices [39], content providers profitability.Additionally,understandingthese trendscanhelp
still seek ways to significantly cut transit costs. Furthermore, improve Internet research, such as growth models [11] and
ascontentprovidersdeployanincreasingnumberofSoftware traffic flow analysis [21].
as a Service (SaaS), such as elastic computing, collaboration
tools, storage and even complete content delivery networks II. RELATEDWORK
(CDNs), they seek to reduce reliance on transit providersthat Several recent papers study the emerging change in the
were reported to perform traffic shaping [6], [9]. Internet ecosystem, which is driven mostly by large scale
As a result of these trends, the Internet was reported to be contentproviders.Oneoftheearlyobservationsofthischange
forming a flatter and denser network [17], [21], [11], mostly was made by Gill et al. [17]. The authors showed that large
2
content providers bypass many tier-1 ISPs by pushing their suffers from vantage point churn, making its observed topol-
networks closer to the users, and suggested that such a trend ogy more “noisy” and susceptible to measurement artifacts.
can possibly flatten the Internet. iPlane on the other hand, is more stable, both in the number
Kuai et al. [20], He et al. [19], [18], and more recently of measuring vantage points and the target IPs, however its
Augustinetal.[5]studiedtheAS-graph,anddiscussindetails topology is smaller.
methodsfor discoveringIXP participants. These works report In this paper we are interested in global trends observed in
significantlyhighernumberofpeeringrelationshipsdiscovered the ecosystem of large Internet players. Since these networks
among ASes that are IXP participants than among ASes that are well observed by both DIMES and iPlane, we expect
are not connecting via an IXP. bothplatformstocapturesimilartrendsthateffecttheInternet
Dhamdhere and Dovrollis [11] presented a new Internet ecosystem, even if the exact numbers are somewhat different.
model that captures the Internet transition from a hierarchy Therefore, when analyzing trends of the entire AS-level
of transit providers to a flatter interconnection of peers. graph, we use DIMES data, which brings a more accurate
Most recently, Labovitz et al. [21] performed a large-scale view of the topology, while the noise gets smoothed due to
two-year study of the inter-domain traffic, showing that the the large amount of data. When analyzing specific ASes, we
amount of traffic originated from content providers is rising, use iPlane’s data, since it is more stable. Indeed, in most
andmostofitisroutedoutsideofthetraditionalInternetcore. cases, both infrastructuresresulted in the same overalltrends,
Specifically, more than 5% of all inter-domain traffic in July assuring that the observationswe make are indeed due to real
2009 originated from Google’s networks. Additionally, they topology and routing characteristics and are not the results of
showed that content providers often use their own networks some measurement bias. Whenever the two datasets do not
forsavingtransitcosts.Asanexample,YouTubeinter-domain agree, we present both results and discuss the differencesand
trafficwasshowntodecayrapidlybymergingitintoGoogle’s their causes.
own peering and data centers. Itisimportanttonotethatsinceweareinterestedintrends,
the exact numbers we obtain (for number of connections,
III. METHODOLOGY clusteringcoefficient,etc.)arenotimportant.Weareinterested
For the purpose of this study we build the Internet AS in the their scale and especially in their evolution over time.
graph everythree monthssince January 2006 untilJuly 2010. Thus, the effect of measurementerrors [4], errors in the IP to
The AS graph is built by traversing IP-level traceroutes and AStranslation[28],[27],[29],andsimilarinaccuracies,which
resolvingeach IP into its correspondingAS. We resolve IP to arediscussedinthemeasurementliterature,arenotsignificant
AS using the published iPlane nano [24] database, which is a for this study since they can not effect trends.
setofmappingsbetweenIPprefixesintoASes,collectedfrom
all major BGP monitors. This set contains 326,102 prefixes IV. AUTONOMOUS SYSTEMS CONNECTIVITY
mapped to 30,779 ASes. IPs that were mapped into AS sets The simplest way to measure the connectivity of an AS
or multi-origin ASes (MOAS) were treated as unresolved. is the number of neighboringASes it is connected to, i.e., its
Additionally,we used InternetExchange(IXP) prefix map- degreeintheASgraph.Fig.1aandFig.1cshowtheexpected
ping provided from [5]. The IXP list is comprised of prefix vast difference between the degree of content providers and
lists collected from Packet Clearing House (PCH) [2], Peer- transit networks. However, while Fig. 1a shows that content
ingDB [3] and additional manually collected sources. This providers are increasing their connectivity over time, the
list provides us with 393 prefixes belonging to 278 IXPs connectivity of transit networks depicted in Fig. 1c exhibits
worldwide.WeusethesameIXPlist,obtainedinlate2009for a slow decrease (except Qwest).
allyears,assumingthattheassignmentofprefixestoIXPsdid Fig. 1b shows that the average degree of the neighboring
not decreased overtime, i.e., even if an IXP becomes defunct ASesofthethree‘veteran’contentprovidersslowlydecreases
[2], there are only a few or no cases that a prefix, which was over time, while the new ones, Amazon and Facebook, start
assignedtoanIXPin2009,wasassignedtoarealASinother withahighaverageneighbordegree,whichisquicklyreduced
times. to match the other three. This indicates that the content
Each AS graph is built by traversing AS traces of a single providers start by connecting mostly to tier-1 providers, but
month, creating a link between two ASes that follow each asthey expendtheyconnectto additionalproviderswhich are
other in the AS trace, or have an IP that belongs to an IXP mostly not tier-1 (see additional discussion below). Fig. 1d
prefixbetweenthem(thelatterfollowsthetechniquedescribed showsthattheaverageneighbordegreeofthetransitnetworks
in [18] and extended in [5]). increase. This can be an indication that small customers are
FortheIP-leveltracesweusetwodatasets,DIMES[35]and disconnecting from the transit networks, or alternatively, that
iPlane [23]. DIMES is a community-based Internet mapping the tier-1 neighbors are becoming better connected, hence
effort,measurementfromthousandsofvantagepoints,located increasetheirdegree.Bothreasonsimplythattransitnetworks
at user homes and since 2010 also in PlanetLab [8]. iPlane are loosing some of their dominance in Internet connectivity.
uses PlanetLab nodes and traceroute servers, measuring from To further understand the reasons behind these trends, we
arelativelystablesetof300servers.AlthoughDIMES,dueto classify the neighbors of each AS. We use the classification
itsdiversedistribution,uncoversmorelinksthaniPlane[36],it providedbyDhamdhereandDovrolis[10], inwhich anAS is
3
120 2000
MSN MSN
Google 1800 Google
100 Yahoo e Yahoo!
e1600
Amazon gr Amazon
80 Facebook de1400 Facebook
gree 60 ghbor 1200
De nei1000
e
40 g 800
a
er
v 600
20 A
400
0 200
08/200162/200064/200078/200172/200074/200088/200182/200084/200098/200192/200094/2010 08/2006 02/2007 08/2007 02/2008 08/2008 02/2009 08/2009 02/2010
Date [month/year] Date [month/year]
(a)Content (b)Content neighbors
4000 50
ATT
3500 LQewveesl3t e45
e
3000 SGpbrlixnt degr40
gree2500 ghbor 35
De2000 nei
e
1500 erag30 ALeTvTel3
v
1000 A25 QSpwreinstt
Gblx
500 20
08/2006 02/2007 08/2007 02/2008 08/2008 02/2009 08/2009 02/2010 08/2006 02/2007 08/2007 02/2008 08/2008 02/2009 08/2009 02/2010
Date [month/year] Date [month/year]
(c)Transit (d)Transitneighbors
Fig. 1: The degree of transit and content networks, and their average neighbor degree in the AS-graph
classified as an enterprize customer (EC), a small or large Internet connectivity.
transit provider (STP and LTP), or a content, access and Finally, we measurethe assortativity coefficientvalues [31]
hostingprovider(CAHP).Theauthorsbasetheirclassification of the complete AS graph, using data from DIMES. These
on the average customer and peer degrees of the AS over its coefficients are positive when vertices tend to connect with
entire lifetime within a 10 year longitudinal study, and claim similar-degree vertices and negative otherwise. The AS-graph
to reach over 80% accuracy. was shown to be disassortative [31], [25], meaning comprise
Fig. 2a depicts the distribution of the AS types of the mostly of radial links, connecting ASes towards the tier-1
neighbors of the content providers as captured during April ASes. Looking at the trend of the assortativity coefficient
in four consecutive years: 2007 (left-most bar in each AS) depicted in Fig. 3 shows that the AS-graph becomes less
till 2010 (right-most bars). The figure shows that the number disassortative, indicating that ASes increase their connections
of LTP neighborsdid notsignificantly change,however,since with other similar-degree ASes.
the number of neighbors increased over time for all content Overall, these findings agree with previous findings [17],
providers, the overall LTP percentage decreased. The most [21], [11], and show a trend of large content providers that
drastic change is observed for Amazon and Facebook, with shift from relying mostly on large transit providers towards
a 2/3 reduction. Amazon and Facebook greatly increased the a flatter topology, interconnecting with smaller networks for
number of CAHP and STP neighbors. gaining transit and direct access to last-mile customers. Con-
Similar analysis on transit providers revealed that they tent providers that have been around longer, such as Google,
mostly interconnect with EC and CAHPs. These neighbors Yahoo!,andMSNdonotincreasethenumberofLTPs,mainly
exhibit the largest reduction, indicating a market loss in their sincetheyalreadyexploitthebenefitofconnectingwiththem.
core business, probably impacting their dominance in the The “younger”contentprovidersfollow this trend and mostly
4
100 100
N/A Others
CAHP Canada
80 LTP 80 UK
ors STP ors S.Korea
b EC b HK
h h
g 60 g 60 Japan
nei nei Germany
of of US
er 40 er 40
b b
m m
u u
N N
20 20
0 0
MSN Google Yahoo Amazon Facebook MSN Google Yahoo Amazon Facebook
ASes ASes
(a)Neighbors types (b)Neighbors countries
Fig. 2: Content providers neighbor types and countries
Yahoo!sustainarelativeconstantnumberofUS-neighborsand
0.19 increase the number of non-US neighbors, whereas Amazon
(cid:0)
and Facebook, which grew faster, significantly increase the
0.20
(cid:0) number of US-neighbors. AT&T, Qwest and Sprint have a
0.21 majority of neighbors in the US, whereas Level3 and Glbx
(cid:0)
y
ativit(cid:0)0.22 have a lower percentage of US-neighbors.
ssort(cid:0)0.23 B. Density and Clustering
A
Theconnectivityofthe ASesin thegraphdirectlyaffectits
0.24
(cid:0)
densityandclustering.Givena graphG=(V,E), thedensity
(cid:0)0.25 D of the graph is defined as the number of existing links out
of the number of potential links, i.e.:
0.26
(cid:0)01/200056/200096/200016/200057/200097/200017/200058/200098/200018/200059/200099/200019/200150/200190/200110/2011 D = 2|E|
Date [month/year] |V|(|V|−1)
Fig. 3: Assortativity coefficient of the AS-graphs produced
Fig.4showsthatthedensityoftheAS-graphdecreasesover
using DIMES dataset
time,mainlysincenewASesthatjointheInternetsignificantly
increase the potential of links, however, they connect only to
a small portionof the already existing ASes. This is expected
use STPs and CAHP. This trend reveals an overall decrease
sinceextremelylargedegreesareobservedonlyinarelatively
inthedominanceoftier-1networksintheInternetecosystem.
small number of ASes, and even these connect to a few
This observation repeats in the following sections, when we
thousands ASes.
study the centrality of these networks.
In order to better understand the local connectivity of the
ASes, we look at their clustering coefficient (CC), which is a
A. Geographical Connectivity
measure of the local density of an AS based on its neighbors.
We further measure how the spatial connectivity of the More formally, the CC of a vertex in a graph is the number
contentprovidersevolves.EachneighboringASisresolvedto of triangles it forms with its immediate neighbors out of the
itsmajorcountry,i.e.,thecountrythathostsmostofitsknown potential number of triangles.
IP addresses, using data from MaxMind [1]. This database is Fig. 5 depicts the CC of content and transit providers.
considered accurate in the country level of resolution [38], Fig. 5a shows that the CC of content providers is mostly
[37]. decreasing, which is a result of the increasing number of
Fig. 2b depicts, for each AS, the number of neighbors in neighbors having few or no links amongst themselves. On
a set of countries, over a period of four years (the bars of the other hand, transit networks exhibit an increasing CC,
each AS are for 2007, 2008, 2009, and 2010, from left to meaningthattheyareloosingneighborswithlowconnectivity
right). The content providers we analyze originate from the and maybe hints at an increase in neighbor interconnections.
US, hence it is not surprising that the largest percentage of These two observations strongly indicate that content
neighbors were found to be in the US. MSN, Google, and providers are increasing connectivity with access providers
5
0.060×10−2 1.0
MSN
0.9 Google
0.055 Yahoo
nt0.8 Amazon
e
ci Facebook
y0.050 effi0.7
Densit0.045 ering co00..65
st
u
Cl0.4
0.040
0.3
0.035 0.2
08/200162/200064/200078/200172/200074/200088/200182/200084/200098/200192/200094/2010 08/200162/200064/200078/200172/200074/200088/200182/200084/200098/200192/200094/2010
Date [month/year] Date [month/year]
Fig. 4: AS graph density (number of link out of the potential (a)Contentproviders
number of link)
0.016
AT&T
0.014 Level3
Qwest
that are not in the core and are thus not interconnected. On
the other hand, transit networks lose small customers and the cient0.012 SGpbrlixnt
remaining customers increase their interconnections that are effi0.010
o
usedtobypassthecore[21],whichcausesthetransitnetworks g c
n0.008
to gradually lose their centrality. eri
st
u0.006
C. IXPs Cl
0.004
IXPs are a convenient method for ASes to interconnect,
since it providesa shared facility and infrastructure[5]. ASes 0.002
thatseek toexpendtheirconnectivityhaveanincentivetouse 08/200162/200064/200078/200172/200074/200088/200182/200084/200098/200192/200094/2010
such facilities as it enables them to connect to a wide range
Date [month/year]
of other ASes. As such, content providers can leverage IXP (b)Transitproviders
connectivitytograduallyincreasethenumberofpeeringASes
Fig. 5: Clustering coefficient
withminimalsetupcosts.InthisanalysisweuseDIMESdata
since it manages to detect more IXP links than iPlane.
Fig.6showsthenumberofIXPsusedbycontentandtransit
of content networks connectivity is high.
providers. Fig. 6a shows that content providers gradually
increased their adoption of IXPs mainly since early 2009.
V. AUTONOMOUS SYSTEMS HIERARCHY
Most of the content providers increased the number of IXPs
theyconnectthroughbymorethan100%in onlya fewyears, The hierarchical structure of the Internet has been studied
emphasizingthe importantrole that IXPs play in the Internet. extensively, classifying ASes into the classical three-tiered
Fig. 6b shows that transit providers use more IXPs, but model [16], [13], and understanding the valley-free packet
theirpercentageoutoftheoverallneighborsis extremelylow. routingrules [15]. A differentmethod of hierarchicalanalysis
Additionally,althoughtransitprovidersincreasetheirusageof is k-pruning [7], which decomposes graphs into shells, based
IXPs, the number of IXPs they connect through grows much on the node connectivity towards the graph center. In the AS
slower than content providers. graph,ASesinthefirstshellarethosewhohaveonlyonelink
Fig. 7 depicts the number of AS-links that use an IXP, i.e., leading to the ‘center’ of the graph, whereas ASes in the kth
foreachAS,thenumberofotherASesthatitconnectsthrough shell have k-connectivity towards the center. The nucleus (or
an IXP. Although both transit and content providers exhibit core)istheshellwiththehighestindex,whichisconsideredto
a rising trend, the percentage of IXP links used by content contain top level providers,mostly tier-1 transit networks[7].
providersis significantlyhigher,reachingalmost40%of their Fig.8showstheshellindexofcontentandtransitproviders.
links, indicating that content providers indeed embrace IXPs Fig. 8b shows that all of the examinedtransit networksare in
as method for increasing connectivity. the same shell, which is the nucleus. This is expected, since
Interestingly, the number of IXP links is not significantly these ASes are the top-levelprovidersof the Internet, making
higherthanthenumberofIXPs,meaningthateachIXPisused them extremely central [7], [36].
for connecting with only a few ASes. Since more ASes are Fig. 8a shows that Google, Yahoo!, and MSN have a very
expectedtojointhesesharedfacilities[5],thegrowthpotential high shell index, and are either in the nucleus or in a very
6
30 60
MSN AT&T
Google Level3
25 50
Yahoo Qwest
Amazon Sprint
20 Facebook 40 Gblx
s s
XP15 XP30
I I
10 20
5 10
0 0
03/200077/200171/200073/200087/200181/200083/200097/200191/200093/201007/201101/201003/2011 03/200077/200171/200073/200087/200181/200083/200097/200191/200093/201007/201101/201003/2011
Date [month/year] Date [month/year]
(a)Content providers (b)Transitproviders
Fig. 6: Number of IXPs used by content and transit providers
100 120
MSN AT&T
Google Level3
100
80 Yahoo Qwest
Amazon Sprint
Facebook 80 Gblx
s 60 s
k k
n n
P li P li 60
IX 40 IX
40
20
20
0 0
03/200077/200171/200073/200087/200181/200083/200097/200191/200093/201007/201101/201003/2011 03/200077/200171/200073/200087/200181/200083/200097/200191/200093/201007/201101/201003/2011
Date [month/year] Date [month/year]
(a)Contentproviders (b)Transitproviders
Fig. 7: Number of IXP links used by content and transit providers
closeshell.FacebookandAmazonexhibitadramaticincrease nucleus, reveals that unlike previously thought [7], there is
in their shell index. Facebook enjoyed increasing popularity, a significantportionof STPs (roughly40%)and evenCAHPs
which drive it into connecting with high tier networks, im- (15-25%) in the nucleus. As content providers increase the
provingitsconnectivityandservicelevels.Amazon’sincrease portion of neighbors that reside in the core, they manage to
reveals the market shift that the company had, from being an increase their shell index, thus reduce their dependency on
online store that resides in a low shell into a major cloud top-tier transit providers. We note that the nucleus index has
service provider, hosting numerous application. This change slightly increased since 2006 (roughly 5% increase in iPlane
mandatessignificantlymoreconnectionswith tier-1networks, and 10% in DIMES). An increase in this index indicates a
resulting in the increase witnessed in 2009. richer interconnection within the Internet core.
For a content provider, being close to the nucleus has
direct capital effect, because the higher the shell a network
is located in, the less it needs to be a paying customer of VI. MEASURING CENTRALITY
transit networks [15], mainly since it does not need transit
services in order to reach most other customer networks. In this section we seek to understand the ways that the
Looking back at the degree of content providers in Fig. 1, changing connectivity of content providers affects their cen-
their degree is significantly lower than tier-1 transit networks. trality, and whether these changes actually change the cen-
Since tier-1 networks are expected to comprise the nucleus trality oflargetransitproviders.We approximatethe potential
[7], it is unexpected that content providers manage to reach load on the ASes, and use a web-based centrality metric for
high shells. However, examining the types of ASes in the assessing the centrality of the networks.
7
traverse the Internet in valley-free paths [15], BC can only
50 serve as an approximation for node centrality. To check the
validity of this approximation, we first calculated the BC on
40 the undirected AS graph, ignoring valley-free paths. Then, in
ordertoaccounttothevalley-freerules,wecalculatedtheBC
ex30 of each AS directly from the probed paths of each month,
d
hell in20 bthyedmiviidddinlegothfethneumtrbaceer)ofbytratcheerotouttaelsnthuamtbtrearveorfsetraacneAroSut(eisn.
S
MSN This method is more accurate than adding directions to the
Google
AS-graph, since inferring commercial relationships between
10 Yahoo
Amazon ASes was shown to contain errors, with up to 20% mistakes
Facebook
in peer-to-peer relationships [41], [12]. We found that the
0
08/200162/200064/200078/200172/200074/200088/200182/200084/200098/200192/200094/2010 masecaasulcruelmateinntgisnhfoerrteensctepaotfhcoennttrhaelituyndgiarveectesidmAilSargBraCphv,awluieths
Date [month/year]
themeasurementinferenceexhibitingslightlylowerBCvalues
(a)Contentproviders
andsignificantlymorenoise.Sinceweareinterestedintrends
wheretheexactvaluesarelesssignificant,weusethecommon
41
ATT and easier method,which is less accurate but providesa clear
40 Level3 view of the trends.
Qwest
39 Sprint Fig. 9 shows the normalized BC of the selected content
Gblx
x38 and transit networks. As expected, the BC values of the tier-
e
nd 1 transit networks are significantly higher (two orders of
ell i37 magnitude) than content network, as the latter are usually the
h
S36
last hop of the routes.
35
Interestingly, Fig. 9a shows an increasing trend in the
34 BC of Google and Yahoo!. We validated this increase with
the DIMES dataset and observed the same trend, with the
33
08/2006 02/2007 08/2007 02/2008 08/2008 02/2009 08/2009 02/2010 dWifefefroelnlocweetdhathteGAoSo-glleevewlittrnaecsesseosfaYashliogoh!tlayndhiGghoeorgliencdrueraisneg.
Date [month/year]
weeks39and40of2010,andfoundthatYahoo!AS10310and
(b)Transiftproviders
GoogleAS15169appearmid-traceinroughly22%and14%of
Fig. 8: Shell index of content and transit providers
thetracestheyappearin,respectively.Wefurtherlookedatthe
tracesthemselves,andfoundthatbothnetworksalwaysappear
as the hop-before-last,and are siblingsof the last hop(owned
A. Approximation of Load
by either Google or Yahoo!). In Google’s case, the majority
Approximating the potential load on an AS is an estimate of traces terminate with YouTube (AS36561). Yahoo! on the
of its importance, since it indicates that an AS serves many other hand is comprised of several different ASes, such as
other ASes for routing packets [25]. This approximation is Yahoo! Japan (AS23926), Yahoo! US (AS7233), and Yahoo!
commonly achieved using betweenness centrality (BC). In Backbone (AS24018).
graph theory, BC measures the number of appearances of a
These two networks exhibit a similar behavior: they both
vertexin the shortestpaths betweenall other vertices,relative
provide transit for sibling ASes as well as content, but only
to the total number of shortest paths. Given a vertex v ∈ V,
for data belongingto their networks.Yahoo!’smajorAS does
its bc B(v) is calculated using:
this because their network is comprised of several regional
σ (v) and probably specialized networks. Google provides transit
st
B(v)= X to companies they purchase, leveraging their major AS’s
σ
st
s6=v6=t∈V connectivity.Inbothcases,theendresultisthesame–content
where σ is the number of shortest paths from s to t and providers, which were once mainly stub networks that termi-
st
σ (v) is the number of shortest paths from s to t that pass nate routes, are becoming more central by providing transit
st
through v. towards other sibling and peering ASes, enabling network
Usually, BC is normalized by dividing it by the number operatorstosave costsbyleveragingtheirgraduallyimproved
of possible pairs, to enable comparison between graphs of connectivity.Thisobservationisalsobackedfromthefindings
different sizes. Given the number of vertices in the graph of Labovitz et al. [21], who showed a decreasing traffic trend
n=|V|,inundirectedgraphsthenumberofpairsisn(n−1). observed from YouTube AS and an increasing traffic trend
BC is commonly applied to the AS-graph for measuring from Google’s AS. This is the result of Google’s AS acting
the possible load that an AS sustains. However, since packets as the major transit provider for YouTube’s traffic.
8
than content providers. However, while the PR of transit
0.25×10−2 networksslowlydecreases,contentproviderswitnessagradual
MSN
Google increase, with MSN being an exception.
y0.20 Yahoo
alit Amazon
centr0.15 Facebook 0.06×10−2
s MSN
s
e Google
n 0.05
n0.10 Yahoo
e
e Amazon
w
et 0.04 Facebook
B0.05 k
n
a
er0.03
g
0.008/0200162/200064/200078/200172/200074/200088/200182/200084/200098/200192/200094/2010 Pa0.02
Date [month/year] 0.01
(a)Contentproviders
0.00
0.25 ATT 08/200162/200064/200078/200172/200074/200088/200182/200084/200098/200192/200094/2010
Level3 Date [month/year]
y0.20 Qwest (a)Contentproviders
alit Sprint
ntr Gblx
e0.15 0.030
c
s ATT
s
e Level3
n
n0.10 0.025 Qwest
e
e Sprint
w
et Gblx
B0.05 k0.020
n
a
er
g
00.08/02006 02/2007 08/2007 02/2008 08/2008 02/2009 08/2009 02/2010 Pa0.015
Date [month/year] 0.010
(b)Transitproviders
Fig. 9: Betweenness centrality 0.0008/52006 02/2007 08/2007 02/2008 08/2008 02/2009 08/2009 02/2010
Date [month/year]
B. PageRank Centrality (b)Transitproviders
PageRank (PR), which was proposed initially for scoring Fig. 10: PageRank centrality
web-pages[34], is a measure of the importanceof a vertex in
a network[30], [22]. PR assigns relative scores to all vertices
inthenetworkbasedontheiterativeprinciplethathigh-scoring VII. DISCUSSION
neighbors contribute more to the score of a vertex than low-
The AS-level graph exposes the trend that is changing the
scoring neighbors. More formally, given a graph G=(V,E),
Internet – content providers become key players in the In-
the PR of a vertex u∈V is iteratively computed using:
ternet. The connectivity trends indicate that content providers
PR(v) increase the number of neighbors and diversify their types
PR(u)= X degree(v) and geographicalspread. They mostly make new connections
v∈V, (v,u)∈E withsmalltransitandaccessproviders,enablingthemabetter
wherethe initialPR ofallverticesis setuniformly,meaning: reach to worldwide customers, while minimizing high-tier
∀u ∈ V, PR(u) = 1/|V|. Notice that this is a very simple transit costs. These changes are also witnessed in the way
formulation of PR, and a more complete form was used, content providers are gradually climbing towards the core of
however, the exact details are not essential for understanding theInternet,actuallyreformingitsothatitincludesnon-transit
its meaning. networks.
WeappliedPRontheASgraphsforextractingthecentrality IXPs are also gaining popularity, and they seem to be
of ASes. Fig. 10 shows the PR score of content and transit underutilized,meaningthatthegrowthpotentialusingpeering
providers, strengthening the above observations. The PR of agreements is high. As they provide an easy method for
transitprovidersistwoordersofmagnitudelargerthancontent interconnecting with other networks, they are likely to be
providers, indicating that they are significantly more central better utilized in the future.
9
The centrality of content providersis a strong indication to [15] L.Gao. Oninferring autonomous system relationships in the Internet.
their increasingdominance,andon the otherhand,the chang- IEEE/ACMTrans.onNetworking, 9(6):733–745, 2001.
[16] Z.Ge,D.Figueiredo,S.Jaiwal,andL.Gao.Onthehierarchicalstructure
ingroleoftop-tiertransitproviders.Thelatter,despiteoffering
ofthelogical internet graph. InSPIEITCOM,Aug.2001.
attractive wholesale pricing plans [39], lose their dominance, [17] P.Gill,M.Arlitt,Z.Li,andA.Mahanti.TheflatteningInternettopology:
as content providers, which are amongst the highest paying natural evolution, unsightly barnacles or contrived collapse? In PAM,
pages1–10.Springer-Verlag, 2008.
customersdue to the amountoftraffic they produce[21], find
[18] Y. He, G. Siganos, M. Faloutsos, and S. Krishnamurthy. Lord of
ways to reduce redundancy on tier-1 networks, while better the links: A framework for discovering missing links in the Internet
utilizingtheirownresources.Inadditiontosavingtransitcosts, topology. IEEE/ACMTrans.onNetworking,17(2):391–404,Apr.2009.
[19] Y.He,G.Siganos,M.Faloutsos,andS.V.Krishnamurthy.Asystematic
content providers increase their offerings to customers by
frameworkforunearthingthemissinglinks:Measurementsandimpact.
providingSoftwareasaService(SaaS),suchashosting,cloud InNSDI,2007.
services, collaboration tools and others, without worrying for [20] X.Kuai,D.Zhenhai,Z.Zhi-Li,andC.Jaideep.Onpropertiesofinternet
exchange points and their impact on AStopology andrelationship. In
network neutrality violations or other traffic shaping that can
Networking, 2004.
interfere with the service grade they provide [9], [6], [40]. [21] C.Labovitz,S.Iekel-Johnson,D.McPherson,J.Oberheide,andF.Jaha-
nian. Internet inter-domain traffic. InSIGCOMM, pages 75–86. ACM,
VIII. CONCLUSION 2010.
[22] J. Liu, Y. Huang, and S. Li. Video search re-ranking via multi-graph
This paper studies the topological trends of large content propagation. InACMMultimedia, 2007.
providersin the AS-levelgraph.We show thatthese networks [23] H. V. Madhyastha, T. Isdal, M. Piatek, C. Dixon, T. E. Anderson,
A.Krishnamurthy,andA.Venkataramani. iPlane:Aninformationplane
increase their connectivity with other networks, mostly using
fordistributed services. InOSDI,pages 367–380,2006.
IXPs. As a result, the centrality of these networks increase, [24] H. V. Madhyastha, E. Katz-Bassett, T. Anderson, A. Krishnamurthy,
as they manage to reduce their dependency in tier-1 transit and A. Venkataramani. iPlane Nano: path prediction for peer-to-peer
applications. InNSDI,pages 137–152,2009.
networks, which lose dominance in the Internet ecosystem.
[25] P.Mahadevan,D.Krioukov,M.Fomenkov,X.Dimitropoulos,kcclaffy,
Throughwell-designed selection of peers, usage of IXPs, and and A. Vahdat. The Internet AS-level topology: three data sources
acting as transit networks for traffic originating from sibling andonedefinitivemetric. ACMSIGCOMMComputerCommunications
Review, 36(1):17–26, 2006.
ASes,thelargecontentprovidersmanagetoclimbtheInternet
[26] G. Maier, A. Feldmann, V. Paxson, and M. Allman. On dominant
hierarchy,enablingthemtoimproveservicetotheircustomers characteristics of residential broadband Internet traffic. In IMC, pages
while cutting down transit costs. 90–102.ACM,2009.
[27] Z.Mao,D.Johnson,J.Rexford,J.Wang,andR.H.Katz. Scalableand
REFERENCES accurate identification of AS-level forwarding paths. In INFOCOM,
2004.
[1] Maxmind, geolocation and online fraud prevention, [28] Z. Mao, J. Rexford, J. Wang, and R. H. Katz. Towards an accurate
(http://www.maxmind.com/). AS-level traceroute tool. InSIGCOMM, 2003.
[2] Packet Clearing House,http://www.pch.net/. [29] Z.M.Mao,L.Qiu,J.Wang,andY.Zhang. Onas-levelpathinference.
[3] PeeringDB, http://www.peeringdb.org/. InSIGMETRICS’05,pages 339–349,2005.
[4] B.Augustin,X.Cuvellier, B.Orgogozo,F.Viger,T.Friedman,M.Lat- [30] V. Mehta, C. Bartzis, H.Zhu, E.Clarke, andJ. Wing. Ranking attack
apy, C. Magnien, and R.Teixeira. Avoiding traceroute anomalies with graphs.InProceedingsofRecentAdvancesinIntrusionDetection,2006.
paris traceroute. InIMC’06,pages153–158, 2006. [31] M. E. J. Newman. Assortative mixing in networks. Physical Review
[5] B. Augustin, B. Krishnamurthy, and W. Willinger. IXPs: mapped? In Letters, 89(20),Oct.2002.
IMC’09,pages336–349.ACM,2009. [32] W.Norton. Drpeering.net. http://drpeering.net.
[6] R. Beverly, S. Bauer, and A. Berger. The Internet is not a big truck: [33] R.V.Oliveira,D.Pei,W.Willinger,B.Zhang,andL.Zhang. Insearch
Towardquantifying networkneutrality. InPAM,pages 135–144,2007. oftheelusivegroundtruth:theinternet’sAS-levelconnectivitystructure.
[7] S. Carmi, S. Havlin, S. Kirkpatrick, Y. Shavitt, and E. Shir. A model InACMSIGMETRICS,2008.
of Internet topology using k-shell decomposition. Proceedings of the [34] L.Page,S.Brin,R.Motwani,andT.Winograd. ThePageRankcitation
NationalAcademyofSciencesUSA(PNAS),104(27):11150–11154,July ranking: Bringing order to the web. Technical report, Stanford Digital
32007. LibraryTechnologies Project, 1998.
[8] B.Chun,D.Culler,T.Roscoe,A.Bavier,L.Peterson,M.Wawrzoniak, [35] Y. Shavitt and E. Shir. DIMES: Let the Internet measure itself. ACM
and M. Bowman. Planetlab: An overlay testbed for broad-coverage SIGCOMMComputer Communications Review,35(5),2005.
services. ACM SIGCOMM Computer Communications Review, 33(3), [36] Y. Shavitt and U. Weinsberg. Quantifying the importance of vantage
July2003. pointsdistributionininternettopologymeasurements. InIEEEJournal
[9] A.DhamdhereandC.Dovrolis.CanISPsbeprofitablewithoutviolating onSelected AreasinCommunications, Oct.2011.
“networkneutrality”? InProceedingsofthe3rdinternationalworkshop [37] Y. Shavitt and N. Zilberman. A geolocation databases study. IEEE
onEconomicsofnetworkedsystems,pages13–18,NewYork,NY,USA, JournalonSelected AreasinCommunications. toappear.
2008.ACM. [38] S. S. Siwpersad, B. Gueye, and S. Uhlig. Assessing the geographic
[10] A.DhamdhereandC.Dovrolis.TenyearsintheevolutionoftheInternet resolution of exhaustive tabulation for geolocating Internet hosts. In
ecosystem. InIMC,pages 183–196.ACM,2008. PAM.Springer-Verlag, 2008.
[11] A. Dhamdhere and C. Dovrolis. The Internet is flat: modeling the [39] V. Valancius, C. Lumezanu, N. Feamster, R. Johari, and V. Vazirani.
transition from a transit hierarchy to a peering mesh. In CoNext ’10. Howmanytiers? pricing intheInternet transit market. InSIGCOMM,
ACM,2010. 2011.
[12] X. Dimitropoulos, D. Krioukov, M. Fomenkov, B. Huffaker, Y. Hyun, [40] U. Weinsberg, A. Soule, and L. Massoulie´. Inferring traffic shaping
kcclaffy,andG.Riley.ASrelationships:Inferenceandvalidation. ACM andpolicy parameters usingendhostmeasurements. InInfocom (min-
SIGCOMMComputer Communications Review, 37:2007,2006. conference). IEEE,2011.
[13] X.Dimitropoulos, D.Krioukov,G.Riley,andK.Claffy. Revealing the [41] J.XiaandL.Gao. Ontheevaluation ofASrelationship inferences. In
autonomoussystemtaxonomy:Themachinelearningapproach.InPAM, IEEEGlobecom, Dallas,TX,USA,2004.
2006.
[14] C.Fraleigh,S.Moon,B.Lyles,C.Cotton,M.Khan,D.Moll,R.Rockell,
T. Seely, and C. Diot. Packet-Level Traffic Measurements from the
SprintIPBackbone. IEEENetwork, 17(6):6–16, 2003.