Balanced Overlay Networks (BON): Decentralized Load Balancing via Self-Organized Random Networks

0.36 MB·English

by  Jesse S. A. Bridgewater

1
Balanced Overlay Networks (BON): An Overlay Technology for
Decentralized Load Balancing
Jesse S.A. Bridgewater, Vwani P. Roychowdhury and P. Oscar Boykin, Member, IEEE
Abstract—We present a novel framework, called bal- its unused resources by adding and removing edges
anced overlay networks (BON), that provides scalable, when resources are freed and consumed as depicted
6 decentralized load balancing for distributed computing in Figs. I and II. As we will show, this topology
0 using large-scale pools of heterogeneous computers. Fun- makes it possible to efficiently locate nodes with
0 damentally, BON encodes the information about each
the most free resources, which in turn enables load
2 node’s available computational resources in the structure
balancing with no central server.
n of the links connecting the nodes in the network. This
a distributed encoding is self-organized, with each node This work makes several novel contributions to
J
managing its in-degree and localconnectivity via random- distributed computing and resource sharing. First,

6
walksampling.Assignmentofincomingjobstonodeswith BON is decentralized and scalable with known
1
the most free resources is also accomplished by sampling
lower bounds on balancing performance. While
] the nodes via short random walks. Extensive simulations
other decentralized load-balancing algorithms (e.g.,
C show that the resulting highly dynamic and self-organized
D graphstructurecanefficientlybalancecomputationalload Messor; see also Section II for more detailed
. throughoutlarge-scalenetworks.Thesesimulationscovera comparisons) have been proposed in the literature,
s
c wide spectrumofcases, including significantheterogeneity performance and scalability analyses for such al-
[ in available computing resources and high burstiness in gorithms, which guarantee almost-optimal perfor-

incoming load. We provide analytical results that prove
2 mance as the number of nodes becomes very large,
v BON’s scalability for truly large-scale networks: in par-
have been lacking. Under certain ideal conditions,
6 ticular we show that under certain ideal conditions, the
4 network structure convergesto Erdo¨s-Re´nyi (ER) random we show that the network structure converges to a
0 random graph that is at least as regular and balanced
graphs; our simulation results, however, show that the
1
algorithm does much better, and the structures seem to as Erdo¨s-Re´nyi (ER) graphs. Secondly, the algo-
1
4 approach the ideal case of d-regular random graphs. rithms and protocols for both network maintenance
0 We also make a connection between highly-loaded BONs and job allocation are based only on local infor-
/ and the well-known ball-bin randomized load balancing
s mation and actions: each node decides the amount
c framework.
: of resource or computing power it wants to share,
v
i and it embeds this information into the network
X
I. INTRODUCTION structure via short random walks; similarly, jobs
r
a are distributed based only on information available
Distributed computing was one of the earliest
through local explorations of the overlay network.
applications of computer networking and many dif-
Thus, BON is a truly self-organized dynamical
ferent methods have been developed to harness the
system. Thirdly, since the BON algorithm produces
collective resources of networked computers. Some
dynamic random graph topologies, these resulting
important architectures include centralized client-
networks are very resilient to random damage and
server systems, DHT-based systems, and diffusive
also have no central point of failure. Finally, we
algorithms. Here we introduce the concept of bal-
make a connection between the performance of
anced overlay networks (BON) which takes the
BON in some regimes with ball-bin random load
novel approach of encoding the resource balancing
balancing problems [1].
algorithm into the evolution of the network’s topol-
It is also important to note that BON is a novel
ogy. Each node’s in-degree is kept proportional to
paradigm for for resource sharing of any kind and
Jesse S.A. Bridgewater and Vwani P. Roychowdhury are with its applicability is not limited to only distributed
theElectricalEngineeringDepartment, UniversityofCalifornia,Los computing. The in-degree of a node can be made
Angeles, CA 90095
to correspond to any type of shareable resource.
P.OscarBoykiniswiththeDepartmentofElectricalandComputer
Engineering, University of Florida, Gainesville, FL 32611 Then one can exploit the fact that BON networks
2
Newly Edge
Assigned To Be
4 1 Job 4 1 Deleted 4 1
0 1 0 1 1 1
2 3 2 3 2 3
New 2 3 2 3 2 3
Incoming
Job
a) b) c)
Fig. 1. New jobs are assigned by using a greedy random walk. The large nodes depict computers in a schematic BON network while the
small filled nodes are jobs running on the node to which they are connected. The label for each of the computer nodes denotes the current
number of jobsitisrunning. Parta) shows new loadentering thenetwork. Inb)weseethatthenode whereload arrivesinitiatesarandom
walk which keeps track of the degree(free resources) of each visited node. The largest degree node(most free resources) is selected to run
the new load. To compensate for the additional load, the node which accepted the new load deletes one of its incoming edges to account
for its diminished resources. The resulting network is depicted in c).
have low diameters associated with random graphs, BON algorithm. The system could allow a huge
which makes them easy to sample using short number of software users to participate in providing
random walks. Extensive simulation results support download mirrors.
the efficacy of this approach in networks with a This paper is organized as follows. Section II
wide range of resource and load distributions.These describes prior related load balancing research. Sec-
simulations show that the actual performance of the tion III introduces the BON architecture. Section IV
algorithm far exceeds the lower bounds mentioned discusses theoretical analysis of BON’s scalability.
above. Section V provides a description of the simulation
setup and results. Finally Section VI deals with
BON is a very simple, realistic and easily im-
practical considerations for implementing BON.
plementable algorithm using standard networking
protocols. The completely decentralized nature of
the algorithm makes it very well-suited to massive II. RELATED WORK
applications encompassing very large ensembles of
The authors have previously considered
nodes. The following are a few examples of appli-
topologically-based load balancing with a simpler
cations for which BON is very well suited.
model than BON which is amenable to analytical
Single-System Image (SSI) LAN/WAN clusters: study [6]. In that work each node’s resources were
BON can be used for single-system image (SSI) proportional to in-degree and load was distributed
clusters in the same way that Mosix [2] is used but by performing a short random walk and migrating
without the need for all nodes to be aware of each load to the last node of the walk; this method
other as is the case in Mosix. This can allow BON produces Erdo¨s-Re´nyi (ER) random graphs and
to scale to very large system sizes. exhibits good load-balancing performance. As
Public Resource Computing: BON is also applica- we demonstrate in the current work, performing
ble to @HOME-style projects [3]. This projects are more complex functions on the random walk can
typically special purpose for each application. The significantly improve performance.
decentralized nature of BON will allow multiple The majority of distributed computing research
projects to share the same pool of computers. has focused on central server methods, DHT ar-
Grid Computing: BON also has the potential to be chitectures, agent-based systems, randomized algo-
integrated with GRID [4], [5] systems for efficient rithms and local diffusive techniques [1], [7]–[12].
resource discovery and load distribution across vir- Some of the most successful systems to date [3],
tual organizations (VOs). [13] have used a centralized approach. This can
Web Mirroring: Distributed web mirroring is an be explained by the relatively small scale of the
example of a non-computational application of the networked systems or by special properties of the
3
Job
Finishes 4 1 3 1 3 1
1 1 1 1 1 1
2 3 2 3 2 3
2 3 2 3 2 3
New Edge
Origin
a) b) c)
Fig. 2. When a running job finishes, the host node may need to increase its connectivity to advertise its increased resources. Subpart a)
shows a job finishing and thus leaving the network. In b) the node where load finishes initiates a random walk. The last node on the walk
willbetheoriginof anew edge incident onthewalkinitiatorasseeninc).Thisnew edgerepresents theincreaseinavailableresourceson
the node where the job just completed.
workload experienced by these systems. However [18] and others. Many of these systems focus on
since a central server must have O(N) bandwidth providing a specific desired level of service for
capacity and CPU power, systems that depend on jobs. This contrasts to the approach taken by BON,
central architectures are unscalable [14], [15]. Re- Mosix and others in which processes are migrated
liability is also a concern since a central server is to nodes where they will have the most resources
a single point of failure. BON addresses both of applied to them rather than a specific level of
these issues by using O(logN) maximum commu- resources. The other systems are mostly based on
nications scaling and no single points of failure. DHT architectures and provide for querying based
Furthermore since the networks created by the BON on arbitrary node attributes and link qualities. For
algorithm are random graphs, they will be highly complex distributed applications where each partic-
robust to random failures. ipating node must have a certain level of resources
The Messor project [9] in particular has the same and where the connectivity between the nodes must
goal as BON which is to provide self-organized, have prescribed latencies, these DHT systems will
distributed load balancing. The agent-based design be the most suitable platform. For many types
of Messor also involves performing random walks of parallel scientific computing however, BON’s
onanetworktodistributeload.HoweverBONisde- objective of placing a job where it will finish as
signed specifically to reshape the network structure quickly as possible is appropriate and desirable.
soitcanbeefficientlysampled.Messorwasinspired
by the notion of a swarm of ants that wander around
the network picking up and dropping off load. Thus
it is not clear how long the ant agents will need
to walk while performing the load balancing. It is BON is designed to be deployed on extremely
the focus on topology that distinguishes BON from large ensembles of nodes. This is a major similarity
other similar efforts. BON endeavors to reshape withBOINC [3]. TheEinstein@homeprojectwhich
the network topology to make resource discovery processes gravitation data and Predictor@home
feasible with O(logN) length random walks. A which studies protein-related disease are based on
simplified version of BON can be analyzed and thus BOINC, the latest infrastructure for creating public-
we can put performance bounds on its behavior. resource computing projects. Such projects are
Messor, while very intriguing, provides no analyti- single-purpose and are designed to handle massive,
cal treatment. embarrassingly parallel problems with tens or hun-
Within the large body of research some tech- dreds of thousands of nodes. BON should scale to
niques have been implemented including Mosix, networks of this scale and beyond while providing
Messor, BOINC, Condor, SWORD, Astrolabe, a dynamic, multi-user environment instead of the
INS/Twine, Xenosearch [2], [3], [8], [9], [13], [16]– special purpose environment provided by BOINC.
4
40 1.25
Average Degree
35 Minimum Node Degree
System Load
e 30 System Capacity
gre 25 1 ad
e o
D L
e 20 m
g e
a 15 st
er 0.75 y
v S
A 10
5
0 0.5
0 200 400 600 800 1000 1200 1400
Time
Fig.3. TherelationshipbetweenloadandnodedegreeisthebasisfortheBONalgorithm; anodewithhighin-degreeismorelikelytobe
visited on a random walk and thus more likely to be the recipient of new load than a node with low in-degree. As the total load increases,
hki decreases until the load becomes clipped. In the load clipped regime the algorithm remains the same but the mechanism behind the
performancechangestobecomesaball-binloadbalancingproblemwithlogN choices.Thischangeisduetothefactthatthereisnolonger
a connection between free resources and in-degree.
III. THE BON ARCHITECTURE maximum in-degree, k(max).
i
A. BON Topology
The concept underlying BON is that the load B. BON Algorithm
characteristics of a distributed computing system Each node’s load, s (t), can change as new load
i
can be encoded in the topology of the graph that arrives in the network or when existing work is
connects the computational nodes. done. When new load arrives at v , a short random
i
In schematic terms, an edge in a BON graph walk is initiated to locate a suitable execution site.
represents some unit of unused capacity on the node Contained in this random walk is a BON resource
to which the edge points. Consequently when a discovery message(BRDM) which stores the merit
node’s resources are being exhausted, its in-degree function information for the most capable node
will decline as seen in Fig. I. Conversely when visited so far on the walk. The fact that random
a node’s available resources are increasing, its in- walks will preferentially sample nodes with high-
degree will rise as seen in figure II. degree motivates the mapping of node in-degree to
Formally a BON is a dynamic, directed graph, free resources. The simplest approach to choosing a
D = (E,V), where each node vi ∈ V maintains node on the walk is to select the last node inserted
k(min) ≤ k (t) ≤ k(max) incoming edges. The into the BRDM. This case has been previously
i i
maximum incoming edges that a node can have, explored [6]. While this simple approach can be
k(max), is proportional to the computational power studied analytically, simulation results indicate that
i
of v . Each node, v , has a scalar metric s (t) large improvements to the balancing performance
i i i
which is kept inversely proportional to k (t). As are possible by always keeping the most capable
i
s (t) changes with time, v severs or acquires new node’s information.
i i
incoming links to maintain the relationship. In the Instead of performing a simple random walk and
context of distributed computing, s (t) is a scalar selecting the last node to receive incoming load, the
i
representation of the current load experienced by node on the walk with the largest power per load
node v . This means that each node will endeavor to will be the target (see Algorithm 1). Due to the
i
keep its in-degree proportional to its free resources mapping between load and in-degree this greedy
or inversely proportional to its load. Idle nodes will random walk selects the least loaded node on the
have a relatively large in-degree while overloaded walk to receive new load which is the same as
nodes will have a small in-degree. The total un- choosing the highest degree node when the network
loaded resources of a node are proportional to it’s is not load clipped. This clipping occurs when a
5
node has the minimum allowable in-degree. We are sampled. However prior results [6] show that a
will discuss the case when the network is load modified BON is more amenable to analysis.
clipped in Section V-D. For time sharing systems Rather than selecting the node on the BRDM
the concept of overloading is not well-defined since walk that can process an incoming job the fastest,
anodewithLjobswillapply1/Lofitsresources to one can simply select the last node of the walk. In
each job. In the context of the BON algorithm load this model the average number of absent edges,J, in
clippingsimplymeansthatnodeshavetheminimum the N-node graph is identified as the total number
allowablein-degreeandthusarenolongerbalancing of jobs running. The maximumnumber of incoming
load based on preferential sampling. In practice a edges that a node can have will be called C and
node in the clipped regime will be under very heavy the number of incoming edges to a given node is
computational load, but fundamentally it can still denoted as i. For the case when the average number
accept new jobs. ofjobsremainsconstantwecandescribethissystem
as a simple Markov process with state-dependent
Algorithm 1 PickTarget(source): A new job en- arrival and service rates; it can be denoted by the
tering the network initiates a random walk that standard queueing notation as M/M/∞//M. The
maintain information about the node on the walk arrival rate of new jobs is proportional to the free
where the job would run the most quickly. The resources,i/(NC−J), ofeachnodesincejobsarrive
job is then assigned to that node when the walk preferentially based on in-degree. Assuming that
is complete. jobs terminate uniformly randomly, the departure
.
rateis(C−i)/J.SolvingthebirthanddeathMarkov
1: ttl ← clogN, v ← source, hops ← 0
process we obtain for the degree distribution:
23:: owbhjimleaxh←opsvv.<.LPooatwdt(el)r+(d)1o, target ← v C J n J C−n
p = 1− (1)
4: v ← RandomOutNeighbor(v) n (cid:18)n(cid:19)(cid:20) NC(cid:21) (cid:20)NC(cid:21)
5: objtemp ← vv..LPooawd(e)r+()1 Defining the normalized load as α = J/NC,
6: if objtemp > objmax then
the binomial distribution means that for each node,
7: objmax ← objtemp
each unit of capacity is occupied with probability
8: target ← v
α. If C = N − 1, this model recovers the degree
9: end if
distribution for ER graphs:
10: hops ← hops+1
11: end while N −1 E n E C−n
12: return target pn = 1−
(cid:18) n (cid:19)(cid:20)N(N −1)(cid:21) (cid:20) N(N −1)(cid:21)
(2)
Where E = N(N −1)−J.
IV. ANALYSIS For a non-clipped network with uniform resource
The performance of BON walk selection will be distribution, the variance of the degree distribution
bounded below by the performance of the stan- maps directly onto the variance of the load balanc-
dard walk selection. Therefore although we do not ing. This is because each incoming edge represents
present a calculation of the load distribution for freeresources.Inaperfectlybalancednetwork,each
BON graphs, we can state that it has the same scal- node will have the same free resources. This ideally
ability as the standard walk case described below. balanced network would be a regular graph and
The exact BON algorithm is difficult to analyze, thus the variance of the degree distribution would
howeveritispossibletoplaceboundsonthebalanc- be zero. For the simple case mentioned above the
ing performanceby simplifyingtheload distribution degree distribution is binomial and thus it has a
protocol. We also calculate the bandwidth used by small but non-vanishing variance.
the algorithmand compare it to a centralized model. When the highest-degree (most free resources)
node on a random walk is selected to receive
A. Scalability
incomingload,thatnode’sresourcesmustbegreater
The BON algorithm is difficult to study analyt- than or equal to the resource of the last node on the
ically due to the way in which the random walks walk.
6
In addition to this queueing model, it has been round. For N , ∀i ∈ {1,···,N} the bandwidth
i
shown by information theoretic arguments [6] that consumed will be B(i) = β[A+L] which is O(1).
T
the simplified rewiring protocol described here cre- 2) BON: For the decentralized BON algorithm,
ates ER random graphs. the network topology is now more complex than
for the central server. While the graph of the central
model was a star, BON will look approximately like
B. Communications Complexity
a random regular graph. Initially we will assume
An important metric of performance for dis-
that we begin with a correctly-formed BON. As
tributed computing is the network bandwidth re-
with the central model we assume that Nβ jobs
quired for a protocol. It is clear that the architecture
begin and end at random nodes in each time unit.
that requires the least total bandwidth is a central
Since there is not a central server, each node that
server. However the maximum bandwidth that any
initiates a new job must send a BRDM to find
node must consume in a central system will not
a node to run the job. Every node on the walk
be the least. And while the total consumption of
will need to replace the value of obj = Power() ,
bandwidth is important, the bandwidth that any Load()+1
(L bytes of data), in the BRDM if obj is larger
singlenodeconsumescanbeasignificantbottleneck
than the objective function that is currently in the
for large central networks. Below we compare the
BRDM (see Algorithm 1. Since the random walk
bandwidth required by a centralized algorithm and
will be O(logN) steps long, the total bandwidth
by BON.
of the walk will be B = LlogN. Likewise when
1) Centralized: The simplest non-trivial central- w
a job finishes, another walk will be used to find
ized architecture for a computing network is the
a replacement for the removed edge. Factoring in
case where initially the central node, denoted C,
the cost of transmitting the program to the target
knows the power and load of each of the N nodes
node and needed handshaking protocols, the total
that it controls. When a job on one of the nodes
bandwidth consumed by BON is
completes, that node will notify C so that it can
update its load state information for the network.
B(BON) = Nβ[A+L{logN +2}]. (4)
Obviously C keeps track of assignments of new T
load to each of the nodes. This method does away Therefore we can see that the total bandwidth
with the need to periodically probe every node in cost of BON is O(logN) greater than the cen-
the network, however it is clear that the bandwidth, tral model. However a more important metric in
memory and CPU cost that C has to bear is still many situations will be the maximum bandwidth
O(N). Further assume a steady-state network load consumed by any of the nodes. In BON each node
and that in every time unit, Nβ, jobs begin and the will on consume bandwidth in proportion to how
same number terminate. We further assume that all many jobs it initiates and how powerful it is. Thus
the jobs will start at one of the computational nodes if all of the nodes use the network equally then each
and that they will then be sent to C for assignment. node will consume B(BON)/N bandwidth, which is
T
Now assume that for every job that is started a
logarithmicinthesizeofthenetwork.Thiscontrasts
relatively large A-byte packet, including the size of
to the O(N) bandwidth needs of the central server.
the program code and input data, must be sent from
C to N , i ∈ {1,···,N} and that relatively small
i V. SIMULATIONS
L-byte packets must be sent to the central server in
response to changes in load. Therefore C must send A. Simulation Description
NβA bytes per unit time which consumes kernel For the simulations,each node in a BON network
resourcesandrequiresbandwidththatincreaseswith is a computer with power equal to its maximum
N. The total bandwidth consumed by the entire degree minus its minimum degree, P = k(max) −
i i
centralized network is k(min). One unit of power can process a unit of load
B(C) = Nβ[A+L]. (3) in each unit of time. Jobs run on these computers
T in a time-sharing fashion with each of the L jobs
This is also the same amount that C must con- of a computer equally sharing the node’s power at
sume since it is involved in every communications each time step. The simulations deal only with CPU
7
power as the objective function of the balancing.
LS Fit
Other features such as memory ushering will not 10
be simulated but will be added as features in the
8
reference implementation. Simulations of the BON
r
e
system were performed using the Netmodeler pack- et
m 6
age. Two type of experiments were done. a
Di
The first experiments are very idealized using
4
uniform node power, uniform job arrival rates and
Poisson-distributed job sizes. Equation 5 indicates 2
that all nodes have k(max) = 71, the size of each
1 2 3 4 5 6
job is Poisson distributed and that at each time step
ln(N)/ln(<k>)
β jobs are created. For different simulations β and
ν will have different values in order to show a wide Fig.4. BONgraphsobeytherandomgraphscalingrelationshipbe-
range of system behavior. While this setup is very tweendiameter and average degree: Diameter∝lnN/lnhki.This
BON graph was generated using the uniform simulation parameters
idealized, it might apply to cluster computing.
from Eqn. 5.
1, k = 71
P(k) = ,
(cid:26)0, k 6= 71 completely ordered network with O(N) diameter,
νje−ν the graph will quickly converge to the low-diameter
P (j) = ,
ν structure depicted in these simulations.
!j
1, b = β
P (b) = (5)
β (cid:26)0, b 6= β B. Graph Structure
The idea at the heart of BON is that the graph
The second type of simulation (Eqn. 6) uses
structure can capture the load state of a computa-
power-law distributions for all parameters includ-
tional network. In section IV we discussed prior
ing job arrival rate, node power and job size.
theory results that describe the structure of graphs
This configuration represents a situation where ev-
formed using algorithms similar to BON. We now
ery important system parameter is distributed in a
present simulation results for both the uniform and
bursty, heavy-tailed way. Heavy-tailed distributions
heavy-tailed systems described above. The degree
are common in many real systems [19] includ-
distribution for a balanced overlay network matches
ing networks and thus these simulations provide a
the resources of the constituent nodes for both
fairly realistic idea of how the system will perform
uniform and power-law resource distributions as
under real loads. Most importantly for simulation
seen in Figs. 11 and 12.
performance the computing power ranges from 1
Figs. 4 and 5 show that BONs maintain a low
unit of power to 300 units of power. This is at
diameter and exhibit the property of random graphs
least ten times the range of performance seen in
that the diameter is proportional to lnN/lnhki. The
commonly used CPUs. As we will revisit in the
changes in connectivity can be seen in Fig. 6.
performanceevaluation,havingmanynodesthatcan
It is important that BON graphs remain at least
only accept a few processes prior to being load-
weakly-connected as they evolve. All simulations
clipped will impact the balance distribution simply
indicate that BONs remain weakly-connected, but
due to quantization effects. This issue will have
that when they are load-clipped they can acquire a
design implications for the implementation.
complex strongly-connected structure. As the load
P ∝ k−1, k(min) +1 ≤ k ≤ 304, surpasses 1 (the clipping threshold) and the network
k
P ∝ j−1, 32 ≤ j ≤ 1024, becomes a k(min)-regular graph, the number of
j
P ∝ b−1, 1 ≤ b ≤ b (6) strongly-connectedcomponents(SCC)increases.As
b max shown in Fig. 7, the number of SCCs falls back
In all of these simulations we begin with a to unity when an overloaded network becomes less
randomly-connected network subject to the initial loaded. It is also important to note that the SCCs in
degree distribution. However if one starts with a an overloaded BON can change due to rewiring of
8
the network. So while every node will not be able to
2000
communicate with every other node at each instant
of time, the out-component of each node in the
graph can change with time. Also the network does 1500
x
remain weakly-connected even when the network e
d
has many SCCs. e In 1000
d
o
N
C. Load Balancing Performance 500
When discussing load balancing performance we
want metrics which measure how closely load fol-
0
lows capacity. When all the nodes are equally capa-
ble, standard deviation is a convenient measure of
balancing, when nodes are heterogeneous, correla- 35
tion coefficient is what we use.
1) Simple Idealized System: For the uniform
25
simulation model, Fig. 8 shows that the ensemble >
k
<
standard deviation of the node load is just below
15
1% when the network is in the under-loaded regime.
When the network is clipped the standard deviation
of the load is slightly higher than in the under- 5
loaded regime but still quite close to 1%. This dif- 0 500 1000 1500 2000
ference in performance is likelydue to the transition Time
from the degree-correlated load-balancing that is in
effect when the network is under-loaded to the ball- Fig. 6. As a BON network evolves there is significant turnover in
connections.Inthetopsub-figurethein-degreesofanarbitrarynode,
bin load balancing that takes over when the network
v0,aredepictedasafunctionoftime.Averticalline,t=a,intersects
is clipped. zeroormorepoints,A0,whichisthesetofnodesthathavedirected
Another important measure of performance is edges incident to v0. This illustrates that the structure of the graph
changes significantly even when the macroscopic properties such as
how well BON performs in comparison to a central
average degree are not changing.
system that places new jobs at the least loaded node
in the network. In the uniform configuration after
1000 iterations the central system has completed
501314 jobs compared with 501238 jobs being
completed by BON. This indicates that BON’s job
throughput is only about 0.01% worse than the
12
LS Fit optimal schedule.
11
10 2) Power-Law System: The power-law simula-
9 tions illustrate an important design criterion for
r
e
et 8 practical implementations. For these simulations the
m
a 7 power distribution of the nodes is a power-law
Di
6 given in Eqn. 6. The minimum power is 1 and the
5 maximum power is 300. Therefore there are many
4 nodes that have very low power resources. This
3 means that for many values of the load it will be
1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
impossible to get close to optimal balancing. For
ln(N)/ln(<k>)
this reason the correlation between degree and free
Fig.5. Onecanseethatthissystemobeystherandomgraphscaling resources is used to evaluate performance as shown
relationship between diameter and average degree: Diameter ∝ in Fig. 10. A good example is a node with P = 2.
lnN/lnhki. This BON graph was generated using the power-law
Because the load is defined to be P/L, where L is
simulation parameters from Eqn..6
the number of running jobs, the load is limited to be
9
1.25 0.03
1
Load 0 0.7.55 ad Dev. 0.02
0.25 LoStd. 0.01
0
15 0
4
C
10
C
S
5 ad ean
oM 2
L
1
12 0
er 0 500 1000 1500 2000
met 8 Time
a
Di
4 Fig. 8. This uniform resource, 2048-node BON under increasing
overload also shows that the standard deviation of the load is low
0 at about 1%. The difference in performance as the network enters
the clipped regime can alsobe seen. At the clipping transition point
45 the standard deviation experiences a spike which is likely due to
jobs accumulating in a small SCC before additional rewiring can
> 30 rebalance the load. After a short time this load imbalance dissipates
k
< astherewiringallowsloadtobedistributedthroughout thenetwork.
15
0
1000 2000 3000 4000 5000 tions the central system has completed 585872 jobs
Time compared with 585788 jobs being completed by
BON. As with the uniform configuration, BON’s
job throughput in the heavy-tailed configuration is
Fig. 7. In simulation we observe that BONs are always at least
only about 0.01% worse than the optimal schedule.
weakly connected directed random graphs. This 2048-node BON
with heavy-tailed parameters remains weakly connected even in the Please note that this result ignores the effects of
clippedregime.Thenumberofstrongly-connectedcomponents(SCC) job distribution latency on total throughput but it
does increase under heavy load, but the number of SCCs returns to
indicates that job placement is very close to optimal
unity when the clipping condition passes.
when communications delays are ignored.
D. Ball-Bin Regime
non-negative integer multiples of 1/2. Thus if the
network is 75% loaded then this low-powered node Every node in the graph must maintain a mini-
is equally unbalanced whether one or two jobs are mum degree to ensure that the graph stays at least
running. By selecting a suitable minimum power, weakly-connected. For these experiments each node
one can bound this finite size effect. For example maintains at least 4 incoming edges which means
if the least powerful node has P = 5 then it can thatifthenetwork’sloadbecomesclippedthenthere
always get within 10% of the optimal value. This is no longer a correlation between a node’s degree
finite size effect appears as cyclical behavior of the and its resources. For this reason the real metricthat
load standard deviation and can clearly be seen in is sampled on the walk is the amount of computing
Fig. 9. power that the next incoming process can expect on
As was done for the uniform simulation configu- a given node. When the network is not clipped this
ration, we compare centrally scheduled job through- is the same as choosing the highest-degree node on
put to BON throughput with the same load trace. the walk. However for a clipped network it selects
In the heavy-tailed configuration, after 1000 itera- thenodeonthewalk that has thelargest valueofthe
10
1
0.999
r2 0.998
0.997
0 500 1000 1500 2000
d 900 t r2=0.9975
a
o
L
600 r2=0.9981
2
300 r =0.9979
2
r =0.9996
0
0 100 200 300
Power
Fig.10. Forthecasewherewedohavemany nodes withalow maximumdegree thecorrelationbetweennode power in-degreeisamore
appropriate measure of performance than standard deviation. and load. For a network that is getting increasingly loaded the load vs. power
is plotted at instants and the correlation is also calculated. Even when the network is 3X the load clipping threshold, r2 >0.99.
1
T=20
0.75 T=100
T=200
)
(k 0.5 T=300
P
T=400
0.25 T=500
0
5 15 25 35 45 55 65 75 85 95
k
0.75
<k>
60 s
k
45 0.5
>
k k
< 30 s
0.25
15
0 0
0 100 200 300 400 500 600 700
Time
Fig. 11. When all nodes have the same resources, BONs will be approximately regular graphs. These degree distribution snapshots (top
graph) of an evolving network that is getting progressively more loaded show the regular nature of the graph over a wide range of load
conditions.
expected power for the next incoming job as shown
in algorithm 1. Now considerthat a clipped network
hki > k(min), preferential sampling
is approximately a regular random graph and thus (7)
(cid:26)hki = k(min), ball-bin sampling
a short random walk will sample uniformly from
the nodes in the network. This problem now shows In ball-bin systems a ball is uniformly randomly
itself to be very similar to ball-bin load balancing assigned to one of N bins. As this process is
[1], [20]. repeated a distribution of bin population emerges