Virtual Network Migration on the GENI Wide-Area SDN-Enabled Infrastructure

1.1 MB·

by  Yimeng Zhao

Virtual Network Migration on the GENI Wide-Area
SDN-Enabled Infrastructure
Yimeng Zhao, Samantha Lo, Ellen Zegura, Mostafa Ammar Niky Riga
School of Computer Science Raytheon BBN Technologies
Georgia Institute of Technology Cambridge, MA, USA
Email: {yzhao389,samantha,ewz,ammar}@cc.gatech.edu Email: [email protected]
Abstract—A virtual network (VN) contains a collection of vir- ResearchintoVNmigrationpolicyismotivatedbyspecificob-
tualnodesandlinksassignedtounderlyingphysicalresourcesin jectives. These have included: efficient utilization of dynamic
7 anetworksubstrate.VNmigrationistheprocessofremappinga
resources [2], [3], recovery from failure [4], [5], defending
1 VNslogicaltopologytoanewsetofphysicalresourcestoprovide
against attacks [6], and reducing energy consumption [7].
0 failure recovery, energy savings, or defense against attack. Pro-
2 vidingVNmigrationthatistransparenttorunningapplicationsis
asignificantchallenge.Efficientmigrationmechanismsarehighly Our focus in this paper is on VN migration mechanisms.
n
a dependent on the technology deployed in the physical substrate. Thedevelopmentofsuchmechanismscanbequitechallenging
Prior work has considered migration in data centers and in the
J because of the desire to make them transparent or seamless
PlanetLab infrastructure. However, there has been little effort
6 – informally, with minimal impact on running applications. A
targeting an SDN-enabled wide-area networking environment –
animportantbuildingblockoffuturenetworkinginfrastructure. further challenge is that a VN migration mechanism is highly
]
I In this work, we are interested in the design, implementation dependent on the technology deployed in the substrate. There
N and evaluation of VN migration in GENI as a working example is, therefore, no generic mechanism that can be used univer-
of such a future network. We identify and propose techniques
. sally. Previous research has developed mechanisms for VN
s to address key challenges: the dynamic allocation of resources
c migrationindifferentenvironments.Inthedatacentercontext,
during migration, managing hosts connected to the VN, and
[ flow table migration sequences to minimize packet loss. We Ghorbani et al. develop migration methods within their LIME
1 find that GENI’s virtualization architecture makes transparent architecturethatprovablymeetatransparencydefinitionbased
v and efficient migration challenging. We suggest alternatives that onvalidbehaviorsinamigration-freesetting[8].Inthewide-
2 mightbeadoptedinGENIandareworthyofadoptionbyvirtual area context, Lo et al. [9] develop a tool for VN migration in
0 network providers to facilitate migration. PlanetLab [10], a well-known shared infrastructure used for
7
network experimentation. Their PL-VNM tool implements a
1 I. INTRODUCTION
migration schedule heuristic that minimizes packet loss under
0
. Virtualization is well-recognized as a technique to share ideal conditions, but in practice cannot ensure zero loss, due
1
physical resources, providing the appearance of dedicated in part to coarse timing control in PlanetLab.
0
resources and isolation from others sharing the same physi-
7
1 cal resources. Virtual networks run over a physical network In this paper we focus on developing VN migration mech-
: substrate, with an allocation of physical network resources anisms for GENI, a recently developed infrastructure for
v
i (e.g., routers, switches, links, paths, or portions thereof) to sharing wide-area network resources [11]. A key technol-
X
the virtual network. A virtual network (VN) thus contains a ogy included in GENI is software-defined networking (SDN)
r collectionofvirtualnodesandvirtuallinksassignedtoasubset wherethepacket-processingrulesinswitchesareinstalledand
a
of the underlying physical resources. A virtual link spans one modified from a logically-centralized controller [12]. SDNs
or more physical links in the substrate, and a substrate node offer a number of advantages including ease of network
can host multiple virtual nodes. managementandtheopportunityforincreasedinnovation.Our
Network virtualization allows significant flexibility in net- focus on GENI is motivated by the fact that SDN-enabled
work operation. Most important are the flexibility in the VN’s wide-areanetworksarelikelytobecomeanimportantbuilding
placement (the specific mapping of VNs elements to substrate block of future networking and GENI represents a fully-
resources [1]) and VN agility (the ability to remap the VN to functionalinstantiationofthistechnology.Assuch,techniques
a different set of substrate resources over time). Our interest developed for GENI will have wider applicability. Further,
in this paper is on enabling VN agility through VN migration because SDN technology is at a stage where its future can
mechanisms. This refers to the process of remapping some be influenced, lessons we learn about the capability of such
or all of a VN’s logical topology to a new set of physical technology in supporting network agility can have significant
resources. value on future developments.
VNmigrationresearchconsidersbothpolicy,whenandwhy
a VN is migrated, and mechanism, how a VN is migrated. OurworkfocusesonmigratinganentireVNfromtheinitial
TABLE I: GENI Context vs. Virtual Components
Component GENIContext
Substratenetworks GENItestbed
VirtualNetwork(s) GENIslice
Physicallocation GENIaggregate
VirtuallinkswithinaVN LANs
VirtuallinksbetweenVNs SharedVLAN
Virtuallinksconnectingdifferentphysicallocations Stitchedlinks
MappingbetweenVNtophysicalsubstrate Rspecfile
to their mitigation.
The remainder of the paper is structured as follows. In
Section II we develop a framework for enabling VN agility
within the context of the GENI substrate technology. We
Fig.1:VNmigrationprocessfromVN1toVN2:Instep1,setupVN1
andconnectvirtualswitchesinVN1totheSDNcontroller.Instep2, highlight a decision regarding VN migration related to the
setupVN2andconnectvirtualswitchesinVN2totheSDNcontroller. allocation of resources in or across GENI slices. Section III
In step 3, the migration controller clones flow tables from VN1 to proposes mechanisms to address the challenges associated
VN2 based on the mapping. In step 4, connect VN2 with the hosts with VN migration on GENI to meet the goals of efficiently
and disconnect VN1.
and transparency and our solutions. We develop a controller
placement to the final placement, without moving the hosts 1. architectureanddescribethedeploymentofourVNmigration
Figure 1 illustrates the migration steps assuming an SDN- mechanism within GENI in Section IV. In Section V we
enabled infrastructure. A migration controller interacts with present results from experiments conducted using our proto-
the SDN controller to initialize and schedule the migration type with the aim of evaluating its performance. Section VI
process. Prior to migration, virtual switches on the VN1 discusses GENI limitations that were exposed by our research
are controlled by the client application running on the SDN and approaches to address them. Related Work is covered in
controller,andVN1isusedtodelivertraffic(Step1).Whenthe Section VII. We conclude the paper in Section VIII.
migration starts, VN2 is setup (Step 2) and flow tables on the
II. ENABLINGFULLVNAGILITYINGENI
virtual switches in VN1 are cloned to the virtual switches in
VN2 based on the mapping (Step 3). The migration controller GENI is relatively mature shared infrastructure. As such
issuescommandstoreconnecthostsfromVN1toVN2inStep GENI provides support for for sharing, isolation, resource
4 and to disconnect VN1. allocation, and multiple physical substrate “owners”. GENI,
This paper addresses several challenges in realizing the however, was not designed specifically to support our desired
basic VN migration steps above. In addressing these chal- transparent and efficient VN agility. Our work, therefore,
lenges we make the following contributions: (1) We develop involves the development and evaluation of options to support
approaches that enable VN agility in the SDN-enabled GENI VN agility within GENI. This section explores some critical
infrastructure; (2) Develop and evaluate options for dealing aspects of providing agility support. Because GENI uses its
withthedynamicallocationofresourcesinherentinmigration, own unique terminology, Table I summarizes how the general
where the initial mapping to physical resources is known but VN terminology maps to GENI terms.
the future mappings necessitated by migration may not be;
(3) Propose an approach for managing the hosts that connect A. Allocating VNs to Slice(s)
to the VN and will remain in place when the VN migrates; GENI is a “sliced” platform that allows concurrent ex-
(4) Develop techniques to mitigate the disruption caused by periments on shared infrastructure. A GENI slice is a unit
live VN migration, to minimize packet loss observed by the that contains all the resources for an experiment, including
application in the data plane and to maintain the topological computingresourcesandnetworklinks.Thisisalreadyaform
view in the control plane (as observed by the application ofnetworkvirtualizationusedprimarilytoisolateexperiments
running on the SDN controller). We carefully manage process inGENI.Inareal-worldGENI-likesubstrate,slicingwouldbe
steps and flow table migration sequences to achieve this; (5) usedtoisolatecommercialnetworkproviderssharingthesame
Evaluate, using an implementation running on GENI, how the physicalsubstrate.Slicesexistinoneormoreaggregates;each
performance of live VN migrations as a function of design aggregate is an independent collection of physical resources
decisions and network parameters; and (6) Expose some often operated by the same entity (e.g., a specific university).
limitationsoftheGENIinfrastructureandproposeapproaches Figure 2 shows two slices on the GENI infrastructure. Slice 1
has three virtual nodes in Aggregate A, while Slice 2 has six
1 Notethatthismodelforhostsdiffersfromthedatacentercontextwhere virtual nodes across Aggregate A and Aggregate B connected
hostsaremigratedwiththeVN.Insharedwide-areainfrastructure,weassume
withstitchedlinkthroughthebackbonenetwork.Eachsliceis
thehostsarecustomer-premiseequipmentandremaininplacewhentheVN
moves. an isolated environment where virtual nodes and virtual links
translated to a request Rspec file by Rspec generation tools,
and GENI aggregates automatically allocate resources to the
requested VN based on the Rspec file.
WhileGENIaggregate’sautomaticassignmentofresources
can meet the requirements of most experiments, it may be
necessary to have the flexibility of mapping virtual nodes to
specific physical resources for VN migration research. Al-
thoughRspecgeneratingtoolsdonotdirectlysupportresource
assignment, we are able to map a virtual node to a specific
Fig. 2: An example of GENI architecture machinebymanuallymodifyingtherequestRspec.TheOmni
tool [13] provides commands to obtain information about
can be added. Each virtual node in a slice can be accessed by all reserved and available resources at a specific aggregate,
the user who creates the slice with corresponding SSH key. including machine ID, machine status, hardware types, and
Slices are meant to be deployed for the long term and OS types. We can locate the component name for a specific
are thus not agile. To enable agility, VNs will need to be physical machine in the resource information returned by
deployed within slices as an additional layer of virtualization. Omni and copy its component ID to the request Rspec file
We consider two options for mapping VNs to slices with to achieve a fixed mapping.
an eye to migration. The first option is to build all VNs
(original and future) and hosts for migration within the same C. Assigning VNs to Substrates
slice. This approach follows the common usage model for In VN migration, it might be necessary to migrate be-
GENI to include all resources for an experiment within a tween different physical substrates, or aggregates in GENI
single slice. However, this option has three disadvantages: 1) terminology. A GENI aggregate comprises a set of resources
ThereisnoclearisolationbetweenthedifferentVNs.2)Most under common control including storage resources, comput-
GENIresourcescannotbemodifiedafterthereservation.Once ing nodes, and OpenFlow switches [11]. Experimenters can
resources are reserved on a slice, no partial modification (e.g, reserve multiple aggregates within the same slice and connect
add a virtual link or a virtual node) is allowed. In the case of them with stitched links (See Figure 2). It is also possible to
migration, this restriction requires us to reserve all resources allocate each VN to a different aggregate and connect them
for hosts and VNs, including those that will be migrated to withbothsharedVLANandstitchedlinks.Wewillshowhow
in the future, at the outset. 3) When a VN or a host fails, we tousesharedVLANandstitchedlinkstogetherinSectionIII.
need to rebuild the whole virtual topology.
Alternatively, it is possible to allocate a single VN to a III. DEALINGWITHVNMIGRATIONCHALLENGES
slice, starting with the original VN and later allocating a VN After VN agility is enabled in GENI as discussed in the
tomigrateto.DeployingaVNononesliceisstraightforward. previous section, we are still faced with several challenges as
The challenge for deploying and migrating a VN between we strive to meet the goals of efficiency and transparency.
two slices is caused by the difficulty to enable the inter-slice In this section we investigate three distinct challenges and
communication during migration. We cannot create a virtual propose mechanisms to deal with them. The challenges are:
link to connect virtual components in different slices directly. (1)howtomanageinter-slicecommunicationthatconnectsthe
Instead,wecansetupaVLAN,abroadcastdomainatthedata hosts to both VNs temporarily during the migration; (2) how
link layer, to connect virtual components in different slices. tominimizepacketlossbyschedulingtheflowtablemigration
All virtual components in the same VLAN will receive the sequence;and(3)howtoprovideaseamlessinterfacetoSDN
broadcasting packets even when the virtual components are in applications during and after migration. The first challenge is
separateslices.ComparedwithdeployingallVNsononeslice, specific to a sliced platform like GENI and the other two
thisseconddesignprovidesclearseparationbetweenVNsand challenges can be generalized to other SDN environment.
gives more flexibility in resource reservation. We can reserve In the section following this one we use the solutions for
one VN first and create another VN when needed. However, each challenge to inform the design of a migration controller
it complicates the virtual topology during migration. We will architecture.
talkfurtheraboutsharedVLANsandmigrationinSectionIII.
A. Inter-slice Connection
B. Mapping Virtual Switches to Physical Machines
WedescribedtwoVN-to-SliceallocationoptionsinSection
GENI uses a Resource Specification (RSpec) document to II-A. In the first option, all hosts, the old VN, and the new
describearequestedvirtualtopologyanditsphysicalsubstrate, VN are located within the same slice. We will not discuss
including ID of the virtual machines (VM), virtual network the first design in detail since it follows the common usage
interface configuration, VLAN tags assigned to links, and of the GENI testbed. We will focus on the second design,
corresponding hardware information. The RSpec is submitted where the old VN, the new VN, and the hosts are assigned
with the GENI aggregate manager API to aggregates in to three different slices. The challenge in the second design
order to reserve resources. The requested virtual topology is is to direct traffic from one slice to another given the current
GENI constraints which do not support virtual links between
slices.Itshouldbenotedthatthedynamictunnelimplemented
in LIME [8] does not apply for our case. The tunnel uses the
controlplanefordatalinksandcannotguaranteeperformance
such as bandwidth and latency. Moreover, the control plane is
a shared channel on GENI and should not be used to send a
large amount of data.
1) Broadcasting problem in a virtualization environment:
To enable inter-slice communication, it may seem natural to
use a shared VLAN to connect a host slice to VN slices. The
trafficisbroadcastwithinthesameVLAN,nomatterinwhich
slice a switch is located. The connection/disconnection of the
VNs with the hosts are controlled by turning up/down the
network interfaces on virtual switches. Figure 3 presents an
Fig. 4: an example of a VN and its substrate
example of this approach. The topology includes three hosts,
the old VN (VN1), the new VN (VN2), and the controller
VN1 with the hosts by turning up the network interfaces eth1
slice. In our virtual topology, each host connects to both VN1
and eth2 and disconnect VN2 by turning down the network
and VN2 with a shared VLAN. When host1 sends data to
interfaces eth1’ and eth2’. The data from host1 to host2 is
host2, the data will be broadcast to both OVS1 and OVS1’.
broadcast by Rack SW1 to eth1 and eth2, and then broadcast
WhenVN1isinuse,thenetworkinterfacesofOVS1isupand
byRack SW2toeth2’andhost2.Althoughweturndownthe
thenetworkinterfacesofOVS1’isdown.Afterthemigration,
virtual network interface eth1’ and eth2’, a small number of
we redirect traffic from VN1 to VN2 by turning down the
packets are still stored in the XEN domain 0.
interfaces of OVS1 and turning up the interfaces of OVS1’.
During the migration, we switch from VN1 to VN2 by
turning up eth1’ and eth2’ and turning down eth1 and
eth2. Previously buffered packets in domain 0 are transferred
througheth1’andeth2’tothevirtualmachinethathostsSW2.
Thesepacketshavethesamematchingfields(e.g,samesource
and destination IP) but request different actions(e.g, send
through different ports). In the standard SDN implementation
of the learning switch, this is considered as an error. The
switchwillinstallrulestodropallpacketsforseveralseconds,
resulting in a much longer disconnection time than normal
migration process. In the worst case, when conflicting rules
Fig. 3: An example of using shared VLAN to connect the host and
are installed on the openflow switch, the switch may stop
the VNs
forwarding packets, which requires manual configuration to
Unfortunately, this approach can violate the correctness
recover.
of the migration in a virtualized environment. GENI uses
XEN [14] as a virtual machine monitor to allow multiple 2) Mitigatethebroadcastingproblem–gatewaydesign: To
virtual machines to share the same hardware resources. Xen avoid the broadcasting problem, we propose a gateway design
only allows a privileged virtual machine called domain 0 to which establishes additional SDN switches as ‘gateways’ to
access the physical Network Interface Card (NIC). Domain switch traffic from the old VN to the new VN. The gateways
0 communicates with other virtual machines through a set are layer 2 devices that sit between hosts and VNs, hiding
of back-end interfaces. All the packets destined to a virtual changesinVNsfromendhosts.Figure5presentsanexample
machinewillbefirsttransferredtodomain0andthendestined of the gateway design that enables migration within the same
tothevirtualmachine.Thepacketsstoredindomain0arenot aggregate. Each host is connected with a gateway, and each
dropped when the network interfaces in the virtual machine gateway uses two different shared VLANs to connect to the
is turned down. When the virtual network interface goes up two VNs. The gateway switch is responsible for forwarding
again, these buffered packets will be copied from domain 0 packets from hosts to a certain VN. In the process of VN
memory to the receiver virtual machine’s memory. migration, the migration controller issues commands to the
Weillustratewhythissmallnumberofbufferedpacketscan gateway switches, asking them to redirect traffic from VN1
be a problem through a one-node VN example as shown in to VN2 after all flow tables in VN1 are cloned to VN2. The
Figure 4. In our virtual topology, host1 connects two switches controller sends SDN commands to the gateway switches to
with shared VLAN1 and host2 connects two switches with update the flow tables, redirecting traffic from VN1 to VN2.
shared VLAN2. In actual substrate network, there is a rack The gateway design can be extended to enable migration
switch residing in the shared VLAN to broadcast packets to across substrates. As mentioned earlier, we use GENI ag-
allswitchesinthesameVLAN.Beforemigration,weconnect gregates to represent substrates in different locations. Virtual
Fig. 5: Gateway design
Fig. 7: The topology of two-node VN on GENI
Before migration, GW1 directs f from in-port 1 to out-port
1,2
2, directs f from in-port 2 to in-port 1, and drops any traffic
2,1
fromin-port3todisconnectVN2.ThesameappliesforGW2
to control traffic from/to host2. When the migration begins,
our migration controller issues commands to GW1 and GW2
and updates their flow tables to redirect traffic from VN1 to
VN2. We assume GW1 finishes update at time t and GW2
1,2
finishes at time t . We define d as the latency from GW1
2,1 1
to GW2 and d as the latency from GW2 to GW1. The data
2
rate of f is r and the data rate of f is r . We therefore
1,2 1 2,1 2
calculate the number of dropped data c for f and c for
1,2 1,2 2,1
f as follows:
Fig. 6: Topology to enable cross-aggregate migration 2,1
components in different aggregates are connected with a (cid:40)
speciallinkcalledstitchedlinks.Unfortunately,astitchedlink c = (t2,1−t1,2)−d1)×r1, if t2,1−t1,2 ≥d1
1,2
cannot be part of a shared VLAN. We use additional nodes to (t −t )+d )×r , otherwise
1,2 2,1 1 1
serveasabridgetoconnectstitchedlinksandsharedVLANs.
A cross-aggregates example is shown in Figure 6. The hosts,
VN1,andVN2arelocatedinthreedifferentaggregates.GENI (cid:40)
(t −t )−d )×r , if t −t ≥d
does not provide inter-slice stitched links to connect gateway c2,1 = 1,2 2,1 2 2 1,2 2,1 2
(t −t )+d )×r , otherwise
switches in the host slice with SDN switches in two VNs 2,1 1,2 2 2
directly. To connect gateways with VN1, we put three more
It is obvious that t −t =d (d >=0) and t −t =
additional nodes in the host slice. These three nodes are in 2,1 1,2 1 1 1,2 2,1
d (d >= 0) cannot be both satisfied. At least one of c
the same aggregate with VN1 and we use the stitched link to 2 2 1,2
and c is larger than 0, which means additional packet loss
connect them with the gateway switches. Then we use shared 2,1
is unavoidable in this setting.
VLANs to connect those three nodes to the virtual switches
1) Flow Migration Sequence: SDN shows promise to en-
in VN1. Those three additional nodes serve as a bridge to
able lossless migration with an optimized sequence of rule
connect the host slice with VN1. We do the same to connect
installation. We propose a scheduling sequence to remove the
the hosts with VN2.
additionalpacketdropintroducedbyVNmigration.Algorithm
B. Minimizing Packet Loss 1 shows pseudocode for the traffic redirection process. We
In our migration mechanism, packet loss may occur when install rules to let traffic coming from the new VN to go
themigrationcontrollerissuescommandstogatewayswitches through the gateway switches. Then we update rules on
to disconnect the old VN and reconnect the new VN. In a tra- gateway switches to direct traffic from hosts to the new VN.
ditional network without SDN features, unicast Reverse Path Finally, we insert drop rules to disconnect the old VN. By
Forwarding (uRPF) [15] in strict mode drops traffic received following this sequence, we avoid dropping packets buffered
onaninterfacethatisnotusedtoforwardthereturntraffic.We in the old VN.
illustratewhyVNmigrationalwaysintroducespacketslossin Algorithm1alsoappliestopartialVNmigrationwhenonly
symmetric routing through a two-node topology. partoftheoldVNisremappedtodifferentphysicalmachines.
In Figure 7, there are two hosts and two VNs, each VN In the partial VN migration, the traffic redirection occurs at
containingtwovirtualnodes.EachhostconnectstobothVN1 the neighboring nodes of the partial network instead of the
andVN2throughagatewayswitch.Wedefinef asthetraffic gateways. In this case, all neighboring nodes of the partial
1,2
flowfromhost1tohost2,andf asthetrafficflowfromhost2 network should be treated as gateway switches and the same
2,1
to host1. We migrate the virtual network from VN1 to VN2. algorithm can be applied to minimize the packet loss.
Algorithm 1 Traffic Redirection Algorithm
1: for gateway ∈gatewayList do
2: Portsh ←Ports on gateway that point to hosts
3: for Port∈Portsh do
4: installnewruler wherer.inPort=PortToVN2
and r.outPort = Port
5: for gateway ∈gatewayList do
6: for Port∈Portsh do
7: updateruler setr.outPort=PortToVN2where
r.outPort = PortToVN1 and r.inPort = Port
8: for gateway ∈gatewayList do
9: for Port∈Portsh do
10: update rule r set r.action = dropPkt where
r.outPort = PortToVN1 and r.inPort = Port
Fig. 8: Migration controller architecture
2) Remote Scheduling Methods: We have two implemen- the flow tables contain the port information, our migration
tations for issuing migration commands to disconnect the old controller modifies the flow tables based on the port mapping
VNandconnectthenewVN.ThefirstoptionistocontrolVN whenitclonesflowtablesfromoldswitchestonewswitches.
connection by turning up/down network interfaces using SSH
sessions. We refer this type of scheduling as SSH scheduling. IV. MIGRATIONCONTROLLERARCHITECTURE
With this scheduling method, there is a lag between the
The migration controller stands in the center of our migra-
time when the command is issued and the time when the
tion architecture and is responsible for the migration process.
command is actually executed in the remote node. Besides,
It clones the flow tables from the old switches to the new
GENIrequiresSSHkey-basedauthenticationbeforeexecuting
switches, schedules the migration sequences and switches
commands on a node, which might lead to a longer lag time.
traffic between VNs. We implement our migration controller
The second implementation, called OpenFlow-message
on GENI using the POX controller platform [16]. The mi-
scheduling, redirects traffic by installing flows on gateways
gration controller runs on the POX controller while other
based on the OpenFlow messages from the migration con-
client applications keep operating normally. The controller
troller. This method does not support complicated operations
architecture is shown in Figure 8.
such as executing a monitoring script. We expect this method
tobefasterthanSSHschedulingbecauseitdoesnotintroduce Mapping Module: specifies how to map the switches in
authentication overhead. the old VN to the switches in the new VN. It also includes
mapping of the virtual network interfaces in the old switches
C. Seamless Migration and in the new switches. When reserving resources on GENI,
Our migration mechanism should ensure the illusion of a we cannot specify virtual network interfaces in the request
consistent VN for the client SDN applications during and Rspec file and GENI aggregate arbitrarily assigns virtual
after the migration. We discuss possible inconsistencies in the network interfaces to VMs. We need to query the virtual
migration process and our solutions. network interface corresponding to a certain IP address and
1) TopologyChanges: AsmentionedinSectionIII,boththe store that information in the Mapping Module.
old and the new VN are connected during the migration. To Flow Table Manager: When a request for VN migration
present the client SDN applications with a consistent view of is initiated, the Flow Table Manager polls switches that are
a single VN, our migration controller intercepts all OpenFlow affected by migration, translates flow tables from the old VN
events, changes the datapath ID of the events based on the based on the mapping information stored in the Mapping
mapping between the old and the new VN, and passes the Module, and installs the flows into the new switches.
modified events to the client SDN applications. No events Scheduler:calculatesthesequenceofruleinstallationbased
about the topology changes in the VN2 should be passed to onourtrafficredirectionalgorithmtominimizethepacketloss.
the client SDN applications. Traffic Redirector: After all flows are successfully in-
2) Switch State Changes: A switch maintains information stalled,theFlowTableManagernotifiestheTrafficRedirector
aboutitsportsandflowtables.Ideally,aswitchshouldpresent togeneratetrafficredirectioncommands.ThetrafficRedirector
the same switch Information including datapath ID, serial retrieves the sequence of rule installation from the Scheduler
number and ports in the old and the new VN. Unfortunately, and redirects the traffic from the old VN to the new VN.
GENIdoesnotallowuserstoassignvirtualnetworkinterfaces VN Presenter: intercepts events from switches, translates
to the virtual switch and the ports number are randomly them based on mapping information from the Mapping Mod-
assigned during reservation stage. It is highly likely that the ule, and presents a consistent VN topology to client applica-
virtual switch has different ports status in the new VN. Since tions. This module hides all migration process from clients.
Fig. 9: CDF of time difference between command issued and execu- Fig. 10: Migration time as flow table size per switch grows
tion
rules.Themigrationtimedependsonthenumberofrulesand
Status Monitor: collects dynamic network statistics and
the number of switches but is independent of the topology.
decides where and when to migrate based on the VN place-
ment algorithm. Our focus is to migration mechanisms, thus
we have not implemented the Status Monitor.
B. Packet Loss During Migration
Migration API: Migration APIs are similar to OpenFlow
controller APIs so that client applications adapt to the new 1) Move a Complete VN within a Substrate: We build a
APIs easily. The migration APIs allow client SDN applica-
prototype of the basic migration controller using the POX
tions to configure migration parameters such as migration
controller platform [16] and evaluate its performance through
destinations and triggering requirements. The client SDN
experiments on the topology illustrated in Figure 5 with
applicationsshouldusemigrationAPItoretrievevirtualswitch
three hosts and six virtual switches. All virtual switches are
information, the connections to virtual switches, and events
Open vSwitches [17]. We use iperf to generate UDP traffic
from virtual switches to get a consistent view of the VN.
for 10 seconds between all pairs of hosts and migrate VN
from its initial position to final position at time t=5s. We
V. PERFORMANCEEVALUATION
vary the data sending rate to see whether our migration
In this section, we evaluate the performance of our migra-
controller works well in relatively high data rate. We perform
tion mechanism in terms of the migration time, packet loss
three sets of experiments: (a) a baseline experiment where
during migration, latency, and the controller overhead. More
no migration occurs, (b) migration with symmetric routing,
evaluationresultscanbefoundintheaccompanyingtechnical
where traffic redirection commands are issued at the same
report [].
timebycontroller,and(c)migrationwithasymmetricrouting,
A. Migration Time wheretrafficredirectioncommandsareissuedinanoptimized
sequence.Werepeattheexperimentsfor30timesforeachdata
1) SSH vs. OpenFlow-messaging: We evaluate whether
rate and measure the migration time and data loss rate.
OpenFlow-message scheduling performs better than SSH
scheduling in terms of the time difference between com- We only present results for forwarding and reverse flows
mand issue and command execution. We use our migration between host1 and host3 due to space constraints. In Fig-
controller to issue command to turn down/up a network ure 11, the percentage of average packet loss on y-axis is
interface using SSH session for 50 times. We repeat the based on the measurement of UDP traffic for 10s, and the
sameexperimentswithOpenFlow-messagescheduling.Figure x-axis shows baseline experiment, symmetric routing, and
9 shows the CDF of the time difference between the time asymmetric routing for different data sending rates. For both
when migration commands are issued by the controller and the forwarding and the reserve flows, the packet loss rate in
the time when the migration finishes with SSH scheduling asymmetricroutingisalmostthesamewiththatinamigration-
and OpenFlow-message scheduling. All OpenFlow-message free setting. It demonstrate that asymmetric routing prevents
scheduling completes within 0.1 second, but about 50% of hosts from experiencing significant increase in packet loss
SSH scheduling takes 1s or longer to finish. This confirms during migration.
our earlier assessment that OpenFlow-Message scheduling is 2) ImpactofRTT: Duringthemigration,packetlossoccurs
much faster and has lower variance than SSH scheduling. when packets buffered in the old VN are dropped by gateway
2) Migration Duration: Figure 10 shows how migration switchesbecauseofthetrafficredirection.AsshowninFigure
timechangesastheflowtablesizegrowswith95%confidence 12, the performance of the symmetric routing is much worse
level upper and lower bands. The migration time is negligible than that of the asymmetric routing, especially when the flow
whenflowtablesizeissmall.Ittakeslessthan1stofinishthe tablesizeislarge.Theaveragepacketdroprateforsymmetric
migration when the flow table size is smaller than 1000. The routing increases linearly with the increase of the RTT while
migrationdurationincreasesroughlylinearlywiththenumber the packet drop rate for asymmetric routing is very close to
of rules per switch and can take 7s when there are 10,000 zero for any RTT values.
C. Control-Plane Overhead
Our migration controller intercepts the events, modifies the
datapath ID based on the mapping between the old and the
new virtual switches, and passes the new events to the client
application.Theseoperationscauseoverheadatthecontroller.
To evaluate the controller performance, we use the cbench
program [18], which creates and sends a large number of
OpenFlow messages to the controller. Figure 13 shows per-
formanceoftheunmodifiedPOXcontrollerandourmigration
controller from one switch to 64 switches. The y-axis shows
thenumberofflowsthata switchcanhandlewithinasecond.
Our migration controller processes roughly 3% fewer flows
(a) From host1 to host3
per second than the unmodified controller does.
VI. MITIGATINGGENILIMITATIONS
When we started our work, our goal was to design, im-
plement, and evaluate an efficient VN migration mechanism
in GENI as an example of a future SDN-enabled wide-area
network. While it is possible to deploy virtual networks on
GENI and use proper remote scheduling implementation to
enable live migration, we observe that some GENI limitations
complicatethedesign.Theseconstraintsarenotonlyparticular
to our VN migration research, but may also apply to other
types of experimentation. We summarize the features that are
(b) From host3 to host1
not well supported by GENI. This will aid in future GENI
Fig.11:Averagepacketlossaspercentageoftotalpackets(baseline,
development and also in informing the designs of GENI-
symmetric routing, optimized scheduling) at different data rates for
the flow between host1 and host3 inspired SDN-enabled wide area infrastructure.
A. Interaction with the Substrate Network
GENI deploys virtualization architectures to share physical
resources for simultaneous experiments. It provides multiple
models of virtualization to cater for different levels of per-
formance and isolation requirements. However, experimenters
areonlyfreetoselectfromtheprovidedvirtualizationmodels,
and do not have the privilege to modify the model or build an
alternativeone.Inparticular,wehavethefollowingconstraints
if our experimentation explores the interaction between the
virtualized architectures and the substrate networks.
Fig. 12: The performance of symmetric routing and asymmetric a) Littleknowledgeaboutsubstratenetworks: Underthe
routing for different RTTs current GENI context, we only have access to limited infor-
mation about the substrate network such as the geographical
information about GENI aggregates and VM load on each
physical machine. Without sufficient real-time information
about the physical substrate, it is difficult or impossible to
implement an algorithm that has interaction with the sub-
strate network. For example, some VN migration research
may require real-time statistics about the substrate network
to determine when to trigger the migration and where to
migrate a VN. More generally, to support experimentation
where the placement of the virtual topology is affected by
the performance of the substrate network, we expect GENI
to expose more network statistics such as link utilization,
throughput, and latency.
Fig. 13: Controller performance for different switch numbers
b) Difficulty in debugging: The virtualization techniques to reconsider the shared VLAN. The restriction in resource
are deployed on GENI to support simultaneous experiments reservation makes it difficult to scale up an experiment.
on limited physical infrastructure. However, virtualized ar-
chitecture not only implies a trade-off between performance VII. RELATEDWORK
and isolation, but also makes debugging challenging. The Some of the VN embedding solutions suggest reconfigura-
virtualization architecture may bring unexpected problems, tion or remapping of the VN [2]–[6]. However, all of those
and the limited access to physical substrate further increases works use simulation to demonstrate the effectiveness of their
the difficulty in debugging. In our VN migration research, solutions.Itremainsachallengingtaskfornetworkresearchers
we had a hard time finding the cause of the duplicated to move their experiments to a real infrastructure when there
packets when shared VLANs are used. We can only debug is a lack of effective migration mechanism.
by observing the traffic in virtual topology and infer what There has been some work addressing the challenges of
is happening in the physical substrate. We expect GENI to VN migration in a real infrastructure. Prior work [9] proposes
develop efficient debugging tools to make the debugging an orchestration mechanism for VN migration on PlanetLab,
process easier. Besides, it is impossible to debug without using the same technology to move a single virtual router
a deep understanding of the mechanisms (e.g., how shared withoutdisruptingcurrenttraffictraversingthevirtualnetwork
VLAN works). Most GENI tutorials only introduce how to presented in [19]. Other work [8], [20] shows how to migrate
use their features. It would be helpful if GENI can include VN within software defined networks. Pisa et al. considers
more architecture design of GENI features in its tutorials. the basic migration scenario for migrating virtual network
c) No control of substrate networks: We have flexibility in traditional network and software defined network [20].
to assign bandwidth to our virtual links in the reservation Ghorbani et al. [8] move the whole SDN network with the
stage, but we cannot adjust parameters for the substrate hosts in the data center context. It concentrates on low level
network. Therefore, it is difficult to evaluate an algorithm configuration including packet-in events, traffic statistics and
with bandwidth constraints. This constraint makes it difficult rule timeout to handle correctness violation [21].
toobservehowdynamicsinphysicalsubstratesuchaschanges
in bandwidth or latency can affect the performance of a VIII. CONCLUSION
virtualized architecture.
In this paper we consider the design, implementation and
evaluationofvirtualnetworkmigrationonGENIasaworking
B. Multi-domain Network Research
example of a future SDN-enabled wide-area infrastructure.
In GENI, a slice is a unit that contains all computing and VN migration adds network agility to the repertoire of net-
networking resources for an experiment. The common usage work functions and provides a mechanism that enables the
ofGENIistoreserveallresourcesforanexperimentwithinan deployment of important policies for resource management,
sliceandisolatedifferentslices.Onepossibledesignformulti- energy conservation and attack defense. We show how agility
domain network experiment is to place all domains within can be enabled on top of GENI’s slicing approach, enumerate
the same slice and build different administrative controllers to and address challenges in the design of efficient VN migra-
handle different domains. The disadvantage is obvious: there tion mechanisms, and develop and deploy an implementation
isnoisolationamongdomains,andweareunabletoaddmore of a controller architecture that achieves our VN migration
domains to dynamically scale up the networks. Alternatively, objectives. We perform a set of experiments that help us
we can place one domain on one slice with isolated adminis- understand the implications of various design decisions and
tration. To enable inter-slice communication, we need to use network parameters on VN migration performance. Our work
sharedVLANstoconnectslices,whichcomplicatethevirtual also exposes some limitations on the current design of GENI.
topology and makes it difficult to scale up. Neither of the
REFERENCES
two designs are ideal solutions for experiments that involves
multiple domains. [1] A.Fischer,J.F.Botero,M.T.Beck,H.DeMeer,andX.Hesselbach,
“Virtualnetworkembedding:Asurvey,”IEEECommunicationsSurveys
&Tutorials,2013.
C. Dynamic Resource Reservation [2] J.FanandM.H.Ammar,“Dynamictopologyconfigurationinservice
overlay networks: A study of reconfiguration policies.” in INFOCOM,
The GENI platform requires experimenters to reserve all 2006.
resources on GENI slices before running their experiments. [3] I.Fajjari,N.Aitsaadi,G.Pujolle,andH.Zimmermann,“Vnralgorithm:
A greedy approach for virtual networks reconfigurations,” in GLOBE-
Most GENI resource does not provide flexibility to partially
COM,2011.
modify the resources. In our work, we take advantage of [4] Y. Tang, E. Al-Shaer, and R. Boutaba, “Efficient fault diagnosis using
the shared VLAN feature to make resource reservation more incremental alarm correlation and active investigation for internet and
overlaynetworks,”NetworkandServiceManagement,IEEE,2008.
dynamic.Thisresourcereservationmethodrequirestheexper-
[5] F. Gillani, E. Al-Shaer, M. Ammar, and M. Demirci, “Fine-grain
imenterstoconsiderwhichvirtuallinksinthefirstsliceshould diagnosis of overlay performance anomalies using end-point network
be converted to shared VLAN at the beginning when they experiences,”inCNSM,2012.
[6] F. Gillani, E. Al-Shaer, S. Lo, Q. Duan, M. Ammar, and E. Zegura,
designtheirexperiments.Eachdesignisparticulartoaspecific
“Agile virtualized infrastructure to proactively defend against cyber
topology: whenever we need a new virtual topology, we need attacks,”inINFOCOM,2015.
[7] V.Eramo,E.Miucci,andM.Ammar,“Studyofreconfigurationcostand
energy aware vne policies in cycle-stationary traffic scenarios,” IEEE
JournalonSelectedAreasinCommunications,2016.
[8] S.Ghorbani,C.Schlesinger,M.Monaco,E.Keller,M.Caesar,J.Rex-
ford,andD.Walker,“Transparent,livemigrationofasoftware-defined
network,”intheACMSymposiumonCloudComputing,2014.
[9] S.Lo,M.Ammar,E.Zegura,andM.Fayed,“Virtualnetworkmigration
onrealinfrastructure:Aplanetlabcasestudy,”in2014IFIPNetworking.
[10] B.Chun,D.Culler,T.Roscoe,A.Bavier,L.Peterson,M.Wawrzoniak,
and M. Bowman, “Planetlab: an overlay testbed for broad-coverage
services,”ACMSIGCOMMComputerCommunicationReview,2003.
[11] M.Berman,J.S.Chase,L.Landweber,A.Nakao,M.Ott,D.Raychaud-
huri,R.Ricci,andI.Seskar,“GENI:Afederatedtestbedforinnovative
networkexperiments,”ComputerNetworks,2014.
[12] N.McKeown,T.Anderson,H.Balakrishnan,G.Parulkar,L.Peterson,
J. Rexford, S. Shenker, and J. Turner, “Openflow: enabling innovation
incampusnetworks,”ACMSIGCOMM,2008.
[13] “Omni,” 2015. [Online]. Available: http://groups.geni.net/geni/wiki/
GENIExperimenter/Tutorials/HelloOmni
[14] P.Barham,B.Dragovic,K.Fraser,S.Hand,T.Harris,A.Ho,R.Neuge-
bauer,I.Pratt,andA.Warfield,“Xenandtheartofvirtualization,”ACM
SIGOPSOperatingSystemsReview,2003.
[15] Y.K.DalalandR.M.Metcalfe,“Reversepathforwardingofbroadcast
packets,”CommunicationsoftheACM,1978.
[16] “Pox wiki,” 2015. [Online]. Available: https://openflow.stanford.edu/
display/ONL/POX+Wiki
[17] B.Pfaff,J.Pettit,K.Amidon,M.Casado,T.Koponen,andS.Shenker,
“Extendingnetworkingintothevirtualizationlayer.”inHotnets,2009.
[18] R. Sherwood and Y. KOK-KIONG, “Cbench: an open-flow controller
benchmarker,”2010.
[19] Y. Wang, E. Keller, B. Biskeborn, J. van der Merwe, and J. Rexford,
“Virtual routers on the move: live router migration as a network-
management primitive,” ACM SIGCOMM Computer Communication
Review,2008.
[20] P.S.Pisa,N.C.Fernandes,H.E.Carvalho,M.D.Moreira,M.E.M.
Campista, L. H. M. Costa, and O. C. M. Duarte, “Openflow and
xen-based virtual network migration,” in Communications: Wireless in
DevelopingCountriesandNetworksoftheFuture,2010.
[21] S.GhorbaniandB.Godfrey,“Towardscorrectnetworkvirtualization,”
inHotSDN,2014.