OSCAR: A Collaborative Bandwidth Aggregation System Karim Habak Khaled A. Harras Moustafa Youssef School of Computer Science Computer Science Department Department of CS. and Eng. College of Computing School of Computer Science Egypt-Japan University of Georgia Institute of Technology Carnegie Mellon University Science and Technology (E-JUST) Email: [email protected] Email: [email protected] Email: [email protected] Abstract—The exponential increase in mobile data demand, and railway stations. Fortunately, the widespread deployment coupled with growing user expectation to be connected in of various wireless technologiescoupled with the exponential all places at all times, have introduced novel challenges for increaseofmulti-interfaceenableddevicesareprovidingusers researcherstoaddress.Fortunately, thewidespread deployment with many alternatives for sending and receiving data. of various network technologies and the increased adoption of multi-interface enabled devices have enabled researchers to A potential approach for solving the data tsunami problem develop solutions for those challenges. Such solutions aim to as well as expensive data roaming charges for mobile users exploit available interfaces on such devices in both solitary and is exploiting all available communication interfaces available collaborative forms.Thesesolutions,however, havefacedasteep on modern mobile devices in both solitary and collaborative deployment barrier. forms. In the solitary form, the goal is to exploit any direct Inthispaper,wepresentOSCAR,amulti-objective,incentive- based, collaborative, and deployable bandwidth aggregation Internet connectivity on any of the available interfaces by system. We present the OSCAR architecture that does not distributing applications data across them in order to achieve introduce any intermediate hardware nor require changes to higherthroughput,minimizeenergyconsumption,and/ormin- current applications or legacy servers. The OSCAR architec- imizecost.Inthecollaborativeform,thegoalistoenableand ture is designed to automatically estimate the system’s context, incentivize mobile devices to utilize their neighbors’ under- dynamically schedule various connections and/or packets to different interfaces, be backwards compatible with the current utilized bandwidth in addition to their own direct Internet Internet architecture, and provide the user with incentives for connections. collaboration. We also formulate the OSCAR scheduler as a Utilizing the mobile devices interfaces in these two forms multi-objective, multi-modal scheduler that maximizes system have been investigated over the past few years [4]. The focus throughput while minimizing energy consumption or financial has largely been on the solitary form developing a variety cost. We evaluate OSCAR via implementation on Linux, as well as via simulation, and compare our results to the current opti- ofmulti-interfacebandwidthaggregationsolutionsatdifferent mal achievable throughput, cost, and energy consumption. Our layersof the protocolstack [5]–[17].Other attempts to utilize evaluation shows that, in the throughput maximization mode, theseinterfacesinthecollaborativeformwerealsointroduced we provide up to 150% enhancement in throughput compared [18]–[22]. These approaches either deal with a small scale to current operating systems, without any changes to legacy collaborative community managed by a single authority [19], servers. Moreover, this performance gain further increases with the availability of connection resume-supporting, or OSCAR- [20], or utilize proxy servers to handle and guarantee such enabledservers, reachingthemaximumachievableupper-bound collaboration [18], [21]. Overall, despite the fact that current throughput. smart phones, tablets, and other mobile devices are equipped with multiple network interfaces, current operating systems I. INTRODUCTION mainly allow users to utilize only one interface at a time, or The Federal Communications Commission (FCC) has in- enable limited bandwidth sharing options through tethering. dicated an expected data tsunami problem predicting a 25- In other words, there has been a high deploymentbarrier and 50× increase in mobile data traffic by the year 2015 [1], solutions have focused on bandwidth maximization while not [2]. This expected explosive demand for mobile data, along paying sufficient attention to energy and/or cost efficiency. with expensive data roaming charges and users’ expectations In this paper, we present OSCAR, a multi-objective, toremainconnectedinallplacesatalltime,arecreatingnovel incentive-based, collaborative, and deployable bandwidth ag- challengesfor service providersand researchers to solve. The gregationsystem. OSCAR fulfills the followingrequirements: recentdeploymentandadoptionofFON[3],asystemthaten- (1)Itis easily deployablewithoutrequiringchangesto legacy ables users to share their home bandwidth for obtaining WiFi servers, applications, or network infrastructure (i.e. adding coverageviaotherworldwideFONusers,reflectsarecentpush new hardware like proxies and routers); (2) It is able to for novel solutions to help solve a subset of these challenges. seamlessly exploit available network interfaces in solitary Other novel solutions are needed, however, when users are and collaborative forms; (3) It adapts to real-time Internet on the run or in public places like malls, airports, or bus characteristics and the system parameters to achieve efficient utilization of these interfaces; (4) It is equipped with an Suchtechniquesrequirechangestotheclientoperatingsystem, incentive system that encourages users to share their band- legacy servers, and/or applications; and hence have a high width with others; (5) It adopts an optimal multi-objective deploymentbarrier. Finally, the majority of the network layer and multi-modal scheduler that maximizes the overall system solutionshidethevariationininterfacesfromtherunningTCP throughput, while minimizing cost and energy consumption protocol [5]–[7]. Chebrolu et al. [5] require having a proxy based on the user’s requirements and system’s status; (6) server that communicates with the client and is aware of the It leverages incremental system adoption and deployment to client’s multiple interfaces. Others implement their system at further enhance performance gains. both connection-ends, which makes their deployment rely on Our contributions in this paper are: (1) Designing the updating the legacy servers [6]. On the other hand, MAR [7] OSCAR system architecture fulfilling the stated requirements requires having a special router as well as an optional proxy above.(2)FormulatingOSCAR’sdataschedulerasanoptimal server. multi-objective,multi-modalschedulerthattakesuserrequire- Despite allthiswork,modernoperatingsystemsonlyallow ments, device context information, as well as application users to use one of the available interfaces at a time, even if requirements into consideration while distributing application multiple of them are connected to the Internet. This attests to data across multiple interfaces. (3) Developing OSCAR com- the fact that all current proposals for bandwidth aggregation munication protocols in order to enable efficient and secured face a steep deployment barrier. In addition, these solutions communication between the collaborating nodes as well as have largely focused only on maximizing throughput while OSCAR enabled servers. overlookingother user and system goals like minimizing cost We evaluate OSCAR via implementation on Linux, as and energy consumption. well as via simulation, and show its ability to increase the Recently, developing a deployable bandwidth aggregation overallsystem throughputwhile achievingits cost and energy system has been investigated [14], [15], [23]. DBAS focuses efficiency targets. The results show that, with no changes to only on maximizing the overallsystem throughput[15], [23]. currentInternetarchitectures, OSCAR reaches the throughput On the other hand, even though OPERETTA is deployable upper-bound. It also provides up to 150% enhancement in and energy-aware, it does not utilize the interfaces to their throughputcomparedtothecurrentOperatingSystemswithout maximum in case of non-OPERETTA enabled servers [14]. any change to legacy servers. Our results also demonstrate In addition, similar to previous solutions, they only focus OSCAR’sabilitytomaintaincostandtheenergyconsumption on utilizing the available interfaces in the solitary form, levels in the user-defined acceptable ranges. overlooking available connectivity at neighboring devices as Theremainderofthispaperisorganizedasfollows.Section well as cost efficiency. II discusses the related work. A motivational scenario along B. Collaborative Solutions with the intuition of our solutions is presented in Section III. We then present the overall architecture of our system There exists some shy attempts to utilize the available in Section IV. Section V formulates the OSCAR scheduling interfacesinthe collaborativeform[18]–[22].Thesesolutions problem, after which we present OSCAR’s communication are motivated by the increase of the devices density and protocolsin Section VI. In Section VII, we discuss our Linux are aimed at sharing neighboring devices connectivity. Un- OS implementation and evaluate OSCAR’s performance and fortunately,thesesystemshavemanyshortcomings.Examples compare it to the current solutions. Finally, Section VIII include requiring Internet infrastructure updates such as the concludes the paper and provides directions for future work. existenceof proxyservers[18],[19],[21],updatingthe avail- able applications to interface with the new API and specify II. RELATEDWORK their requirements [20], developing applications that are fully A. Solitary Solutions responsible for handling and making use of the collaboration Manysolutionshaveemergedtoaddresssingle-deviceband- for their own benefits [22]. In addition, all these solutions width aggregation at different layers of the protocol stack focus only on maximizing throughput and usually ignore [4].Applicationlayersolutionseitherassumethatapplications developing incentives systems that enable this collaboration. are aware of the existing network interfaces and take the The closest system to OSCAR in terms of incentivized responsibility of utilizing them for their needs [12] or rely bandwidth sharing is FON [3]. FON is a commercial system on modifying the kernel socket handling functions to enable thatenablesuserstosharetheirhomebandwidthforobtaining existing applications use multiple interfaces [12], [13]. Such WiFi coverage via other worldwide FON users. Basically, modifications require changes to legacy servers in order to each user installs a Fonera router in her home network. This supportthese new sockets.In addition,[13]requiresfeedback Fonerarouterisresponsibleforsharinga portionoftheuser’s from the applications about their performance, and hence bandwidthwithotherFONsystemusers.Themaindifferences is not backwards compatible with previous versions of the between FON and OSCAR are: (1) FON does not aggregate applications. theavailablebandwidthneitherinadevicenorincollaborative Manybandwidthaggregationtechniques,however,naturally devices.(2)FONusesspecialrouter(Fonera).(3)FONrouters lieinthetransportlayer[8]–[11].ThesesolutionsreplaceTCP are static while OSCAR targets mobile devices. (4) The user withmechanismsandprotocolsthathandlemultipleinterfaces. is unable to connect to more than one Fonera. (5) FON users Virtual Bank John Alice BVailratunacle MAlaicrek Applications: WiFi Connection-Oriented OSCAR OSCAR Client Bluetooth Context Awareness Packet-Oriented Incentives Manager Legacy Server Managger Scheduling Manager The Internet 3G Packet-Oriented OSCAR User Interface MMooddee DDeetteeccttiioonn Tendering Price John Resume Module Module Determiner Supporting WiFi Mark LSeegravecyr ACpphalicr.a Etiosntim Taratoffric Resume Module aTnedn Fdieltresr iCngo llMecotdiounle 3G OSCAR-Enabled Server Neighbor Discovery Contracts Module Assessment Module Connection #1 Packets John’s Incentives Packets CCoonnnneeccttiioonn ##23 PPaacckkeettss MAalicrke’’ss IInncceennttiivveess PPaacckkeettss NCethwaorr. kE Isnttiemrafatocers Balance Handler Scheduler Battery Sensor Reservation Manager Fig.1. OSCARscenario are incetivized by the need of being connected to the Internet Backward Compatibility Manager Packet Reordering Network Address Connection while being away from their home network. OSCAR builds Module Translation Module Redirection Module on this needas well, butalso addsa tangiblefinancialpricing system to further encourage collaboration. Network Interfaces: III. MOTIVATING SCENARIO AND SOLUTION Most mobile devices today are equipped with multiple Fig.2. OSCARclient architecture heterogeneousnetworkinterfaces.Forexample,John’smobile resumefunctionalitytoschedulethedataatafinergranularity deviceisequippedwithWifi,Bluetoothand3G.WhenJohnis in order to enhance the overall system performance. It would sitting in a mall, waiting for a train at the station, or having a also be equipped with an incentive mechanism in order to meal,hetypicallywatchesyoutubevideos,listenstopodcasts, convince John’s neighbors to share their connectivity with uses facebook to get his social network feeds, and opens his him.Finally,thissystem wouldadoptan efficientandsecured favoritenewscites.Heconnectshismobiledevicetoavailable communication protocol to make the collaboration between Wifi hotspots while being connected to his 3G network and the different users feasible. leaving his Bluetooth idle. In such places John has a lot of Figure 1 shows a scenario, based on OSCAR, in which people around him such as Mark who is not using his 3G John’s device is able utilize all the available connectivity Internet connectivity and Alice who is connected to a high optionsincludingMark’sand Alice’s under-utilizedinterfaces bandwidth WiFi network but is not currently using it. to achieve John’s requirements and enhances his experience. Unfortunately, John’s current operating system assigns all In this scenario, we notice that while communicating with his connections to Wifi. Consequently, John has to wait until the legacyservers the device schedules the connectionsto the theyoutubevideoisbufferedinordertowatchitcontinuously paths in a connection-orientedmode, in which all the packets without disruption; meanwhile other applications are slowly belonging to the same connection utilizes the same path. retrieving their content. With the high contention on the However,whilebeingconnectedtoanOSCARenabledserver Wifi interface, the available bandwidth is degraded, and John orresume-supportinglegacyserver,itusesfine-grainedpacket- disconnects his device from Wifi to utilize his 3G connec- oriented scheduling for further performance enhancements. tion. Although, this increases his throughput, the available We also observe that there is also a virtual bank, which is bandwidth is not sufficient to smoothly retrieve the content, part of the incentive system, that John uses to pay for the he incurs the added overhead of restarting his non-resumable bandwidth he shared from Mark and Alice’s. We now move connections, and pays a higher cost in battery and money for ontodiscussingthearchitectureandsystemforenablingsuch using 3G. scenario. John’s needs can be satisfied by concurrently utilizing his available network interfaces as well as the under-utilized IV. THE OSCAR SYSTEM connections of Mark and Alice. John’s applications traffic In this section, we start by describing our system assump- can be scheduled across different interfaces and neighbors tions. We then provide an overview of OSCAR’s architecture in order to enhance his user experience. Such a scheduler followed by a detailed description of its main components. should take the network interface characteristics, bandwidth A. Assumptions heterogeneity, and different application traffic characteristics into account when scheduling different connections to the OSCAR works in a distributed environment, where a node availableinterfaces.Furthermore,theschedulershouldexploit can share and use the bandwidth available from its OSCAR- servers supportfor the resume functionalityin order to seam- enabled neighbors to connect to both legacy and OSCAR- lessly migrate connections assigned to an interface that got enabled servers. For this, we assume that all OSCAR com- disconnected to another. The scheduler can also leverage this ponentsare trusted. Handling security aspects of the software modulesis outside the scopeof this paper.We furtherassume (implementedthroughthe BackwardCompatibilityManager); that there is a trusted entity, called the virtual bank, that is which enables OSCAR’s clients to make use of the available responsibleforauthenticatingOSCARusersandkeepingtrack connectivity opportunities without requiring any changes to of the virtual currency exchanged by users for the bandwidth legacy servers, network infrastructure, or legacy applications. (as detailed in sections IV-F and IV-I). Bandwidth sharing Third,leveragingconnectionresume-supportinglegacyservers agreement between two nodes is performed in rounds, each andOSCAR-enabledserversforfurtherperformanceenhance- withafixeddurationdeterminedatthebeginningoftheround. ments (implemented through the Packet-Oriented Scheduling To determine the cost of the bandwidth at the selling node, Manager). Fourth, Handling the user incentives and collabo- OSCAR has a module that suggests an initial cost based rating with the trusted bank module in handling the payment on different factors (as detailed in Section IV-F1). The user, transactions implemented through the (Incentive Manager). however,canmanuallyadjustthecost,e.g.toofferbandwidth Finally, optimally scheduling the connections and/or packets sharingforherowndevicesforfree.Oncethepriceisfixed,it onthedifferentpathsinordertoachievetheuserrequirements cannot be negotiated until the next round. The buyer collects (implemented in the OSCAR Scheduler). the different offers from the neighboring buyers and selects Note that, OSCAR server architecture is a subset of this the ones that best fits its needs. A buyer can re-sell her clientarchitecturecontainingthefollowingmodules:(1)Mode bandwidth to other nodes, allowing for multi-hop bandwidth Detection Module, (2) Packet Reordering Module, (3) Sched- sharing. OSCAR modulesaddress the cost and loopingissues uler,(4) Network InterfacesCharacteristics Estimator,and (5) associated with bandwidth re-selling. Network Address Translation Module. For the balance of this section,wedescribeeachofthesecomponentsinmoredetails. B. Communicating Entities D. Context Awareness Manager Figure 1 shows that we have five main communicating entities in our architecture. Firstly, client devices equipped One of the key factors that enables OSCAR to efficiently with multiple network interfaces varying in their available utilize all available connectivity options is its ability to de- bandwidth, energy consumption rates and cost per unit data. termine, store, and utilize the mobile device’s context infor- Each interface can be used to connect directly to the Inter- mation accurately and efficiently. This is the responsibility net and/or through neighboring client devices sharing their of the context-awareness manager. This component consists connectivity options. These options vary in their usage cost of the following five modules: the User Interface Module, and available bandwidth to be shared. We call each of these the Application Traffic Characteristics Estimator, the Neigh- possible ways to connectto the Interneta path. Each of these bor Discovery Module, the Network Interface Characteristics devicesrunsmultipleapplicationsthatvaryin their character- Estimator, and the Battery Sensor. istics. Secondly, a virtual bank which is a trusted third party 1) UserInterfaceModule: OSCARprovidesauser-friendly that handles the payment transactions between the different interface that enables users to set their requirements, prefer- devices as well as the authentication related aspects. Thirdly, ences,andschedulingpolicies.Thisalsoenablesuserstoinput legacyserverswhicharetypicalnon-modifiedInternetservers. other contextual parameters like the interface usage cost (e.g. The clients use the connection-orientedmode to connectwith cost of using 3G or the hotel Wifi). Finally, this module is them as its default mode of operation. In this mode, OSCAR also used to express willingness to share bandwidth, set the schedules different connections to the available paths such amount to be shared, and define the needed revenue in return that a connection can be assigned to only one path. Once for consuming the device’s battery to aid others. assigned, all the packets belonging to this connection utilize 2) Application Traffic Characteristics Estimator: Knowing thesamepath.Fourthly,connectionresume-supportinglegacy theapplicationtrafficcharacteristicssignificantlyimpactsOS- servers, e.g.FTPandHTTPserversthatsupportresumingthe CAR’s performance.However,to be fully deployable,we can connections. OSCAR leverages these servers to enhance the not require any changes to existing applications in order to performance by switching to a packet-oriented mode, where determine this crucial information. Our approach is to auto- each packet or group of packets can be independently sched- matically estimate application characteristics based on their uled on a different path. Finally, OSCAR-enabled servers, operational history on given port numbers. These estimation running our software modules, which OSCAR leverages for techniques enable us to accurately determine the application enabling highly efficient packet-oriented scheduling. traffic characteristics since most popular applications have specific ports reserved for their communication and exhibit C. Architecture Overview relatively consistent behavior. We estimate the average con- Figure 2 shows the OSCAR’s client architecture. The nection data demand in bytes for each port as follows: architecture implements five core functionalities: First, con- After a connection is terminated, the module updates the text awareness (implemented through the Context Awareness estimated values of connection data demand (Cdemand) as: Manager). It enables OSCAR to sense its environment; in- Cdemand(i)=(1−ρ)Cdemand(i−1)+ρTCdemand(i) (1) cluding user requirements, applications traffic characteristics, attached interfaces characteristics, battery status, and nearby where TCdemand(i) is the number of bytes transmitted by OSCAR-enabled devices. Second, backward compatibility the ith connection that uses this port number which has just TABLEI such opportunities is the responsibility of the packet-oriented POWERCONSUMPTIONTABLE[25] scheduling manager. This module exploits the availability of OSCAR-enabledserversorserversthatsupporttheconnection Interface Tech. ∆Power1 Datarate Netgear MA701 Wifi 726mW 11Mbps resumptiontoenhancetheperformanceofOSCAR.Itcontains LinksysWCF12 Wifi 634mW 11Mbps two sub-modules:the mode detection module and the resume BlueCore3 Bluetooth 95mW 723.2Kbps module. finished,Cdemand(i)isthenewestimateoftheconnectiondata 1) Mode Detection Module: This module detects whether demand,andρisasmoothingcoefficient,takenequalto0.125 the end point of the connection supports OSCAR or not toevaluatetheequationefficiently.Thisway,theestimate can in order to enable the optional packet-oriented mode. It is be obtained by an efficient shift operation. implemented as a service that monitors the exchanged data 3) Neighbor Discovery Module: This module is responsi- packetsbetweenthetwo endpointsandaltersthe TCPpacket ble for discovering the existence of nearby OSCAR-enabled header. When the client starts a new connection, the module devices. It selects the interfaces that are going to be used in alters the TCP header adding OSCAR existence flag in the the discovery process, determines the frequency of sending optionpartoftheTCPheader,iftheserverrespondssimilarly, neighbor discovery queries, maintains the lists of the discov- this indicates that OSCAR is supported on the server. Hence, ered neighbors, and communicates with the virtual bank to packet-oriented mode can be utilized. If so, OSCAR will authenticate these neighbors. schedulethepacketsthatbelongtothisTCPconnectiononthe 4) NetworkInterfaceCharacteristicsEstimator: Thismod- available paths. If OSCAR is not supported on the server, the ule is responsible for estimating the characteristics of each client resorts to the default connection-oriented mode unless network interface. In particular, it estimates the available the connection is resumable. bandwidth, energy consumption rates, the used physical data This approach enables OSCAR to efficiently determine the rates,andtheachievablemaximumdatarateforeachinterface. operation mode without incurring extra packet transmission To estimate the available bandwidth, it periodically connects overhead. It also enables us to avoid extra delay since imme- through each interface to various geographically dispersed diate scheduling in connection-orientedmode occurs until we serversthat supportthe packet-pairestimation technique[24]. determine the need to switch to packet-oriented mode. On the other hand, when communicating with OSCAR- 2) Resume Module: This module is utilized when com- enabledservers,themoduleperiodicallysendsthedatapackets municating with legacy servers, with the purpose of enabling in pairs and marks them in order to be used for estimating packet-oriented scheduling even if the server is not OSCAR- theavailablebandwidthusingthepacket-pairtechnique.These enabled. To detect whether a particular server supports the estimates are sufficient since bandwidth bottlenecks typically resume functionalityor not, we build a web service maintain- exist at the client’s end not the server’s, which is typically ing a list of protocols that support the resume functionality. connectedto high bandwidthlinks and designed to scale with OSCAR periodically connects to this web service to update the number of clients. and cache this list of protocols. Each protocol is represented Tocharacterizetheenergyconsumption,TableIshowshow by a template that uniquely identifies the protocol, e.g. its the power consumptionof a networkinterface dependson the port number, as well as how to check with the end server NIC, the technology used, and the physical transmission data that it supports the resume functionality, e.g. checking for rate[25].Hence,inordertomakeOSCARestimatetheenergy the “Accept-Ranges” header option in HTTP. The OSCAR consumptionforitsinterfaces,webuildanonlineservicewith Resume Module uses such protocol information to determine a database containing energy consumption rates for various whether the server supports the resume functionality or not, networkinterfacesatdifferentphysicaltransmissiondatarates. and accordingly enables the optional packet-oriented mode. Once OSCAR runs, it queries the database for the energy The resume module is implemented as an in-device consumption rates of each network interface attached to the proxy responsible for identifying the resumable connections, device. scheduling sub-connections across different device interfaces, Estimatingthephysicaltransmissiondatarateisthesimplest and closing these sub-connections once they terminate. To task since it can be queried using the supported OS API. It minimize the latency and buffer size the application needs is used in additionto the queriedenergyconsumptionrates to to wait for the in-order delivery of data, the module uses determinetheinterface’scurrentenergyconsumptionrate.We twoapproaches:(1)itdividestheconnectionintofixedlength usetheestimatedphysicaltransmissionrateasanestimatefor units, where a unit with an earlier offset from the beginning the maximum achievable data rate for the interface. of the connection is retrieved over the multiple interfaces 5) Battery Sensor: This module senses whether the device before the next unit is retrieved. Each interface is assigned ispluggedto apowersourceorrunningonbatteryalongwith a portion of the unit to retrieve based on weights decided the available battery level. by the scheduler. (2) It sorts the interfaces in a descending order according to their scheduling weight and starts the E. Packet-Oriented Scheduling Manager sub-connection with the larger weight first. This way, larger Utilizing the opportunities of packet-oriented scheduling is amounts of data will arrive in order from the beginning of one of the key features introduced by OSCAR. Detecting the connection. For further optimization, if the resumable connection supports range requests, it does not close the sub- status2 and other system parameters. The device owner can connection once the load is finished but reuses it for the next agree with this initial price suggested by the system or load assigned to that interface. This optimization technique preemptivelychangeitthroughtheUser InterfaceModule.To avoids the extra delay of increasing the congestion windows determine an initial price, multiple cost parameters are taken to its optimal size introduced by TCP slow start. into account including the service provider cost, the energy On the other hand, this module also handles interface consumptioncost,andthemarketstatus. Thiscostisfixedfor disconnectionwhichincreasesinthecollaborativemodedueto a fixed amount of time and needs to be renewed in order to devicemobility.Itdoessobymonitoringeachsub-connection handle mobility and disconnections of devices. ensuring it will receive its portion. If one or more of the sub- a. Service Provider Cost: Since a node may be paying for connections’ paths became unavailable because of disconnec- its own Internetaccess (either through an Internetprovideror tion,itre-schedulesthissub-connectionononeoftheavailable a neighboringsharing node),this node needs to at least cover runningpaths.Theschedulerisresponsiblefordeterminingthe such cost in order to avoid incurring a loss. Hence, the first appropriate path to be used in this case. portion of the incentives price will be ci which is the service provider cost per unit data of using interface i. b. Energy Consumption: Sharing bandwidth by enabling F. Incentives Manager multiple network interfaces will consume extra power, which isanextremelycrucialresourceespeciallyforbatteryoperated The success of a collaborative bandwidth aggregation sys- mobile devices. Hence, a device sharing its bandwidth needs tem is tightly coupled with its ability to encourage users to some compensation for its consumed energy. Noting that a collaborate and share their available bandwidths. Incentive selling device uses two of its interfaces (one connected to systems have been widely studied in literature for a wide the Internet, i, and the other connected to the buyer, j), array of applicationswhere users share resources; approaches calculatingtherequiredcompensationshouldtakeintoaccount proposed generally fall into (1) reputation-based approaches the following parameters: (1) the energy consumed to relay a and (2) credit-based approaches. unit of data for interfaces i and j would equal (e +e ), (2) i j In reputation based incentive approaches [26], [27], nodes the battery capacity (E ), and (3) the remaining battery Capacity earn a reputation based on their collaboration with their ratio(R ).Hencewecalculatetheenergycompensation remaining neighborswhothenspreadthisreputationaccordingly.Hence, factor (EC ) as follows: ij the nodes carry the overhead of monitoring their neighbors e +e and spread their observations in the network to enable other EC =[γ+(1−R )] i j (2) ij remaining nodes to determine their reputation level and act accordingly. ECapacity These approaches,however, are typically suitable for systems Where γ is a binary value reflecting whether the device is with stable memberships that last long enough to earn high connected to a power source or not. If connected, γ = 0, reputations.Theyarenotsuitablefordeployablecollaborative the user still needs to be compensated for the consumed bandwidth aggregation systems, where nodes will randomly energy as part of the charging power is still used in the meetatrandomlocationsatanypointin time(e.g.passengers bandwidth sharing. Therefore, the importance of your energy at an airport or a train station). consumptiondegradestillitreaches0whenthebatteryisfully Credit-based systems are more suitable for such networks charged.Ontheotherhand,ifthedeviceisrunningonbattery since they usually rely on a trusted third party that maintains power, γ = 1 guarantees receiving appropriate compensation credit for the communicating nodes [28], [29]. Like free even if the battery is fully charged. markets, nodes in such systems collaborate with each other c. The Market Status: With potentially multiple devices and pay for the services acquired from one another. requesting and offering bandwidth, the demand and supply The Incentives Manager in our architecture is designed mini-market status amongst these devices is a crucial pricing to enable bandwidth sharing amongst users by adopting a factor. A balanceis requiredin orderto maximizethe sharing credit-basedsystem. Itis particularlyresponsiblefor handling node’sbenefit while providingpriority to users willing to pay virtual currency exchange and bandwidth reservation. These more for important data they need to transmit. We therefore functions are implemented using the following five modules: calculate the market dependent cost (MC) as follows: the Tendering Price Determiner, the Tenders Collection and Filtering Module, the Contracts Assessment Module, the Bal- MCn,i =max[MCn−1,i+ ance Handler, and the Reservation Manager. (1+κ)B −B (3) i,Requested i,Offered η,0 1) Tendering Price Determiner: This module runs at the B seller and is responsible for determining a price to offer (cid:18) i,Offered (cid:19) (cid:21) Where i is the index of the interface connected to the for sharing a device’s bandwidth on a given interface for a fixed amount of time. It proposes an initial price covering Internet that we share its bandwidth, MCn−1,i is the market status cost during the previous estimation session, B the cost for using the shared bandwidth at interface j (which i,Requested is connected to the Internet) while communicating with a 2Environmentalstatusisthecurrentmarketstatusincludingrequestsonthe buyerthroughthesellerinterfaceibasedontheenvironmental sellerandreservations. is the sum of the requested bandwidth for reservation during 4) Balance Handler: This module is responsible for han- the last session from such interface (i.e. demand), B is dling the payment transactions with the virtual bank. It also i,Offered theofferedbandwidthfromthatinterface(i.e.supply),η isthe makessurethatbuyernodedoesnotmakerequeststhatexceed costincreasefactordeterminedbytheuser,andκisarevenue its available credit. multiplier.The reasonbehindusing the revenuemultiplier(κ) 5) Reservation Manager: This module is designed to keep is to allow for increasing the price when the supply is equal track of the reserved bandwidth on each node to avoid using to the demand and is set to a small value (0.1 in our case). aninterfaceaboveitscapacity.Itbuildsareservationtablethat Theintuitionisthataslongasthedemandishigherthanthe contains the neighbor identifier, the reserved bandwidth, and supply, we keep increasing the market status cost value. This time stamp for the most recent usage. A reservation time out increase is a function of the cost variation factor (η) as well (RT ) is used to avoid indefinite reservation that may occur Out as the normalizeddifference between the demandand supply. due to mobility or the existence of a malicious node. The Incentive Price: After discussing the various factors that should be taken into consideration while calculating the G. Backward Compatibility Manager requiredincentivespriceweputthemtogetherinthefollowing In order to make OSCAR deployable in today’s Internet, it equation: shouldbebackwardscompatiblewithboththe currentlegacy servers and applications. This is the responsibility of the I =νEC +MC +c backward compatibility manager that consists of the Packet n,i,j i,j n,i i e +e Reordering Module, the NAT Module, and the Connection i j =ν[γ+(1−Rremaining)] +ci+ Redirection Module. E Capacity + 1) PacketReorderingModule: Thismoduleisactivatedfor (1+κ)B −B MCn−1,i+ i,Requested i,Offered η the packet-oriented mode to handle packet reordering issues B (cid:20) (cid:18) i,Offered (cid:19) (cid:21) introduced by transmitting data using multiple network inter- (4) faces to be compatible with legacy applications. It delays the packetsaswellastheiracknowledgmentsbeforepassingthem Where ν is an energy-cost conversion factor that maps the to the upper layer. This is designed to avoid the performance energycost (EC ) to monetarycost. The valueof ν is set by i,j degradation introduced by using TCP over multiple paths. In the virtual bank based on the current fair market prices. ordertoavoidbeingover-protective,itmaintainsanestimation At the end, this incentive price is proposed to the user and of the TCP’s RTT in order to forward the out-of-order data she can freelychangeit seeking moreor less profit.Although to TCP before the timeout to prevent excessive drops in the setting I >c can be used to avoid loss and loops, a user n,i,j j congestion window. has the ability to set the price even to zero to satisfy cases 2) Network Address Translation Module: When a connec- when she shares bandwidth with other devices belonging to tion go through a sharing neighbor to a legacy server, since her or to family members. The IP of the interface with the a legacy server can see only one IP address for the client, farthest OSCAR device in the chain is always used in the OSCAR needs to change the IP address in the packet to the tenders to detect and avoid loops. neighboring client IP. Otherwise, the reply from the legacy Oncethiscostisdeterminedandsenttothebuyer,thebuyer server will go directly to the client, rather than to the sharing either accepts or rejects it. neighbor,removing the benefit of sharing the bandwidth with 2) Tenders Collection and Filtering Module: This module neighbors. runs at the bandwidth requestor and is responsible for col- To address this, we implement a Network Address Trans- lecting tenders from nearby OSCAR enabled devices. These lation (NAT) module at each node which is activated on the tenders contain the path available bandwidth and the price of sharing(seller) node.This moduledoesthe NATingoperation transmitting a unit of data in this path (as calculated by the forthesentpacketsandreversesthisoperationforthereceived Tendering Price Determiner at the seller). After successfully packets from the legacy server. collectingthetenders,themodulesortsthembasedoncostand 3) ConnectionRedirectionModule: Thismoduleisrespon- forwardsthe least cost ones whose aggregatebandwidthsatu- sibleforredirectingthenewlyrequestedconnectionstothethe rates the requester bandwidth to the scheduler. This way, the Resume Module in order to enable utilizing the connection scheduler can take the available bandwidth throughneighbors resumes while hiding this from legacy applications. in its decision. 3) Contracts Assessment Module: When two neighbors H. Scheduler agree on the set of terms for bandwidth sharing (i.e. cost, reserved bandwidth, and duration), both the buyer and seller This is the heart of the OSCAR system. It takes its monitor the traffic to regulate and ensure these terms. Note input from all the previously mentioned modules in order that this is enforcedby the trusted OSCAR components. This to schedule the connections and/or packets to the different module is also responsible for renewing the contract before it available paths. We discuss the OSCAR scheduler in details expires. in Section V. I. Virtual Banks TABLEII LISTOFSYMBOLSUSED Thisisatrustedthirdpartythatmaintainsuseraccountsand creditbalances.Itisalsoresponsibleforhandlingthepayment Symbol Description process between the different users as well as OSCAR’s T Theoverall systemthroughput L Thecurrent systemload authentication. Li Thecurrent systemloadforstreami Si Determines whether streamiisconnection-based (1) V. OSCAR SCHEDULER orpacket-based (0) Inthissection,weformulatetheoptimalOSCARscheduling bj Theeffective bandwidthofpathj rj Thedatarateofpathj problem. We begin by describing our system model followed aj Difference inpowerbetweenactive andidlestatesofpathj by the optimal scheduling problem. E Theenergyconsumedinordertotransferthesystemload Eavg Theaverage energyconsumedperunit A. System Model datawhiletransferring thesystemload Ej Theenergyconsumedforpathj totransferitsload Table II summarizes the system parameters and notations. cj Thecostperunitdataofpathj We assume a mobile device with m different paths to the C Thesystemloadtransferring cost Cavg Theaverage costperunitdataoftransmitting thesystemload Internet.Eachpathrepresentsaway toconnecttotheInternet Cj Pathj costoftransferring itsload either by using the Interface’s direct connectivity or by using ∆j Pathj neededtimeforfinishingitsload theinterfacetocommunicatewithoneofitsneighborssharing xij Forconnection-oriented streams,equals 1ifstreamiis assignedtointerface j.Equals0otherwise. its Internet connectivity. Each of those paths has its effective wj Theratioofpackets assignedtointerface j bandwidth b and cost per unit data c , which can be the n Numberofactive streamsincluding thenew request j j service providerusage costor the costpaid to the neighborin m Numberofdifferent paths ordertousetheirconnectivity.Inaddition,eachpathusesone mode(SectionV-B4).Finally,wepresentourproblemsolution network interface and it has an energy consumption rate a , j (Section V-B5). wherea equalsthedifferenceinpowerconsumptionbetween j 1) Throughput Maximization Mode: In this mode, the the active and idle states of the used interface. The data rate scheduler’s goal is to maximize the system throughput under of each path interface is denoted as r . The device runs a set j certainenergyandcostconstraints.Inparticular,theuser puts of connections that share these interfaces and varies in their a limit on the average cost per unit data (C ) as well characteristics. avg,Target as a limit on the energyconsumptionper unit data (E ) Ourschedulingunitisaconnectionorapacket.Wereferto avg,Target that should not be exceeded. a standardnetworkconnectionas a stream to avoid confusion Objective Function: withtheschedulingunit.Schedulingdecisionsaretakenwhen Theobjectiveofthescheduleratanydecisioninstanceisto a new stream (number of streams active in the system is n, maximizetheoverallsystemthroughput(T).Giventhesystem including the new stream) is requested from an application. load (L), the objective can be written as: TheModeDetectionModule(SectionIV-E1)andtheResume Module(Section)thendeterminewhethertheoperationmode L MaximizeT = (5) isconnection-based(Sn =1),orpacket-based(Sn =0)when maxj∆j theotherendisOSCAR-enabledorsupportstheresumemode. where ∆ is the time needed for path j to finish all its In the former case, the scheduler’s goal is to determine to j load (connection and packet based). Since L is constant, the which path it should be assigned (sets x = 1 for only nj objective function is equivalent to: one path j). In either case, the percentage of packets to be assigned to each path, i.e. paths relative packet load (w ), j Minimize max∆ should be re-calculated based on the current system load (L). j j Our OSCAR scheduler has three modes of operation based n n on user preferences: (1) throughput maximization, (2) energy =Minimize max i=1(Li(1−Si)wj)+ i=1(LiSixij) minimization, and (3) cost minimization. j (cid:18)P bj P (cid:19) (6) B. Optimal Scheduling Where the left summation represents the packet-oriented In this section, we formulateour schedulerand its different mode load (each term, Li(1−Si)wj, is the numberof bytes modes of operation. Generally, the decision variables are: (1) fromthepacket-orientedstreamiloadassignedtopathj)and If S =1, which path to assign the new stream n to (variable the right summation is the connection-oriented mode load. j x ) and (2) the new values for w , ∀ :1≤j ≤m. Note that any stream i will be either connection-oriented nj j j WestartbypresentingtheformaldefinitionoftheThrough- (Si = 1) or packet-oriented (Si = 0) and thus will appear put Maximization Mode (Section V-B1). Then, we present in only one of the two summations. Dividing the sum of two the formal definition of the Energy Minimization Mode (Sec- loads by the available bandwidth on that path (bj) gives the tion V-B2). After that, we present the formaldefinition of the time needed for path j to finish its load. CostMinimizationMode(SectionV-B3)followedbythecom- Constraints: mon constraints that must be satisfied regardless of operation The following constraints must be satisfied: Target Cost: As we mentioned, the user puts a limit (C ) on the average cost she is willing to pay per unit avg,Target m n a data. Minimize E = j w (L (1−S ))+L S x j i i n n nj r Cavg ≤Cavg,Target (7) j=1 j i=1 ! X X (13) Since Where: aj refers to the energy consumed per unit data. C = mj=1cj[ ni=1(Li(1−Si)wj)+ ni=1LiSixij] wj ni=1(Lrij(1−Si))+LnSnxnj the load assigned to inter- avg L facej exceptthepreviouslyassignedconnectionorientedload P P P (8) (a P/r ) n−1L S x . We removed this part from the equa- Substituting in (7): j j i=1 i i ij tion since it is constant and will not affect the minimization P m n n process. cj wj (Li(1−Si))+ LiSixij ≤LCavg,Target Constraints: ! j=1 i=1 i=1 The following constraints must be satisfied: X X X (9) Target Throughput: As we mentioned, in this mode, the Where c is the averagecost per unitdata while using path j client’s device has to achieve at least a certain level of j. n (L (1−S )w )+ n L S x istheloadassigned i=1 i i j i=1 i i ij throughput(T ). Therefore, to path j as in (6). Target P P Target Energy: Similarly, the client’s device can tolerate L T = ≥T (14) up to a certain level of the average energyconsumed per unit max ∆ Target j j data. where ∆ is the time needed for path j to finish all its load E ≤E (10) j avg avg,Target (L)(connectionandpacketbased).Thethroughputistheratio Since betweentheloadonthesystemandthetimeneededtotransfer m aj [ n (L (1−S )w )+ n L S x ] this load. Eavg = Pj=1 rj Pi=1 i Li j Pi=1 i i i(j11) →∀j, ∆j = ni=1(Li(1−Si)wj)+ ni=1(LiSixij) ≤ L b T Substituting in (10): P j P Target Rearranging: m n n a j (Li(1−Si)wj)+ LiSixij ≤LEavg,Target n n−1 r Lb Xj=W1hejr e Xia=j1is the average energXiy=c1onsumpti!on per unit d(1a2ta) →∀j, wjXi=1(Li(1−Si))+LnSnxnj ≤ TTargjet−Xi=1(Li(S1i5x)ij) rj while using path j. Note that the RHS is constant. 2) Energy Minimization Mode: In this mode, the sched- Target Cost: Similar to the throughputmaximization mode uler’sgoalistominimizingtheoverallenergyconsumedunder (Section V-B1), the target cost constraint is represented by certain throughputand cost constraints. In particular, the user Equation 9 puts a limit on the average cost per unit data to Cavg,Target and 3) Cost Minimization Mode: In this case, the user is requests minimum throughput of Ttarget interested in achieving at least a certain level of throughput Objective Function: T with limitations on the average energy consumed per target The objective of the scheduler at any decision instance is unit data E avg,target to minimize the overall system energy consumption (E). This Objective Function: objective can be written as: The overall objective of the scheduler is to minimize the overall system cost needed (C) to consume the system load: Minimized E = E j Minimized C = C j j X j where, E is the energy consumption of path j. This energy X j consumptioncan be dividedinto two parts:(1)energyneeded where, C is the cost needed for path j. This cost can be j to finish the load of the connection-oriented streams and (2) dividedintotwoparts:(1)costneededtofinishtheloadofthe energyneededtofinishtheloadofthepacket-orientedstreams. connection-oriented streams and (2) energy needed to finish The former is equal to (a /r ) n L S x where a /r is the load of the packet-oriented streams. The former is equal j j i=1 i i ij j j the energy consumed per unit data while using path j and to c n L S x j, where c is the cost per unit data for n−1L S x istheconnectionPorientedloadassignedtopath pathjj ani=d1 ini Li S x j thejamountof data assigned to this j.Si=im1ilairlyi,tihjelaterisequalto(a /r ) n L (1−S )w . path iPn conneci=tio1n-ioriienitedmode.Similarly,the later is equal SPince all the connection-orientedsjtreajms biu=t1theinewlyiarrivj- to c n PL (1−S )w . Since connection-orientedstreams P j i=1 i i j ingonecannotbere-assignedtootherinterfaces,theobjective cannot be re-assigned to other paths, the objective function P function becomes: becomes: C. Discussion m n In some cases, the selected constraints by the user may Minimize C = c w (L (1−S ))+L S x lead to an infeasible solution. For example, it may be the j j i i n n nj j=1 i=1 ! case that there is no assignment strategy that achieves both X X (16) the cost and energy consumption constraints specified by the Constraints: user concurrently. To eliminate this possibility and make the The following constraints must be satisfied: selectionprocessuserfriendly,wedesignaniterativeselection TargetThroughput:Similartothethroughputmaximization policy where the user selects her top priority constraint first mode (Section V-B2), the target cost constraint is represented (e.g.limitoncost).Basedonthisselection,OSCARcalculates by Equation 15 the feasible range for the second constraint (e.g. energy Target Energy: Similar to the throughput maximization consumption) and the user is asked to choose a value from mode (Section V-B1), the target cost constraint is represented this range. by Equation 12 VI. OSCAR COMMUNICATION PROTOCOLS 4) Common Constraints: Here, we present our set of con- strains that must be satisfied regardless of the scheduling In order to implement OSCAR, we developed two sim- mode. ple protocols. First, a collaboration protocol that handles Integral Association: If the new stream is connection- bandwidth sharing amongst collaborating nodes. Second, an oriented, it should be assigned to only one path: OSCARservercommunicationprotocolfordetectingOSCAR- enabled servers and exchanging control information between m the client and server. In this section we give an overview on x +(1−S )=1 (17) nj n our OSCAR communication protocols followed by a detailed Xj=1 description for each protocol. Note that when S = 0, x = 0,∀j, which is the case n nj A. Protocols Overview when the new stream is determined to be packet-oriented by In order to enable the collaboration between nodes, the the Mode Detection Module. OSCARcollaborationprotocolaimsto:(1)efficientlydiscover PacketLoadDistribution:Forpacket-orientedstreams,their the neighbors, (2) authenticate both buyers and sellers, (3) total load should be distributed over all interfaces: enable cost agreement and (5) and guarantee the payment. m On the other hand, the OSCAR Server communication w =1 (18) protocolaimsto(1)discovertheexistenceofOSCAR-enabled j j=1 servers,(2)enabletheservertoknowthelistof(IP,port)pairs X that they can use to send their data to the clients, (3) and VariableRanges:Thetrivialconstraintsfortherangeofthe exchange with the clients the packet scheduling weights. We decision variables give the details of both protocols in the next two subsections. 1≥w ≥0,1≤j ≤m (19) B. OSCAR Collaboration Protocol j Figure 3 shows the state diagram of this protocol. The xnj ∈{0,1},1≤j ≤m (20) communicating entities in this protocol are client devices offering or requesting bandwidth (separated by vertical line) 5) Solution: Ingeneral,thisproblemisamixed0-1Integer and the trusted bank server supporting our incentive system. Programming problem, which is an NP-complete problem. The figure also shows the states generally broken down to However, it has a special structure that allows for an effi- the followingthree phases (separatedby the horizontallines): cient solution. In particular, we have two cases: if the new discovery and authentication, tendering, and payment. stream that triggeredthe schedulingdecisions is packet-based (S =0) and if it is connection-based (S =1). C. Discovery and Authentication n n Solution for the packet-based case: In order to maintain energy-efficient neighbor discovery, a In this case, xnj = 0∀j. The problem becomes a stan- requestingnode sends neighbordiscoveryrequests only when dard linear programming problem. Hence, it can be solved it has active packet-oriented connections or new connection efficiently to determine the different values of wj. requests. For this discovery, we adopt the eDiscovery [30] Solution for the connection-based case: protocol and apply it to the available interfaces to determine In this case, we need to determine the binary variables the best discovery duty cycle for each interface. ∀ x such that only one of them equals 1 and others are Once eDiscovery fires a neighbor discovery event, our j nj 0. Our algorithm sets each one of them to 1 and then solves OSCARprotocolengagesandbroadcastsaDiscoveryRequest the resulting linear programming problem to find ∀ w . The (DReq) packet.This discoverypackethas the header depicted j j value that achieve the best objective is then selected as the in Figure 4(a). It contains the OSCAR version running at optimal decision. the requesting client, header length, the discovery request