Table Of Content

View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by HAL-Rennes 1 JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers Radu Tudoran, Olivier Nano, Ivo Santos, Alexandru Costan, Hakan Soncu, Luc Boug´e, Gabriel Antoniu To cite this version: Radu Tudoran, Olivier Nano, Ivo Santos, Alexandru Costan, Hakan Soncu, et al.. JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers. DEBS’14 - The 8th ACMInternationalConferenceonDistributedEvent-BasedSystems,May2014,Mumbai,India. ACM, pp.23 - 34, 2014, <10.1145/2611286.2611298>. <hal-01090281> HAL Id: hal-01090281 https://hal.archives-ouvertes.fr/hal-01090281 Submitted on 19 Dec 2014 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destin´ee au d´epˆot et `a la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publi´es ou non, lished or not. The documents may come from ´emanant des ´etablissements d’enseignement et de teaching and research institutions in France or recherche fran¸cais ou ´etrangers, des laboratoires abroad, or from public or private research centers. publics ou priv´es. JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers Radu Tudoran Olivier Nano Ivo Santos IRISA/ENSRennes,France MicrosoftResearch–ATL MicrosoftResearch–ATL [email protected] Europe,Munich,Germany Europe,Munich,Germany [email protected] [email protected] Alexandru Costan Hakan Soncu Luc Bougé IRISA/INSARennes,France MicrosoftResearch–ATL ENSRennes,France [email protected] Europe,Munich,Germany [email protected] [email protected] Gabriel Antoniu InriaRennes-Bretagne Atlantique,Rennes,France [email protected] ABSTRACT rate can further be tripled thanks to the use of multi-route streaming. The easily-accessible computation power offered by cloud infrastructures coupled with the revolution of Big Data are CategoriesandSubjectDescriptors expandingthescaleandspeedatwhichdataanalysisisper- formed. In their quest for finding the Value in the 3 Vs of H.3.5[OnlineInformationServices]: Datasharing;D.2.4 Big Data, applications process larger data sets, within and [Distributed Systems]: Distributed applications acrossclouds. Enablingfastdatatransfersacrossgeograph- ically distributed sites becomes particularly important for Keywords applications which manage continuous streams of events in real time. Scientific applications (e.g. the Ocean Observa- Event Streaming, Cloud Computing, Multi Data-Centers, tory Initiative or the ATLAS experiment) as well as com- High Performance Data Management mercialones(e.g. Microsoft’sBingandOffice365large-scale services) operate on tens of data-centers around the globe 1. INTRODUCTION andfollowsimilarpatterns: theyaggregatemonitoringdata, On-demand resource provisioning, coupled with the pay- assesstheQoSorrunglobaldataminingqueriesbasedonin- as-you-go model, have redefined the way data is processed ter site event stream processing. In this paper, we propose nowadays. Scientific applications, simulations, monitoring, a set of strategies for efficient transfers of events between networks of sensors, trading or commercial data mining are clouddata-centersandweintroduceJetStream: aprototype allareaswhichhavebenefitedfromandsupportedtheemer- implementingthesestrategiesasahighperformancebatch- gence of cloud computing infrastructures. Cloud providers basedstreamingmiddleware. JetStreamisabletoself-adapt like Microsoft, Amazon, Google, Rackspace have built their to the streaming conditions by modeling and monitoring a cloud solutions at a global scale with data-centers spread setofcontextparameters. Itfurtheraggregatestheavailable across numerous geographical regions and continents. At bandwidth by enabling multi-route streaming across cloud thesametime,theversatilityofdataanalysishasincreased sites. The prototype was validated on tens of nodes from with applications running on multiple sites, which have to US and Europe data-centers of the Windows Azure cloud process larger data sets coming from remote locations and usingsyntheticbenchmarksandwithapplicationcodefrom distinct sources. The services dealing with such computa- the context of the Alice experiment at CERN. The results tions have to face important challenges and issues regard- showanincreaseintransferrateof250timesoverindividual ingthemanagementofdataacrossthesegeographicallydis- event streaming. Besides, introducing an adaptive transfer tributed environments. strategybringsanadditional25%gain. Finally,thetransfer Stream data processing is becoming one of the most sig- nificant subclass of applications in the world of Big Data. Vastamountsofstreamdataarecollectedatincreasingrates frommultiplesources[20,29]inmanyareas: climatemoni- toring,large-scalesensor-basedsystems,transactionalanal- Permissiontomakedigitalorhardcopiesofallorpartofthisworkfor personalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesare ysis, financial tickers, monitoring and maintenance systems notmadeordistributedforprofitorcommercialadvantageandthatcopies forwebservices,large-scalescienceexperiments. Acquiring, bearthisnoticeandthefullcitationonthefirstpage.Tocopyotherwise,to processing and managing this data efficiently raise impor- republish,topostonserversortoredistributetolists,requirespriorspecific tantchallenges,especiallyiftheseoperationsarenotlimited permissionand/orafee. to a single geographical location. There are several scenar- DEBS’14,May26-29,2014,Mumbai,India. ios which created the need to geographically distribute the Copyright2014ACM978-1-4503-2737-4...$15.00. computation on clouds. The size of the data can be so big andinareallifescenariousingtheMonALISA[33]monitor- that data have to be stored across multiple data-centers. It ing system of the CERN Alice experiment [1]. The experi- isthecaseoftheATLASCERNexperimentwhichgenerates mentsshowperformanceimprovementsof250timesoverin- 40PBofdataperyear. Furthermore,evenincrementalpro- dividualeventstreamingand25%overstaticbatchstream- cessingofsuchadatasetasastreamofeventswilloverpass ing,whilemulti-routestreamingcanfurthertriplethetrans- the capacity of local scientific infrastructure, as it was the fer rate. case of the Higgs boson discovery which had to extend the The rest of the paper is structured as follows. Section 2 computationtotheGooglecloudinfrastructure[2]. Another introducestheterminologyandpresenttheproblemswhich scenario is given by the data sources which can be phys- appearinthecontextofstreamingacrossclouddata-centers. ically distributed in wide geographical locations as in the Section 3 presents our approach for adapting the batch size Ocean Observatory Initiative [5, 12] in which the collected by modeling the latency for clouds, while considering the events are streamed to Nimbus [9] clouds. Finally, the na- cost of multi-routing. We continue with the description ture of the analysis, for an increasing number of services, of the system architecture and its implementation in Sec- requires aggregating streams of data from remote applica- tion 4 and discuss the evaluation of the approach across tioninstances. Large-scaleserviceslikeMicrosoft’sBingand Azure data-centers from different continents (i.e. Europe Office365operateontensofdata-centersaroundtheGlobe. andAmerica)inSection5. InSection6,wepresentanexten- Maintenance,monitoring,assertingtheQoSofthesystemor sivestateoftheartinseveraldomainstangenttostreaming global data mining queries all require (near) real-time inter anddatamanagementandpositionthecontributionsofour siteeventstreamprocessing. Allsuchcomputationscarried approach with respect to these directions. Finally, Section oncontinuousstreamsofeventsacrossresourcesfromdiffer- 7 concludes the paper. ent geographical sites are highly sensitive to the efficiency of the data management. 2. BACKGROUND:EVENTSTREAMINGON An extended survey over thousands of commercial jobs CLOUDS and millions of machine hours of computation, presented in [23], has revealed that the execution of queries is event- We start by defining the terminology related to stream driven. Furthermoretheanalysisshowsthattheinputdata processing on clouds and discuss the main problems which accountsonlyfor20%ofthetotalIO,therestcorresponding need to be addressed in this context. tothereplicationofdatabetweenqueryservicesortothein- 2.1 Context termediatedatapassedbetweenthem. Thisemphasizesthat the processing services, be they distributed, exchange large amounts of data. Additionally, the analysis highlights the Cloud computing is a recent paradigm which enables re- sensitivity of the performance of the processing of streams sources (e.g. compute power, storage capacity, network to the management and transfer of events. This idea is dis- bandwidth)tobeeasilyleasedon-demand,followingapay- cussedalsoin[20],inwhichtheauthorsstresstheneedfora as-you-go pricing model. From the infrastructure point highperformancetransfersystemforreal-timedata. Asim- of view, clouds are built across several data-centers dis- ilarconclusionisdrawnin[10],wheretheissueswhichcome tributedaroundtheglobe. Theproximityofthegeograph- fromthecommunicationoverheadandreplicationareexam- ical repartition of resources to users enables high QoS and inedinthecontextofstate-basedparallelization. Finally,in fastqueryresponsesbymeansofcontentdeliverynetworks [40],theauthorsemphasizetheimportanceofdatafreshness, (CDN). From the user point of view, all these resources which improves the Quality of Service of a stream manage- are harvested transparently by means of virtualization. A ment system and implicitly the quality of the data (QoD). typical cloud usage scenario consists of a user deployment Alltheseresearcheffortssupportandmotivatethe growing through which an application is run within virtual ma- need for a high performance system for event streaming. chines (VMs) on several cloud nodes from a data-center. In this paper, we address challenges like latency, transfer Large-scaleapplicationsandwebservicestypicallyrunmul- rate, throughput and performance variability for support- tiple deployments across several data-centers, enabling ge- ing high performance event transfers in the context of ge- ographical proximity to users and load balancing. Besides ographically distributed applications. Our solution, called the local computations answering to user requests, sev- JetStream, is a novel approach for streaming events across eralglobaloperations(e.g.,monitoring,QoSenhancement, cloud data-centers. In order to enable a high performance globaldatamining)requireinter-sitetransfers betweenge- transfer solution, we leverage batch-based transfers. The ographically distant nodes, most often in (near) real-time. size of the batches and the decision on when to stream the Thecloudsupportforsuchtransfersisratherrudimentary: events are controlled by modeling the latency based on a it relies on the cloud storage (e.g., Azure Blobs, Queues, set of parameters which characterize the streaming in the TablesorAmazonS3,SQS)asaglobalsharedpointwhich context of clouds. To further improve performance, we ag- incurs high transfer latencies [24, 45]. gregate inter-data-center bandwidth as we extend our ap- Stream computing referstotheprocessingofcontinuous proachtoprovidemulti-routestreamingacrosscloudnodes. sequences of relatively small data items [20], otherwise re- Furthermore, we developed a prototype that can be used ferred to as events. According to [17], the characteristics on clouds, either public or private, which is environment- ofdata streams are: 1)astreamisformedofevents,which aware by means of light monitoring. The information gath- aretuplesofrecords;2)itisapotentiallyinfinitesequence eredandinferredabouttheenvironmentiscoupledwiththe of events ordered by time (e.g., acquisition time, genera- streaming model, allowing the system to adapt the transfer tion time); and 3) the events can be produced by multi- decisions to all context changes. The system was validated ple external sources at variable rates, independent of the ontheWindowsAzure[6]cloudusingsyntheticbenchmarks constraints of the stream processing engine. The stream a) Transfer Rate (throughput) b) Average Latency of an Event Sender - Site 1 2500 0.03 Time between events 2000 0.025 MTBE=Rate1Acquisition Events/sec 11050000 Time(sec) 0.000..100512 Batch DestinationSP -rt roSecaietmses in2g 500 0.005 Serializer Network De-SerializerEngine 0 0 1 10 100 250 5001000 1 10 100 250 5001000 Batch Size (#Events) Batch Size (#Events) Figure 1: The performance of transferring events in SEovuerncte LatencyEncoding LatencyDecoding batchesbetweenNorthEuropeandNorthUSAzure LatencyBatching LatencyTransfer data-centers. The measurements show the correla- Figure 2: Breaking down the latency to deliver an tion between (a) the transfer rate and (b) the aver- eventfromsourcetostreamprocessingengineacross age event latency for increasing batch sizes. cloud nodes processingengines aresystemswhichtreattheeventsfrom Environment awareness - The cloud infrastructures are thestreambyapplyingtheuser’squeries. Asitisgenerally subjecttoperformancevariationsduetomulti-tenancy infeasible to store the streams entirely, the queries are ap- and network instability on the communication links plied continuously and iteratively over windows of events. withinandacrosssites. Monitoringanddetectingsuch Thecomputationisevent-driven,thereforemostofthedata performance changes allows the system to react ac- management is done at the granularity of an event. Gen- cordinglyandschedulethetransferofeventsefficiently. erally, the size of an event is small, in the range of bytes to kilobytes, which allows to handle them in-memory, but Decoupling the transfer from processing - The event also makes their transfer sensitive to any overhead. transfermoduleneedstobedesignedasastand-alone component, decoupled from the stream processingen- 2.2 Problemstatement gine. We advocate this solution as it allows seamless integration with any engine running in the cloud. At Achievinghighperformanceeventstreamingacrossdata- thesametime,itprovidessustainableandpredictable centers requires new cloud-based solutions, since currently performance, independent on the usage setup. there are no adequate tools to support data streaming or transfers across sites. Most of the existing stream process- Self-optimization - User configurations do not guarantee ingenginesonlyfocusoneventprocessingandprovidelittle optimal performance, especially in dynamic environ- ornosupportforefficienteventtransfers,simplydelegating ments. Moreover, when it comes to large-scale sys- this functionality to the event source. Today, the typical tems, the tasks of configuring and tuning the service way to transfer events is individually (i.e., event by event). tends to become complex and tedious. The alterna- Thisishighlyinefficient,especiallyacrossWAN,duetothe tive is to design autonomic cloud middleware, able to incurred latencies and overheads at various levels (e.g., ap- self-optimize. Coupledwithaneconomicmodel,these plication, technology tools, virtualization, network). systems could also regulate the resource consumption Abetteroptionistotransfereventsinbatches. Whilethis and enforce service-level agreements (SLAs). improvesthetransferrate,italsointroducesanewproblem, relatedtoselectingtheproperbatchsize. Figure1presents Generic solution - Building specific optimizations which thedependencybetweenthebatchsizeandthetransferrate, target precise applications is efficient, but limits the and the transfer latency per event, respectively. We notice applicabilityofthesolution. Instead,weproposeaset thatthekeychallengehereisthechoiceofanoptimalbatch of techniques which can be applied in any cloud con- size and the decision on when to trigger the batch sending. text, independent of the application semantics. Jet- This choice strongly relies on the streaming scenario and Stream does not depend on the nature of the data, on the sensed environment (i.e. the cloud). We aim to nor on the query types. tackle these problems by proposing an environment-aware 3.2 Zoomontheeventdeliverylatency solution, which enables optimum-sized batch streaming of events in the clouds. To achieve this, we model the latency We express the latency of the events based on a set of oftheeventtransferwithrespecttothecloudenvironment, cloudparameterswhichcanbemonitored. Suchatechnique dynamically adapt the batch size to the context and enable allows to bind the batch size corresponding to the minimal multi-route streaming across clouds nodes. eventlatencybothtothestreamandtheenvironmentinfor- mation. We start by breaking down the end-to-end latency 3. MODELINGTHESTREAMINGOFDATA ofaneventinfourcomponents,depictedinFigure2: form- ing the batch, encoding (e.g., serializing), transferring the INTHECONTEXTOFCLOUDS batchanddecoding(e.g.,de-serializing). Thesetofparam- Thecloudvariabilityandthefluctuatingeventgeneration eters able to describe the context and define the latency is: ratesrequireanappropriatemodelforeventstreaming,able the average acquisition rate (Rate ) or mean time Acquisition to capture the cloud specificities. In this section, we intro- between events (MTBE), the event size (Event ), the SizeMB ducesuchamodelandpresentthedecisionmechanismsfor serialization/deserializationtechnique,thethroughput(thr) selecting the number of events to batch. andthenumberofeventstoputinthebatch(i.e.,batchsize -batch ). Thegoalistodeterminethelatterdynamically. 3.1 DesignPrinciples Size Inthefollowing,wewillmodelthelatencycomponentsusing JetStream relies on the following set of design principles: these parameters. (cid:20)(cid:8)(cid:21)(cid:22)(cid:18)(cid:23)(cid:13)(cid:11)(cid:2)(cid:3)(cid:7)(cid:24)(cid:18)(cid:14)(cid:2)(cid:11)(cid:25)(cid:15)(cid:5)(cid:26) a) Throughput b) Approximating the Throughput (cid:23)(cid:15)(cid:25)(cid:18)(cid:23)(cid:13)(cid:11)(cid:2)(cid:3)(cid:7)(cid:24)(cid:18)(cid:14)(cid:2)(cid:11)(cid:25)(cid:15)(cid:5)(cid:26) 70 100 60 80 (cid:1)(cid:2)(cid:3)(cid:4)(cid:2)(cid:5) (cid:6)(cid:2)(cid:7)(cid:2)(cid:8)(cid:9)(cid:2)(cid:5) Throughput (MB/s) 23450000 Percentage (%) 246000 10 0 (cid:17)(cid:13)(cid:11)(cid:13)(cid:18) (cid:10)(cid:3)(cid:11)(cid:2)(cid:5)(cid:12)(cid:2)(cid:4)(cid:8)(cid:13)(cid:11)(cid:2) (cid:19)(cid:17)(cid:2)(cid:13)(cid:3)(cid:11)(cid:11)(cid:13)(cid:2)(cid:18)(cid:5) 0.00127030.1052554630.1205446312.054426310142.27531552510.29262018051.852417 0.0012730.1052554630.1205446312.05442631S0142i.z275e3 1(5M525B10).29262018051.852417 (cid:19)(cid:2)(cid:3)(cid:11)(cid:2)(cid:5) (cid:14)(cid:15)(cid:4)(cid:2)(cid:16) Size (MB) Number of channels Number of channels 1 2 Figure 3: The proposed schema for multi-route 1 2 3 35 46 streaming across cloud nodes. 4 5 6 Approximation Figure 4: a) The value of the throughput with re- The batching latency correspondstothedelayintroduced spect to the number of routes across cloud Small by an event waiting in the batch for other events to VMs for increasing size of the data chunk. b) Ap- arrive,beforetheyareallsentasabatch. Theparam- proximatingwithonepatternthecloudthroughput, eters which describe this latency are the average ac- independent of the number of routes. The approxi- quisition rate of the events and the number of events mation is built from the averages of the normalized in the batch. As the delay depends on the position throughput functions for each number of routes. of the event in the batch, we chose to use the av- erage latency per event. This can be computed by averaging the sum of the delays of all events in the the intra-data-center throughput (between the nodes within batch: Latency = batchSize . Intuitively, one’s single site deployment) is relatively stable and a pri- batching 2×RateAcquisition ori known, as it is specified by the cloud provider within this corresponds to the latency of the middle event. the SLA (e.g., 100 Mbps for Small Azure VMs). On the The transformation latency groupsthetimestoencode other hand, there are no guarantees when it comes to the and to decode the batch using a specific serialization inter-data-center throughput (betweenthedata-centers). In library. It depends on the used format (e.g. binary, fact,thehighlatencyandgenerallylowthroughputnetwork JSON etc.), the number of bytes to convert and the connecting the sites is not owned by the cloud provider, as numberofeventsinthebatch. Toexpressthis,werep- the packet routing is done across multiple internet service resent the transformation operation as an affine func- providers (ISPs). This makes it more unstable, unknownin tion (i.e. f(x) = ax+b) where a corresponds to the advance and subject to poor performance. conversionrate(i.e. amountofbytesconvertedpersec- Inordertoaddresstheissueoflowinter-data-centerthrough- ond-tDs),whilethebconstantgivesthetimetowrite put, we designed a transfer strategy suitable for a typical themetadata(tHs). Thelatencypereventcanthenbe cloudsetup(i.e.,applicationrunningacrossseveralnodesin expressedas: Latency = tHs+tDs×batchSizeMB, differentclouddata-centersdeployments),whichcanharvest whichholdsbothfortthraensefonrcmoadtiionnganddecobdaticnhgSizoeper- extrabandwidthbyusingadditionalintermediatenodes. The ations. Thisformulacanalsobeappliedtoexpressthe idea is to use multiple routes for streaming across sites, as size of the data to be transferred by simply replacing depicted in Figure 3. We build this strategy on two obser- the time constants with data related constants (i.e. vations: the virtual inter data-center routes do not map to size of the metadata and the compression/extension identical physical paths and the latency of intra-site com- ratio). munication is low (less than 10%) compared to inter site communication [45]. These observations allow to aggregate The transfer latency models the time required to trans- additionalbandwidthbysendingdatafromthesendernodes fer an event between cloud nodes across data-centers. to intermediate nodes within the same deployment (i.e. be- To express it, we consider both the amount of data in longing to the same application space), which will then for- the batch as well as the overheads introduced by the ward it towards the destination. With this approach, the transferprotocol(e.g. HTTP,TCP)-sOtandtheen- system aggregates extra physical bandwidth from the dif- coding technique - sOe. Due to the potentially small ferent paths and routes established when connecting dif- size of data transferred at a given time, the through- ferent nodes across the ISP infrastructures. This generic put between geographically distant nodes cannot be method of aggregating inter-site bandwidth is further dis- expressed as a constant value. It is rather a func- cussed in [45], in the context of bulk multi-hop data trans- tion of the total batch size (SizeTotal =batchSizeMB× fers across multiple cloud data-centers. To integrate this batchSize), since the impact of the high latency be- approachwithinourmodel,weextendthesetofparameters tween data-centers depends on the batch size. The that are used to characterize the stream context with the cloud inter data-center throughput - thr(Size) is dis- number of channels. cussed in more detail in the following section. The Independent of the number of routes used for streaming, average latency for transferring an event can then be the value of the throughput function has to be known at expressed as Latencytransfer = sOt+tshOr(eS+izbeaTtocthaSli)zeMB runtime. In order to limit the number of network samples that need to be performed to obtain it, and to reduce the 3.3 Multi-routestreaming monitoring intrusiveness, we approximate this function. In The throughput is one of the most peculiar aspects when Figure 4 a) we present measurements of the throughput for consideringcommunicationintheclouds. Ontheonehand, a number of routes between North Central US and North (cid:23)(cid:5)(cid:7)(cid:24)(cid:2)(cid:7)(cid:19)(cid:14)(cid:12)(cid:13) "(cid:9)(cid:13)(cid:9)(cid:12)(cid:19) Algorithm 1 The selection of the optimal batch size and #$!(cid:11)(cid:16)(cid:3)(cid:14)(cid:10)(cid:2)(cid:3)(cid:9)!(cid:31)(cid:7)(cid:7)(cid:14)(cid:21)(cid:2)(cid:20)!(cid:22)(cid:2)(cid:3)(cid:9)! (cid:25)(cid:5)(cid:19)(cid:9)(cid:16) the number of channels to be used (cid:28)(cid:2)(cid:3)(cid:8)(cid:29) (cid:4)(cid:9)(cid:12)(cid:19)(cid:9)(cid:7) (cid:2)%(cid:12)$!(cid:19)(cid:18)!(cid:10)(cid:9)(cid:2)(cid:9)(cid:16)(cid:16)(cid:6)(cid:16)(cid:7)(cid:2)(cid:9)(cid:13)!(cid:9)(cid:15)!(cid:29)(cid:4)(cid:7)(cid:14)(cid:5)(cid:26)(cid:9)(cid:6)(cid:13)(cid:29) (cid:6)(cid:3) 1: procedureBatchAndChannelsSelection (cid:30)(cid:7)(cid:2)(cid:8)(cid:20)(cid:9) (cid:25)(cid:5)(cid:19)(cid:9) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) &$!’(cid:12)((cid:6)(cid:14)(cid:7)(cid:9)!(cid:24)(cid:29)(cid:9)(cid:12)!(cid:3)(cid:5)!(cid:16)(cid:9)(cid:12)(cid:19) 2: getMonitoredContextParameters() # & % (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) 3: ESTIMATEMaxBatchedfrom[MaxTimeConstraint] (cid:28)(cid:6)*(cid:9)(cid:7) (cid:11)(cid:21)(cid:9)(cid:12)(cid:3) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) (cid:11)(cid:21)(cid:9)(cid:12)(cid:3) (cid:28)(cid:6)*(cid:9)(cid:7) 4: whilechannels<MaxNodesConsidereddo (cid:4)(cid:9)(cid:12)(cid:19)(cid:9)(cid:7) (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) (cid:22)(cid:9)(cid:8)(cid:9)(cid:14)(cid:21)(cid:9)(cid:7) 5: whilebatchsize<MaxBatcheddo (cid:22)(cid:9)(cid:8)(cid:9)(cid:14)(cid:21)(cid:9)(cid:7) 6: ESTIMATE latency from (cid:4)(cid:9)(cid:7)(cid:14)(cid:2)(cid:20)(cid:14)(cid:26)(cid:9)(cid:7) )(cid:31)(cid:25) (cid:25)(cid:5)(cid:19)(cid:9) (cid:1)(cid:9)(cid:27)(cid:4)(cid:9)(cid:7)(cid:14)(cid:2)(cid:20)(cid:14)(cid:26)(cid:9)(cid:7) [RateAcquisition,batchsize] batching 7: ESTIMATE latency from encoding [overheads,batchsizeMB] (cid:1)(cid:2)(cid:3)(cid:2) (cid:1)(cid:2)(cid:3)(cid:4)(cid:3)(cid:5)(cid:2)(cid:6)(cid:7) (cid:4)(cid:3)(cid:7)(cid:9)(cid:2)(cid:10) (cid:46)Estimatethetransferlatencyfor1channel (cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:9) (cid:31)(cid:19)(cid:2) (cid:3)(cid:14)(cid:21)(cid:9)!(cid:28)(cid:2)(cid:3)(cid:8)(cid:29)!(cid:4)(cid:3)(cid:7)(cid:9)(cid:2)(cid:10)(cid:9)(cid:7) (cid:11)(cid:12)(cid:13)(cid:14)(cid:12)(cid:9) 8: ESTIMATE latencytransfer1 from [batchsizeMB,thrRef,1channel] Figure 5: The architecture and the usage setup of 9: COMPUTE RatioCPU from [latency ,latency ,VM Cores] encoding batching the adaptive batch streamer (cid:46)Preventsidlechannels Europe Azure data-centers. In Figure 4 b) we normalize 10: if RatioCPU ∗ latencybatching × channels < latency +latency then thesevalues(i.e.,%ofthecorrespondingstablethroughput) 11: transfer1ESTIMATEencoding latency from decoding and approximate them using a polynomial function, which [overheads,batchsizeMB] we determined empirically that it gives the most accurate 12: ESTIMATE latency from transfer approximation. Hence such a model can in fact represent [batchsizeMB,thrRef,channels] the throughput pattern, with an error introduced by the 13: ESTIMATE latencyreordering from cloudvariabilityoflessthan15%. Usingthisapproximation, 14: [channels,latenlactyetnracnysfer] =(cid:80)latency the entire function can be extrapolated by measuring only perEvent ∗ 15: if latency <bestLatencythen the asymptotic stable throughput. This will be used as the perEvent 16: UPDATE[bestLatency,Obatch,Ochannels] amplitude which multiplied with the normalized estimation 17: endif will give the throughput for any size. 18: endif Thedownsideofusingmultipleroutesforsendingbatches 19: endwhile isthattheorderingguaranteesofferedbythecommunication 20: endwhile protocol for one link do not hold anymore. This translates 21: endprocedure into batches arriving slightly out of order due to changing conditions on the communication links (e.g., packet drops, congestions etc.). It is mandatory to maintain the integrity size, the maximal end-to-end event latency introduced by ofthecommunicationandavoiddroppingdatajustbecause batching can be unsatisfactory for a user, as it might vio- anotherlinkwasfastersothatbatchesneedtobereordered late application constraints, even if the system operates at atthedestination. Weachievethisbybufferingthebatches optimaltransferrates. Hence,theuserscansetamaximum at the destination until their turn to be delivered to the acceptable delay for an event, which will be converted in a streamingenginearrives. Thisintroducesanewcomponent maximum size for the batch (Line 3). As for the number for the latency: latency for reordering. It can be modeled ofchannelsselection,weestimatehowmanybatchescanbe using the Poisson distribution (Poisson(k,λ) = λk×e−λ) to formed while one is being transferred (Lines 6-8). k! estimate the probability of having k number of batches ar- Once the transfer of a batch is completed, adding ad- riving before the expected batch, which fixes λ to 1, as ditional channels to distribute the batches become useless. we consider as reference the time to transfer one batch . The condition on Line 10 prevents the system from creat- This probability can then be correlated with a penalty as- ing idle routes. Finally, the CPU usage needs to be also signedtoeachunorderedbatch,givenbyitslatency. Finally, integrated in the decision process. Sending frequent small summing these probabilities over the number of channels batches will increase the CPU consumption and artificially and normalizing based on the events will give an estima- decrease the overall performance of the cloud node. We tion of the worst case expected latency: Latency = therefore assign a penalty based on the ratio between the reordering (cid:80)ci=ha2nnels(cid:80)Lj Poisson(j,1)×j×Latency×batch. Here, L gives the time to form a batch and the time used by the CPU to en- ×batch×size×L code it, according to the formula RatioCPU = maximum number of batches (e.g., 10) regarded as poten- latencyencoding . Tosumup,JetStream tially arriving before the reference one through a channel, (latencybatching+latencyencoding)×VM Cores collectsasetofcontextparameters(Line2)andusesthemto givingtheupperlimitforiteratingthekPoissonparameter. estimate the latency components according to the formulas 3.4 Adaptivecloudbatching presented above. Based on these estimations, it selects the optimalbatchsizeandthenumberofchannelsforstreaming. InAlgorithm1wewrapupeverythingandpresentthede- cision mechanism in JetStream for selecting the number of 4. SYSTEMOVERVIEW events to batch and the number of channels to use. The al- gorithmestimatestheaveragelatencypereventforarange We designed JetStream as a proof of concept of the pre- of batch sizes and channels, retaining the best one. As an vious approach. Its conceptual schema is presented in Fig- optimization, a simulating annealing technique can be used ure5. Theeventsarefedintothesystematthesenderside, toperformthesearchfaster,byprobingthespacewithlarge assoonastheyareproduced,andtheyarethendeliveredto steps and performing exhaustive searches only around local the stream processing engine at the destination. The pro- optima. Depending on the magnitude of the optimal batch totypewhichimplementsthisarchitecturewasdevelopedin C#usingthe.NET4.5framework. Thedistributedmodules Latency components and their functionality are described bellow: 0.014 0.012 The BfourffJeertSitsreuasmed. bTothheassenadneirnpauptplaicnadtioountpourttehnedepvoeinntt me (sec) 00..000.000681 source adds the events to be transferred, as they are Ti 0.004 produced,whilethereceiverapplication(i.e.,thestream 0.002 0 processing engine) pops (synchronously) the events or 1 5 10 25 50 100 200 400 500 750 1000 Batch Size (#Events) it is notified (asynchronously) when the events have Latency been transferred. The module relies on two queues Total Batching Transfer for synchronization purposes in a producer-consumer Transformation fashion. TheBufferisalsoinchargeofmonitoringthe Figure 6: The latency components per event with input stream in order to assert in real time the acqui- respecttothebatchsizeforasinglestreamingchan- sition rate of the events and their sizes. nel, for inter-sites event transfers Avro (Microsoft HDInsight), but others modules can The Batch Oracle staysatthecoreofJetStream,asiten- be easily integrated. This module can host additional forces the environment-aware decision mechanism for functionalities (e.g., data compression or deduplica- adaptively selecting the batch size and the amount of tion) in the future. channels to use. It implements Algorithm 1 and col- lects the monitoring information from the Buffer and the Transfer Module. It further controls the moni- 5. EVALUATION toring intrusiveness by adjusting the frequency of the monitorsamplesaccordingtotheobservedvariability. Thegoaloftheexperimentalevaluationpresentedinthis section is to validate the JetStream system in a real cloud The Transfer Module implementsthemulti-routestream- setup and discuss the main aspects that impact on its per- ing. On the intermediate nodes, its role is to forward formance. the packets towards the destination. Currently, the 5.1 Experimentalsetup modulesisagnosticofanystreamingscenarioandcan integrateanytransferprotocol. Itprovidesseveralim- The experiments were run in the Microsoft’s Windows plementations on top of TCP: synchronous and asyn- Azure cloud in the North-Central US and the North Eu- chronous,single-ormulti-threaded. Itisalsoincharge rope data-centers. We chose these sites in order to create a ofprobingthenetworkandmeasuringthethroughput geographical-distributedsetupwhichpermitstoanalyzethe and its variability. The batches are sent in a round- abilityofthesystemtoadapttochangingfactorsgenerated robin manner across the channels, balancing the load. both by the cloud itself (i.e., multi-tenancy, virtualization) As future work, the transfer performance can be im- andthewideareacommunicationacrossmultipleISPs. All proved by integrating additional transfer techniques theexperimentswererunwithSmallWebRoleVMshaving amongwhichthesystemcanautomaticallyswitchbased 1 CPU, 1.75 GB of memory, 225 GB local storage and 100 onthestreamingpatternandthecontextinformation. Mbps bandwidth. When using multi-route streaming, up to 5 additional nodes were used within the sender deploy- The Event Sender coordinatestheeventtransfersbyman- ment, implementing the transfer scheme discussed in Sec- aging the interaction between all modules. It queries tion 3.3. Each experiment sends between 100,000 and 3.5 the Batch Oracle about when to start the transfer of million events, which, given the size of an event, translates the batch. Next, it setups the batch by getting the intoatotalamountofdatarangingfromtensofMBsto3.5 eventsfromtheBuffer andaddingthemetadata(e.g., GB.Eachsampleiscomputedastheaverageofatleastten batch ID, streams IDs etc.). The batch is then serial- independent runs of the experiment performed at various ized by the Serialization module and the data trans- moments of the day (morning, afternoon and night). ferred across data-centers by the Transfer Module. The performance metrics considered are the transfer rate The Event Receiver isthecounterpartoftheEventSender and the average latency of an event. The transfer rate is module. The arriving batches are de-serialized, re- computed as the ratio between a number of events and the ordered and delivered to the application as a stream time it takes to transfer them. More specifically, we mea- of events, in a transparent fashion for the stream pro- sured,atthedestinationside,thetimetotransferafixedset cessing engine. The module issues acknowledgements ofevents. Fortheaveragelatencyofanevent,wemeasured to the sender or makes requests for re-sending lost or thenumberofeventsinthesenderbuffer, thetransfertime delayed batches. Alternatively, based on users’ poli- andreportedthenormalizedaveragepereventbasedonthe cies, it can decide to drop late batches, supporting latency formulas described in Section 3.2. theprogressofthestreamprocessingdespitepotential Theevaluationisperformedusingasyntheticbenchmark cloud-related failures. Users configure when such ac- and a real-life application. We created a synthetic bench- tionsareperformedbymeansofwaitingtimes,number markinordertohavefullcontroloverthesizesandthegen- ofbatches,markersorcurrenttimeincrements(CTI). eration rate of the events. The generation rates are varied between hundred of events per second to tens of thousands Serialization/De-Serialization has the role of convert- ofeventspersecond,asindicatedbythescaleofthetransfer ing the batch to raw data, which can be afterwards rates. Such a predictable and configurable setup allows us sent over the network. We integrate in our prototype to analyze and to understand the results better when test- several libraries: Binary (native), JSON (scientific) or ing the behavior of JetStream in several scenarios. Finally, Real Measured Model-based Estimation Low Events Arrival Rate High Events Arrival Rate 4 4 Transfer Rate Transfer Rate 1200 3 3 16000 Time (sec) 12 Time (sec) 12 Events/sec 1046800000000 Events/sec 128000000 4000 200 0 50 100 500 1000 2500 0 50 100 500 1000 2500 0 1 Route 3 Routes 5 Routes 0 1 Route 3 Routes 5 Routes Batch Size (#Events) Batch Size (#Events) Average Latency per Event Average Latency per Event 2 3 4 5 2 3 4 5 6 Number of channels 6 Number of channels 0.002 0.0002 Fcaigseurlaete7n:cCyo(mi.ep.,arthinegptehnealetyst)imofautinoonrdoefrethdebwatocrhsets- me (sec) 000...000000011826 me (sec) 00..800E00−00011526 with actual measurements, while increasing the Ti 0.0004 Ti 4E−05 number of channels used for the transfers 0 1 Route 3 Routes 5 Routes 0 1 Route 3 Routes 5 Routes Batch Size Batch Size 1 10 100 1 10 100 we use the events collected by the MonALISA monitoring 1000 5000 JetStream 1000 5000 JetStream systems from the Alice LHC experiment to assert the gains Figure8: Comparingtheperformance(transferrate brought by our approach in a real setup. - top and average latency per event - bottom) of in- dividualeventstreamingandstaticbatcheswiththe 5.2 Accuracy adaptivebatchofJetStreamfordifferentacquisition rates, while keeping the size of an event fixed (224 We depict in Figure 6 the total latency and its compo- bytes). Thelatencyandtransferratesforindividual nents, per event, with respect to the number of batched event streaming (i.e., batch of size 1) are between events. Selecting the optimal size of the batch comes down 50 to 250 times worse than the ones of JetStream to finding the value corresponding to the minimal latency (e.g. ∼ 200 for the scenario illustrated in Figure 6). The Transfer Rate Average Latency per Event searchforthebatchsizethatminimizesthelatencyperevent 12000 0.0002 0.00018 is at the core of the JetStream algorithm presented in Sec- 10000 0.00016 tomifoarnteipo3nr.4es.seaTnbtohauettiosetnlhefecotreiontnvhierooflnattmheenencoytp(tiseim.cgoa.mlnvpeautlwuteeodrfkrfrootmhmrtothuhigsehtepysupttie-, Events/sec 468000000000 Time (sec) 00..06800.EE000−−0000011055241 CPU),whichmaynotbeexact. Indeed,thecloudvariability 2000 4E−05 canleadtodeviationsaroundtheoptimalvalueintheselec- 2E−05 0 0 tion process of the amount of events to batch. Considering 1 Route 3 Routes 5 Routes 1 Route 3 Routes 5 Routes thattheperformancearoundtheoptimumisratherflat(as Batch Size Batch Size seen in Figure 6), JetStream delivers more than 95% of the 1 10 1 10 100 1000 100 1000 optimal performance even in the case of big and unrealistic 5000 JetStream 5000 JetStream shifts from the optimum batch size (e.g. selecting a batch Figure 9: The performance (transfer rate – left and sizeof150or250insteadof200inFigure6willdecreasethe averagelatencyperevent–right)ofindividualevent performance with 3%). The solution for further increasing streaming, static batches and JetStream for events the accuracy, when selecting the batch size, is to monitor of size 800 bytes. The latency and transfer rates for the cloud more frequently. These observations show that individual event streaming (i.e. batch of size 1) are accuracy can be traded for monitoring intrusiveness at the from 40 up to 150 times worse than JetStream level of cloud resources (e.g. network, memory and CPU). In order to validate the penalty model proposed for the 5.3 Individualvs. batcheventtransfers latency of reordering when using multiple routes, we com- pare the estimations of our approach with actual measure- The goal of this set of experiments is to analyze the per- ments. TheresultsarepresentedinFigure7. Thedelayfor formance of individual event streaming compared to batch- unordered batches was measured as the time between the based streaming, with static and adaptive (i.e., JetStream) momentwhenanoutoforderbatch(nottheonenextinse- sizes. Inthestaticcase,thenumberofeventstobebatched quence) arrives, and the moment when the actual expected is fixed a priori, with a batch of size 1 representing inde- one arrives. The experiment shown in Figure 7 presents pendent event streaming. These setups are compared to the average of this maximum unordered delay over tens of JetStream, which adapts the batch size to the context at independent trials in which a fixed amount of events (i.e. runtime, for varying event sizes or generation rates. 1 million events of size 2.5 KB) is transferred. We notice The experiments presented in Figure 8 use an event of thattheproposedmodelgivesagoodestimationofthisde- size 224 bytes and evaluate the transfer strategies consider- lay, which can be used as a penalty latency for multi route ing low (left) and high (right) event generation rates. The streaming. The accuracy in this case is 90%–95%, because experiments were repeated for different number of routes the penalty model for multi-routing does not integrate the for streaming: 1, 3 and 5, and measure the transfer rates variability of the cloud. Yet, such errors can be tolerated (top)andaveragelatencyperevent(bottom). Thefirstob- as they are not determinant when selecting the number of servation is that batch-based transfers clearly outperform channels to use. individual event transfers for all the configurations consid- ered. The results confirm the high overhead and the low (cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:6)(cid:7)(cid:2)(cid:8)(cid:3)(cid:9)(cid:7) 35000 throughput with small sizes for the transfers between data- 30000 ctoenhtuernsd.rGedrooufptinimgetshe(uepvetnots2i5n0crteimaseesstfohreJtreatSntsrfeeramra)tewtheinles Ba51t00c00h00Size ents/sec 122505000000000 decreasing the average latency per event. Two aspects de- 1Je0t0Stream Ev 10000 EventArrivalRate 5000 terminetheperformance: thesizeofthebatchwithrespect 0 to the stream context and the performance changes of the 1 3 5 7 T9ime11int1e3rva15l(11T07imsee1c9oinnt2de1srv)a23l(1205se27con2d9s) cloud. Static batches cannot handle these variations, mak- 30000 7000 i(aneng.dgc.heribgtaahticnehvbeeasnttochfacssqiizzueeiss1igt0iooogndivreiantpeoonaoenrdcpogneortfoeoxdrtmpaeanrndfcoebrmafdoarninc1eortfohouertr5es Events/sec 112205055000000000000000 Events/sec 123456000000000000000000 routes and low acquisition rates). Selecting the correct size 0 5 6 7 0 7 8 9 atruntimebringsanadditionalgainbetween25%and50% Timeinterval(10seconds) Timeinterval(10seconds) over static batch configurations. We repeated the same set BatchSize BatchSize 5000 1000 100 JetStream 5000 1000 100 JetStream of experiments for bigger event sizes, all showing the same behavior. Due to space constraints, we only illustrate the Figure10: Theevolutionofthetransferrateintime transfer rate and average event latency for an event size of for variable event rates with JetStream and static 800 bytes in Figure 9. batches transfer strategies Transfer Rate 5.4 Adaptingtothecontextchanges 40000 Thedataacquisitionprocessinastreamingscenarioisnot necessarilyuniform. Fluctuationsintheeventratesofanap- 30000 c poflitchaetidonatrausnonuirncge,inthtehveirctluoualdizceadninafprpasetarrudctuueretoorthtehencaltouurde nts/se 20000 e performancevariability[16]. ToanalyzethebehaviorofJet- Ev 10000 Stream in such scenarios, we performed an experiment in which the event generation rate randomly changes in time. 0 For the sake of understanding, we present in Figure 10 a 1 3 5 Number of streaming routes snapshot of the evolution of the transfer rate in which we use fine grain intervals (10 seconds) containing substantial 100 Batch Size JetStream rate changes. JetStream is able to handle these fluctua- Figure 11: The transfer rate for an increasing num- tions by appropriately adapting the batch size. In contrast, ber of routes used for streaming static batch transfers either are introducing huge latency fromwaitingfortoomanyevents,especiallywhentheevent cisely, a larger bandwidth allows to send more data, and acquisition rate is low (e.g., batches of size 1000 or 5000 at implicitly, the additional data carried with each batch does time moment 9) or are falling behind the acquisition rate not throttle the network anymore. This brings the trans- which leads to increasing amount of memory used to buffer ferrateofsmaller,andconsequentlymorefrequent,batches the untransferred events (e.g., batch of size 100 at moment closertothemaximumpotentialeventthroughput. Itisthe 5). Reacting fast to such changes is crucial for delivering caseofthestaticbatchofsize100onFigure11,deliveringa high performance in the context of clouds; this is also the throughput close to JetStream for higher number of routes. reasonwhywechosetoconsidersuchsuddenrateschanges. With higher throughput and a lower overhead, the optimal The number of used resources (e.g. the extra nodes which batch size can be decreased. When selecting it, JetStream enable multiple route streaming) is reduced by 30% when isabletodecreasetheendtoendlatency. Weconcludethat takingintoaccountthestreamcontext. Theseresourcesav- sustaininghightransferratesunderfixedtimeconstraintsis ingsarejustifiedbythefactthatloweventacquisitionrates possiblebyimposingupperboundsforthebatchsizes,which do not require multiple route streaming as higher ones do. enables JetStream to integrate users’ time constraints for Additionally, as shown in [23], the fluctuations in the ap- maximum delay. As discussed in Algorithm 1, it does this plication loads have certain patterns across the week days. by considering a maximum limit on the batch. Hence,inlongrunningapplications,ourapproachwillmake 5.6 ExperimentingwithALICE:areal-life substantialsavingsbydetectingthesedailyorhourlytrends and using such knowledge to scale up/down the number of HighEnergyPhysicsapplication additional nodes used for transfers. In a second phase, our goal was to assess the impact of JetStream in a real-life application. We opted for ALICE 5.5 Benefitsofmultipleroutestreaming (A Large Ion Collider Experiment) [1], one of four LHC Figure 11 shows the gain obtained in transfer rate with (Large Hadron Collider) experiments at CERN (European respecttothenumberofroutesusedforstreaming,forJet- OrganizationforNuclearResearch),asitsscale,volumeand Stream and for a static batch of a relatively small size (i.e. geographical distribution of data require appropriate tools 100events). Multipleroutestreamingpaysoffwhenincreas- for efficient processing. Indeed, the ALICE collaboration, ing the amount of data sent for both strategies (i.e. static consisting of more than 1,000 members from 29 countries andadaptive). Infact,byaggregatingextrabandwidthfrom and 86 institutes, is strongly dependent on a distributed the intermediate nodes, we are able to decrease the impact computingenvironmenttoperformitsphysicsprogram. The of the overhead (batch metadata, communication headers, experiment collects data at a rate of up to four petabytes serialization headers etc.) on smaller batches. More pre- per year, produce more than 109 data files per year, and require tens of thousands of CPUs to process and analyze a) Latency b) Transfer Rate them. TheCPUandstoragecapacitiesaredistributedover 2500 1600 morethan80computingcentersworldwide. Theseresources 2000 1400 1200 atoreOohpueertrefaortocinugegs,nseiynosuttsehmienseaanlsledarisbepasetcoctfhse,qxfurpoeeumriinmCgePnsUotsft,mwisoadroeen.lathnedmcoounnit- Time (sec) 11050000 Events/sec 1068000000 toringinformationcollectedinreal-timeaboutallALICEre- 500 400 200 sources. WeusedtheMonALISA[33]servicetoinstrument 0 0 timheenhtu.geMaomreoutnhtanof3m50onMitoorniAngLIdSaAtasiesrsvuiecdesfraorme rtuhnisneinxgpeart- 10 100 1000 1000J0etStream 1 10 100 1000 100J0et0Stream sitesaroundtheworld,collectinginformationaboutALICE Batch Size (#Events) Batch Size (#Events) computingfacilities,localandwideareanetworktraffic,and Figure 12: The total latency (a) and the trans- thestateandprogressofthemanythousandsofconcurrently fer rate (b) measured when transferring 1.1 million running jobs. This yields more than 1.1 million parameters eventsfromMonALISAmonitoringtheAliceexper- published in MonALISA, each with an update frequency of iment. The latency for independent event transfer one minute. Using ALICE-specific filters, these raw pa- (i.e. batchofsize1)ina)isnotrepresentedbecause rameters are aggregated to produce about 35,000 system- itwouldmodifythescalegreatlydueitsmagnitude, overview parameters in real time. These overviews are usu- having a value of 30900 seconds as opposed to 80 ally sufficient to identify problems or to take global actions seconds for JetStream inthesystemandtheyarethefuelfortheJetStreamevent streaming platform. The MonALISA framework and its latencyoftheeventsasdepictedinFigure12a. Asourap- highfrequencyupdatesforlargevolumesofmonitoringdata proach selects the appropriate batch size at each moment, matched closely with JetStream’s architecture so the pieces itconsequentlyreducestheamountofeventswaitinginthe fitnaturallytogether,butitalsoprovidedusamajoroppor- senderqueueanddecreasestheoveralllatencyoftheevents. tunity to fullfill the system’s initial goal: using the (moni- Compared to the static batch strategies providing constant toring)datatotakeautomateddecisionsinrealtime(inthis transfer performance, the latency of JetStream is decreased context, improve the observed system). from2.2(100-eventbatches)downto17times(10000-event Based on the monitoring data collected and stored by batches). MonALISA as of December 2013, we have replayed a se- quenceof1.1millioneventsconsideringtheircreationtimes at the rate they were generated by Alice. The measure- 6. RELATEDWORK mentswereperformedusing2additionalVMsasintermedi- Theexistingworkscanbegroupedinto5categoriesbased ate nodes at the sender side (i.e. 3 streaming routes). Ini- ontheirapplicationdomainsandfeatures. Wediscussthem tially,theexperimentalsetupconsidered5streamingroutes. in this section. However,duringthetransferofthedatausingJetStream,we observedthatmaximum3suchrouteswereused,asthesys- Systemsfordatatransfers. tem determined that the performance cannot be increased Severalsolutionshavebeenproposedinthelastyearsfor beyond this point. The same number of nodes is recom- managing data transfers at large scale. StorkCloud [30] in- mendedifwequerythesystembasedonthestreamcontext tegrates multi-protocol transfers in order to optimize the (i.e. event size and acquisition rate). The accuracy of this end-to-end throughput based on a set of parameters and decisionwasinfactvalidatedasouradaptiveapproachwas policies. It adapts the parallel transfers based on the clus- obtaining the same transfer performance using 3 nodes as ter link capacity, disk rate and CPU capacity, using the the static batch configurations which were using 5. Fur- algorithm proposed in [47]. The communication between thermore, the static configurations also obtained the same StorkCloud components is done using textual data repre- transfer performances when switched to 3 nodes, showing sentation. Similarly, our system optimizes the streaming that indeed this streaming context was not requiring more betweendata-centersbymodelingasetofparameters. How- than 3 streaming routes. This observation shows the im- ever,JetStreamremainstransparenttothedataformatand portance of an environment-aware adaptive approach, not limits the annotation of events or batches with metadata, subject to arbitrary human configuration choices. reducing the overhead of the transfer, which becomes cru- Figure 12 displays the total latency of the events at the cial in the context of continuous streams of data. Other sender(Figure12a)andthetransferrate(Figure12b)when systems, like NetSticher [32], target bulk transfers between comparing JetStream with static configurations for varying data-centers, as we do. The authors exploit the day/night batchsizes. Thetransferperformancesofstaticbatcheswith patternpeaksofusageofadata-centerandleveragethere- more than 100 events are similar with JetStream. Consid- mainingunutilizedbandwidth. ThoughNetSticherisuseful ering that the generation rate of the events varies from low for backups and checkpoints, it does not work for real-time to high, sub-optimal batch sizes will in fact lead to an ac- systems. GlobusOnline[3]isanothersystemwhichprovides cumulation of the events in the sender queue during the data transfers over WAN, targeting data sharing between peak rates. These buffered events will artificially increase different infrastructures (e.g. grids, institution infrastruc- the performance, at the expense of extra memory, during tures,communityclouds). Unlikeoursystemwhichhandles theperiodswhentheacquisitionrateofeventsislow. Allin streams of events with a light resource fingerprint, Globu- all,thisbehaviorwillproduceastabletransferperformance sOnline manages only file transfers and has substantial re- overawiderangeofstaticbatchsizes,asitcanbeobserved sourcerequirements. Otherrecenteffortsforenablingparal- in Figure 12 b; but on the other hand, it will increase the leldatatransfersleadtothedevelopmentofthemulti-path

Description:
Luc Bougé, Gabriel Antoniu. To cite this version: tory Initiative or the ATLAS experiment) as well as com- mercial ones Stream data processing is becoming one of the most sig- . JetStream relies on the following set of design principles: toring) data to take automated decisions in real time (in
ebook img

JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers PDF

13 Pages·2016·1.24 MB·English
by  
Save to my drive
Quick download
Download
Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers

View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by HAL-Rennes 1 JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers Radu Tudoran, Olivier Nano, Ivo Santos, Alexandru Costan, Hakan Soncu, Luc Boug´e, Gabriel Antoniu To cite this version: Radu Tudoran, Olivier Nano, Ivo Santos, Alexandru Costan, Hakan Soncu, et al.. JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers. DEBS’14 - The 8th ACMInternationalConferenceonDistributedEvent-BasedSystems,May2014,Mumbai,India. ACM, pp.23 - 34, 2014, <10.1145/2611286.2611298>. <hal-01090281> HAL Id: hal-01090281 https://hal.archives-ouvertes.fr/hal-01090281 Submitted on 19 Dec 2014 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destin´ee au d´epˆot et `a la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publi´es ou non, lished or not. The documents may come from ´emanant des ´etablissements d’enseignement et de teaching and research institutions in France or recherche fran¸cais ou ´etrangers, des laboratoires abroad, or from public or private research centers. publics ou priv´es. JetStream: Enabling High Performance Event Streaming across Cloud Data-Centers Radu Tudoran Olivier Nano Ivo Santos IRISA/ENSRennes,France MicrosoftResearch–ATL MicrosoftResearch–ATL [email protected] Europe,Munich,Germany Europe,Munich,Germany [email protected] [email protected] Alexandru Costan Hakan Soncu Luc Bougé IRISA/INSARennes,France MicrosoftResearch–ATL ENSRennes,France [email protected] Europe,Munich,Germany [email protected] [email protected] Gabriel Antoniu InriaRennes-Bretagne Atlantique,Rennes,France [email protected] ABSTRACT rate can further be tripled thanks to the use of multi-route streaming. The easily-accessible computation power offered by cloud infrastructures coupled with the revolution of Big Data are CategoriesandSubjectDescriptors expandingthescaleandspeedatwhichdataanalysisisper- formed. In their quest for finding the Value in the 3 Vs of H.3.5[OnlineInformationServices]: Datasharing;D.2.4 Big Data, applications process larger data sets, within and [Distributed Systems]: Distributed applications acrossclouds. Enablingfastdatatransfersacrossgeograph- ically distributed sites becomes particularly important for Keywords applications which manage continuous streams of events in real time. Scientific applications (e.g. the Ocean Observa- Event Streaming, Cloud Computing, Multi Data-Centers, tory Initiative or the ATLAS experiment) as well as com- High Performance Data Management mercialones(e.g. Microsoft’sBingandOffice365large-scale services) operate on tens of data-centers around the globe 1. INTRODUCTION andfollowsimilarpatterns: theyaggregatemonitoringdata, On-demand resource provisioning, coupled with the pay- assesstheQoSorrunglobaldataminingqueriesbasedonin- as-you-go model, have redefined the way data is processed ter site event stream processing. In this paper, we propose nowadays. Scientific applications, simulations, monitoring, a set of strategies for efficient transfers of events between networks of sensors, trading or commercial data mining are clouddata-centersandweintroduceJetStream: aprototype allareaswhichhavebenefitedfromandsupportedtheemer- implementingthesestrategiesasahighperformancebatch- gence of cloud computing infrastructures. Cloud providers basedstreamingmiddleware. JetStreamisabletoself-adapt like Microsoft, Amazon, Google, Rackspace have built their to the streaming conditions by modeling and monitoring a cloud solutions at a global scale with data-centers spread setofcontextparameters. Itfurtheraggregatestheavailable across numerous geographical regions and continents. At bandwidth by enabling multi-route streaming across cloud thesametime,theversatilityofdataanalysishasincreased sites. The prototype was validated on tens of nodes from with applications running on multiple sites, which have to US and Europe data-centers of the Windows Azure cloud process larger data sets coming from remote locations and usingsyntheticbenchmarksandwithapplicationcodefrom distinct sources. The services dealing with such computa- the context of the Alice experiment at CERN. The results tions have to face important challenges and issues regard- showanincreaseintransferrateof250timesoverindividual ingthemanagementofdataacrossthesegeographicallydis- event streaming. Besides, introducing an adaptive transfer tributed environments. strategybringsanadditional25%gain. Finally,thetransfer Stream data processing is becoming one of the most sig- nificant subclass of applications in the world of Big Data. Vastamountsofstreamdataarecollectedatincreasingrates frommultiplesources[20,29]inmanyareas: climatemoni- toring,large-scalesensor-basedsystems,transactionalanal- Permissiontomakedigitalorhardcopiesofallorpartofthisworkfor personalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesare ysis, financial tickers, monitoring and maintenance systems notmadeordistributedforprofitorcommercialadvantageandthatcopies forwebservices,large-scalescienceexperiments. Acquiring, bearthisnoticeandthefullcitationonthefirstpage.Tocopyotherwise,to processing and managing this data efficiently raise impor- republish,topostonserversortoredistributetolists,requirespriorspecific tantchallenges,especiallyiftheseoperationsarenotlimited permissionand/orafee. to a single geographical location. There are several scenar- DEBS’14,May26-29,2014,Mumbai,India. ios which created the need to geographically distribute the Copyright2014ACM978-1-4503-2737-4...$15.00. computation on clouds. The size of the data can be so big andinareallifescenariousingtheMonALISA[33]monitor- that data have to be stored across multiple data-centers. It ing system of the CERN Alice experiment [1]. The experi- isthecaseoftheATLASCERNexperimentwhichgenerates mentsshowperformanceimprovementsof250timesoverin- 40PBofdataperyear. Furthermore,evenincrementalpro- dividualeventstreamingand25%overstaticbatchstream- cessingofsuchadatasetasastreamofeventswilloverpass ing,whilemulti-routestreamingcanfurthertriplethetrans- the capacity of local scientific infrastructure, as it was the fer rate. case of the Higgs boson discovery which had to extend the The rest of the paper is structured as follows. Section 2 computationtotheGooglecloudinfrastructure[2]. Another introducestheterminologyandpresenttheproblemswhich scenario is given by the data sources which can be phys- appearinthecontextofstreamingacrossclouddata-centers. ically distributed in wide geographical locations as in the Section 3 presents our approach for adapting the batch size Ocean Observatory Initiative [5, 12] in which the collected by modeling the latency for clouds, while considering the events are streamed to Nimbus [9] clouds. Finally, the na- cost of multi-routing. We continue with the description ture of the analysis, for an increasing number of services, of the system architecture and its implementation in Sec- requires aggregating streams of data from remote applica- tion 4 and discuss the evaluation of the approach across tioninstances. Large-scaleserviceslikeMicrosoft’sBingand Azure data-centers from different continents (i.e. Europe Office365operateontensofdata-centersaroundtheGlobe. andAmerica)inSection5. InSection6,wepresentanexten- Maintenance,monitoring,assertingtheQoSofthesystemor sivestateoftheartinseveraldomainstangenttostreaming global data mining queries all require (near) real-time inter anddatamanagementandpositionthecontributionsofour siteeventstreamprocessing. Allsuchcomputationscarried approach with respect to these directions. Finally, Section oncontinuousstreamsofeventsacrossresourcesfromdiffer- 7 concludes the paper. ent geographical sites are highly sensitive to the efficiency of the data management. 2. BACKGROUND:EVENTSTREAMINGON An extended survey over thousands of commercial jobs CLOUDS and millions of machine hours of computation, presented in [23], has revealed that the execution of queries is event- We start by defining the terminology related to stream driven. Furthermoretheanalysisshowsthattheinputdata processing on clouds and discuss the main problems which accountsonlyfor20%ofthetotalIO,therestcorresponding need to be addressed in this context. tothereplicationofdatabetweenqueryservicesortothein- 2.1 Context termediatedatapassedbetweenthem. Thisemphasizesthat the processing services, be they distributed, exchange large amounts of data. Additionally, the analysis highlights the Cloud computing is a recent paradigm which enables re- sensitivity of the performance of the processing of streams sources (e.g. compute power, storage capacity, network to the management and transfer of events. This idea is dis- bandwidth)tobeeasilyleasedon-demand,followingapay- cussedalsoin[20],inwhichtheauthorsstresstheneedfora as-you-go pricing model. From the infrastructure point highperformancetransfersystemforreal-timedata. Asim- of view, clouds are built across several data-centers dis- ilarconclusionisdrawnin[10],wheretheissueswhichcome tributedaroundtheglobe. Theproximityofthegeograph- fromthecommunicationoverheadandreplicationareexam- ical repartition of resources to users enables high QoS and inedinthecontextofstate-basedparallelization. Finally,in fastqueryresponsesbymeansofcontentdeliverynetworks [40],theauthorsemphasizetheimportanceofdatafreshness, (CDN). From the user point of view, all these resources which improves the Quality of Service of a stream manage- are harvested transparently by means of virtualization. A ment system and implicitly the quality of the data (QoD). typical cloud usage scenario consists of a user deployment Alltheseresearcheffortssupportandmotivatethe growing through which an application is run within virtual ma- need for a high performance system for event streaming. chines (VMs) on several cloud nodes from a data-center. In this paper, we address challenges like latency, transfer Large-scaleapplicationsandwebservicestypicallyrunmul- rate, throughput and performance variability for support- tiple deployments across several data-centers, enabling ge- ing high performance event transfers in the context of ge- ographical proximity to users and load balancing. Besides ographically distributed applications. Our solution, called the local computations answering to user requests, sev- JetStream, is a novel approach for streaming events across eralglobaloperations(e.g.,monitoring,QoSenhancement, cloud data-centers. In order to enable a high performance globaldatamining)requireinter-sitetransfers betweenge- transfer solution, we leverage batch-based transfers. The ographically distant nodes, most often in (near) real-time. size of the batches and the decision on when to stream the Thecloudsupportforsuchtransfersisratherrudimentary: events are controlled by modeling the latency based on a it relies on the cloud storage (e.g., Azure Blobs, Queues, set of parameters which characterize the streaming in the TablesorAmazonS3,SQS)asaglobalsharedpointwhich context of clouds. To further improve performance, we ag- incurs high transfer latencies [24, 45]. gregate inter-data-center bandwidth as we extend our ap- Stream computing referstotheprocessingofcontinuous proachtoprovidemulti-routestreamingacrosscloudnodes. sequences of relatively small data items [20], otherwise re- Furthermore, we developed a prototype that can be used ferred to as events. According to [17], the characteristics on clouds, either public or private, which is environment- ofdata streams are: 1)astreamisformedofevents,which aware by means of light monitoring. The information gath- aretuplesofrecords;2)itisapotentiallyinfinitesequence eredandinferredabouttheenvironmentiscoupledwiththe of events ordered by time (e.g., acquisition time, genera- streaming model, allowing the system to adapt the transfer tion time); and 3) the events can be produced by multi- decisions to all context changes. The system was validated ple external sources at variable rates, independent of the ontheWindowsAzure[6]cloudusingsyntheticbenchmarks constraints of the stream processing engine. The stream a) Transfer Rate (throughput) b) Average Latency of an Event Sender - Site 1 2500 0.03 Time between events 2000 0.025 MTBE=Rate1Acquisition Events/sec 11050000 Time(sec) 0.000..100512 Batch DestinationSP -rt roSecaietmses in2g 500 0.005 Serializer Network De-SerializerEngine 0 0 1 10 100 250 5001000 1 10 100 250 5001000 Batch Size (#Events) Batch Size (#Events) Figure 1: The performance of transferring events in SEovuerncte LatencyEncoding LatencyDecoding batchesbetweenNorthEuropeandNorthUSAzure LatencyBatching LatencyTransfer data-centers. The measurements show the correla- Figure 2: Breaking down the latency to deliver an tion between (a) the transfer rate and (b) the aver- eventfromsourcetostreamprocessingengineacross age event latency for increasing batch sizes. cloud nodes processingengines aresystemswhichtreattheeventsfrom Environment awareness - The cloud infrastructures are thestreambyapplyingtheuser’squeries. Asitisgenerally subjecttoperformancevariationsduetomulti-tenancy infeasible to store the streams entirely, the queries are ap- and network instability on the communication links plied continuously and iteratively over windows of events. withinandacrosssites. Monitoringanddetectingsuch Thecomputationisevent-driven,thereforemostofthedata performance changes allows the system to react ac- management is done at the granularity of an event. Gen- cordinglyandschedulethetransferofeventsefficiently. erally, the size of an event is small, in the range of bytes to kilobytes, which allows to handle them in-memory, but Decoupling the transfer from processing - The event also makes their transfer sensitive to any overhead. transfermoduleneedstobedesignedasastand-alone component, decoupled from the stream processingen- 2.2 Problemstatement gine. We advocate this solution as it allows seamless integration with any engine running in the cloud. At Achievinghighperformanceeventstreamingacrossdata- thesametime,itprovidessustainableandpredictable centers requires new cloud-based solutions, since currently performance, independent on the usage setup. there are no adequate tools to support data streaming or transfers across sites. Most of the existing stream process- Self-optimization - User configurations do not guarantee ingenginesonlyfocusoneventprocessingandprovidelittle optimal performance, especially in dynamic environ- ornosupportforefficienteventtransfers,simplydelegating ments. Moreover, when it comes to large-scale sys- this functionality to the event source. Today, the typical tems, the tasks of configuring and tuning the service way to transfer events is individually (i.e., event by event). tends to become complex and tedious. The alterna- Thisishighlyinefficient,especiallyacrossWAN,duetothe tive is to design autonomic cloud middleware, able to incurred latencies and overheads at various levels (e.g., ap- self-optimize. Coupledwithaneconomicmodel,these plication, technology tools, virtualization, network). systems could also regulate the resource consumption Abetteroptionistotransfereventsinbatches. Whilethis and enforce service-level agreements (SLAs). improvesthetransferrate,italsointroducesanewproblem, relatedtoselectingtheproperbatchsize. Figure1presents Generic solution - Building specific optimizations which thedependencybetweenthebatchsizeandthetransferrate, target precise applications is efficient, but limits the and the transfer latency per event, respectively. We notice applicabilityofthesolution. Instead,weproposeaset thatthekeychallengehereisthechoiceofanoptimalbatch of techniques which can be applied in any cloud con- size and the decision on when to trigger the batch sending. text, independent of the application semantics. Jet- This choice strongly relies on the streaming scenario and Stream does not depend on the nature of the data, on the sensed environment (i.e. the cloud). We aim to nor on the query types. tackle these problems by proposing an environment-aware 3.2 Zoomontheeventdeliverylatency solution, which enables optimum-sized batch streaming of events in the clouds. To achieve this, we model the latency We express the latency of the events based on a set of oftheeventtransferwithrespecttothecloudenvironment, cloudparameterswhichcanbemonitored. Suchatechnique dynamically adapt the batch size to the context and enable allows to bind the batch size corresponding to the minimal multi-route streaming across clouds nodes. eventlatencybothtothestreamandtheenvironmentinfor- mation. We start by breaking down the end-to-end latency 3. MODELINGTHESTREAMINGOFDATA ofaneventinfourcomponents,depictedinFigure2: form- ing the batch, encoding (e.g., serializing), transferring the INTHECONTEXTOFCLOUDS batchanddecoding(e.g.,de-serializing). Thesetofparam- Thecloudvariabilityandthefluctuatingeventgeneration eters able to describe the context and define the latency is: ratesrequireanappropriatemodelforeventstreaming,able the average acquisition rate (Rate ) or mean time Acquisition to capture the cloud specificities. In this section, we intro- between events (MTBE), the event size (Event ), the SizeMB ducesuchamodelandpresentthedecisionmechanismsfor serialization/deserializationtechnique,thethroughput(thr) selecting the number of events to batch. andthenumberofeventstoputinthebatch(i.e.,batchsize -batch ). Thegoalistodeterminethelatterdynamically. 3.1 DesignPrinciples Size Inthefollowing,wewillmodelthelatencycomponentsusing JetStream relies on the following set of design principles: these parameters. (cid:20)(cid:8)(cid:21)(cid:22)(cid:18)(cid:23)(cid:13)(cid:11)(cid:2)(cid:3)(cid:7)(cid:24)(cid:18)(cid:14)(cid:2)(cid:11)(cid:25)(cid:15)(cid:5)(cid:26) a) Throughput b) Approximating the Throughput (cid:23)(cid:15)(cid:25)(cid:18)(cid:23)(cid:13)(cid:11)(cid:2)(cid:3)(cid:7)(cid:24)(cid:18)(cid:14)(cid:2)(cid:11)(cid:25)(cid:15)(cid:5)(cid:26) 70 100 60 80 (cid:1)(cid:2)(cid:3)(cid:4)(cid:2)(cid:5) (cid:6)(cid:2)(cid:7)(cid:2)(cid:8)(cid:9)(cid:2)(cid:5) Throughput (MB/s) 23450000 Percentage (%) 246000 10 0 (cid:17)(cid:13)(cid:11)(cid:13)(cid:18) (cid:10)(cid:3)(cid:11)(cid:2)(cid:5)(cid:12)(cid:2)(cid:4)(cid:8)(cid:13)(cid:11)(cid:2) (cid:19)(cid:17)(cid:2)(cid:13)(cid:3)(cid:11)(cid:11)(cid:13)(cid:2)(cid:18)(cid:5) 0.00127030.1052554630.1205446312.054426310142.27531552510.29262018051.852417 0.0012730.1052554630.1205446312.05442631S0142i.z275e3 1(5M525B10).29262018051.852417 (cid:19)(cid:2)(cid:3)(cid:11)(cid:2)(cid:5) (cid:14)(cid:15)(cid:4)(cid:2)(cid:16) Size (MB) Number of channels Number of channels 1 2 Figure 3: The proposed schema for multi-route 1 2 3 35 46 streaming across cloud nodes. 4 5 6 Approximation Figure 4: a) The value of the throughput with re- The batching latency correspondstothedelayintroduced spect to the number of routes across cloud Small by an event waiting in the batch for other events to VMs for increasing size of the data chunk. b) Ap- arrive,beforetheyareallsentasabatch. Theparam- proximatingwithonepatternthecloudthroughput, eters which describe this latency are the average ac- independent of the number of routes. The approxi- quisition rate of the events and the number of events mation is built from the averages of the normalized in the batch. As the delay depends on the position throughput functions for each number of routes. of the event in the batch, we chose to use the av- erage latency per event. This can be computed by averaging the sum of the delays of all events in the the intra-data-center throughput (between the nodes within batch: Latency = batchSize . Intuitively, one’s single site deployment) is relatively stable and a pri- batching 2×RateAcquisition ori known, as it is specified by the cloud provider within this corresponds to the latency of the middle event. the SLA (e.g., 100 Mbps for Small Azure VMs). On the The transformation latency groupsthetimestoencode other hand, there are no guarantees when it comes to the and to decode the batch using a specific serialization inter-data-center throughput (betweenthedata-centers). In library. It depends on the used format (e.g. binary, fact,thehighlatencyandgenerallylowthroughputnetwork JSON etc.), the number of bytes to convert and the connecting the sites is not owned by the cloud provider, as numberofeventsinthebatch. Toexpressthis,werep- the packet routing is done across multiple internet service resent the transformation operation as an affine func- providers (ISPs). This makes it more unstable, unknownin tion (i.e. f(x) = ax+b) where a corresponds to the advance and subject to poor performance. conversionrate(i.e. amountofbytesconvertedpersec- Inordertoaddresstheissueoflowinter-data-centerthrough- ond-tDs),whilethebconstantgivesthetimetowrite put, we designed a transfer strategy suitable for a typical themetadata(tHs). Thelatencypereventcanthenbe cloudsetup(i.e.,applicationrunningacrossseveralnodesin expressedas: Latency = tHs+tDs×batchSizeMB, differentclouddata-centersdeployments),whichcanharvest whichholdsbothfortthraensefonrcmoadtiionnganddecobdaticnhgSizoeper- extrabandwidthbyusingadditionalintermediatenodes. The ations. Thisformulacanalsobeappliedtoexpressthe idea is to use multiple routes for streaming across sites, as size of the data to be transferred by simply replacing depicted in Figure 3. We build this strategy on two obser- the time constants with data related constants (i.e. vations: the virtual inter data-center routes do not map to size of the metadata and the compression/extension identical physical paths and the latency of intra-site com- ratio). munication is low (less than 10%) compared to inter site communication [45]. These observations allow to aggregate The transfer latency models the time required to trans- additionalbandwidthbysendingdatafromthesendernodes fer an event between cloud nodes across data-centers. to intermediate nodes within the same deployment (i.e. be- To express it, we consider both the amount of data in longing to the same application space), which will then for- the batch as well as the overheads introduced by the ward it towards the destination. With this approach, the transferprotocol(e.g. HTTP,TCP)-sOtandtheen- system aggregates extra physical bandwidth from the dif- coding technique - sOe. Due to the potentially small ferent paths and routes established when connecting dif- size of data transferred at a given time, the through- ferent nodes across the ISP infrastructures. This generic put between geographically distant nodes cannot be method of aggregating inter-site bandwidth is further dis- expressed as a constant value. It is rather a func- cussed in [45], in the context of bulk multi-hop data trans- tion of the total batch size (SizeTotal =batchSizeMB× fers across multiple cloud data-centers. To integrate this batchSize), since the impact of the high latency be- approachwithinourmodel,weextendthesetofparameters tween data-centers depends on the batch size. The that are used to characterize the stream context with the cloud inter data-center throughput - thr(Size) is dis- number of channels. cussed in more detail in the following section. The Independent of the number of routes used for streaming, average latency for transferring an event can then be the value of the throughput function has to be known at expressed as Latencytransfer = sOt+tshOr(eS+izbeaTtocthaSli)zeMB runtime. In order to limit the number of network samples that need to be performed to obtain it, and to reduce the 3.3 Multi-routestreaming monitoring intrusiveness, we approximate this function. In The throughput is one of the most peculiar aspects when Figure 4 a) we present measurements of the throughput for consideringcommunicationintheclouds. Ontheonehand, a number of routes between North Central US and North (cid:23)(cid:5)(cid:7)(cid:24)(cid:2)(cid:7)(cid:19)(cid:14)(cid:12)(cid:13) "(cid:9)(cid:13)(cid:9)(cid:12)(cid:19) Algorithm 1 The selection of the optimal batch size and #$!(cid:11)(cid:16)(cid:3)(cid:14)(cid:10)(cid:2)(cid:3)(cid:9)!(cid:31)(cid:7)(cid:7)(cid:14)(cid:21)(cid:2)(cid:20)!(cid:22)(cid:2)(cid:3)(cid:9)! (cid:25)(cid:5)(cid:19)(cid:9)(cid:16) the number of channels to be used (cid:28)(cid:2)(cid:3)(cid:8)(cid:29) (cid:4)(cid:9)(cid:12)(cid:19)(cid:9)(cid:7) (cid:2)%(cid:12)$!(cid:19)(cid:18)!(cid:10)(cid:9)(cid:2)(cid:9)(cid:16)(cid:16)(cid:6)(cid:16)(cid:7)(cid:2)(cid:9)(cid:13)!(cid:9)(cid:15)!(cid:29)(cid:4)(cid:7)(cid:14)(cid:5)(cid:26)(cid:9)(cid:6)(cid:13)(cid:29) (cid:6)(cid:3) 1: procedureBatchAndChannelsSelection (cid:30)(cid:7)(cid:2)(cid:8)(cid:20)(cid:9) (cid:25)(cid:5)(cid:19)(cid:9) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) &$!’(cid:12)((cid:6)(cid:14)(cid:7)(cid:9)!(cid:24)(cid:29)(cid:9)(cid:12)!(cid:3)(cid:5)!(cid:16)(cid:9)(cid:12)(cid:19) 2: getMonitoredContextParameters() # & % (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) 3: ESTIMATEMaxBatchedfrom[MaxTimeConstraint] (cid:28)(cid:6)*(cid:9)(cid:7) (cid:11)(cid:21)(cid:9)(cid:12)(cid:3) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) (cid:15)(cid:7)(cid:2)(cid:12)(cid:16)(cid:17)(cid:9)(cid:7) (cid:11)(cid:21)(cid:9)(cid:12)(cid:3) (cid:28)(cid:6)*(cid:9)(cid:7) 4: whilechannels<MaxNodesConsidereddo (cid:4)(cid:9)(cid:12)(cid:19)(cid:9)(cid:7) (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) (cid:18)(cid:5)(cid:19)(cid:6)(cid:20)(cid:9) (cid:22)(cid:9)(cid:8)(cid:9)(cid:14)(cid:21)(cid:9)(cid:7) 5: whilebatchsize<MaxBatcheddo (cid:22)(cid:9)(cid:8)(cid:9)(cid:14)(cid:21)(cid:9)(cid:7) 6: ESTIMATE latency from (cid:4)(cid:9)(cid:7)(cid:14)(cid:2)(cid:20)(cid:14)(cid:26)(cid:9)(cid:7) )(cid:31)(cid:25) (cid:25)(cid:5)(cid:19)(cid:9) (cid:1)(cid:9)(cid:27)(cid:4)(cid:9)(cid:7)(cid:14)(cid:2)(cid:20)(cid:14)(cid:26)(cid:9)(cid:7) [RateAcquisition,batchsize] batching 7: ESTIMATE latency from encoding [overheads,batchsizeMB] (cid:1)(cid:2)(cid:3)(cid:2) (cid:1)(cid:2)(cid:3)(cid:4)(cid:3)(cid:5)(cid:2)(cid:6)(cid:7) (cid:4)(cid:3)(cid:7)(cid:9)(cid:2)(cid:10) (cid:46)Estimatethetransferlatencyfor1channel (cid:4)(cid:5)(cid:6)(cid:7)(cid:8)(cid:9) (cid:31)(cid:19)(cid:2) (cid:3)(cid:14)(cid:21)(cid:9)!(cid:28)(cid:2)(cid:3)(cid:8)(cid:29)!(cid:4)(cid:3)(cid:7)(cid:9)(cid:2)(cid:10)(cid:9)(cid:7) (cid:11)(cid:12)(cid:13)(cid:14)(cid:12)(cid:9) 8: ESTIMATE latencytransfer1 from [batchsizeMB,thrRef,1channel] Figure 5: The architecture and the usage setup of 9: COMPUTE RatioCPU from [latency ,latency ,VM Cores] encoding batching the adaptive batch streamer (cid:46)Preventsidlechannels Europe Azure data-centers. In Figure 4 b) we normalize 10: if RatioCPU ∗ latencybatching × channels < latency +latency then thesevalues(i.e.,%ofthecorrespondingstablethroughput) 11: transfer1ESTIMATEencoding latency from decoding and approximate them using a polynomial function, which [overheads,batchsizeMB] we determined empirically that it gives the most accurate 12: ESTIMATE latency from transfer approximation. Hence such a model can in fact represent [batchsizeMB,thrRef,channels] the throughput pattern, with an error introduced by the 13: ESTIMATE latencyreordering from cloudvariabilityoflessthan15%. Usingthisapproximation, 14: [channels,latenlactyetnracnysfer] =(cid:80)latency the entire function can be extrapolated by measuring only perEvent ∗ 15: if latency <bestLatencythen the asymptotic stable throughput. This will be used as the perEvent 16: UPDATE[bestLatency,Obatch,Ochannels] amplitude which multiplied with the normalized estimation 17: endif will give the throughput for any size. 18: endif Thedownsideofusingmultipleroutesforsendingbatches 19: endwhile isthattheorderingguaranteesofferedbythecommunication 20: endwhile protocol for one link do not hold anymore. This translates 21: endprocedure into batches arriving slightly out of order due to changing conditions on the communication links (e.g., packet drops, congestions etc.). It is mandatory to maintain the integrity size, the maximal end-to-end event latency introduced by ofthecommunicationandavoiddroppingdatajustbecause batching can be unsatisfactory for a user, as it might vio- anotherlinkwasfastersothatbatchesneedtobereordered late application constraints, even if the system operates at atthedestination. Weachievethisbybufferingthebatches optimaltransferrates. Hence,theuserscansetamaximum at the destination until their turn to be delivered to the acceptable delay for an event, which will be converted in a streamingenginearrives. Thisintroducesanewcomponent maximum size for the batch (Line 3). As for the number for the latency: latency for reordering. It can be modeled ofchannelsselection,weestimatehowmanybatchescanbe using the Poisson distribution (Poisson(k,λ) = λk×e−λ) to formed while one is being transferred (Lines 6-8). k! estimate the probability of having k number of batches ar- Once the transfer of a batch is completed, adding ad- riving before the expected batch, which fixes λ to 1, as ditional channels to distribute the batches become useless. we consider as reference the time to transfer one batch . The condition on Line 10 prevents the system from creat- This probability can then be correlated with a penalty as- ing idle routes. Finally, the CPU usage needs to be also signedtoeachunorderedbatch,givenbyitslatency. Finally, integrated in the decision process. Sending frequent small summing these probabilities over the number of channels batches will increase the CPU consumption and artificially and normalizing based on the events will give an estima- decrease the overall performance of the cloud node. We tion of the worst case expected latency: Latency = therefore assign a penalty based on the ratio between the reordering (cid:80)ci=ha2nnels(cid:80)Lj Poisson(j,1)×j×Latency×batch. Here, L gives the time to form a batch and the time used by the CPU to en- ×batch×size×L code it, according to the formula RatioCPU = maximum number of batches (e.g., 10) regarded as poten- latencyencoding . Tosumup,JetStream tially arriving before the reference one through a channel, (latencybatching+latencyencoding)×VM Cores collectsasetofcontextparameters(Line2)andusesthemto givingtheupperlimitforiteratingthekPoissonparameter. estimate the latency components according to the formulas 3.4 Adaptivecloudbatching presented above. Based on these estimations, it selects the optimalbatchsizeandthenumberofchannelsforstreaming. InAlgorithm1wewrapupeverythingandpresentthede- cision mechanism in JetStream for selecting the number of 4. SYSTEMOVERVIEW events to batch and the number of channels to use. The al- gorithmestimatestheaveragelatencypereventforarange We designed JetStream as a proof of concept of the pre- of batch sizes and channels, retaining the best one. As an vious approach. Its conceptual schema is presented in Fig- optimization, a simulating annealing technique can be used ure5. Theeventsarefedintothesystematthesenderside, toperformthesearchfaster,byprobingthespacewithlarge assoonastheyareproduced,andtheyarethendeliveredto steps and performing exhaustive searches only around local the stream processing engine at the destination. The pro- optima. Depending on the magnitude of the optimal batch totypewhichimplementsthisarchitecturewasdevelopedin C#usingthe.NET4.5framework. Thedistributedmodules Latency components and their functionality are described bellow: 0.014 0.012 The BfourffJeertSitsreuasmed. bTothheassenadneirnpauptplaicnadtioountpourttehnedepvoeinntt me (sec) 00..000.000681 source adds the events to be transferred, as they are Ti 0.004 produced,whilethereceiverapplication(i.e.,thestream 0.002 0 processing engine) pops (synchronously) the events or 1 5 10 25 50 100 200 400 500 750 1000 Batch Size (#Events) it is notified (asynchronously) when the events have Latency been transferred. The module relies on two queues Total Batching Transfer for synchronization purposes in a producer-consumer Transformation fashion. TheBufferisalsoinchargeofmonitoringthe Figure 6: The latency components per event with input stream in order to assert in real time the acqui- respecttothebatchsizeforasinglestreamingchan- sition rate of the events and their sizes. nel, for inter-sites event transfers Avro (Microsoft HDInsight), but others modules can The Batch Oracle staysatthecoreofJetStream,asiten- be easily integrated. This module can host additional forces the environment-aware decision mechanism for functionalities (e.g., data compression or deduplica- adaptively selecting the batch size and the amount of tion) in the future. channels to use. It implements Algorithm 1 and col- lects the monitoring information from the Buffer and the Transfer Module. It further controls the moni- 5. EVALUATION toring intrusiveness by adjusting the frequency of the monitorsamplesaccordingtotheobservedvariability. Thegoaloftheexperimentalevaluationpresentedinthis section is to validate the JetStream system in a real cloud The Transfer Module implementsthemulti-routestream- setup and discuss the main aspects that impact on its per- ing. On the intermediate nodes, its role is to forward formance. the packets towards the destination. Currently, the 5.1 Experimentalsetup modulesisagnosticofanystreamingscenarioandcan integrateanytransferprotocol. Itprovidesseveralim- The experiments were run in the Microsoft’s Windows plementations on top of TCP: synchronous and asyn- Azure cloud in the North-Central US and the North Eu- chronous,single-ormulti-threaded. Itisalsoincharge rope data-centers. We chose these sites in order to create a ofprobingthenetworkandmeasuringthethroughput geographical-distributedsetupwhichpermitstoanalyzethe and its variability. The batches are sent in a round- abilityofthesystemtoadapttochangingfactorsgenerated robin manner across the channels, balancing the load. both by the cloud itself (i.e., multi-tenancy, virtualization) As future work, the transfer performance can be im- andthewideareacommunicationacrossmultipleISPs. All proved by integrating additional transfer techniques theexperimentswererunwithSmallWebRoleVMshaving amongwhichthesystemcanautomaticallyswitchbased 1 CPU, 1.75 GB of memory, 225 GB local storage and 100 onthestreamingpatternandthecontextinformation. Mbps bandwidth. When using multi-route streaming, up to 5 additional nodes were used within the sender deploy- The Event Sender coordinatestheeventtransfersbyman- ment, implementing the transfer scheme discussed in Sec- aging the interaction between all modules. It queries tion 3.3. Each experiment sends between 100,000 and 3.5 the Batch Oracle about when to start the transfer of million events, which, given the size of an event, translates the batch. Next, it setups the batch by getting the intoatotalamountofdatarangingfromtensofMBsto3.5 eventsfromtheBuffer andaddingthemetadata(e.g., GB.Eachsampleiscomputedastheaverageofatleastten batch ID, streams IDs etc.). The batch is then serial- independent runs of the experiment performed at various ized by the Serialization module and the data trans- moments of the day (morning, afternoon and night). ferred across data-centers by the Transfer Module. The performance metrics considered are the transfer rate The Event Receiver isthecounterpartoftheEventSender and the average latency of an event. The transfer rate is module. The arriving batches are de-serialized, re- computed as the ratio between a number of events and the ordered and delivered to the application as a stream time it takes to transfer them. More specifically, we mea- of events, in a transparent fashion for the stream pro- sured,atthedestinationside,thetimetotransferafixedset cessing engine. The module issues acknowledgements ofevents. Fortheaveragelatencyofanevent,wemeasured to the sender or makes requests for re-sending lost or thenumberofeventsinthesenderbuffer, thetransfertime delayed batches. Alternatively, based on users’ poli- andreportedthenormalizedaveragepereventbasedonthe cies, it can decide to drop late batches, supporting latency formulas described in Section 3.2. theprogressofthestreamprocessingdespitepotential Theevaluationisperformedusingasyntheticbenchmark cloud-related failures. Users configure when such ac- and a real-life application. We created a synthetic bench- tionsareperformedbymeansofwaitingtimes,number markinordertohavefullcontroloverthesizesandthegen- ofbatches,markersorcurrenttimeincrements(CTI). eration rate of the events. The generation rates are varied between hundred of events per second to tens of thousands Serialization/De-Serialization has the role of convert- ofeventspersecond,asindicatedbythescaleofthetransfer ing the batch to raw data, which can be afterwards rates. Such a predictable and configurable setup allows us sent over the network. We integrate in our prototype to analyze and to understand the results better when test- several libraries: Binary (native), JSON (scientific) or ing the behavior of JetStream in several scenarios. Finally, Real Measured Model-based Estimation Low Events Arrival Rate High Events Arrival Rate 4 4 Transfer Rate Transfer Rate 1200 3 3 16000 Time (sec) 12 Time (sec) 12 Events/sec 1046800000000 Events/sec 128000000 4000 200 0 50 100 500 1000 2500 0 50 100 500 1000 2500 0 1 Route 3 Routes 5 Routes 0 1 Route 3 Routes 5 Routes Batch Size (#Events) Batch Size (#Events) Average Latency per Event Average Latency per Event 2 3 4 5 2 3 4 5 6 Number of channels 6 Number of channels 0.002 0.0002 Fcaigseurlaete7n:cCyo(mi.ep.,arthinegptehnealetyst)imofautinoonrdoefrethdebwatocrhsets- me (sec) 000...000000011826 me (sec) 00..800E00−00011526 with actual measurements, while increasing the Ti 0.0004 Ti 4E−05 number of channels used for the transfers 0 1 Route 3 Routes 5 Routes 0 1 Route 3 Routes 5 Routes Batch Size Batch Size 1 10 100 1 10 100 we use the events collected by the MonALISA monitoring 1000 5000 JetStream 1000 5000 JetStream systems from the Alice LHC experiment to assert the gains Figure8: Comparingtheperformance(transferrate brought by our approach in a real setup. - top and average latency per event - bottom) of in- dividualeventstreamingandstaticbatcheswiththe 5.2 Accuracy adaptivebatchofJetStreamfordifferentacquisition rates, while keeping the size of an event fixed (224 We depict in Figure 6 the total latency and its compo- bytes). Thelatencyandtransferratesforindividual nents, per event, with respect to the number of batched event streaming (i.e., batch of size 1) are between events. Selecting the optimal size of the batch comes down 50 to 250 times worse than the ones of JetStream to finding the value corresponding to the minimal latency (e.g. ∼ 200 for the scenario illustrated in Figure 6). The Transfer Rate Average Latency per Event searchforthebatchsizethatminimizesthelatencyperevent 12000 0.0002 0.00018 is at the core of the JetStream algorithm presented in Sec- 10000 0.00016 tomifoarnteipo3nr.4es.seaTnbtohauettiosetnlhefecotreiontnvhierooflnattmheenencoytp(tiseim.cgoa.mlnvpeautlwuteeodrfkrfrootmhmrtothuhigsehtepysupttie-, Events/sec 468000000000 Time (sec) 00..06800.EE000−−0000011055241 CPU),whichmaynotbeexact. Indeed,thecloudvariability 2000 4E−05 canleadtodeviationsaroundtheoptimalvalueintheselec- 2E−05 0 0 tion process of the amount of events to batch. Considering 1 Route 3 Routes 5 Routes 1 Route 3 Routes 5 Routes thattheperformancearoundtheoptimumisratherflat(as Batch Size Batch Size seen in Figure 6), JetStream delivers more than 95% of the 1 10 1 10 100 1000 100 1000 optimal performance even in the case of big and unrealistic 5000 JetStream 5000 JetStream shifts from the optimum batch size (e.g. selecting a batch Figure 9: The performance (transfer rate – left and sizeof150or250insteadof200inFigure6willdecreasethe averagelatencyperevent–right)ofindividualevent performance with 3%). The solution for further increasing streaming, static batches and JetStream for events the accuracy, when selecting the batch size, is to monitor of size 800 bytes. The latency and transfer rates for the cloud more frequently. These observations show that individual event streaming (i.e. batch of size 1) are accuracy can be traded for monitoring intrusiveness at the from 40 up to 150 times worse than JetStream level of cloud resources (e.g. network, memory and CPU). In order to validate the penalty model proposed for the 5.3 Individualvs. batcheventtransfers latency of reordering when using multiple routes, we com- pare the estimations of our approach with actual measure- The goal of this set of experiments is to analyze the per- ments. TheresultsarepresentedinFigure7. Thedelayfor formance of individual event streaming compared to batch- unordered batches was measured as the time between the based streaming, with static and adaptive (i.e., JetStream) momentwhenanoutoforderbatch(nottheonenextinse- sizes. Inthestaticcase,thenumberofeventstobebatched quence) arrives, and the moment when the actual expected is fixed a priori, with a batch of size 1 representing inde- one arrives. The experiment shown in Figure 7 presents pendent event streaming. These setups are compared to the average of this maximum unordered delay over tens of JetStream, which adapts the batch size to the context at independent trials in which a fixed amount of events (i.e. runtime, for varying event sizes or generation rates. 1 million events of size 2.5 KB) is transferred. We notice The experiments presented in Figure 8 use an event of thattheproposedmodelgivesagoodestimationofthisde- size 224 bytes and evaluate the transfer strategies consider- lay, which can be used as a penalty latency for multi route ing low (left) and high (right) event generation rates. The streaming. The accuracy in this case is 90%–95%, because experiments were repeated for different number of routes the penalty model for multi-routing does not integrate the for streaming: 1, 3 and 5, and measure the transfer rates variability of the cloud. Yet, such errors can be tolerated (top)andaveragelatencyperevent(bottom). Thefirstob- as they are not determinant when selecting the number of servation is that batch-based transfers clearly outperform channels to use. individual event transfers for all the configurations consid- ered. The results confirm the high overhead and the low (cid:1)(cid:2)(cid:3)(cid:4)(cid:5)(cid:6)(cid:7)(cid:2)(cid:8)(cid:3)(cid:9)(cid:7) 35000 throughput with small sizes for the transfers between data- 30000 ctoenhtuernsd.rGedrooufptinimgetshe(uepvetnots2i5n0crteimaseesstfohreJtreatSntsrfeeramra)tewtheinles Ba51t00c00h00Size ents/sec 122505000000000 decreasing the average latency per event. Two aspects de- 1Je0t0Stream Ev 10000 EventArrivalRate 5000 terminetheperformance: thesizeofthebatchwithrespect 0 to the stream context and the performance changes of the 1 3 5 7 T9ime11int1e3rva15l(11T07imsee1c9oinnt2de1srv)a23l(1205se27con2d9s) cloud. Static batches cannot handle these variations, mak- 30000 7000 i(aneng.dgc.heribgtaahticnehvbeeasnttochfacssqiizzueeiss1igt0iooogndivreiantpeoonaoenrdcpogneortfoeoxdrtmpaeanrndfcoebrmafdoarninc1eortfohouertr5es Events/sec 112205055000000000000000 Events/sec 123456000000000000000000 routes and low acquisition rates). Selecting the correct size 0 5 6 7 0 7 8 9 atruntimebringsanadditionalgainbetween25%and50% Timeinterval(10seconds) Timeinterval(10seconds) over static batch configurations. We repeated the same set BatchSize BatchSize 5000 1000 100 JetStream 5000 1000 100 JetStream of experiments for bigger event sizes, all showing the same behavior. Due to space constraints, we only illustrate the Figure10: Theevolutionofthetransferrateintime transfer rate and average event latency for an event size of for variable event rates with JetStream and static 800 bytes in Figure 9. batches transfer strategies Transfer Rate 5.4 Adaptingtothecontextchanges 40000 Thedataacquisitionprocessinastreamingscenarioisnot necessarilyuniform. Fluctuationsintheeventratesofanap- 30000 c poflitchaetidonatrausnonuirncge,inthtehveirctluoualdizceadninafprpasetarrudctuueretoorthtehencaltouurde nts/se 20000 e performancevariability[16]. ToanalyzethebehaviorofJet- Ev 10000 Stream in such scenarios, we performed an experiment in which the event generation rate randomly changes in time. 0 For the sake of understanding, we present in Figure 10 a 1 3 5 Number of streaming routes snapshot of the evolution of the transfer rate in which we use fine grain intervals (10 seconds) containing substantial 100 Batch Size JetStream rate changes. JetStream is able to handle these fluctua- Figure 11: The transfer rate for an increasing num- tions by appropriately adapting the batch size. In contrast, ber of routes used for streaming static batch transfers either are introducing huge latency fromwaitingfortoomanyevents,especiallywhentheevent cisely, a larger bandwidth allows to send more data, and acquisition rate is low (e.g., batches of size 1000 or 5000 at implicitly, the additional data carried with each batch does time moment 9) or are falling behind the acquisition rate not throttle the network anymore. This brings the trans- which leads to increasing amount of memory used to buffer ferrateofsmaller,andconsequentlymorefrequent,batches the untransferred events (e.g., batch of size 100 at moment closertothemaximumpotentialeventthroughput. Itisthe 5). Reacting fast to such changes is crucial for delivering caseofthestaticbatchofsize100onFigure11,deliveringa high performance in the context of clouds; this is also the throughput close to JetStream for higher number of routes. reasonwhywechosetoconsidersuchsuddenrateschanges. With higher throughput and a lower overhead, the optimal The number of used resources (e.g. the extra nodes which batch size can be decreased. When selecting it, JetStream enable multiple route streaming) is reduced by 30% when isabletodecreasetheendtoendlatency. Weconcludethat takingintoaccountthestreamcontext. Theseresourcesav- sustaininghightransferratesunderfixedtimeconstraintsis ingsarejustifiedbythefactthatloweventacquisitionrates possiblebyimposingupperboundsforthebatchsizes,which do not require multiple route streaming as higher ones do. enables JetStream to integrate users’ time constraints for Additionally, as shown in [23], the fluctuations in the ap- maximum delay. As discussed in Algorithm 1, it does this plication loads have certain patterns across the week days. by considering a maximum limit on the batch. Hence,inlongrunningapplications,ourapproachwillmake 5.6 ExperimentingwithALICE:areal-life substantialsavingsbydetectingthesedailyorhourlytrends and using such knowledge to scale up/down the number of HighEnergyPhysicsapplication additional nodes used for transfers. In a second phase, our goal was to assess the impact of JetStream in a real-life application. We opted for ALICE 5.5 Benefitsofmultipleroutestreaming (A Large Ion Collider Experiment) [1], one of four LHC Figure 11 shows the gain obtained in transfer rate with (Large Hadron Collider) experiments at CERN (European respecttothenumberofroutesusedforstreaming,forJet- OrganizationforNuclearResearch),asitsscale,volumeand Stream and for a static batch of a relatively small size (i.e. geographical distribution of data require appropriate tools 100events). Multipleroutestreamingpaysoffwhenincreas- for efficient processing. Indeed, the ALICE collaboration, ing the amount of data sent for both strategies (i.e. static consisting of more than 1,000 members from 29 countries andadaptive). Infact,byaggregatingextrabandwidthfrom and 86 institutes, is strongly dependent on a distributed the intermediate nodes, we are able to decrease the impact computingenvironmenttoperformitsphysicsprogram. The of the overhead (batch metadata, communication headers, experiment collects data at a rate of up to four petabytes serialization headers etc.) on smaller batches. More pre- per year, produce more than 109 data files per year, and require tens of thousands of CPUs to process and analyze a) Latency b) Transfer Rate them. TheCPUandstoragecapacitiesaredistributedover 2500 1600 morethan80computingcentersworldwide. Theseresources 2000 1400 1200 atoreOohpueertrefaortocinugegs,nseiynosuttsehmienseaanlsledarisbepasetcoctfhse,qxfurpoeeumriinmCgePnsUotsft,mwisoadroeen.lathnedmcoounnit- Time (sec) 11050000 Events/sec 1068000000 toringinformationcollectedinreal-timeaboutallALICEre- 500 400 200 sources. WeusedtheMonALISA[33]servicetoinstrument 0 0 timheenhtu.geMaomreoutnhtanof3m50onMitoorniAngLIdSaAtasiesrsvuiecdesfraorme rtuhnisneinxgpeart- 10 100 1000 1000J0etStream 1 10 100 1000 100J0et0Stream sitesaroundtheworld,collectinginformationaboutALICE Batch Size (#Events) Batch Size (#Events) computingfacilities,localandwideareanetworktraffic,and Figure 12: The total latency (a) and the trans- thestateandprogressofthemanythousandsofconcurrently fer rate (b) measured when transferring 1.1 million running jobs. This yields more than 1.1 million parameters eventsfromMonALISAmonitoringtheAliceexper- published in MonALISA, each with an update frequency of iment. The latency for independent event transfer one minute. Using ALICE-specific filters, these raw pa- (i.e. batchofsize1)ina)isnotrepresentedbecause rameters are aggregated to produce about 35,000 system- itwouldmodifythescalegreatlydueitsmagnitude, overview parameters in real time. These overviews are usu- having a value of 30900 seconds as opposed to 80 ally sufficient to identify problems or to take global actions seconds for JetStream inthesystemandtheyarethefuelfortheJetStreamevent streaming platform. The MonALISA framework and its latencyoftheeventsasdepictedinFigure12a. Asourap- highfrequencyupdatesforlargevolumesofmonitoringdata proach selects the appropriate batch size at each moment, matched closely with JetStream’s architecture so the pieces itconsequentlyreducestheamountofeventswaitinginthe fitnaturallytogether,butitalsoprovidedusamajoroppor- senderqueueanddecreasestheoveralllatencyoftheevents. tunity to fullfill the system’s initial goal: using the (moni- Compared to the static batch strategies providing constant toring)datatotakeautomateddecisionsinrealtime(inthis transfer performance, the latency of JetStream is decreased context, improve the observed system). from2.2(100-eventbatches)downto17times(10000-event Based on the monitoring data collected and stored by batches). MonALISA as of December 2013, we have replayed a se- quenceof1.1millioneventsconsideringtheircreationtimes at the rate they were generated by Alice. The measure- 6. RELATEDWORK mentswereperformedusing2additionalVMsasintermedi- Theexistingworkscanbegroupedinto5categoriesbased ate nodes at the sender side (i.e. 3 streaming routes). Ini- ontheirapplicationdomainsandfeatures. Wediscussthem tially,theexperimentalsetupconsidered5streamingroutes. in this section. However,duringthetransferofthedatausingJetStream,we observedthatmaximum3suchrouteswereused,asthesys- Systemsfordatatransfers. tem determined that the performance cannot be increased Severalsolutionshavebeenproposedinthelastyearsfor beyond this point. The same number of nodes is recom- managing data transfers at large scale. StorkCloud [30] in- mendedifwequerythesystembasedonthestreamcontext tegrates multi-protocol transfers in order to optimize the (i.e. event size and acquisition rate). The accuracy of this end-to-end throughput based on a set of parameters and decisionwasinfactvalidatedasouradaptiveapproachwas policies. It adapts the parallel transfers based on the clus- obtaining the same transfer performance using 3 nodes as ter link capacity, disk rate and CPU capacity, using the the static batch configurations which were using 5. Fur- algorithm proposed in [47]. The communication between thermore, the static configurations also obtained the same StorkCloud components is done using textual data repre- transfer performances when switched to 3 nodes, showing sentation. Similarly, our system optimizes the streaming that indeed this streaming context was not requiring more betweendata-centersbymodelingasetofparameters. How- than 3 streaming routes. This observation shows the im- ever,JetStreamremainstransparenttothedataformatand portance of an environment-aware adaptive approach, not limits the annotation of events or batches with metadata, subject to arbitrary human configuration choices. reducing the overhead of the transfer, which becomes cru- Figure 12 displays the total latency of the events at the cial in the context of continuous streams of data. Other sender(Figure12a)andthetransferrate(Figure12b)when systems, like NetSticher [32], target bulk transfers between comparing JetStream with static configurations for varying data-centers, as we do. The authors exploit the day/night batchsizes. Thetransferperformancesofstaticbatcheswith patternpeaksofusageofadata-centerandleveragethere- more than 100 events are similar with JetStream. Consid- mainingunutilizedbandwidth. ThoughNetSticherisuseful ering that the generation rate of the events varies from low for backups and checkpoints, it does not work for real-time to high, sub-optimal batch sizes will in fact lead to an ac- systems. GlobusOnline[3]isanothersystemwhichprovides cumulation of the events in the sender queue during the data transfers over WAN, targeting data sharing between peak rates. These buffered events will artificially increase different infrastructures (e.g. grids, institution infrastruc- the performance, at the expense of extra memory, during tures,communityclouds). Unlikeoursystemwhichhandles theperiodswhentheacquisitionrateofeventsislow. Allin streams of events with a light resource fingerprint, Globu- all,thisbehaviorwillproduceastabletransferperformance sOnline manages only file transfers and has substantial re- overawiderangeofstaticbatchsizes,asitcanbeobserved sourcerequirements. Otherrecenteffortsforenablingparal- in Figure 12 b; but on the other hand, it will increase the leldatatransfersleadtothedevelopmentofthemulti-path

Description:
Luc Bougé, Gabriel Antoniu. To cite this version: tory Initiative or the ATLAS experiment) as well as com- mercial ones Stream data processing is becoming one of the most sig- . JetStream relies on the following set of design principles: toring) data to take automated decisions in real time (in
See more

The list of books you might like

Upgrade Premium
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.
DMCA & Copyright: Dear all, most of the website is community built, users are uploading hundred of books everyday, which makes really hard for us to identify copyrighted material, please contact us if you want any material removed.
Copyright @ PDFdrive.to, 2022 | We love our users