Table Of Content

: A Declarative Data Ingestion System I B NGEST ASE Alekh Jindal Jorge-Arnulfo Quiane´-Ruiz Samuel Madden Microsoft QCRI MIT [email protected] [email protected] [email protected] Abstract—Big data applications have fast arriving data that to static and hard-coded data ingestion pipelines in traditional must be quickly ingested. At the same time, they have specific databases as well as in big data systems like Hadoop. Hadoop needs to preprocess and transform the data before it could be Distributed File System (HDFS), for instance, chunks input put to use. The current practice is to do these preparatory data into fixed sized blocks, replicates each block (three times 7 transformationsoncethedataisalreadyingested,howeverthisis by default), and stores them on different machines for fault 1 expensive to run and cumbersome to manage. As a result, there tolerance. Some efforts have tried to add additional steps to 0 isaneedtopushdatapreprocessingdowntotheingestionitself. this static pipeline, such as indexing [1], co-partitioning [2], 2 In this paper, we present a declarative data ingestion system, anderasurecoding[3].However,eachoftheseforksoutanew n scpalelceidfyINthGeEirSTdBatAaSEin,gteostailolnowloagpicpliincataiomnodreeveslyospteemrsattoicpmlaannannerd. storage system with one additional feature at a time and does a We introduce the notion of ingestions plans, analogous to query notofferfullflexibilitytospecifyarbitraryapplication-specific J plans, and present a declarative ingestion language to help de- ingestion logic. 1 velopers easily build sophisticated ingestion plans. INGESTBASE 2 providesanextensibleingestionoptimizertorewriteandoptimize The current practice to deal with application-specific in- ingestion plans by applying rules such as operator reordering gestion needs is to additionally deploy so called cooking ] B and pipelining. Finally, the INGESTBASE runtime engine runs jobs to prepare the data, i.e., use a query processor to run the optimized ingestion plan in a distributed and fault-tolerant preprocessingjobsoncethedataisalreadyingested.However, D manner. Later, at query processing time, INGESTBASE supports this forces the users to spend additional time and money, as . ingestion-aware data access and interfaces with upstream query well as introduces another dependency before the data could s processors, such as Hadoop MapReduce and Spark, to post- c be put to use. Cooking jobs are also hard to share because [ process the ingested data. We demonstrate through a number of they contain custom ad-hoc logic not necessarily understood experiments that INGESTBASE: (i) is flexible enough to express by others, i.e., they lack a formal data ingestion language. 1 a variety of ingestion techniques, (ii) incurs a low ingestion v overhead, (iii) provides efficient access to the ingested data, and In addition, we now have both the ingested as well as the 3 (iv) has much better performance, up to 6 times, than preparing cooked data at the same time, i.e., data duplication, and in a 9 data as an afterthought, via a query processor. fault-tolerance manner (e.g., replicated). Finally, the cooking 0 jobs end up overloading the compute clusters, even when 6 data ingestion typically runs on separate capacity which is 0 I. INTRODUCTION often underutilized. Thus, with cooking jobs, the users end up . 1 Modern big data applications witness massive amounts of creating additional data pipelines, which are tedious to build, 0 continuously and quickly arriving data. At the same time, this expensive to run, and cumbersome to manage. 7 data needs to be preprocessed and optimized in a specific 1 manner before it could be put to use. Examples of such appli- In this paper, we identify data ingestion as an explicit step : cations include: (i) data exploration over periodically arriving that needs to be specified and planned in a more systematic v i scientific datasets, e.g., astronomy; (ii) analyzing service logs, manner. We present INGESTBASE, a flexible and declarative X e.g., from cloud services, which need to be quickly ingested data ingestion system to quickly prepare the incoming data r for real time debugging; (iii) approximate query processing for application specific requirements. At the same time, IN- a over fast arriving social network data, e.g., tweets, to get the GESTBASE hides the ingestion processing complexity from the users, similar to databases hiding the query processing latest trends; (iv) quality checking and cleaning commodity data, e.g., news content, before selling it on the data market; complexity. To do this, INGESTBASE exposes a declarative language interface, the ingestion language, to easily express and(v)archivinghighvelocitytelecomdata,e.g.,phonecalls, arbitraryingestionlogic,essentiallyanoperatorDAGconnect- for security purposes. In all these scenarios, data needs to be ing raw data sources to application ready data in the storage consumed as soon as it arrives and so once it gets ingested thereislittleroomforfurtherpreprocessing,whichisanyways system.INGESTBASEusesanoptimizertorewriteandcompile declarativeingestionstatementsintoanefficientingestionplan. prohibitively expensive due to the massive data volumes. INGESTBASE has a runtime engine that runs the optimized As a result, applications developers need to carefully ingestion plan in a distributed and fault-tolerant manner over design their data ingestion pipelines and push the application a cluster of machines. Finally, INGESTBASE provides a data specific data preparation logic down to data ingestion itself. access kernel to support ingestion aware query processing via For instance, applying multi-dimensional partitioning to slice higher level substrates, such as MapReduce and Spark. and dice the data in different ways for data exploration, or detecting and fixing data quality violations for data market In summary, our key contributions are as follows: applications, or considering different erasure codes to reduce (1) We introduce the notion of ingestion plans, analogous to the storage footprint during data archiving. This is in contrast query plans in relational databases, to specify a sequence of transformations that should be applied to raw data as it is linestatus, i.e., products shipped on the same date have the ingestedintoastoragesystem.Toeasilybuildingestionplans, same linestatus. This would require to partition the data on we describe a declarative data ingestion language to express shipdate, iterate over every pair of tuples in each partition, complex ingestion logic by composing a variety of ingestion and check whether or not there is a functional dependency operators and their data flow (Section IV). violation. Subsequently, the violated records could be output (2) We present an extensible, rule-based ingestion optimizer to a violations file (for further correction) or even discarded. to rewrite and optimize ingestion plans, via techniques such Denial Constraint Checks. Consider again the TPC-H as operator reordering and pipelining. We further describe the lineitemdataset.Supposetheuserwantstocheckthefollowing INGESTBASE runtime engine to efficiently run the optimized denial constraint (DC): each item sold in quantity less than ingestion plan in a distributed and fault-tolerant manner. In 3 does not have a discount of more than 9%. This requires particular, we show how the system allows users to control to scan the lineitem table, check each tuple for this denial the fault-tolerance mechanism for their data based on their constraint, and then store both the violating tuples as well as ingestion plans (Sections V and VI). the original data. (3) Wedescribethe INGESTBASE supportforingestion-aware data access, i.e., leveraging the ingest processing for efficient Single-pass Repair. Besides detecting violations to data qual- data access via upstream query processors. Specifically, we ity rules, users may also want to perform single-pass repairs. show how our prototype implementation works with two For example, consider a tax dataset having a country code storage-compute combinations, namely HDFS-MapReduce attribute.Incasethecountrycodeisnotvalid,usersmaywant and HDFS-Spark. (Sections VII and VIII). to correct the code using a dictionary (e.g., changing a value (4) Finally, we present experimental evaluation over TPC- “mexico”toitscorrespondingcode“MX”).Thiswouldrequire H dataset to: (i) show the overhead of INGESTBASE and parsing the country code attribute in the dataset, checking if compare it with plain HDFS upload times, (ii) compare the the code is valid, and looking up in the dictionary in case the effectiveness of ingest-aware query processing compared to code is invalid. Only the corrected values are finally stored. both MapReduce and Hive, (iii) contrast INGESTBASE with ingestion via cooking jobs (using Hive), and (iv) show the B. Data Sampling fault-tolerance behavior of INGESTBASE (Section IX). Sampling is a common technique to gather quick insights Intheremainderofthepaper,wefirstdiscussfourdifferent from very large datasets [7]. Samples can be used to quickly case study scenarios to understand the data ingestion pain evaluate statistical properties of data (e.g., approximate aver- in modern applications (Section II). Then, we describe an ages or counts in certain subgroups), or to get a representative overviewofthe INGESTBASE architecture(SectionIII)before subset of the data. A key problem in using samples is the presenting our core contributions (Sections IV– IX). processofgeneratingsamplesthemselves:producingasample requires an entire pass over the data. Rather, the users want II. THEDATAINGESTIONPAIN to collect samples as the data is being ingested, with minimal overhead. We discuss a few scenarios below. Let us first see the data ingestion pain in modern applications and understand the need for a declarative ingestion Random Sampling. Users may want to create Bernoulli system. Below we describe four case study scenarios, namely samples by probabilistically replicating some of the tuples in data cleaning, data sampling, data analytics, and data storage, the dataset and collecting them into a separate physical file, to highlight the preprocessing needs in modern big data i.e.,inadditiontocollectingalltuplesintoabasefileanyways. applications and motivate a more systematic approach to it. Likewise, users may also want to create reservoir samples by addingeachtupleintoareservoir,removingtuplesfromitwith A. Data Cleaning a given probability, and then finally emitting the reservoir as samples in the end. Data cleaning is traditionally done as an afterthought, Stratified Sampling. Besides pure random sampling, users i.e., the data cleaning process starts once a dataset has already may also generate stratified samples, where rare subgroups been uploaded [4], [5], [6]. This means that users have to are over-represented vs common sub-groups. For example, in applytediousandtimeconsumingdatacleaningtransformation a dataset about people by state, a larger fraction of records before the data could be put to use. In contrast, cleaning data fromNorthDakotamightbeincludedthanfromCalifornia,to whileingestingthedatasetswouldspeeduptheentirecleaning ensure that enough records about North Dakota are present to process. Users want to detect the portions of the data that achieve a target level of statistical confidence. Such samples violatetheirbusinessrules,oftenanexpensivestepinthedata are commonly used in databases to produce statistical approx- cleaning process [5], and apply simple repairs. Though data imations [8], [7]. This requires to partition the data on the repair may be an iterative process, users could apply one-pass stratification attribute and randomly pick records from each repairsontheirdatasets.Below,wediscussexamplesofafew strata (partition). The number of records picked from each data cleaning operations. partition is proportional to the partition size. Functional Dependency Checks. Consider the TPC-H lineitem table, which includes one entry per item per order C. Data Analytics in a business analytics application. This table includes a shipdate field (the date the item shipped) and a linestatus Data analytics often requires special data formats for field (whether the item has shipped or not). We may want to good performance. The typical practice is to either create enforceafunctionaldependency(FD)thatshipdatedetermines these formats once the data has been already uploaded to a Ingestion Runtime Engine Ingestion Plan Optimizer Declarative Ingestion Language storage system [9], or to modify the storage system with the Query Hive Pig rk application-specific logic [1], [2]. In contrast, the developers a Processor MapReduce p wouldwanttosimplyspecifytheirformats(declaratively)and S let the ingestion system take care of creating appropriate files in the storage system. e IDngaetas tAiocnc-easwsare PDraetdai cAactec ePsus sHhdaondwlen Fault Tolerance s a Co-partitioning. Users may want to apply custom data par- B Storage t titioning when ingesting data. For example, users can mimic es CaretotprHaibratuidttoeioonapirne[g2s.]tU,orsweedhrsetrcoeoguetwltdhoefur,dratehtnaearbsselaitnmsgpwleeiftfithhceiaetnwctoomjoreminlaostniown“istjhoaoinnud”t Ing IPnrgoecsetsioson r DeIcInnlgagereasstttiiivooenn I RnPgulaennst tiOmiopent EiLmnangizigneureage Fault Tolerance evaluate the skew in the join attribute, creating more balanced Input Data co-partitions. Fig.1. IngestBaseSystemArchitecture Layouts & Indexes. Users may want to plug-in alternate data data and replication for hot data). The recovery mechanisms, layouts and indexes, e.g., RCFile [10] (the default Hive [11] however,shouldworkwithbotherasurecodesandreplication. layout in HDFS), Trojan Layouts [12] (i.e., a different data layoutforeachdatareplica)andHail[1](i.e.,adifferentindex E. Remarks for each data replica). Creating Trojan Layouts, for instance, would require to create data blocks, replicate each block three We see that the above applications require custom data times, and serialize each block replica differently (e.g., row, ingestionlogic,i.e.,userswouldliketoprovidedifferentclean- column, and RCFile). Users may also want to create different ingrules,samplingtechniques,datalayouts,anderasurecodes. layouts for different parts of data replicas, i.e., sub-divide Employing cooking jobs for such applications is tedious, time the data blocks within a data replica and create a different taking, and inefficient. Rather the users would want to specify layout for each of them. Such hybrid replicas improve query howthedatashouldbetransformedasitgetsingested,without robustness as more queries are likely to see at least some of incurring additional cooking jobs or worrying about the low- the data blocks in favorable layouts. level details. Thus, we need a systematic and declarative data ingestion system. Data Placement. With large data centers, users may want to control how the data is placed in them, e.g., placing hot III. INGESTBASEOVERVIEW and cold data blocks differently. This requires looking at the contents of each data block when placing it into a cluster. The goal of INGESTBASE is to allow developers to easily Suchcontentbaseddataplacementcouldbefurtherusefulfor: express and efficiently run arbitrary data ingestion logic. In (ii) improving data locality, (iii) isolating concurrent queries our earlier works, we demonstrated flexible data upload to to different nodes, and (i) utilizing a portion of the cluster to HDFS [13], as well as we showed how it could be used save energy or to multiplex resources. for scalable violation detection [6] and robust data partitioning [14], [15]. This paper describes a full-fledged, declarative data ingestion system that could work with arbitrary storage D. Data Storage and query processor substrates. Figure 1 illustrates the archi- Despite the plummeting price of disks, storage space still tecture of our system. remains a concern in replicated storage systems with large TheinputtobeingestedusingINGESTBASEisrepresented datasets. Below we describe two scenarios on how users may as data items, referred to as ingest data items. At the very want to optimize the storage space. beginning,theingestdataitemsaresimplytherawinputfiles. Replicated Storage. Users may want to control both what However, these could be later broken into smaller ingest data parts of the data are replicated and how many times. This items,suchasfilechunksorrecordsforfine-grainedingestion control becomes crucial when different parts of the data have logic, e.g., applying chunk level replication or detecting null different relative importance. For example, a user storing values in each record. Each ingest data item is further associ- weblogs might replicate the most recent logs (hot data) more ated with a list of labels denoting its lineage during ingestion. frequently for higher availability, compared to the massive Finally, ingestion operators specify the logic to transform the olderlogs(colddata).Thiswouldrequirepartitioningthedata ingestdataitems,i.e.,IngestOp:LID→LID’,whereLID on date (could be trivial in case of time series), and then andLID’areinputandoutputsetoflabelledingestdataitems. applying the replication selectively. For example, the single-pass repair from Section II-A would output only a repaired tuple, i.e., SinglePassRepair : Erasure-coded Storage. Erasure coding is an alternative to t → t |φ. Ingestion operator follows the iterator model repaired replication for handling failures. The advantage of erasure with the following API: coding over replication is that it provides the same degree • initialize: initialize an operator for the first time. of redundancy as replication at lower storage overhead (but • setInput: assign the set of input ingest data items. with a higher access cost in the event of failure). Creating an • hasNext: check whether next output is available. arbitrary erasure code would require dividing the input blocks • next: get the next output labelled ingest data item. into stripes and applying erasure coding for each stripe. As • finalize: cleanup the ingestion operator in the end. with data layouts and indexes, user may want to use different erasure codes, or a mix of replication and erasure codes, for Withingestdataitemsandtheingestoperatorsasthebuild- different portions of the data (e.g., erasure codes for cold ing blocks, INGESTBASE allows to create arbitrary operator DAGs, called ingestion plans. An ingestion plan can further couldapplymulti-levelpartitioning(acrossandwithinchunks) controlthedataflowbyselectivelychoosingwhichingestdata as follows: items go to which portions of the DAG. INGESTBASE makes s3 = FORMAT s1 it easier for the users to build sophisticated ingestion plans by PARTITION BY top-level-partition providing a declarative ingestion language. The ingestion plan CHUNK BY chunk PARTITION BY intra-chunk-partition is then optimized via the ingestion optimizer, which choses to ORDER BY order push-downorpush-uptheingestionoperators,pipelinethedata SERIALIZE AS serializer; flow across several ingest operators, and block the data flow Finally, where to ingest is specified using the following wherever needed. Finally, the INGESTBASE runtime engine STORE statement: runstheresultingoptimizedingestionplaninadistributedand fault-tolerant manner. s4 = STORE s3 LOCATE USING locator UPLOAD TO target; IV. DATAINGESTIONLANGUAGE Thelocator operatorspecifieswhichingestdataitemsmustbe co-located(oranti-located),whilethetarget operatorspecifies In this section, we describe the declarative ingestion lan- the final storage substrate. Note that target only points to the guage in INGESTBASE. In contrast to the current practice of using query processing language to cook the data, to the best registeredstoragelocation;theactualbindingof INGESTBASE with the storage system is a bit more involved, as described of our knowledge, this is the first work to propose primitives in Section VIII. foraningestionlanguage.Therearetwopartstoouringestion language: (i) the declarative ingestion operators to specify the application-specific data transformation during ingestion, and B. Ingestion Dataflow (ii) the declarative ingestion data flow to control (via the use In the previous section, we saw the declarative statements of labels) which data items flow through different parts of the for specifying and chaining ingestion operators. Our ingestion ingestion plan. We describe these two below. language further allows the users to control the ingestion data flow, i.e., selectivity feed different portions of the ingest data A. Ingestion Operators items to different ingest operators. To do so, we define a data flow stage as a set of ingest operators operating on a set of Theingestionoperatorshelpaddressthreeingestionneeds: ingest data items. Recall, that the ingest data items have an what to ingest, where to ingest, and how to ingest, similar to associated set of labels denoting the transformations applied the what, where, and how of data storage proposed in [16]. to them so far. We use these labels to filter the relevant data Foragivenapplication,theseingestionneedscouldbederived items for each stage. using storage optimizer tools [16], [17]. Users can then define what to ingest using a SELECT statement as follows: CREATE STAGE a USING s1,s2,..,sm s1 = SELECT projection WHERE l_op1=v1,l_op2=v2,..,l_opn=vn FROM LID USING parser WHERE filter In the above, we define a stage a with the ingest operators in REPLICATE BY replicator; s −s (whichcouldbeeitheroftheSELECT,FORMAT,and 1 m STORE) and operating on ingest data items that have labels While the above syntax is very similar to standard SQL l = v . For example, consider ingesting hourly data and selectstatement(exceptreplicationandresultassignment),the op i assiume that the parser operator assigns the file creation projection,parser,filter,andreplicatorcanalsobeprovidedas timestampasthelabelforeachingestdataitem,thefollowing customingestoperators.Forinstance,wemayprojectmachine stage ingests only the last hour of data each time: learning features for each tuple or replicate the ingest data itemsprobabilistically.Oncompilation,theingestionoperators s1 = SELECT * FROM input; in the SELECT statement are chained as follows: CREATE STAGE a USING s1 LID→parser→filter→projection→replicator WHERE l_parser > now-1; For a SELECT statement to be valid, the output and input Multiple stages could be chained to each other using the ingest data items of consecutive ingest operators must match, CHAIN STAGE statement as shown below: i.e., they should have the same granularity and the same schema.NextweshowtheFORMATstatementtodescribehow CHAIN STAGE b TO a1,a2,..,ak to ingest the data. USING s1,s2,..,sm WHERE l_op1=v1,l_op2=v2,..,l_opn=vn s2 = FORMAT s1 PARTITION BY partition Note that the above statement performs a union all on the CHUNK BY chunk outputs from stages a ,a ,..,a before feeding it to stage b. 1 2 k ORDER BY order SERIALIZE AS serializer; By defining stages on top of the ingestion operators, users Operators in the above FORMAT statement are chained in can selectively process different ingest data items in different the order in which they appear, i.e., the chaining in s2 is parts of the ingestion plan. Such selective ingestion capability partition → chunk → order → serialize. Users can create is useful for: (i) handling heterogeneous data where different alternate chains by changing the order of these operators, portions of the data have different characteristics and hence e.g., ordering before chunking will create a global sort order theyneeddifferentdataingestionlogic,(ii)supportingmultiple as opposed to per-chunk sort order in s2. The operators in workload types, e.g., graph and relational analytics, each re- FORMATstatementcouldalsoappearmultipletimes,e.g.,users quiringthedatatobeshoehorneddifferently,and(iii)reducing Output Data Output Data Output Data Stage g Uploader Stage g Uploader Block 5 Uploader Disjoint Random Disjoint Random Disjoint Random Stage f Locator Locator Stage f Locator Locator Block 3 Locator Block 4 Locator Stage e Stage e Stage d Sorted Row RCFile PAX Stage d Sorted Row RCFile PAX Sorted Row RCFile PAX replica 1a replica 1b replica 1a replica 1b replica 1a replica 1b Pa1r0ti0timonbe r Pa1r0ti0timonbe r 2x Replicator Pa1r0ti0timonbe r Block 2 2x Replicator Pa1r0ti0timonbe r 100mb 100mb Stage b 2x Replicator Hash Partitioner Stage c Stage b Partitioner Hash Partitioner Stage c Partitioner Hash Partitioner replica 1 replica 2 replica 1 replica 2 replica 1 replica 2 2x Replicator 2x Replicator 2x Replicator Parser Parser Parser Stage a Stage a Block 1 Input Dataset Input Dataset Input Dataset (a) ExampleLogIngestionPlan. (b) Operatorreordering. (c) Operatorpipelining. Fig.2. Illustratingingestionplananditsoptimization. the risk of picking the wrong ingestion logic, e.g., due to allow users to easily control and selectively process the ingest changes in the workload, by applying multiple logic in the dataitems.Together,thedeclarativeingestionoperatorsandthe first place. data flow primitives allow users to quickly stitch sophisticated ingestion plans for their applications. C. Example: Log Analytics Letusnowillustrateourlanguageviaanexample.Consider V. INGESTIONOPTIMIZER a log analytics scenario where large volumes of logs are collected from a cloud service. These logs need to ingested INGESTBASE takes the declarative ingestion statements quickly with a low overhead. Later, in case of any problems andcompilesthemintoaDAG,theingestionplan,asshownin with the cloud service, e.g., disruption or slow performance, Figure2(a).TheingestionoptimizertakesthisDAGandemits the service administrators need to quickly search the relevant an optimized ingestion plan. To do so, the optimizer supports log lines. Each log line contains a combination of structured rule-basedtreetransformationstoidentifysubtreepatternsand (e.g., timestamp, machine name) and unstructured (e.g., the transform them into alternate subtrees. An ingestion plan sub- error stack, manual user commands) data items. tree is represented as an ingestion operator expression, which consists of the root operator and its descendants (recursively). For such an application, developers may create the fol- An optimizer rule (for tree transformations) operates on the lowing ingestion logic: create three data replicas and apply a ingestion operator expression via the following two methods: differentsetofoperatorstoeachofthem;firsttwoofthethree check:IngestOpExpr→true/false replicas differ only in their layout (sorted row and RCFile), apply:IngestOpExpr→IngestOpExpr(cid:48) while the third replica uses logical partitioning in addition to the physical partitioning. As a result, the first two replicas The check method verifies whether a rule is applicable to an are suitable for selection and projection queries, while the ingestionoperatorexpressionandtheapplymethodproduces third replica is suitable for join and aggregation queries. The the modified ingestion operator expression. The optimizer ingestion statements for these are as follows: performsapreordertraversalovertheingestionDAGandfires matching rules wherever applicable, i.e., larger subtrees are s1 = SELECT * FROM input USING parser REPLICATE BY 2; matched for relevant rules first. The rules are matched in the s2 = SELECT * FROM s1 REPLICATE BY 2; s3 = FORMAT s2 CHUNK BY 100mbBlocks; same sequence as provided in the ordered rule set, and they s4 = FORMAT s3 SERIALIZE AS sortedRow; are applied iteratively until none of the rules match any of the s5 = FORMAT s3 SERIALIZE AS rcFile; s6 = FORMAT s1 PARTITION BY hash CHUNK BY 100mbBlocks ingestionoperatorexpressionsinthetree.Wenowdescribetwo SERIALIZE AS pax; rules,namelyoperatorreorderingandpipelining,toreducethe s7 = STORE s4,s5 LOCATE USING disjointLocator; data volume and the materialization cost respectively. s8 = STORE s6 LOCATE USING randomLocator; s9 = STORE s7,s8 UPLOAD TO hdfsStorage; Operator Reordering. This rule rearranges ingestion opera- Thecorrespondingingestiondataflowisdescribedasfollows: tors in order to reduce the data volume in flight, i.e., push- CREATE STAGE a USING s1; down data reducing operators, e.g., filter, while push-up the CHAIN STAGE b TO a USING s2,s3 WHERE l_replicate1=1; data expanding operators, e.g., replicate. In order to preserve CHAIN STAGE c TO a USING s6,s8 WHERE l_replicate1=2; the semantics of the ingestion plan, we only rearrange the CHAIN STAGE d TO b USING s4 WHERE l_replicate2=1; CHAIN STAGE e TO b USING s5 WHERE l_replicate2=2; ingestionoperatorsinthesamedataflowstage,i.e.,thereisno CHAIN STAGE f TO d,e USING s7; conditionalprocessingofthedataitemsinvolved.Oneinstance CHAIN STAGE g TO c,f USING s9; of this rule could push replicate operator at the very end of Finally, Figure 2(a) depicts the resulting log ingestion the stage, i.e., replicate data as late as possible, as shown in logic, as described above. As also noted in [16], we see that stage b of Figure 2(b). Another instance could swap the filter thinking in terms of what, where, and how in INGESTBASE and projection operator depending on which provides more makesitmoreintuitivetoreasonaboutarbitrarydataingestion data reduction, i.e., whether we reduce data volume more by operations. Also, the data flow primitives in INGESTBASE filtering the rows or by filtering the columns. Thus, operator reordering rules could be useful in reducing the data traffic dataitemsindifferentthreads.Tosupportsuchmulti-threaded while executing the ingestion plans. parallelism, the ingest operator implementation has a parallel mode, in addition to the default serial mode, to process input Operator Pipelining. By default, all output ingest data items ingest data items using a thread pool. These threads are later arecollected(i.e.,materialized)fromaningestoperatorbefore synchronized in the finalize method of the ingestion operator. being fed to the next one. Internally, this is done by adding The parallel mode is turned on by default for CPU heavy a materialize operator after each ingest operator. An obvious operators such as serialize. However, users could provide optimization is to pipeline the data items between operators additional optimizer rules to control the serial/parallel modes. as much as possible and materialize only when really needed. Operator pipelining rule removes materialization between op- Parallel ingestion allows INGESTBASE to significantly re- erators that process ingest data items of the same granularity duce the overhead of transforming the data. We demonstrate (detected by looking at the data types). We materialize only this experimentally in Section IX. whenthegranularityoftheingestdataitemchanges,e.g.,from tuples to blocks. To illustrate, Figure 2(c) shows the log B. Efficient Distributed I/O ingestion plan from Section IV-C with operator pipelining. We can see that stages a − g of the plan, as shown in In the previous section, we described how we can paral- Figure 2(a), have been transformed to five pipelined blocks lelizetheingestionplanandprocessdatalocallyoneachnode. 1−5inFigure2(c).Otherinstancesoftheoperatorpipelining However, several ingestion plans require to move data around. rules could consider materializing long pipelines in between In this section, we describe how the INGESTBASE runtimes for fault-tolerance, or for early access to the incoming data. engine handles distributed I/O efficiently. Below we describe the three major data movement scenarios, namely shuffling, Thus, we see that the ingestion optimizer provides an placement, and replication. extensiblewaytotransformandoptimizetheingestionDAGs. Shuffling. An ingestion plan may require to shuffle inter- VI. INGESTBASERUNTIMEENGINE mediate ingest data items in order to produce the final data items. For example, to gather stratified samples, we need to Recallthatthemodernbigdataapplicationsneedtoingest grouptheentiredatasetacrossallnodesandthenpicksamples the incoming data quickly and with low overhead. As a result, it is critical to have an efficient runtime engine for these from each group1. INGESTBASE runtime engine handles this using a distributed file system by first creating local groups applications. In this section, we describe the INGESTBASE on each node and then copying them to the distributed file runtimeenginewhich(i)runsaningestionplaninparallelona system in parallel. While copying, the data is organized into clusterofmachines,(ii)efficientlyhandlesdistributeddataI/O directories, one for each group, such that data belonging to during ingestion, and (iii) handles fault-tolerance both during the same group is in the same directory. Finally, each node and after ingestion. We describe each of these below. reads back and processes the group-directories, one at a time, A. Parallel Ingestion from the distributed file system. Essentially, the INGESTBASE runtime engine leverages the remote data access mechanism Givenaningestionplanandaclusterofmachines,INGEST- of the distributed file system to shuffle data across nodes. BASE runtime engine exploits two kinds of parallelism: inter- node and intra-node parallelism. We describe these below. Placement. INGESTBASE allows users to reason data placement at a logical level, i.e., using the locator operator to map Inter-node Parallelism. When a user submits an ingestion eachingestdataitemtoalocationID,withoutgettingintothe plan on one of the nodes (the client) for execution, the lowleveldataplacementpolicies.Asaresult,userscaneasily INGESTBASE runtime engine copies the resulting optimized make data placement decisions, such as which portions of the plan to all nodes (specified via a slaves configuration file) in datashouldresideonwhichnodes;or,whichdataitemsshould the cluster and executes it over the local data on each node. be co-located and which data items should not be co-located. This makes sense because the raw data is typically generated To enforce these decisions, the INGESTBASE runtime engine on multiple nodes in the first place, e.g., log data, and it simply looks at the location IDs of each ingest data item, e.g., is cumbersome to bring all of this data to a single node. a data block, and copies items with the same location ID to Therefore, instead of bringing data to the ingestion plan, we the same node in the cluster. The mapping from location IDs shiptheplantothedataitself.Thisissimilartoshippingquery to nodes can either be provided by the user, or the runtime plans in distributed query processing. INGESTBASE runtime self-assigns the location IDs to nodes in the same order (in a engine launches remote shell to start the ingestion plans on round robin manner) as they appear in the slaves file. all nodes in parallel and waits for them to finish before it terminates. Replication.Replicationisusuallydoneforfault-toleranceand ittypicallyinvolvesmovingeachreplicatoadifferentnode.In Intra-node Parallelism. Besides parallelizing the ingestion contrast, INGESTBASE completely decouples data replication process across different nodes, the INGESTBASE runtime en- andplacement,andallowsuserstotakeindependentdecisions ginealsoparallelizespartoftheingestionplanacrossdifferent for the two. As a result, users can choose to replicate data threads on the same node. For example, the serialize at different granularities and/or may not place the replicas on operator is CPU bound and so the INGESTBASE runtime different nodes. For example, users may choose to replicate engineforksseveraloperatorinstances(asmanyasthenumber some rows (could be seen as samples) in each data block and of cores by default) at the same time, each serializing a different subset of ingest data items. Likewise, the INGEST- 1Userscouldalsodoper-nodestratifiedsamplingtocomputethesamples BASE runtime engine transforms different replicas of ingest fromthelocalstratumoneachnode. storethemalongwiththedatablockonthesamenode,i.e.,no Detector Recovery Implementation Recovery additional data movement is needed. Impl. Catalog Failed Recovered Detect c/morisruspintged? Failed Blocks Block UDF Recovery BlocksUDF Block Paths C. Fault Tolerance Inthissection,wedescribethefault-tolerancemechanisms Fig.3. Post-ingestionfault-toleranceinINGESTBASE. in INGESTBASE runtimeenginetohandlefailuresbothduring maintains a catalog of detect and recover UDFs for and after the data ingestion. each ingestion plan. Figure 3 depicts this post-ingestion fault- 1)Handling In-flight Failures: The INGESTBASE runtime tolerance mechanism in INGESTBASE. engine can handle two types of failures while running the We built three implementations of the detect and recover ingestion plan. UDFs using the above architecture: Ingestion Operator Failure. In case an ingestion operator Replication based. This fault-tolerance mechanism looks for fails, we need to re-run all pipelined operators that appear be- a replica of the failed data block and increases the replication forethefailedone.However,insteadofrestartingtheingestion factor of the replica by 1. The block placement policy takes plan from scratch, we can resume ingestion from the previous care of storing the new replica on a different node. block of pipelined operators. This is because ingest data items Transformation based. This recovery mechanism is for data arefullymaterializedaftereverypipelinedoperatorblock,and block replicas that are not bitwise identical, i.e., they are therefore each such block serves as a checkpoint. In case of repeated failures (3 times by default) of the same operator2, serialized differently. This mechanism copies and transforms a data block replica so that it has the same serialization as the the INGESTBASE runtime engine replaces the failing operator failed data block. with a dummy pass-through operator. The dummy operator simplyreturnstheinputingestdataitemsandassigneachitem Erasure coding based. This recovery mechanism is for a label of “−1” to denote the failure. Application developers erasure-coded,insteadofreplicated,datablocks.Itfirstfetches can further control the recovery time (e.g., in case they expect all data blocks in the same stripe and then reconstructs the more failures) by adding custom operator pipelining rules to missingdatablock.Thereconstructeddatablockisstoredback forcemorefrequentmaterialization,asdiscussedinSectionV. to the HDFS. Node Failure.Incaseoneofthenodesintheclusterfails,we Thus, we see that INGESTBASE users can: (i) inject cus- simply reschedule the ingestion plan from the failing nodes to tom fault-tolerance logic for their application specific needs, other nodes. However, this still requires the data on the failed e.g., heterogeneous replication [1], [12], (ii) change the fault- node to be available remotely (in case that node is used as the tolerance over time as the application needs evolve, e.g., mi- data node as well). For node failures during data shuffling, we grating from replication to erasure coding [3], and (iii) have check which of the groups directories in the distributed file different fault-tolerance mechanisms for different ingestion systemarecorruptandwecopythemagain,assumingthatthe plans, i.e., fault-tolerance mechanism is not tied to the storage distributedfilesystemstillworkswithonelessnode.Tohandle system anymore. data placement, we reassign the location ID of the failed node to the next node (in slaves file) in the round robin sequence. VII. INGESTIONAWAREDATAACCESS 2)Handling Post-ingestion Failures: Given that data is INGESTBASE allows users to apply ad-hoc data transfor- ingested with custom application-specific logic, it may also mations while ingesting their datasets. However, the system need custom fault-tolerance logic. INGESTBASE allows users also needs to keep track of these transformations in order to control the fault-tolerance mechanism for their data based to leverage them later for query processing. Essentially, we on their ingestion plans. To do so, INGESTBASE provides need to track three pieces of information: (i) which ingestion two fault-tolerance UDFs to define how to detect and recover operators were used to preprocess the dataset; (ii) how were failing data items (typically data blocks). theingestionoperatorscomposed;and(iii)theoperatorlineage detect:f →{r1,r2,..,rn} andthetransformationappliedtoeachoutputdataitem.For(i) recover:{Br1,Br2,..,Brn}→Bf and (ii), we simply serialize the ingestion plan in the storage system. Note that we do not serialize the operator instances, Here f is the failing block id while {r ,r ,..,r } are the 1 2 n rather we store the instance parameters and re-instantiate the recovery blocks; B is the corresponding block. The above i operators whenever needed. For (iii), we make use of labels UDFs essentially address two key questions: (1) Which data assigned to the ingest data items, as described below. blocks are needed to recover a failed data block? (2) How to reconstruct a failed data block from the recovery data blocks? Recall that each ingestion operator assigns a label to every data item that is processed. One could imagine storing all As soon as it finishes executing the ingestion plan, the such labels for every ingest data item. However, this would INGESTBASE runtime engine launches a fault-tolerance dae- resultinahugeamountofmetadata.Instead,theINGESTBASE mon that polls the storage system for failing data blocks (the runtime engine collects the labels common to all data items detect) and invokes the user-supplied recovery UDFs for that are materialized and preserves them as the name of every failed block detected (the recover). INGESTBASE the physical file. Thus, each physical file in INGESTBASE is named as follows: label1 label2 label3 label4 .....labeln. 2The INGESTBASE runtime engine detects recurring failures by keeping The labels in the above filename have the same relative track of the execution status of each ingestion operator, i.e., whether or not theypassedthefinalizemethodsuccessfully. sequence as the corresponding operators in the ingestion plan. Thus, the filename of a physical file in INGESTBASE acts Output Data as a signature, or the lineage of the preprocessing applied Standard Reduce Aggregation Reducer to it. For example, the name of a physical file produced by Map/Reduce Map Hash Join Mapper Hash Join Mapper the ingestion plan of log analytics (Figure 2(a)) might be: parseID replicaID hashID fileID paxID locationID uploadID. As Record Readershipdate>1997-12-01 reploicradIeDr=da1te>r1e9p9l7ic-a1I2D-0=12 mktsegment=automobile a result of these label encoded filenames, INGESTBASE does Index Access Index Access Post Filtering not need to maintain any additional metadata files. IngCeasrttBilaagsee   lineitem orders orders customer DDataat aA Acccceessss Co-group Input Format Co-group Input Format Once the data is ingested using INGESTBASE, users want Input Splitter replicaID=1 replicaID=2 to access it from their applications. INGESTBASE provides Replica ID Filter Path Filter ingestion-aware access methods to query data from arbitrary Lineitem Orders Customer query processors. Again, the INGESTBASE access methods address three key questions: (i) what to access, (ii) where to Fig.4. Ingestion-awaredataaccessforTPC-HQ3usingHadoopMapReduce. access,and(iii)howtoaccess.Wedescribethesethreebelow. What to access? INGESTBASE allows developers to retrieve could use these access methods with two popular query pro- a subset of a dataset, based on the labels applied to the cessing engines, namely Hadoop MapReduce and Spark. ingesteddataitems.Todoso,INGESTBASEprovidestwofilter operators:onethatfiltersdatareplicasandonethatfiltersdata VIII. INTEGRATINGINGESTBASE blocks in a particular replica: We now describe how INGESTBASE works with two dif- filterReplica (IngestOp filterOperator, Label operatorLabel) ferent combination of storage and compute substrates. filterBlock (IngestOp filterOperator, Label operatorLabel) Asanexample,considerasamplingingestoperatorthatlabels A. HDFS & MapReduce every data item as either 1 (denoting a sample) or 0 (denoting LetusfirstlookathowINGESTBASEinteractswithHDFS. the original data item). Also assume that the ingestion plan First of all, INGESTBASE needs to map the ingest data items physicallypartitionsthesampledandoriginalingestdataitems to physical HDFS files, i.e., collect the output ingest data into different physical files. To access only the samples, we items from an ingestion plan and store them in HDFS. To can use filterReplica to filter the files that have label 1. This do so, the last transformation in an ingestion plan must be narrowsdowndataaccessonlytotherelevantportionsofdata. upload. If the ingestion plan contains a physical partitioner, Wheretoaccess?Inadditiontofiltering,INGESTBASEallows the upload operator maps each physical partition to a HDFS developers to define: (i) the data access parallelism by setting file. Otherwise, it collects all data items into a single HDFS the number of tasks to run in parallel; and (ii) the amount file. INGESTBASE furthercontrolsseveralstoragedecisionsof of data each computation task has to read. This is done by the HDFS files it creates. For instance, it can replicate each assigningdatablockstocomputationtasks. INGESTBASE API physical file if the ingestion plan already contains a replica- allows key-based splitting as well as co-splitting two or more tion operator, split files into subfiles and choose a different datasets on their respective keys. replication for each subfile, or let HDFS do the standard 3x replication. Likewise, INGESTBASE controls data placement splitByKey (Key key [, Int maxSplitSize]) by assigning location IDs to physical partitions and mapping coSplitByKey (Key key1, Dataset d2, Key key2, ..) each location ID to a particular data node, via a custom data placementpolicy.Similarly,theupload couldpipelinethedata For example, if the ingestion plan partitioned the data on items produced by a ingestion plan directly to HDFS files, an attribute into ranges, developers can distribute different without first collecting them on local disk. It can also bulk rangepartitionstodifferentmachinesandincreasedataaccess parallelism; or to the same machines to improve data locality. load the data items to HDFS files. Finally, INGESTBASE can manipulatethefault-tolerancemechanism,e.g.,transformdata How to access? Finally, INGESTBASE allows developers to layouts when recovering failed blocks (Section VI-C). Thus, deserializetheretrievedblocksandapplyfurtherselection/pro- even though INGESTBASE sits on top of HDFS, it could be jection predicates while reading them. tightly integrated with the storage decisions in HDFS. deserialize(Projection p, Selection s) For MapReduce query processing, we bake INGESTBASE accessmethodsusingtheHadoopInputFormats.TheInputFor- Note that the actual deserialization depends on the se- matallowsuserstospecifyapathfilter tofilterinputbasedon rialization operator in the ingestion plan. The built in IN- the HDFS file path, a splitter to split the data logically, and GESTBASE library provides deserialize operators for all of a record reader to actually read the data. We implemented the serialize operators it provides (PAX, RCFile, SortedFile, custom functionality for these three methods in order to use ColumnGroup, etc.) These implementations take into account theselection/projectionpredicateswhiledeserializingthedata. the INGESTBASE access methods in Hadoop. For example, to implement the filterReplica, we created a path filter which Forexample,theymaydeserializeonlytheprojectedattributes retrieves all physical files having a particular label in their (in case of column layout), or perform index access (in case filename3. We also implemented additional helper methods, the data is sorted). e.g. filterReplicaById, filterReplicaByPartitioning, and filter- Ingestion-aware data access pushes down one or more ReplicaByLayout, for the ease of programming. query predicates before producing the input for the upstream query processor. The following section describes how one 3Recallthatwepersistthelabelsinthefilenames. publicJavaRDDLike<?,?>groupwiseAnalytics(StringingestFilepath){ A. Data Ingestion Scenarios JavaSparkContextctx=newJavaSparkContext(SPARK MASTER,‘‘myJob”, SPARK HOME,SPARK JAR); In this section, we describe the performance of INGEST- // IngestBasedataaccess BASE on the four ingestion scenarios described in Section II. IngestBaseDatasetd=newIngestBaseDataset(ctx,ingestFilepath); d.filterBlock(SamplingOperator,SAMPLE ID); 1)Data Cleaning: We start by evaluating INGESTBASE d.filterReplicaByPartitioning (PARTKEY); on data cleaning operations, when uploading the TPC-H d.splitByPartitionKey(PARTKEY); d.deserializeProject(PARTKEY,SUPPKEY); lineitem table into HDFS. // standardSparktransformations Setup. We consider the data quality rules described in Sec- return d.RDD() tion II-A: (i) a functional dependency (FD) stating that any .map(newGroupbyKeyMap()) .reduceByKey(newGroupbyKeyReduce()); two tuples having the same ship_date must have the same } line_status; and (ii) a denial constraint (DC) stating Listing1. Ingestion-awaredataaccessforgroup-wiseanalyticsusingSpark. that any tuple having quantity smaller than 3 must have discount smaller than 9%. We measure the runtime of three different transformations: (i) detecting FD violations, To illustrate, Figure 4 shows how the INGESTBASE access (ii) detecting DC violations, and (iii) repairing (in addition methods can be used to run TPC-H Q3 (which consists of to detecting) DC violations. two joins and a GROUP BY) in a single MapReduce job, in contrast to two jobs in standard Hadoop. This is possible Discussion. Figure 5(a) shows the results. We observe that because INGESTBASE co-groups all three TPC-H relations. detectingviolationsfortheDCrulewhileingestingdataincurs Note that the output of INGESTBASE access methods is fed to only a 25% overhead over standard HDFS. This is because standard map/reduce data flows. Thus, in addition to allowing the detection process is simply piggy-backed into the process users to easily preprocess and transform their datasets, IN- of uploading data into main-memory. This is also why DC GESTBASE access methods also allows developers to quickly repairincursalmostnoextraoverhead.However,thisoverhead build efficient query processors. increases when the data quality rules require more complex data transformations. For instance, the ingestion plan to detect B. HDFS & Spark FD violations takes double the standard HDFS upload time. This is because the FD requires grouping the entire dataset Spark runs over HDFS as well as it uses the same Hadoop on ship_date, which results in shuffling the data across all InputFormats to read data from HDFS. As a result, we could nodes.Still,detectingviolationswheningestingdatasets(using easily run Spark jobs on top of INGESTBASE ingested data. INGESTBASE) is much better than detecting violations after To illustrate, Listing 1 shows the data access for group-wise a dataset is ingested (using e.g., Hive), as we shall see in analytics over sampled data in Spark. This data access plan Section IX-B. selects the replica of the sampled data which is partitioned on PARTKEY, co-locates (splits) values of PARTKEY, and projects 2)Data Sampling: We now look at the performance of PARTKEY and SUPPKEY attributes. Finally, we get an RDD INGESTBASE for computing samples during data ingest. from the ingested dataset and can apply standard Spark trans- Setup. We consider five different sampling techniques: formation over it. We see that using the INGESTBASE data Bernoulli, simple random, systematic random, local stratified, accessplans,developerscaneasilynarrowdowntheiranalysis andglobalstratified.Herethelocalstratifiedsamplingcollects to the most relevant portions of the data, without dealing with samplesfromthelocalstrataoneachnode,whereastheglobal the actual physical data representation used to store the data. stratified sampling collects samples from the global strata. Discussion.Figure5(b)showstheresultsoftheseexperiments. IX. EXPERIMENTS Weobservethat INGESTBASE hasaverysmalloverhead(less WerananumberofexperimentstoevaluateINGESTBASE. than 10%) for all methods except global stratified sampling. Our goal was to answer two key questions: (i) how efficiently Thissmalloverheadreflectsthetimetowritethedatasamples does INGESTBASE allow users to perform data transforma- todisk.Inthecaseofglobalstratifiedsampling, INGESTBASE tions? and (ii) is transform-as-you-upload in INGESTBASE upload time is nearly twice of HDFS upload time. The reason better than other possible transformation approaches? To is the same as for the DC rule in the previous experiments: evaluate these questions, we ran INGESTBASE ingestion plans global stratified sampling requires shuffling the entire dataset for the four different data ingestion scenarios described in across all nodes to collect samples from each subgroup. SectionII.Forallexperiments,wemeasureunmodifiedHDFS However, these experiments show that most types of sampling data upload times as the default baseline, and Hive (a widely can be done efficiently, as data is being ingested, with little used SQL-based database that runs on Hadoop MapReduce additional overhead (no additional passes over the entire data and HDFS) as an additional baseline wherever possible. Ad- set). ditionally, we evaluate the data access times INGESTBASE for 3)DataAnalytics: Inthissection,weanalyzethe INGEST- several common relational operations. All experiments were BASE ingest times when preparing datasets for different data done on a cluster of 10 nodes. Each node has 1.07 GHz analytical tasks. with 32-core Xeon running on Ubuntu 12.04, 256 GB main- memory, and 11 TB of disk storage. We experimented with Setup. We create different layouts for each replica (using the TPC-H data at scale factor 1000 (1TB in total), and generated schemetheTrojanLayouts[12]paper).Wedenotethisscheme data on all 10 nodes in parallel. We run INGESTBASE on top asPer-replicaLayouts.Itworksbycreatingabinaryrow,PAX, of Hadoop 2.0.6-alpha and use Hive 0.13.1. and compressed PAX layouts for each of the three replicas, 1000010000100In00geI1sn0tgB0e0a0sstIeBn gPaeslaesnt PBsInlaagsneess PtBlaansse Plans 80008000 8000Inge80Isn0tgB0easstIeBn gPaeslaesnt PBsInlaagsneess PtBlaansse Plans 100001000010000 10000 IngeIsntgBeasstIeBn gPaeslaesnt PBsInlaagsneess PtBlaansse Plans 80008000 8000 8000 IngeIsntgBeasstIeBn gPaeslaesnt PBsInlaagsneess PtBlaansse Plans StanSdtaarndd HSarDtdaFn HSdDaSrFdtSa HndDaFrSd HDFS StanSdtaarndd HSarDtdaFn HSdDaSrFdtSa HndDaFrSd HDFS StanSdtaarndd HSarDtdaFn HSdDaSrFdtSa HndDaFrSd HDFS StanSdtaarndd HSarDtdaFn HSdDaSrFdtSa HndDaFrSd HDFS 80008000 8000 8000 64006400 6400 6400 80008000 8000 8000 64006400 6400 6400 Time (seconds) 46Time (seconds)0000004600Time (seconds)0000 460000Time (seconds)00 46000000 Time (seconds)3428Time (seconds)0000342800Time (seconds)00 34280000Time (seconds)34280000 Time (seconds) 46Time (seconds)0000004600Time (seconds)0000 460000Time (seconds)00 46000000 Time (seconds)3428Time (seconds)0000342800Time (seconds)00 34280000Time (seconds)34280000 16001600 1600 1600 20002000 2000 2000 16001600 1600 1600 20002000 2000 2000 BernoBuellrinouSBlilmierpnSloeiu mllSBipyelesrS tneioSmmuyplasliltteiec m SSaiyLmtisocptc elaeml L aoStciycas lGt elLomobacaGtali cllo  baLlo Gclaolb al  Global  Per-rePpelri-crae pPliecra- repHlyiPcbaer riH-dr yebprliicdaC H oynbtreCidno tn-HbteyanbsCter-ioddbn  aCtseoenndt-tC ebConaontsn-tebetdean nstC-teb-odban astseeednd t-C boansteedn t-based  FlexibFllee xibleF lexiEbrlea sFulEererxa isbuler eE rasFulreex EibFrlalees xuirbele F leRxeibplelic RFaeletpixoliincb alRet ieopnl icatRioenp lication  DC DDetCe cDtieotDneCcDt iDoCne RteDeDcpCtCia o RiDrneepFDteaDCcir tDRioeFentDpe acDDitrCieot nFeRDcetp iDoaneirteFctDio Dnetection RandRomandoRmRanadnRodmaonmdRSoamtRnraadtnoiSfidmetordamtRifiSaetSnrdatdrtaoiSfitmietfirdeadtSifiterSadttriafietidfiedStratified LayouLtasyoutsLayRouetpsliLcRaaeyspoluictasRsePpalicrtaitsPiRoanerpitniltgiicoaPnsianrgtiPtiolancPienPamgrlateictnieotmnPienlngatcemePnltacement ReplicRaetpiolincaRtieopnlicaCtoRiodenipnClgiocdatiniognCEroadsiunErgera CCsuoorddeEii nnCrgagosduEirrneag sECurEoarerdsa iuHnsrugyerb eCEr iHrodadysibunrrgied EHryabsruidre Hybrid (a) DataCleaning (b) DataSampling (c) DataAnalytics (d) DataStorage Fig.5. IngestionruntimeengineoverheadofINGESTBASEoverdifferentapplications,comparedtouploadingtostandardHDFS(withoutanypreprocessing). respectively.Inaddition,weimplementthreenewdatastorage users apply both replication and erasure coding, on different schemes, as described in Section II-C: (i) Hybrid Replicas, portions of the data: we replicate the first range partition 10 which store subsets of the blocks of a replica in different times and apply erasure coding for the remaining partitions (3 layouts,(ii)Content-basedPartitioning,whichchunksthedata parity blocks for every 10 data blocks). basedonitscontentinsteadofphysicalsize,and(iii)Content- Discussion. Figure 5(d) shows the results. Interestingly, we basedPlacement,whichplacesdatablocksbasedontheircon- tent. For Hybrid Replicas, we create the same three layouts as observe that INGESTBASE outperforms HDFS in the Flexible inPer-replicaLayoutsandweletHDFShandlethereplication. Replication case. This is because INGESTBASE creates fewer data replicas than HDFS and hence stores less data. Erasure For the other two schemes, we use a logical partitioner to coding, on the other hand, stores 30% more data as well as generate10rangepartitions(fortheContent-basedPartitioner) incurs more CPU costs. As a result, it is has almost 40% and place all those data blocks having the same range on the same data node (for the Content-based Placement). higher runtime than standard HDFS. However, INGESTBASE allows developers to flexibly choose the erasure codes as well Discussion. Figure 5(c) illustrates the results, which overall as freely combine erasure coding with replication. Hence, confirm the trend we observed in the previous section: the INGESTBASEcanbeeffectivelyusedtooptimizestoragespace overhead is directly proportional to the time spent by IN- by transforming the physical data representation in a variety GESTBASE intransferringorprocessingdata.Weobservethat of ways. when creating a different data layout per data replica (Per- replicaLayouts),theINGESTBASEingesttimeisnearlydouble B. Comparison with Hive Cooking Jobs the HDFS upload time. This is mainly because INGESTBASE has to deal with data replication outside HDFS. When IN- The typical practice is to prepare a dataset once it is GESTBASE pushes data replication to HDFS itself (such as already ingested into HDFS using query processing tools or in Hybrid Replicas, Content-based Partitioning, and Content- MapReduce over HDFS. Let us now compare INGESTBASE based Placement), we see a decrease of the INGESTBASE with such an approach. overhead. In particular, we observe that for the Content-based Partitioning and Content-based Placement schemes, which Setup. We preload the data into HDFS and create an external are less CPU demanding than the other two schemes, the Hive table to contain the data. We then run HiveQL queries to overheaddecreasesevenmore,tojustover20%.Theseresults do three data transformations, namely functional dependency, showtheefficiencyof INGESTBASE atapplyingarbitrarydata denialconstraintchecking,andrandomsampling(thesearethe transformations at ingest time, and without any modifications only transformations that are easy to represent in HiveQL). to the HDFS. Transformation Hive(s) INGESTBASE(s) Improvement FunctionalDependency 8,100 4,516 1.8x 4)Storage Space Optimization: Finally, we evaluate IN- DenialConstraint 2,616 1,011 2.6x GESTBASE in scenarios where optimizing the storage space is RandomSampling 2,274 371 6.1x the primary goal of the user. TABLEI. INGESTIONOVERHEADINHIVEANDINGESTBASE. Setup. We consider four different scenarios. First, the case Discussion Table I shows the ingestion overhead, above the where users over-replicate the hot data, for better availabil- standard HDFS upload time, in Hive and INGESTBASE. We ity, and under-replicate the cold data, for preserving storage can see that INGESTBASE has 1.8× less overhead than Hive space (Flexible Replication). To do so, we create 10 range forcheckingfunctionaldependencyandalmost6.1×lessover- partitions and consider the first partition to be hot (replicated head for random sampling. The reason is that INGESTBASE 10 times) and the remaining partitions to be cold (replicated piggy-backs these operations onto the ingestion process. For 2 times). Second, we consider using erasure coding instead of example, to generate random samples, INGESTBASE incurs replication. We erasure code the data with 3 parity blocks for a single data read from disk while ingesting it into HDFS. every 10 data blocks. Third, we consider the case where users Hive on the other hand, needs to re-read the entire dataset applydifferenterasurecodestodifferentportionsoftheirdata twice. Furthermore, it is tedious to run more complex trans- (Flexible Erasure Coding): we create 3 parity blocks for every formations such as stratified sampling in Hive, and physical 5 data blocks of the first range partition, while the remaining transformations such as erasure coding are not possible at all. range partitions are encoded as before, i.e. 3 parity blocks Thus, INGESTBASE has utility both in terms of performance for every 10 data blocks. Finally, we consider the case where aswellasflexibilityiningestingdatasetsinanad-hocmanner. �1�1 �1 �1