Data-Intensive Workflow Management: For Clouds and Data-Intensive and Scalable Computing Environments (Synthesis Lectures on Data Management)

181 Pages·2019·4.477 MB·English

by  Daniel C. M. de Oliveira, Ji Liu, Esther Pacitti

Series ISSN: 2153-5418
D
E
O
L
I
V
E
I
R
A
•
E
Series Editor: H.V. Jagadish, University of Michigan
T
Founding Editor: M. Tamer Özsu, University of Waterloo A Data-Intensive
L

Data-Intensive Workflow Management
FDoarn iCel lCou. Mds. daen Od lDiveairtaa, -UInnivteernsisdiavdee Faendder aSl cFalulambinlee nCseomputing Environments D Workflow Management
A
Ji Liu, University of Montpellier, Inria & CNRS T
A
Esther Pacitti, University of Montpellier, Inria & CNRS -
IN For Clouds and Data-Intensive
T
Workflows may be defined as abstractions used to model the coherent flow of activities in the context of an in silico scientific E
experiment. They are employed in many domains of science such as bioinformatics, astronomy, and engineering. Such workflows N
S
usually present a considerable number of activities and activations (i.e., tasks associated with activities) and may need a long time I
V
for execution. Due to the continuous need to store and process data efficiently (making them data-intensive workflows), high- E and Scalable Computing
performance computing environments allied to parallelization techniques are used to run these workflows. At the beginning of W
the 2010s, cloud technologies emerged as a promising environment to run scientific workflows. By using clouds, scientists have O
R
expanded beyond single parallel computers to hundreds or even thousands of virtual machines.
K
More recently, Data-Intensive Scalable Computing (DISC) frameworks (e.g., Apache Spark and Hadoop) and F Environments
L
environments emerged and are being used to execute data-intensive workflows. DISC environments are composed of processors O
and disks in large-commodity computing clusters connected using high-speed communications switches and networks. The main W
advantage of DISC frameworks is that they support and grant efficient in-memory data management for large-scale applications, M
such as data-intensive workflows. However, the execution of workflows in cloud and DISC environments raise many challenges A
such as scheduling workflow activities and activations, managing produced data, collecting provenance data, etc. N
A
Several existing approaches deal with the challenges mentioned earlier. This way, there is a real need for understanding how to G
manage these workflows and various big data platforms that have been developed and introduced. As such, this book can help E
M
researchers understand how linking workflow management with Data-Intensive Scalable Computing can help in understanding
E
and analyzing scientific big data. N
In this book, we aim to identify and distill the body of work on workflow management in clouds and DISC T Daniel C. M. de Oliveira
etmuhnsaaivnnti rgaao grfneerm amemxeeenencwttus oi.t neWr kdDs ei nIssSu taacC rhs ti e nabnsgyv lA iedr poissinactcmeuh saeesnni Sndtgsp ,m a tarhunkeld t aib n-wasdsei ti icept srpc eelrsoixenutncedtins,p s iltineaos knd iosen.ftg ad iaal,dt savo-alinuntttaieognnes siov tfeh psacrtoi eevnnetanibfialnce c wteh.o eAr koffltpeotriwmwsai. zrNded,e w xetex, ewgcoeu ttpioorwensa eorndf tst htwweo owr kwoflrokorflwkofl wow s Ji Liu
About SYNTHESIS Esther Pacitti
This volume is a printed version of a work that appears in the Synthesis M
Digital Library of Engineering and Computer Science. Synthesis O
R
books provide concise, original presentations of important research G
A
and development topics, published quickly, in digital and print formats.
N
&
C
L
A
Y
P
store.morganclaypool.com O
O
H.V. Jagadish, Series Editor
L

Data-Intensive
Workflow Management
For Clouds and Data-Intensive and
Scalable Computing Environments
Synthesis Lectures on Data
Management
Editor
H.V.Jagadish,UniversityofMichigan
FoundingEditor
M.TamerÖzsu,UniversityofWaterloo
SynthesisLecturesonDataManagement iseditedbyH.V.JagadishoftheUniversityofMichigan.
Theseriespublishes80–150pagepublicationsontopicspertainingtodatamanagement.Topics
includequerylanguages,databasesystemarchitectures,transactionmanagement,data
warehousing,XMLanddatabases,datastreamsystems,widescaledatadistribution,multimedia
datamanagement,datamining,andrelatedsubjects.
Data-IntensiveWorkflowManagement:ForCloudsandData-IntensiveandScalable
ComputingEnvironments
DanielC.M.deOliveira,JiLiu,andEstherPacitti
2019
TransactionProcessingonModernHardware
MohammadSadoghiandSpyrosBlanas
2019
DataManagementinMachineLearningSystems
MatthiasBoehm,ArunKumar,andJunYang
2019
Non-VolatileMemoryDatabaseManagementSystems
JoyArulrajandAndrewPavlo
2019
ScalableProcessingofSpatial-KeywordQueries
AhmedR.MahmoodandWalidG.Aref
2019
DataExplorationUsingExample-BasedMethods
MatteoLissandrini,DavideMottin,ThemisPalpanas,andYannisVelegrakis
2018
iii
DataProfiling
ZiawaschAbedjan,LukaszGolab,FelixNaumann,andThorstenPapenbrock
2018
QueryingGraphs
AngelaBonifati,GeorgeFletcher,HannesVoigt,andNikolayYakovets
2018
QueryProcessingoverIncompleteDatabases
YunjunGaoandXiaoyeMiao
2018
NaturalLanguageDataManagementandInterfaces
YunyaoLiandDavoodRafiei
2018
HumanInteractionwithGraphs:AVisualQueryingPerspective
SouravS.Bhowmick,ByronChoi,andChengkaiLi
2018
OnUncertainGraphs
ArijitKhan,YuanYe,andLeiChen
2018
AnsweringQueriesUsingViews
FotoAfratiandRadaChirkova
2017
DatabasesonModernHardware:HowtoStopUnderutilizationandLoveMulticores
AnatasiaAilamaki,ErietaLiarou,PınarTözün,DanicaPorobic,andIraklisPsaroudakis
2017
InstantRecoverywithWrite-AheadLogging:PageRepair,SystemRestart,Media
Restore,andSystemFailover,SecondEdition
GoetzGraefe,WeyGuy,andCaetanoSauer
2016
GeneratingPlansfromProofs:TheInterpolation-basedApproachtoQuery
Reformulation
MichaelBenedikt,JulienLeblay,BaldertenCate,andEfthymiaTsamoura
2016
VeracityofData:FromTruthDiscoveryComputationAlgorithmstoModelsof
MisinformationDynamics
LaureBerti-ÉquilleandJavierBorge-Holthoefer
2015
iv
DatalogandLogicDatabases
SergioGrecoandCristinaMolinaro
2015
BigDataIntegration
XinLunaDongandDiveshSrivastava
2015
InstantRecoverywithWrite-AheadLogging:PageRepair,SystemRestart,andMedia
Restore
GoetzGraefe,WeyGuy,andCaetanoSauer
2014
SimilarityJoinsinRelationalDatabaseSystems
NikolausAugstenandMichaelH.Böhlen
2013
InformationandInfluencePropagationinSocialNetworks
WeiChen,LaksV.S.Lakshmanan,andCarlosCastillo
2013
DataCleaning:APracticalPerspective
VenkateshGantiandAnishDasSarma
2013
DataProcessingonFPGAs
JensTeubnerandLouisWoods
2013
PerspectivesonBusinessIntelligence
RaymondT.Ng,PatriciaC.Arocena,DenilsonBarbosa,GiuseppeCarenini,LuizGomes,Jr.,
StephanJou,RockAnthonyLeung,EvangelosMilios,RenéeJ.Miller,JohnMylopoulos,Rachel
A.Pottinger,FrankTompa,andEricYu
2013
SemanticsEmpoweredWeb3.0:ManagingEnterprise,Social,Sensor,andCloud-based
DataandServicesforAdvancedApplications
AmitShethandKrishnaprasadThirunarayan
2012
DataManagementintheCloud:ChallengesandOpportunities
DivyakantAgrawal,SudiptoDas,andAmrElAbbadi
2012
QueryProcessingoverUncertainDatabases
LeiChenandXiangLian
2012
v
FoundationsofDataQualityManagement
WenfeiFanandFlorisGeerts
2012
IncompleteDataandDataDependenciesinRelationalDatabases
SergioGreco,CristianMolinaro,andFrancescaSpezzano
2012
BusinessProcesses:ADatabasePerspective
DanielDeutchandTovaMilo
2012
DataProtectionfromInsiderThreats
ElisaBertino
2012
DeepWebQueryInterfaceUnderstandingandIntegration
EduardC.Dragut,WeiyiMeng,andClementT.Yu
2012
P2PTechniquesforDecentralizedApplications
EstherPacitti,RezaAkbarinia,andManalEl-Dick
2012
QueryAnswerAuthentication
HweeHwaPangandKian-LeeTan
2012
DeclarativeNetworking
BoonThauLooandWenchaoZhou
2012
Full-Text(Substring)IndexesinExternalMemory
MarinaBarsky,UlrikeStege,andAlexThomo
2011
SpatialDataManagement
NikosMamoulis
2011
DatabaseRepairingandConsistentQueryAnswering
LeopoldoBertossi
2011
ManagingEventInformation:Modeling,Retrieval,andApplications
AmarnathGuptaandRameshJain
2011
vi
FundamentalsofPhysicalDesignandQueryCompilation
DavidTomanandGrantWeddell
2011
MethodsforMiningandSummarizingTextConversations
GiuseppeCarenini,GabrielMurray,andRaymondNg
2011
ProbabilisticDatabases
DanSuciu,DanOlteanu,ChristopherRé,andChristophKoch
2011
Peer-to-PeerDataManagement
KarlAberer
2011
ProbabilisticRankingTechniquesinRelationalDatabases
IhabF.IlyasandMohamedA.Soliman
2011
UncertainSchemaMatching
AvigdorGal
2011
FundamentalsofObjectDatabases:Object-OrientedandObject-RelationalDesign
SuzanneW.DietrichandSusanD.Urban
2010
AdvancedMetasearchEngineTechnology
WeiyiMengandClementT.Yu
2010
WebPageRecommendationModels:TheoryandAlgorithms
SuleGündüz-Ögüdücü
2010
MultidimensionalDatabasesandDataWarehousing
ChristianS.Jensen,TorbenBachPedersen,andChristianThomsen
2010
DatabaseReplication
BettinaKemme,RicardoJimenez-Peris,andMartaPatino-Martinez
2010
RelationalandXMLDataExchange
MarceloArenas,PabloBarcelo,LeonidLibkin,andFilipMurlak
2010
vii
User-CenteredDataManagement
TizianaCatarci,AlanDix,StephenKimani,andGiuseppeSantucci
2010
DataStreamManagement
LukaszGolabandM.TamerÖzsu
2010
AccessControlinDataManagementSystems
ElenaFerrari
2010
AnIntroductiontoDuplicateDetection
FelixNaumannandMelanieHerschel
2010
Privacy-PreservingDataPublishing:AnOverview
RaymondChi-WingWongandAdaWai-CheeFu
2010
KeywordSearchinDatabases
JeffreyXuYu,LuQin,andLijunChang
2009
Copyright©2019byMorgan&Claypool
Allrightsreserved.Nopartofthispublicationmaybereproduced,storedinaretrievalsystem,ortransmittedin
anyformorbyanymeans—electronic,mechanical,photocopy,recording,oranyotherexceptforbriefquotations
inprintedreviews,withoutthepriorpermissionofthepublisher.
Data-IntensiveWorkflowManagement:
ForCloudsandData-IntensiveandScalableComputingEnvironments
DanielC.M.deOliveira,JiLiu,andEstherPacitti
www.morganclaypool.com
ISBN:9781681735573 paperback
ISBN:9781681735580 ebook
ISBN:9781681735597 hardcover
DOI10.2200/S00915ED1V01Y201904DTM060
APublicationintheMorgan&ClaypoolPublishersseries
SYNTHESISLECTURESONDATAMANAGEMENT
Lecture#60
SeriesEditor:H.V.Jagadish,UniversityofMichigan
FoundingEditor:M.TamerÖzsu,UniversityofWaterloo
SeriesISSN
Print2153-5418 Electronic2153-5426