Data Science of Apache Spark - Databricks

144 Pages·2015·9.83 MB·English

http://training.databricks.com/workshop/datasci.pdf
Data Science Training
Spark
“I want to die on Mars— b Eutlo nno Mt ouns ki,m ipntaecrtv” iew with Chris Anderson
tenta“tTivhee" CI cfso "oyhmTonrheupce uwltuotrdeserrt i u ogaMrunreeee t–d shin asetot ,hde tedafhat sTaece r lat ofsanen,srr geato iecl netntilho oyheu-n -iy gsn-!mp-ht-oe-,o iWtrtshpw wteero eivaslrtlarik asecl” uo,tn in-atoo-hfbtn ee Jglsse eos.c " irt cnooo-go umaF itrnnoareyi egh toShaedfine rontigyhgcu m.dehs" a – oNltt eahHui aierban pytlBk z V ierttsau’osrcrn ilahaaeeengt r s,�� !!
ADVANCED: DATA SCIENCE WITH APACHE SPARK
Data Science applications with Apache Spark combine the scalability of Spark and the
distributed machine learning algorithms.
This material expands on the “Intro to Apache Spark” workshop. Lessons focus on
industry use cases for machine learning at scale, coding examples based on public
data sets, and leveraging cloud-based notebooks within a team context. Includes
limited free accounts on Databricks Cloud.
Prerequisites:
Intro to Apache Spark workshop or
Topics covered include:
equivalent (e.g., Spark Developer Certificate)
Experience coding in Scala, Python, SQL
Data transformation techniques based on both Spark SQL and functional
Have some familiarity with Data Science
programming in Scala and Python.
topics (e.g., business use cases)
Predictive analytics based on MLlib, clustering with KMeans, building classifiers
with a variety of algorithms and text analytics – all with emphasis on an
iterative cycle of feature engineering, modeling, evaluation.
Visualization techniques (matplotlib, ggplot2, D3, etc.) to surface insights.
Understand how the primitives like Matrix Factorization are implemented in a
distributed parallel framework from the designers of MLlib
Several hands-on exercises using datasets such as Movielens, Titanic, State Of
the Union speeches, and RecSys Challenge 2015.
Agenda
• 
Detailed)agenda)In)Google)doc)
• 
https://docs.google.com/document/d/
1T9AkXUmL6gDYTpAEEgqsy9hfJtqGlavjnGzMqzUwDf
E/edit)
Goals
•  Patterns):)Data)wrangling)(Transform,)Model)&)Reason))with)Spark)
o  Use)RDDs,)Transformations)and)Actions)in)the)context)of)a)Data)Science)Problem,)an)Algorithms))&)a)
Dataset)
•  Spend)time)working)through)MLlib))
•  Balance)between)internals)&)handsWon)
o  Internals)from)Reza,)the)MLlib)lead)
•  ~65%)of)time)on)Databricks)Cloud)&)Notebooks)
o  Take)the)time)to)get)familiar)with)the)Interface)&)the)Data)Science)Cloud)
o  Make*mistakes,*experiment,…*
•  Good)Time)for)this)course,)this)version)
o  Will)miss)many)of)the)gory)details)as)the)framework)evolves)
•  Summarized)materials)for)a)3)day)course)
o  Even)if)we)don’t)finish)the)exercises)today,)that)is)fine)
o  Complete)the)work)at)home)W)There*are*also**homework*notebooks*
o  Ask)us)questions)@ksankar,@pacoid,*@reza_zadeh,*@mhfalaki,*@andykonwinski,*@xmeng,*
@michaelarmbrust,*@tathadas*
Tutorial)Outline:)
morning' a)ernoon'
o  Welcome)+)Ge3ng)Started)(Krishna)) o  Ex)3):)Clustering)B)In)which)we)explore)SegmenFng)
o  Databricks)Cloud)mechanics)(Andy))) Frequent)InterGallacFcHoppers)(Krishna))
o  Ex)0:)PreBFlight)Check)(Krishna)) o  Ex)4):)RecommendaFon)(Krishna))
o  DataScience)DevOps)B)IntroducFon)to)Spark)(Krishna)) o  Theory):)Matrix)FactorizaFon,)SVD,…)(Reza))
o  Ex)1:)MLlib):)StaFsFcs,)Linear)Regression)(Krishna)) o  OnBline)kBmeans,)spark)streaming)(Reza))
o  MLlib)Deep)Dive)–)Lecture)(Reza))
o  Design)Philosophy,)APIs)
o  Ex)2:)In)which)we)explore)Disasters,)Trees,) o  Ex)5):)Mood)of)the)UnionBText)AnalyFcs(Krishna))
ClassificaFon)&)the)Kaggle)CompeFFon)(Krishna)) o  In)which)we)analyze)the)Mood)of)the)naFon)from)
inferences)on)SOTU)by)the)POTUS)(State)of)the)Union)
o  Random)Forest,)Bagging,)Data)DeBcorrelaFon)
Addresses)by)The)President)Of)the)US))
o  Deepdive)B)Leverage)parallelism)of)RDDs,)sparse) o  Ex)99):)RecSys)2015)Challenge)(Krishna))
vectors,)etc)(Reza))
o  Ask)Us)Anything)B)Panel)
Introducing:)
Andy Konwinski
@andykonwinski
Hossein Falaki Michael Armbrust
@mhfalaki @michaelarmbrust
Reza Zadeh
@Reza_Zadeh
Tathagata Das
Paco Nathan
@tathadas
@pacoid
Xiangrui Meng
@xmeng
Krishna Sankar
@ksankar
About Me
o Chief Data Scientist at BlackArrow.tv
o Have been speaking at OSCON, PyCon, Pydata,
The)Nuthead)band)!)
Strata et al
o Reviewer “Machine Learning with Spark”
o Picked up co-authorship Second Edition of “Fast
Data Processing with Spark”
o Have done lots of things:
•  (cid:13)(cid:41)(cid:39)(cid:1)(cid:15)(cid:33)(cid:51)(cid:33)(cid:1)(cid:5)(cid:27)(cid:37)(cid:51)(cid:33)(cid:41)(cid:43)(cid:7)(cid:1)(cid:13)(cid:41)(cid:46)(cid:41)(cid:45)(cid:38)(cid:46)(cid:49)(cid:44)(cid:33)(cid:51)(cid:41)(cid:35)(cid:50)(cid:7)(cid:1)(cid:17)(cid:41)(cid:45)(cid:33)(cid:45)(cid:35)(cid:41)(cid:33)(cid:43)(cid:7)(cid:1)(cid:12)(cid:36)(cid:29)(cid:37)(cid:35)(cid:40)(cid:7)(cid:9)(cid:9)(cid:6)(cid:1)
•  (cid:32)(cid:49)(cid:41)(cid:51)(cid:51)(cid:37)(cid:45)(cid:1)(cid:13)(cid:46)(cid:46)(cid:42)(cid:50)(cid:1)(cid:5)(cid:32)(cid:37)(cid:34)(cid:1)(cid:11)(cid:9)(cid:10)(cid:7)(cid:1)(cid:32)(cid:41)(cid:49)(cid:37)(cid:43)(cid:37)(cid:50)(cid:50)(cid:7)(cid:1)(cid:20)(cid:33)(cid:53)(cid:33)(cid:7)(cid:58)(cid:6)(cid:1)
•  (cid:28)(cid:51)(cid:33)(cid:45)(cid:36)(cid:33)(cid:49)(cid:36)(cid:50)(cid:1)(cid:5)(cid:32)(cid:37)(cid:34)(cid:1)(cid:28)(cid:37)(cid:49)(cid:53)(cid:41)(cid:35)(cid:37)(cid:7)(cid:1)(cid:14)(cid:43)(cid:46)(cid:52)(cid:36)(cid:6)(cid:7)(cid:1)(cid:28)(cid:46)(cid:44)(cid:37)(cid:1)(cid:54)(cid:46)(cid:49)(cid:42)(cid:1)(cid:41)(cid:45)(cid:1)(cid:12)(cid:19)(cid:1)
•  (cid:18)(cid:52)(cid:37)(cid:50)(cid:51)(cid:1)(cid:21)(cid:37)(cid:35)(cid:51)(cid:52)(cid:49)(cid:37)(cid:49)(cid:1)(cid:33)(cid:51)(cid:1)(cid:23)(cid:33)(cid:53)(cid:33)(cid:43)(cid:1)(cid:25)(cid:18)(cid:1)(cid:28)(cid:35)(cid:40)(cid:46)(cid:46)(cid:43)(cid:7)(cid:58)(cid:1)
•  (cid:25)(cid:43)(cid:33)(cid:45)(cid:45)(cid:41)(cid:45)(cid:39)(cid:1)(cid:22)(cid:33)(cid:50)(cid:51)(cid:37)(cid:49)(cid:50)(cid:1)(cid:14)(cid:46)(cid:44)(cid:47)(cid:52)(cid:51)(cid:33)(cid:51)(cid:41)(cid:46)(cid:45)(cid:33)(cid:43)(cid:1)(cid:17)(cid:41)(cid:45)(cid:33)(cid:45)(cid:35)(cid:37)(cid:1)(cid:46)(cid:49)(cid:1)(cid:28)(cid:51)(cid:33)(cid:51)(cid:41)(cid:50)(cid:51)(cid:41)(cid:35)(cid:50)(cid:1)(cid:1)
•  (cid:31)(cid:46)(cid:43)(cid:52)(cid:45)(cid:51)(cid:37)(cid:37)(cid:49)(cid:1)(cid:33)(cid:50)(cid:1)(cid:27)(cid:46)(cid:34)(cid:46)(cid:51)(cid:41)(cid:35)(cid:50)(cid:1)(cid:20)(cid:52)(cid:36)(cid:39)(cid:37)(cid:1)(cid:33)(cid:51)(cid:1)(cid:17)(cid:41)(cid:49)(cid:50)(cid:51)(cid:1)(cid:21)(cid:37)(cid:39)(cid:46)(cid:1)(cid:43)(cid:37)(cid:33)(cid:39)(cid:52)(cid:37)(cid:1)(cid:32)(cid:46)(cid:49)(cid:43)(cid:36)(cid:1)(cid:14)(cid:46)(cid:44)(cid:47)(cid:37)(cid:51)(cid:41)(cid:51)(cid:41)(cid:46)(cid:45)(cid:50)(cid:1)
o  @ksankar, doubleclix.wordpress.com [e-mail]
Pre-requisites
①  Register)&)Download)data)from)Kaggle.)
We)cannot)distribute)Kaggle)data.)
Moreover)you)need)an)account)to)submit)entries)
a)  Setup)an)account)in)Kaggle)(www.kaggle.com))
b)  We)will)be)using)the)data)from)the)competition)“Titanic:)
Machine)Learning)from)Disaster”)
c)  Download)data)from)
http://www.kaggle.com/c/titanicWgettingStarted)
②  Register)for)RecSys)2015)Competition)
a)  http://2015.recsyschallenge.com/)
9:00
Welcome +
Getting Started
Getting Started: Step 1
Everyone will receive a username/password for one !
of the Databricks Cloud shards. Use your laptop and browser to login there.
We find that cloud-based notebooks are a simple way to get started using
Apache Spark – as the motto “Making Big Data Simple” states.
Please create and run a variety of notebooks on your account throughout the
tutorial. These accounts will remain open long enough for you to export your
work.
See the product page or FAQ for more details, or contact Databricks to register
for a trial account.
10