Table Of ContentMastering Machine Learning with
Spark 2.x
Create scalable machine learning applications to power a
modern data-driven business using Spark
Alex Tellez
Max Pumperla
Michal Malohlava
BIRMINGHAM - MUMBAI
Mastering Machine Learning with Spark 2.x
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means, without the prior written permission of the
publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the
information presented. However, the information contained in this book is sold without
warranty, either express or implied. Neither the authors, nor Packt Publishing, and its
dealers and distributors will be held liable for any damages caused or alleged to be caused
directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: August 2017
Production reference: 1290817
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78528-345-1
www.packtpub.com
Credits
Author Copy Editor
Alex Tellez Muktikant Garimella
Max Pumperla
Michal Malohlava
Reviewer Project Coordinator
Dipanjan Deb Ulhas Kambali
Commissioning Editor Proofreader
Veena Pagare Safis Editing
Acquisition Editor Indexer
Larissa Pinto Rekha Nair
Content Development Editor Graphics
Nikhil Borkar Jason Monteiro
Technical Editor Production Coordinator
Diwakar Shukla Melwyn Dsa
About the Authors
Alex Tellez is a life-long data hacker/enthusiast with a passion for data science and its
application to business problems. He has a wealth of experience working across multiple
industries, including banking, health care, online dating, human resources, and online
gaming. Alex has also given multiple talks at various AI/machine learning conferences, in
addition to lectures at universities about neural networks. When he’s not neck-deep in a
textbook, Alex enjoys spending time with family, riding bikes, and utilizing machine
learning to feed his French wine curiosity!
First and foremost, I’d like to thank my co-author, Michal, for helping me write this book.
As fellow ML enthusiasts, cyclists, runners, and fathers, we both developed a deeper
understanding of each other through this endeavor, which has taken well over one year to
create. Simply put, this book would not have been possible without Michal’s support and
encouragement.
Next, I’d like to thank my mom, dad, and elder brother, Andres, who have been there every
step of the way from day 1 until now. Without question, my elder brother continues to be
my hero and is someone that I will forever look up to as being a guiding light. Of course,
no acknowledgements would be finished without giving thanks to my beautiful wife,
Denise, and daughter, Miya, who have provided the love and support to continue the
writing of this book during nights and weekends. I cannot emphasize enough how much
you both mean to me and how you guys are the inspiration and motivation that keeps this
engine running. To my daughter, Miya, my hope is that you can pick this book up and one
day realize that your old man isn’t quite as silly as I appear to let on.
Last but not least, I’d also like to give thanks to you, the reader, for your interest in this
exciting field using this incredible technology. Whether you are a seasoned ML expert, or a
newcomer to the field looking to gain a foothold, you have come to the right book and my
hope is that you get as much out of this as Michal and I did in writing this work.
Max Pumperla is a data scientist and engineer specializing in deep learning and its
applications. He currently works as a deep learning engineer at Skymind and is a co-
founder of aetros.com. Max is the author and maintainer of several Python packages,
including elephas, a distributed deep learning library using Spark. His open source
footprint includes contributions to many popular machine learning libraries, such as keras,
deeplearning4j, and hyperopt. He holds a PhD in algebraic geometry from the University of
Hamburg.
Michal Malohlava, creator of Sparkling Water, is a geek and the developer; Java, Linux,
programming languages enthusiast who has been developing software for over 10 years.
He obtained his PhD from Charles University in Prague in 2012, and post doctorate from
Purdue University.
During his studies, he was interested in the construction of not only distributed but also
embedded and real-time, component-based systems, using model-driven methods and
domain-specific languages. He participated in the design and development of various
systems, including SOFA and Fractal component systems and the jPapabench control
system.
Now, his main interest is big data computation. He participates in the development of the
H2O platform for advanced big data math and computation, and its embedding into Spark
engine, published as a project called Sparkling Water.
I would like to thank my wife, Claire, for her love and encouragement.
About the Reviewer
Dipanjan Deb is an experienced analytic professional with over 17 years of cumulative
experience in machine/statistical learning, data mining and predictive analytics across
finance, healthcare, automotive, CPG, automotive, energy, and human resource domains.
He is highly proficient in developing cutting-edge analytic solutions using open source and
commercial software to integrate multiple systems in order to provide massively
parallelized and large-scale optimization.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com. Did
you know that Packt offers eBook versions of every book published, with PDF and ePub
files available? You can upgrade to the eBook version at www.PacktPub.comand as a print
book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
service@packtpub.com for more details. At www.PacktPub.com, you can also read a
collection of free technical articles, sign up for a range of free newsletters and receive
exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt
books and video courses, as well as industry-leading tools to help you plan your personal
development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial
process. To help us improve, please leave us an honest review on this book's Amazon page
at https://www.amazon.com/dp/1785283456.
If you'd like to join our team of regular reviewers, you can e-mail us at
customerreviews@packtpub.com. We award our regular reviewers with free eBooks and
videos in exchange for their valuable feedback. Help us be relentless in improving our
products!
Table of Contents
Preface
1
Chapter 1: Introduction to Large-Scale Machine Learning and Spark
7
Data science 9
The sexiest role of the 21st century – data scientist? 10
A day in the life of a data scientist 11
Working with big data 12
The machine learning algorithm using a distributed environment 13
Splitting of data into multiple machines 14
From Hadoop MapReduce to Spark 15
What is Databricks? 16
Inside the box 17
Introducing H2O.ai 18
Design of Sparkling Water 19
What's the difference between H2O and Spark's MLlib? 20
Data munging 21
Data science - an iterative process 21
Summary 22
Chapter 2: Detecting Dark Matter - The Higgs-Boson Particle
23
Type I versus type II error 23
Finding the Higgs-Boson particle 25
The LHC and data creation 25
The theory behind the Higgs-Boson 26
Measuring for the Higgs-Boson 26
The dataset 27
Spark start and data load 28
Labeled point vector 38
Data caching 38
Creating a training and testing set 40
What about cross-validation? 41
Our first model – decision tree 43
Gini versus Entropy 44
Next model – tree ensembles 51
Random forest model 52
Grid search 54
Gradient boosting machine 56
