Spark: The Definitive Guide: Big Data Processing Made Simple

601 Pages·2018·7.88 MB·English

by  Bill Chambers

Spark: The Definitive Guide
Big Data Processing Made Simple
Bill Chambers and Matei Zaharia
Spark: The Definitive Guide
by Bill Chambers and Matei Zaharia
Copyright © 2018 Databricks. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online
editions are also available for most titles (http://oreilly.com/safari). For more information,
contact our corporate/institutional sales department: 800-998-9938 or [email protected].

Editor: Nicole Tache
Production Editor: Justin Billing
Copyeditor: Octal Publishing, Inc., Chris Edwards, and Amanda Kersey
Proofreader: Jasmine Kwityn
Indexer: Judith McConville
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Rebecca Demarest
February 2018: First Edition
Revision History for the First Edition
2018-02-08: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781491912218 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Spark: The Definitive Guide,
the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. Apache, Spark
and Apache Spark are trademarks of the Apache Software Foundation.
While the publisher and the authors have used good faith efforts to ensure that the information
and instructions contained in this work are accurate, the publisher and the authors disclaim all
responsibility for errors or omissions, including without limitation responsibility for damages
resulting from the use of or reliance on this work. Use of the information and instructions
contained in this work is at your own risk. If any code samples or other technology this work
contains or describes is subject to open source licenses or the intellectual property rights of
others, it is your responsibility to ensure that your use thereof complies with such licenses and/or
rights.
978-1-491-91221-8
[M]
Preface
Welcome to this first edition of Spark: The Definitive Guide! We are excited to bring you the
most complete resource on Apache Spark today, focusing especially on the new generation of
Spark APIs introduced in Spark 2.0.
Apache Spark is currently one of the most popular systems for large-scale data processing, with
APIs in multiple programming languages and a wealth of built-in and third-party libraries.
Although the project has existed for multiple years—first as a research project started at UC
Berkeley in 2009, then at the Apache Software Foundation since 2013—the open source
community is continuing to build more powerful APIs and high-level libraries over Spark, so
there is still a lot to write about the project. We decided to write this book for two reasons. First,
we wanted to present the most comprehensive book on Apache Spark, covering all of the
fundamental use cases with easy-to-run examples. Second, we especially wanted to explore the
higher-level “structured” APIs that were finalized in Apache Spark 2.0—namely DataFrames,
Datasets, Spark SQL, and Structured Streaming—which older books on Spark don’t always
include. We hope this book gives you a solid foundation to write modern Apache Spark
applications using all the available tools in the project.
In this preface, we’ll tell you a little bit about our background, and explain who this book is for
and how we have organized the material. We also want to thank the numerous people who
helped edit and review this book, without whom it would not have been possible.
About the Authors
Both of the book’s authors have been involved in Apache Spark for a long time, so we are very
excited to be able to bring you this book.
Bill Chambers started using Spark in 2014 on several research projects. Currently, Bill is a
Product Manager at Databricks where he focuses on enabling users to write various types of
Apache Spark applications. Bill also regularly blogs about Spark and presents at conferences and
meetups on the topic. Bill holds a Master’s in Information Management and Systems from the
UC Berkeley School of Information.
Matei Zaharia started the Spark project in 2009, during his time as a PhD student at UC
Berkeley. Matei worked with other Berkeley researchers and external collaborators to design the
core Spark APIs and grow the Spark community, and has continued to be involved in new
initiatives such as the structured APIs and Structured Streaming. In 2013, Matei and other
members of the Berkeley Spark team co-founded Databricks to further grow the open source
project and provide commercial offerings around it. Today, Matei continues to work as Chief
Technologist at Databricks, and also holds a position as an Assistant Professor of Computer
Science at Stanford University, where he does research on large-scale systems and AI. Matei
received his PhD in Computer Science from UC Berkeley in 2013.
Who This Book Is For
We designed this book mainly for data scientists and data engineers looking to use Apache
Spark. The two roles have slightly different needs, but in reality, most application development
covers a bit of both, so we think the material will be useful in both cases. Specifically, in our
minds, the data scientist workload focuses more on interactively querying data to answer
questions and build statistical models, while the data engineer job focuses on writing
maintainable, repeatable production applications—either to use the data scientist’s models in
practice, or just to prepare data for further analysis (e.g., building a data ingest pipeline).
However, we often see with Spark that these roles blur. For instance, data scientists are able to
package production applications without too much hassle and data engineers use interactive
analysis to understand and inspect their data to build and maintain pipelines.
While we tried to provide everything data scientists and engineers need to get started, there are
some things we didn’t have space to focus on in this book. First, this book does not include in-
depth introductions to some of the analytics techniques you can use in Apache Spark, such as
machine learning. Instead, we show you how to invoke these techniques using libraries in Spark,
assuming you already have a basic background in machine learning. Many full, standalone books
exist to cover these techniques in formal detail, so we recommend starting with those if you want
to learn about these areas. Second, this book focuses more on application development than on
operations and administration (e.g., how to manage an Apache Spark cluster with dozens of
users). Nonetheless, we have tried to include comprehensive material on monitoring, debugging,
and configuration in Parts V and VI of the book to help engineers get their application running
efficiently and tackle day-to-day maintenance. Finally, this book places less emphasis on the
older, lower-level APIs in Spark—specifically RDDs and DStreams—to introduce most of the
concepts using the newer, higher-level structured APIs. Thus, the book may not be the best fit if
you need to maintain an old RDD or DStream application, but should be a great introduction to
writing new applications.
Conventions Used in This Book
The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program elements such as
variable or function names, databases, data types, environment variables, statements, and
keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
Constant width italic
Shows text that should be replaced with user-supplied values or by values determined by
context.
TIP
This element signifies a tip or suggestion.
NOTE
This element signifies a general note.
WARNING
This element indicates a warning or caution.
Using Code Examples
We’re very excited to have designed this book so that all of the code content is runnable on real
data. We wrote the whole book using Databricks notebooks and have posted the data and related
material on GitHub. This means that you can run and edit all the code as you follow along, or
copy it into working code in your own applications.
We tried to use real data wherever possible to illustrate the challenges you’ll run into while
building large-scale data applications. Finally, we also include several larger standalone
applications in the book’s GitHub repository for examples that it does not make sense to show
inline in the text.
The GitHub repository will remain a living document as we update based on Spark’s progress.
Be sure to follow updates there.
This book is here to help you get your job done. In general, if example code is offered with this
book, you may use it in your programs and documentation. You do not need to contact us for
permission unless you’re reproducing a significant portion of the code. For example, writing a
program that uses several chunks of code from this book does not require permission. Selling or
distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a
question by citing this book and quoting example code does not require permission.
Incorporating a significant amount of example code from this book into your product’s
documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title, author,
publisher, and ISBN. For example: “Spark: The Definitive Guide by Bill Chambers and Matei
Zaharia (O’Reilly). Copyright 2018 Databricks, Inc., 978-1-491-91221-8.”
If you feel your use of code examples falls outside fair use or the permission given above, feel
free to contact us at [email protected].
O’Reilly Safari
Safari (formerly Safari Books Online) is a membership-based training and reference platform for
enterprise, government, educators, and individuals.
Members have access to thousands of books, training videos, Learning Paths, interactive
tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard
Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press,
Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress,
Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New
Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others.
For more information, please visit http://oreilly.com/safari.
How to Contact Us
Please address comments and questions concerning this book to the publisher:
O’Reilly Media, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472
800-998-9938 (in the United States or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)
To comment or ask technical questions about this book, send email to
[email protected].
For more information about our books, courses, conferences, and news, see our website at
http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
Acknowledgments
There were a huge number of people that made this book possible.
First, we would like to thank our employer, Databricks, for allocating time for us to work on this
book. Without the support of the company, this book would not have been possible. In particular,
we would like to thank Ali Ghodsi, Ion Stoica, and Patrick Wendell for their support.
Additionally, there are numerous people that read drafts of the book and individual chapters. Our
reviewers were best-in-class, and provided invaluable feedback.
These reviewers, in alphabetical order by last name, are:
Lynn Armstrong
Mikio Braun
Jules Damji
Denny Lee
Alex Thomas
In addition to the formal book reviewers, there were numerous other Spark users, contributors,
and committers who read over specific chapters or helped formulate how topics should be
discussed. In alphabetical order by last name, the people who helped are:
Sameer Agarwal
Bagrat Amirbekian
Michael Armbrust
Joseph Bradley
Tathagata Das
Hossein Falaki
Wenchen Fan
Sue Ann Hong
Yin Huai
Tim Hunter
Xiao Li
Cheng Lian
Xiangrui Meng
Kris Mok
Josh Rosen
Srinath Shankar
Takuya Ueshin
Herman van Hövell
Reynold Xin
Philip Yang
Burak Yavuz
Shixiong Zhu
Lastly, we would like to thank friends, family, and loved ones. Without their support, patience,
and encouragement, we would not have been able to write the definitive guide to Spark.