Learning Tensorflow: A Guide to Building Deep Learning Systems

242 Pages·2017·12.64 MB·English

by  Tom Hope

Learning
TensorFlow
A GUIDE TO BUILDING DEEP LEARNING SYSTEMS
Tom Hope, Yehezkel S. Resheff & Itay Lieder

Learning TensorFlow
A Guide to Building Deep Learning Systems
Tom Hope, Yehezkel S. Resheff, and Itay Lieder
BBeeiijjiinngg BBoossttoonn FFaarrnnhhaamm SSeebbaassttooppooll TTookkyyoo
Learning TensorFlow
by Tom Hope, Yehezkel S. Resheff, and Itay Lieder
Copyright © 2017 Tom Hope, Itay Lieder, and Yehezkel S. Resheff. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/insti‐
tutional sales department: 800-998-9938 or [email protected].
Editor: Nicole Tache Indexer: Judith McConville
Production Editor: Shiny Kalapurakkel Interior Designer: David Futato
Copyeditor: Rachel Head Cover Designer: Karen Montgomery
Proofreader: Sharon Wilkey Illustrator: Rebecca Demarest
August 2017: First Edition
Revision History for the First Edition
2017-08-04: First Release
2017-09-15: Second Release
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning TensorFlow, the cover image,
and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the authors have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.
978-1-491-97851-1
[LSI]
Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Going Deep 1
Using TensorFlow for AI Systems 2
TensorFlow: What’s in a Name? 5
A High-Level Overview 6
Summary 8
2. Go with the Flow: Up and Running with TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Installing TensorFlow 9
Hello World 11
MNIST 13
Softmax Regression 14
Summary 21
3. Understanding TensorFlow Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Computation Graphs 23
What Is a Computation Graph? 23
The Benefits of Graph Computations 24
Graphs, Sessions, and Fetches 24
Creating a Graph 25
Creating a Session and Running It 26
Constructing and Managing Our Graph 27
Fetches 29
Flowing Tensors 30
Nodes Are Operations, Edges Are Tensor Objects 30
Data Types 32
iii
Tensor Arrays and Shapes 33
Names 37
Variables, Placeholders, and Simple Optimization 38
Variables 38
Placeholders 39
Optimization 40
Summary 49
4. Convolutional Neural Networks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Introduction to CNNs 51
MNIST: Take II 53
Convolution 54
Pooling 56
Dropout 57
The Model 57
CIFAR10 61
Loading the CIFAR10 Dataset 62
Simple CIFAR10 Models 64
Summary 68
5. Text I: Working with Text and Sequences, and TensorBoard Visualization. . . . . . . . . . . 69
The Importance of Sequence Data 69
Introduction to Recurrent Neural Networks 70
Vanilla RNN Implementation 72
TensorFlow Built-in RNN Functions 82
RNN for Text Sequences 84
Text Sequences 84
Supervised Word Embeddings 88
LSTM and Using Sequence Length 89
Training Embeddings and the LSTM Classifier 91
Summary 93
6. Text II: Word Vectors, Advanced RNN, and Embedding Visualization. . . . . . . . . . . . . . . 95
Introduction to Word Embeddings 95
Word2vec 97
Skip-Grams 98
Embeddings in TensorFlow 100
The Noise-Contrastive Estimation (NCE) Loss Function 101
Learning Rate Decay 101
Training and Visualizing with TensorBoard 102
Checking Out Our Embeddings 103
Pretrained Embeddings, Advanced RNN 105
iv | Table of Contents
Pretrained Word Embeddings 106
Bidirectional RNN and GRU Cells 110
Summary 112
7. TensorFlow Abstractions and Simplifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Chapter Overview 113
High-Level Survey 115
contrib.learn 117
Linear Regression 118
DNN Classifier 120
FeatureColumn 123
Homemade CNN with contrib.learn 128
TFLearn 131
Installation 131
CNN 131
RNN 134
Keras 136
Pretrained models with TF-Slim 143
Summary 151
8. Queues, Threads, and Reading Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
The Input Pipeline 153
TFRecords 154
Writing with TFRecordWriter 155
Queues 157
Enqueuing and Dequeuing 157
Multithreading 159
Coordinator and QueueRunner 160
A Full Multithreaded Input Pipeline 162
tf.train.string_input_producer() and tf.TFRecordReader() 164
tf.train.shuffle_batch() 164
tf.train.start_queue_runners() and Wrapping Up 165
Summary 166
9. Distributed TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
Distributed Computing 167
Where Does the Parallelization Take Place? 168
What Is the Goal of Parallelization? 168
TensorFlow Elements 169
tf.app.flags 169
Clusters and Servers 170
Replicating a Computational Graph Across Devices 171
Table of Contents | v
Managed Sessions 171
Device Placement 172
Distributed Example 173
Summary 179
10. Exporting and Serving Models with TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
Saving and Exporting Our Model 181
Assigning Loaded Weights 182
The Saver Class 185
Introduction to TensorFlow Serving 191
Overview 192
Installation 193
Building and Exporting 194
Summary 201
A. Tips on Model Construction and Using TensorFlow Serving. . . . . . . . . . . . . . . . . . . . . . . 203
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
vi | Table of Contents
Preface
Deep learning has emerged in the last few years as a premier technology for building
intelligent systems that learn from data. Deep neural networks, originally roughly
inspired by how the human brain learns, are trained with large amounts of data to
solve complex tasks with unprecedented accuracy. With open source frameworks
making this technology widely available, it is becoming a must-know for anybody
involved with big data and machine learning.
TensorFlow is currently the leading open source software for deep learning, used by a
rapidly growing number of practitioners working on computer vision, natural lan‐
guage processing (NLP), speech recognition, and general predictive analytics.
This book is an end-to-end guide to TensorFlow designed for data scientists, engi‐
neers, students, and researchers. The book adopts a hands-on approach suitable for a
broad technical audience, allowing beginners a gentle start while diving deep into
advanced topics and showing how to build production-ready systems.
In this book you will learn how to:
1. Get up and running with TensorFlow, rapidly and painlessly.
2. Use TensorFlow to build models from the ground up.
3. Train and understand popular deep learning models for computer vision and
NLP.
4. Use extensive abstraction libraries to make development easier and faster.
5. Scale up TensorFlow with queuing and multithreading, training on clusters, and
serving output in production.
6. And much more!
This book is written by data scientists with extensive R&D experience in both indus‐
try and academic research. The authors take a hands-on approach, combining practi‐
cal and intuitive examples, illustrations, and insights suitable for practitioners seeking
to build production-ready systems, as well as readers looking to learn to understand
and build flexible and powerful models.
vii
Prerequisites
This book assumes some basic Python programming know-how, including basic
familiarity with the scientific library NumPy.
Machine learning concepts are touched upon and intuitively explained throughout
the book. For readers who want to gain a deeper understanding, a reasonable level of
knowledge in machine learning, linear algebra, calculus, probability, and statistics is
recommended.
Conventions Used in This Book
The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program ele‐
ments such as variable or function names, databases, data types, environment
variables, statements, and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
Constant width italic
Shows text that should be replaced with user-supplied values or by values deter‐
mined by context.
Using Code Examples
Supplemental material (code examples, exercises, etc.) is available for download at
https://github.com/Hezi-Resheff/Oreilly-Learning-TensorFlow.
This book is here to help you get your job done. In general, if example code is offered
with this book, you may use it in your programs and documentation. You do not
need to contact us for permission unless you’re reproducing a significant portion of
the code. For example, writing a program that uses several chunks of code from this
book does not require permission. Selling or distributing a CD-ROM of examples
from O’Reilly books does require permission. Answering a question by citing this
book and quoting example code does not require permission. Incorporating a signifi‐
cant amount of example code from this book into your product’s documentation does
require permission.
We appreciate, but do not require, attribution. An attribution usually includes the
title, author, publisher, and ISBN. For example: “Learning TensorFlow by Tom Hope,
viii | Preface