Table Of Content

Thushan Ganegedara M A N N I N G TensorFlow in Action THUSHAN GANEGEDARA MANNING SHELTER ISLAND For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: [email protected] ©2022 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Manning Publications Co. Development editor: Patrick Barb 20 Baldwin Road Technical development editor: Joel Kotarski PO Box 761 Review editor: Aleksandar Dragosavljevic´ Shelter Island, NY 11964 Production editor: Andy Marinkovich Copy editor: Michele Mitchell Proofreader: Melody Dolab Technical proofreader: Ninoslav Cerkez Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781617298349 Printed in the United States of America To my wife, Thushani brief contents PART 1 FOUNDATIONS OF TENSORFLOW 2 AND DEEP LEARNING .....................................................................1 1 ■ The amazing world of TensorFlow 3 2 ■ TensorFlow 2 19 3 ■ Keras and data retrieval in TensorFlow 2 47 4 ■ Dipping toes in deep learning 80 5 ■ State-of-the-art in deep learning: Transformers 119 PART 2 LOOK MA, NO HANDS! DEEP NETWORKS IN THE REAL WORLD..................................................147 6 ■ Teaching machines to see: Image classification with CNNs 149 7 ■ Teaching machines to see better: Improving CNNs and making them confess 194 8 ■ Telling things apart: Image segmentation 243 9 ■ Natural language processing with TensorFlow: Sentiment analysis 296 10 ■ Natural language processing with TensorFlow: Language modeling 349 v vi BRIEF CONTENTS PART 3 ADVANCED DEEP NETWORKS FOR COMPLEX PROBLEMS.................................................................385 11 ■ Sequence-to-sequence learning: Part 1 387 12 ■ Sequence-to-sequence learning: Part 2 433 13 ■ Transformers 453 14 ■ TensorBoard: Big brother of TensorFlow 511 15 ■ TFX: MLOps and deploying models with TensorFlow 554 contents preface xiv acknowledgments xvi about this book xvii about the author xxi about the cover illustration xxii PART 1 FOUNDATIONS OF TENSORFLOW 2 AND DEEP LEARNING............................................1 1 The amazing world of TensorFlow 3 1.1 What is TensorFlow? 4 An overview of popular components of TensorFlow 6 Building and deploying a machine learning model 8 1.2 GPU vs. CPU 9 1.3 When and when not to use TensorFlow 10 When to use TensorFlow 10 ■ When not to use TensorFlow 12 1.4 What will this book teach you? 14 TensorFlow fundamentals 14 ■ Deep learning algorithms 14 Monitoring and optimization 14 1.5 Who is this book for? 15 1.6 Should we really care about Python and TensorFlow 2? 16 vii viii CONTENTS 2 TensorFlow 2 19 2.1 First steps with TensorFlow 2 20 How does TensorFlow operate under the hood? 24 2.2 TensorFlow building blocks 28 Understanding tf.Variable 29 ■ Understanding tf.Tensor 32 Understanding tf.Operation 35 2.3 Neural network–related computations in TensorFlow 39 Matrix multiplication 39 ■ Convolution operation 41 Pooling operation 43 3 Keras and data retrieval in TensorFlow 2 47 3.1 Keras model-building APIs 48 Introducing the data set 49 ■ The Sequential API 52 The functional API 56 ■ The sub-classing API 61 3.2 Retrieving data for TensorFlow/Keras models 65 tf.data API 66 ■ Keras DataGenerators 72 tensorflow-datasets package 75 4 Dipping toes in deep learning 80 4.1 Fully connected networks 81 Understanding the data 82 ■ Autoencoder model 85 4.2 Convolutional neural networks 90 Understanding the data 90 ■ Implementing the network 92 4.3 One step at a time: Recurrent neural networks (RNNs) 105 Understanding the data 107 ■ Implementing the model 111 Predicting future CO2 values with the trained model 115 5 State-of-the-art in deep learning: Transformers 119 5.1 Representing text as numbers 120 5.2 Understanding the Transformer model 123 The encoder-decoder view of the Transformer 123 ■ Diving deeper 124 ■ Self-attention layer 128 ■ Understanding self- attention using scalars 131 ■ Self-attention as a cooking competition 135 ■ Masked self-attention layers 136 Multi-head attention 138 ■ Fully connected layer 139 Putting everything together 141 CONTENTS ix PART 2 LOOK MA, NO HANDS! DEEP NETWORKS IN THE REAL WORLD ........................................147 6 Teaching machines to see: Image classification with CNNs 149 6.1 Putting the data under the microscope: Exploratory data analysis 150 The folder/file structure 152 ■ Understanding the classes in the data set 155 ■ Computing simple statistics on the data set 158 6.2 Creating data pipelines using the Keras ImageDataGenerator 160 6.3 Inception net: Implementing a state-of-the-art image classifier 165 Recap on CNNs 166 ■ Inception net v1 169 ■ Putting everything together 181 ■ Other Inception models 183 6.4 Training the model and evaluating performance 188 7 Teaching machines to see better: Improving CNNs and making them confess 194 7.1 Techniques for reducing overfitting 195 Image data augmentation with Keras 196 ■ Dropout: Randomly switching off parts of your network to improve generalizability 203 Early stopping: Halting the training process if the network starts to underperform 207 7.2 Toward minimalism: Minception instead of Inception 210 Implementing the stem 211 ■ Implementing Inception-ResNet type A block 216 ■ Implementing the Inception-ResNet type B block 223 ■ Implementing the reduction block 225 ■ Putting everything together 227 ■ Training Minception 229 7.3 If you can't beat them, join ‘em: Using pretrained networks for enhancing performance 232 Transfer learning: Reusing existing knowledge in deep neural networks 232 7.4 Grad-CAM: Making CNNs confess 238