ebook img

Jason Brownlee -Better deep learning PDF

575 Pages·2019·9.42 MB·English
by  
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Jason Brownlee -Better deep learning

Better Deep Learning Train Faster, Reduce Overfitting, and Make Better Predictions Jason Brownlee i Disclaimer The information contained within this eBook is strictly for educational purposes. If you wish to apply ideas contained in this eBook, you are taking full responsibility for your actions. The author has made every effort to ensure the accuracy of the information within this book was correct at time of publication. The author does not assume and hereby disclaims any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from accident, negligence, or any other cause. No part of this eBook may be reproduced or transmitted in any form or by any means, electronic or mechanical, recording or by any information storage and retrieval system, without written permission from the author. Acknowledgements Special thanks to my proofreader Sarah Martin and my technical editors Andrei Cheremskoy, Michael Sanderson, Arun Koshy. Copyright Better Deep Learning © Copyright 2019 Jason Brownlee. All Rights Reserved. Edition: v1.3 Contents Copyright i Contents ii Preface iii Introduction v Welcome v Framework for Better Deep Learning x Diagnostic Learning Curves xix I Better Learning 1 1 Improve Learning by Understanding Optimization 3 1.1 Neural Nets Learn a Mapping Function . . . . . . . . . . . . . . . . . . . . . . . 3 1.2 Learning Network Weights Is Hard . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.3 Key Features of the Error Surface . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.4 Navigating the Non-Convex Error Surface . . . . . . . . . . . . . . . . . . . . . 8 1.5 Implications for Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.6 Components of the Learning Algorithm . . . . . . . . . . . . . . . . . . . . . . . 9 1.7 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2 Configure Capacity with Nodes and Layers 14 2.1 Neural Network Model Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Nodes and Layers Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.3 Model Capacity Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 ii CONTENTS iii 3 Configure Gradient Precision with Batch Size 29 3.1 Batch Size and Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2 Gradient Descent Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3 Batch Size Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4 Configure What to Optimize with Loss Functions 49 4.1 Loss Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.2 Regression Loss Functions Case Study . . . . . . . . . . . . . . . . . . . . . . . 56 4.3 Binary Classification Loss Functions Case Study . . . . . . . . . . . . . . . . . . 64 4.4 Multiclass Classification Loss Functions Case Study . . . . . . . . . . . . . . . . 74 4.5 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 5 Configure Speed of Learning with Learning Rate 87 5.1 Learning Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.2 Learning Rate Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.3 Learning Rate Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 6 Stabilize Learning with Data Scaling 121 6.1 Data Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.2 Data Scaling scikit-learn API . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 6.3 Data Scaling Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 6.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 7 Fix Vanishing Gradients with ReLU 141 7.1 Vanishing Gradients and ReLU . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 7.2 ReLU Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 7.3 ReLU Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 7.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 7.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 7.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 8 Fix Exploding Gradients with Gradient Clipping 167 8.1 Exploding Gradients and Clipping . . . . . . . . . . . . . . . . . . . . . . . . . . 167 8.2 Gradient Clipping Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 8.3 Gradient Clipping Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 8.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 8.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 CONTENTS iv 8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 9 Accelerate Learning with Batch Normalization 180 9.1 Batch Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 9.2 Batch Normalization Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 9.3 Batch Normalization Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . 189 9.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 9.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 9.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 10 Deeper Models with Greedy Layer-Wise Pretraining 201 10.1 Greedy Layer-Wise Pretraining . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 10.2 Greedy Layer-Wise Pretraining Case Study . . . . . . . . . . . . . . . . . . . . . 204 10.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 10.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 10.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 11 Jump-Start Training with Transfer Learning 221 11.1 Transfer Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 11.2 Transfer Learning Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 11.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 11.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 11.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 II Better Generalization 243 12 Fix Overfitting with Regularization 245 12.1 Problem of Model Generalization and Overfitting . . . . . . . . . . . . . . . . . 245 12.2 Reduce Overfitting by Constraining Complexity . . . . . . . . . . . . . . . . . . 247 12.3 Regularization Methods for Neural Networks . . . . . . . . . . . . . . . . . . . . 248 12.4 Regularization Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . 249 12.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250 12.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 13 Penalize Large Weights with Weight Regularization 252 13.1 Weight Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 13.2 Weight Regularization Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . 258 13.3 Weight Regularization Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . 260 13.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 13.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 13.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 14 Sparse Representations with Activity Regularization 272 14.1 Activity Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272 14.2 Activity Regularization Keras API . . . . . . . . . . . . . . . . . . . . . . . . . 276 14.3 Activity Regularization Case Study . . . . . . . . . . . . . . . . . . . . . . . . . 278 CONTENTS v 14.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 14.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 14.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 15 Force Small Weights with Weight Constraints 290 15.1 Weight Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 15.2 Weight Constraints Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 15.3 Weight Constraints Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 15.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 15.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 15.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 16 Decouple Layers with Dropout 305 16.1 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 16.2 Dropout Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310 16.3 Dropout Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314 16.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 16.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 16.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 17 Promote Robustness with Noise 323 17.1 Noise Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 17.2 Noise Regularization Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 17.3 Noise Regularization Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . 330 17.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 17.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 17.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 18 Halt Training at the Right Time with Early Stopping 344 18.1 Early Stopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 18.2 Early Stopping Keras API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 18.3 Early Stopping Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 18.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 18.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 18.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 366 III Better Predictions 367 19 Reduce Model Variance with Ensemble Learning 369 19.1 High Variance of Neural Network Models . . . . . . . . . . . . . . . . . . . . . . 369 19.2 Reduce Variance Using an Ensemble of Models . . . . . . . . . . . . . . . . . . . 370 19.3 How to Ensemble Neural Network Models . . . . . . . . . . . . . . . . . . . . . 371 19.4 Summary of Ensemble Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 375 19.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 19.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377 CONTENTS vi 20 Combine Models From Multiple Runs with Model Averaging Ensemble 379 20.1 Model Averaging Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 20.2 Ensembles in Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 20.3 Model Averaging Ensemble Case Study . . . . . . . . . . . . . . . . . . . . . . . 382 20.4 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 20.5 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 20.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 21 Contribute Proportional to Trust with Weighted Average Ensemble 398 21.1 Weighted Average Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 21.2 Weighted Average Ensemble Case Study . . . . . . . . . . . . . . . . . . . . . . 399 21.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 21.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 21.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 22 Fit Models on Different Samples with Resampling Ensembles 420 22.1 Resampling Ensembles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 22.2 Resampling Ensembles Case Study . . . . . . . . . . . . . . . . . . . . . . . . . 422 22.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440 22.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 22.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442 23 Models from Contiguous Epochs with Horizontal Voting Ensembles 443 23.1 Horizontal Voting Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 23.2 Horizontal Voting Ensembles Case Study . . . . . . . . . . . . . . . . . . . . . . 444 23.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456 23.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 456 23.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 457 24 Cyclic Learning Rate and Snapshot Ensembles 458 24.1 Snapshot Ensembles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 458 24.2 Snapshot Ensembles Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 460 24.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 24.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480 24.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 480 25 Learn to Combine Predictions with Stacked Generalization Ensemble 482 25.1 Stacked Generalization Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . 482 25.2 Stacked Generalization Ensemble Case Study . . . . . . . . . . . . . . . . . . . . 484 25.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500 25.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 500 25.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502 26 Combine Model Parameters with Average Model Weights Ensemble 503 26.1 Average Model Weight Ensemble . . . . . . . . . . . . . . . . . . . . . . . . . . 503 26.2 Average Model Weight Ensemble Case Study . . . . . . . . . . . . . . . . . . . . 505 26.3 Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523 CONTENTS vii 26.4 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523 26.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524 IV Appendix 525 A Getting Help 526 A.1 Applied Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526 A.2 Official Keras Destinations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 526 A.3 Where to Get Help with Keras . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527 A.4 How to Ask Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527 A.5 Contact the Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528 B How to Setup Your Workstation 529 B.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529 B.2 Download Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529 B.3 Install Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 531 B.4 Start and Update Anaconda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 533 B.5 Install Deep Learning Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . 536 B.6 Further Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 B.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 537 V Conclusions 538 How Far You Have Come 539 Preface Modern open source libraries for developing neural network models are amazing. Gone are the days where we might spend weeks debugging the translation of poorly documented mathematics into code in the hopes of getting even the simplest model running. Today, using libraries like Keras, we can define and begin fitting a Multilayer Perceptron, Convolutional or even Recurrent Neural Network model of arbitrary complexity in minutes. While defining and fitting models has become trivial, getting good performance with a neural network model on a specific problem remains challenging. Traditionally, configuring neural networks in order to get good performance was referred to as a dark art. This is because there are no clear rules on how to best prepare data and configure a model for a given problem. Instead, experience must be developed over time from working on many different projects. Nevertheless, neural networks have been used in academia and industry for decades now and there are a suite of standard techniques, tips, and configurations that you can use to greatly increase the likelihood of getting better-than-average performance with a neural network model. I wrote this book to pull together the best classical and modern techniques in order to provide a playbook that you can use to get better performance on your next project using deep learning neural networks. A lot has changed in the last 5 to 10 years and there are activation functions, regularization methods, and even new ensemble methods that result in remarkably faster learning, lower generalization error, and more robust results. I hope that you’re as excited as me about the journey ahead. Jason Brownlee 2019 viii Introduction ix Welcome Welcome to Better Deep Learning. Deep learning neural networks have become easy to define and fit, but it remains challenging to achieve good predictive modeling performance. Neural networks have been studied in academia and used in industry for decades, and there is a wealth of techniques, tips, and model configurations that are known to result in better than average performance. In addition, the last 5 to 10 years has seen the development and adoption of modern network configurations, regularization techniques, and ensemble algorithms that result in superior performance. I designed this book to tie together classical and modern techniques into a single playbook and teach you step-by-step how to improve the performance of deep learning neural network models on your predictive modeling problems. Who Is This Book For? Before we get started, let’s make sure you are in the right place. This book is for developers that know some applied machine learning and some deep learning. Maybe you want or need to start using deep learning on your research project or on a project at work. This guide was written to help you do that quickly and efficiently by compressing years of knowledge and experience into a laser-focused course of hands-on tutorials. The lessons in this book assume a few things about you, such as: ˆ You know your way around basic Python for programming. ˆ You know your way around basic NumPy for array manipulation. ˆ You know your way around basic Keras for deep learning. This guide was written in the top-down and results-first machine learning style that you’re used to from MachineLearningMastery.com. About Your Outcomes This book will teach you how to get results as a machine learning practitioner interested in getting the most out of deep learning models on your own predictive modeling projects. Techniques are demonstrated in the context of small well-understood predictive modeling problems and Multilayer Perceptron neural network models, but can easily be applied to a wide range of problems and with a suite of different types of neural network models. After reading and working through this book, you will know: x xi ˆ A checklist of techniques that you can use to improve the performance of deep learning neural network models on your own predictive modeling problems. ˆ How to accelerate learning through better configured stochastic gradient descent batch size, loss functions, learning rates, and to avoid exploding gradients via gradient clipping. ˆ How to accelerate learning through correct data scaling, batch normalization, and use of modern activation functions such as the rectified linear activation function. ˆ How to accelerate learning through choosing better initial weights with greedy layer-wise pretraining and transfer learning. ˆ A gentle introduction to the problem of overfitting and a tour of regularization techniques. ˆ How to reduce overfitting by updating the loss function using techniques such as weight regularization, weight constraints, and activation regularization. ˆ How to reduce overfitting using techniques such as dropout, the addition of noise, and early stopping. ˆ A gentle introduction to how to combine the predictions from multiple models and a tour of ensemble learning techniques. ˆ How to combine the predictions from multiple different models using techniques such as weighted averaging ensembles and stacked generalization ensembles, also known as blending. ˆ How to combine the predictions from multiple models saved during a single training run with techniques such as horizontal ensembles and snapshot ensembles. This new understanding of applied deep learning methods will impact your practice of working through predictive modeling problems in the following ways: ˆ Confidently diagnose poor model training and problems such as premature convergence and accelerate the model training process using one or a combination of modifications to the learning algorithm. ˆ Confidently diagnose cases of overfitting the training dataset and reduce generalization error using one or a combination of modifications of the model, loss function, or learning algorithm. ˆ Confidently diagnose high variance in a final model and improve the average predictive skill by combining the predictions from multiple models trained over a single or multiple training runs. This book will teach you how to get good results, quickly, but will NOT teach you how to be a research scientist nor will it teach you all the theory behind why specific methods work. For that, I would recommend good research papers and textbooks. See the Further Reading section at the end of each tutorial for a good starting point. xii How to Read This Book This book was written to be read linearly, from start to finish. That being said, if you know the basics and need help with a specific problem type or technique, then you can flip straight to that section and get started. This book was designed for you to read on your workstation, on the screen. Not away from the computer or on a tablet or eReader. My hope is that you have the book open right next to your editor and run the examples as you read about them. This book is not intended to be read passively or be placed in a folder as a reference text. It is a playbook, a workbook, and a guidebook intended for you to learn by doing and then apply your new understanding to your own predictive modeling projects. To get the most out of the book, I would recommend playing with the examples in each tutorial. Extend them, break them, then fix them. Try some of the extensions presented at the end of each lesson and let me know how you do. About the Book Structure This book was designed around three main activities for getting better results with deep learning models: better or faster learning, better generalization to new data, and better predictions when using final models. There are a lot of things you could learn about getting better results from neural network models, from theory to applications to APIs. My goal is to take you straight to getting results with laser-focused tutorials. I designed the tutorials to focus on how to get things done. They give you the tools to both rapidly understand and apply each technique to your own predictive modeling problems. Each of the tutorials are designed to take you about one hour to read through and complete, excluding the extensions and further reading. You can choose to work through the lessons one per day, one per week, or at your own pace. I think momentum is critically important, and this book is intended to be read and used, not to sit idle. I would recommend picking a schedule and sticking to it. The tutorials are divided into three parts: ˆ Part 1: Better Learning. Discover the techniques to improve and accelerate the process used to learn or optimize the weights of a neural network model. ˆ Part 2: Better Generalization. Discover the techniques to reduce overfitting of the training dataset and improve the generalization of models on new data. ˆ Part 3: Better Predictions. Discover the techniques to improve the performance of final models when used to make predictions on new data. Each part targets a specific approach to improving model performance, and each tutorial targets a specific learning outcome for a technique. This acts as a filter to ensure you are only focused on the things you need to know to get to a specific result and do not get bogged down in the math or near-infinite number of configuration parameters. The tutorials were not designed to teach you everything there is to know about each of the techniques. They were designed to give you an understanding of how they work and how to use them on your projects the fastest way I know how: to learn by doing. xiii About Python Code Examples The code examples were carefully designed to demonstrate the purpose of a given lesson. For this reason, the examples are highly targeted. ˆ Models were demonstrated on small contrived datasets to give you the context and confidence to bring the techniques to your own projects. ˆ Model configurations used were discovered through trial and error and are skillful but not optimized. This leaves the door open for you to explore new and possibly better configurations. ˆ Code examples are complete and standalone. The code for each lesson will run as-is with no code from prior lessons or third parties required beyond the installation of the required packages. A complete working example is presented with each tutorial for you to inspect and copy and paste. All source code is also provided with the book and I would recommend running the provided files whenever possible to avoid any copy-paste issues. The provided code was developed in a text editor and intended to be run on the command line. No special IDE or notebooks are required. If you are using a more advanced development environment and are having trouble, try running the example from the command line instead. Neural network algorithms are stochastic. This means that they will make different predictions when the same model configuration is run on the same training data. On top of that, each experimental problem in this book is based around generating stochastic predictions. As a result, this means you will not get exactly the same sample output presented in this book. This is by design. I want you to get used to the stochastic nature of the neural network algorithms. If this bothers you, please note: ˆ You can re-run a given example a few times and your results should be close to the values reported. ˆ You can make the output consistent by fixing the random number seed. ˆ You can develop a robust estimate of the skill of a model by fitting and evaluating it multiple times and taking the average of the final skill score (highly recommended). All code examples were tested on a POSIX-compatible machine with Python 3 and Keras 2. All code examples will run on modest and modern computer hardware and were executed on a CPU. No GPUs are required to run the presented examples, although a GPU would make the code run faster. I am only human and there may be a bug in the sample code. If you discover a bug, please let me know so I can fix it and update the book and send out a free update. About Further Reading Each lesson includes a list of further reading resources. This may include: ˆ Research papers. xiv ˆ Books and book chapters. ˆ Webpages. ˆ API documentation. Wherever possible, I try to list and link to the relevant API documentation for key objects and functions used in each lesson so you can learn more about them. When it comes to research papers, I try to list papers that are first to use a specific technique or first in a specific problem domain. These are not required reading, but can give you more technical details, theory, and configuration details if you’re looking for it. Wherever possible, I have tried to link to the freely available version of the paper on arxiv.org. You can search for and download any of the papers listed on Google Scholar Search, scholar.google.com. Wherever possible, I have tried to link to books on Amazon. I don’t know everything, and if you discover a good resource related to a given lesson, please let me know so I can update the book. About Getting Help You might need help along the way. Don’t worry; you are not alone. ˆ Help with a Technique? If you need help with the technical aspects of a specific model or method, see the Further Reading sections at the end of each lesson. ˆ Help with Keras? If you need help with using the Keras library, see the list of resources in Appendix A. ˆ Help with your workstation? If you need help setting up your environment, I would recommend using Anaconda and following my tutorial in Appendix B. ˆ Help in general? You can shoot me an email. My details are in Appendix A. Next Are you ready? Let’s dive in! In the next tutorial, you will discover a framework that you can use to diagnose problems with your deep learning neural network and techniques that you can use to address each identified problem.

See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.