Lecture 8 - April 26, 2018 Lecture 8

155 Pages·2012·5.33 MB·English

FFeeii--FFeeii LLii && JJuussttiinn JJoohhnnssoonn && SSeerreennaa YYeeuunngg LLeeccttuurree 88 -- 11 AApprriill 2266,, 22001188
Last time
Regularization: Dropout Transfer
Learning
Optimization: SGD+Momentum,
FC-C
Nesterov, RMSProp, Adam
FC-4096 Reinitialize
FC-4096
this and train
MaxPool
Conv-512
Conv-512
MaxPool
Conv-512
Conv-512
MaxPool Freeze these
Conv-256
Regularization: Add noise, then
Conv-256
marginalize out
MaxPool
Conv-128
Train Conv-128
MaxPool
Conv-64
Test
Conv-64
Image
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 22 April 26, 2018
Today
- Deep learning hardware
- CPU, GPU, TPU
- Deep learning software
- PyTorch and TensorFlow
- Static vs Dynamic computation graphs
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 33 April 26, 2018
Deep Learning
Hardware
FFeeii--FFeeii LLii && JJuussttiinn JJoohhnnssoonn && SSeerreennaa YYeeuunngg LLeeccttuurree 88 -- 44 AApprriill 2266,, 22001188
My computer
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 55 April 26, 2018
Spot the CPU!
(central processing unit)
This image is licensed under CC-BY 2.0
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 66 April 26, 2018
Spot the GPUs!
(graphics processing unit)
This image is in the public domain
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 77 April 26, 2018
NVIDIA AMD
vs
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 88 April 26, 2018
NVIDIA AMD
vs
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 99 April 26, 2018
CPU vs GPU
Cores Clock Memory Price Speed CPU: Fewer cores,
Speed but each core is
much faster and
CPU 4 4.2 GHz System $339 ~540 GFLOPs FP32 much more
(Intel Core (8 threads with RAM capable; great at
hyperthreading)
i7-7700k) sequential tasks
GPU 3584 1.6 GHz 11 GB $699 ~11.4 TFLOPs FP32
GPU: More cores,
(NVIDIA GDDR5
but each core is
GTX 1080 Ti) X
much slower and
“dumber”; great for
parallel tasks
Fei-Fei Li & Justin Johnson & Serena Yeung Lecture 8 - 1100 April 26, 2018