A Statistical Machine Learning Perspective of Deep Learning

286 Pages·2017·21.51 MB·English

A Statistical Machine Learning
Perspective of Deep Learning:
Algorithm, Theory, Scalable Computing
Maruan Al-Shedivat, Zhiting Hu, Hao Zhang, and Eric Xing
Petuum Inc
&
Carnegie Mellon University
Element of AI/Machine Learning
Task
Model • Graphical Models • Large-Margin • Deep Learning • Sparse Coding
• Nonparametric • Regularized • Spectral/Matrix • Sparse Structured
Bayesian Models Bayesian Methods Methods I/O Regression
Algorithm • Stochastic Gradient • Coordinate • L-BFGS • Gibbs Sampling • Metropolis-
Descent / Back Descent Hastings
propagation
Implementation …
• Mahout • Mllib • CNTK • MxNet • Tensorflow
(MapReduce) (BSP) (Async)
System
Hadoop Spark MPI RPC GraphLab …
• Network • Network attached • Server machines • RAM • Cloud compute
Platform • Virtual
switches storage • Desktops/Laptops • Flash (e.g. Amazon EC2)
machines
and Hardware • Infiniband • Flash storage • NUMA machines • SSD • IoT networks
• Mobile devices • Data centers
• GPUs, CPUs, FPGA, TPU
• ARM-powered devices
© Petuum,Inc. 1
ML vs DL
© Petuum,Inc. 2
Plan
• Statistical And Algorithmic Foundation and Insight of Deep
Learning
• On Unified Framework of Deep Generative Models
• Computational Mechanisms: Distributed Deep Learning
Architectures
© Petuum,Inc. 3
Part-I
Basics
Outline
• Probabilistic Graphical Models: Basics
• An overview of DL components
• Historical remarks: early days of neural networks
• Modern building blocks: units, layers, activations functions, loss functions, etc.
• Reverse-mode automatic differentiation (aka backpropagation)
• Similarities and differences between GMs and NNs
• Graphical models vs. computational graphs
• Sigmoid Belief Networks as graphical models
• Deep Belief Networks and Boltzmann Machines
• Combining DL methods and GMs
• Using outputs of NNs as inputs to GMs
• GMs with potential functions represented by NNs
• NNs with structured outputs
• Bayesian Learning of NNs
• Bayesian learning of NN parameters
• Deep kernel learning
© Petuum,Inc. 5
Outline
• Probabilistic Graphical Models: Basics
• An overview of DL components
• Historical remarks: early days of neural networks
• Modern building blocks: units, layers, activations functions, loss functions, etc.
• Reverse-mode automatic differentiation (aka backpropagation)
• Similarities and differences between GMs and NNs
• Graphical models vs. computational graphs
• Sigmoid Belief Networks as graphical models
• Deep Belief Networks and Boltzmann Machines
• Combining DL methods and GMs
• Using outputs of NNs as inputs to GMs
• GMs with potential functions represented by NNs
• NNs with structured outputs
• Bayesian Learning of NNs
• Bayesian learning of NN parameters
• Deep kernel learning
© Petuum,Inc. 6
Fundamental questions of probabilistic modeling
• Representation: what is the joint probability distr. on multiple variables?
!(# , # , # , … , # )
$ & ' )
• How many state configurations are there?
• Do they all need to be represented?
• Can we incorporate any domain-specific insights into the representation?
• Learning: where do we get the probabilities from?
• Maximum likelihood estimation? How much data do we need?
• Are there any other established principles?
• Inference: if not all variables are observable, how to compute the conditional
distribution of latent variables given evidence?
• Computing !(+|-) would require summing over 2/ configurations of the unobserved variables
© Petuum,Inc. 7
What is a graphical model?
• A possible world of cellular signal transduction
© Petuum,Inc. 8
GM: structure simplifies representation
• A possible world of cellular signal transduction
© Petuum,Inc. 9