Table Of Content

A Statistical Machine Learning Perspective of Deep Learning: Algorithm, Theory, Scalable Computing Maruan Al-Shedivat, Zhiting Hu, Hao Zhang, and Eric Xing Petuum Inc & Carnegie Mellon University Element of AI/Machine Learning Task Model • Graphical Models • Large-Margin • Deep Learning • Sparse Coding • Nonparametric • Regularized • Spectral/Matrix • Sparse Structured Bayesian Models Bayesian Methods Methods I/O Regression Algorithm • Stochastic Gradient • Coordinate • L-BFGS • Gibbs Sampling • Metropolis- Descent / Back Descent Hastings propagation Implementation … • Mahout • Mllib • CNTK • MxNet • Tensorﬂow (MapReduce) (BSP) (Async) System Hadoop Spark MPI RPC GraphLab … • Network • Network attached • Server machines • RAM • Cloud compute Platform • Virtual switches storage • Desktops/Laptops • Flash (e.g. Amazon EC2) machines and Hardware • Inﬁniband • Flash storage • NUMA machines • SSD • IoT networks • Mobile devices • Data centers • GPUs, CPUs, FPGA, TPU • ARM-powered devices © Petuum,Inc. 1 ML vs DL © Petuum,Inc. 2 Plan • Statistical And Algorithmic Foundation and Insight of Deep Learning • On Unified Framework of Deep Generative Models • Computational Mechanisms: Distributed Deep Learning Architectures © Petuum,Inc. 3 Part-I Basics Outline • Probabilistic Graphical Models: Basics • An overview of DL components • Historical remarks: early days of neural networks • Modern building blocks: units, layers, activations functions, loss functions, etc. • Reverse-mode automatic differentiation (aka backpropagation) • Similarities and differences between GMs and NNs • Graphical models vs. computational graphs • Sigmoid Belief Networks as graphical models • Deep Belief Networks and Boltzmann Machines • Combining DL methods and GMs • Using outputs of NNs as inputs to GMs • GMs with potential functions represented by NNs • NNs with structured outputs • Bayesian Learning of NNs • Bayesian learning of NN parameters • Deep kernel learning © Petuum,Inc. 5 Outline • Probabilistic Graphical Models: Basics • An overview of DL components • Historical remarks: early days of neural networks • Modern building blocks: units, layers, activations functions, loss functions, etc. • Reverse-mode automatic differentiation (aka backpropagation) • Similarities and differences between GMs and NNs • Graphical models vs. computational graphs • Sigmoid Belief Networks as graphical models • Deep Belief Networks and Boltzmann Machines • Combining DL methods and GMs • Using outputs of NNs as inputs to GMs • GMs with potential functions represented by NNs • NNs with structured outputs • Bayesian Learning of NNs • Bayesian learning of NN parameters • Deep kernel learning © Petuum,Inc. 6 Fundamental questions of probabilistic modeling • Representation: what is the joint probability distr. on multiple variables? !(# , # , # , … , # ) $ & ' ) • How many state configurations are there? • Do they all need to be represented? • Can we incorporate any domain-specific insights into the representation? • Learning: where do we get the probabilities from? • Maximum likelihood estimation? How much data do we need? • Are there any other established principles? • Inference: if not all variables are observable, how to compute the conditional distribution of latent variables given evidence? • Computing !(+|-) would require summing over 2/ configurations of the unobserved variables © Petuum,Inc. 7 What is a graphical model? • A possible world of cellular signal transduction © Petuum,Inc. 8 GM: structure simplifies representation • A possible world of cellular signal transduction © Petuum,Inc. 9