TTIC 31230: Fundamentals of Deep Learning

David McAllester, April 2017

Lecture slides for 21 topics in deep learning with pointers into Deep Learning by Goodfellow, Bengio and Courville, as well as pointers to other relevant material.

  1. Multi-Layer Perceptrons (MLPs) and Stochastic Gradient Descent (SGD)
  2. Feed-Forward Computation Graphs, Backpropagation, and the Educational Framework (EDF)
  3. Minibatching in EDF
  4. Variants of SGD
  5. An SGD Progress Theorem
  6. Architecture and Universality
  7. Convolutional Neural Networks (CNNs)
  8. Some Linear Systems and Wavelet Theory
  9. Second Order Optimization Methods
  10. Vanishing Gradients, Xavier Initialization, Batch Normalization and Highway Architectures (Resnets, LSTMs and GRUs)
  11. Regularization
  12. Some Generalization Theory
  13. Interpreting Deep Networks
  14. Sequence to Sequence Models and Attention
  15. Deep Reinforcement Learning
  16. AlphaGo
  17. Deep Graphical Models
  18. Unsupervised and Predictive Learning
  19. Information Theory and Distribution Modeling
  20. Variational Autoencoders
  21. A Rate-Distortion Case Study
  22. Generative Adversarial Networks (GANs)