CMSC 35246 Deep Learning - University of Chicago

Schedule

Week	Date	Topics	Lecture notes/Slides	Additional Resources, Suggested Readings, etc.
1	March 27	Introduction to Deep Learning Course logistics	slides slides(flat)	Suggested Reading: Chapter 1 of Goodfellow, Bengio, Courville Suggested Tutorial: Python/NumPy Tutorial Classic Paper: McCulloch and Pitts "A Logical Calculus of the ideas immanent in nervous activity" Classic Paper: Rosenblatt "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain" Connectionist History: A Sociological Study of the Official History of the Perceptrons Controversy Connectionist History: A Brief History of Connectionism
1	March 29	Machine Learning Review	slides slides(flat)	Suggested Reading: Chapter 5 of Goodfellow, Bengio, Courville Suggested Tutorial: Basic Estimation TensorFlow Tutorials PyTorch Tutorials
2	April 3	Feedforward Deep Networks Backpropagation Start Assignment 1 Out (on Canvas)	slides slides(flat)	Suggested Reading: A Visual Proof that Neural Nets can compute any function by Michael Nielsen Suggested Reading: Representational power of feedforward networks (Cybenko, Barron, Folklore, Kolmogorov) by Matus Telgarsky Classic Paper: Rumelhart, Hinton and Williams "Learning Internal Representations by Error Propagation" Classic Paper: Hinton and Nowlan "How Learning can guide Evolution"
2	April 5	Backpropagation Quiz 1	slides slides(flat)	Suggested Reading: Chapter 6 of Goodfellow, Bengio and Courville Suggested Reading: Chapter 2 of Nielsen Suggested Reading: Calculus on Computational Graphs, Chris Olah Suggested Reading: Andrej Karpathy's notes (linked below in "Additional Linkage")
3	April 10	Regularization Methods	slides slides(flat)	Suggested Reading: Chapter 7 of Goodfellow, Bengio and Courville Suggested Reading: Sparsity and the LASSO, Tibshirani and Wasserman Suggested Reading: Regularization and Complexity Control in Neural Networks, Chris Bishop Suggested Reading: Bagging Regularizes, Poggio et al. Suggested Reading: Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Srivastava et al. Suggested Reading: Adversarial Perturbations of Deep Neural Networks, Warde-Farley and Goodfellow Further Reading: Training with Noise is Equivalent to Tikhonov Regularization, Chris Bishop Further Reading: Rank, Trace-Norm and Max-Norm, Srebro and Shraibman Reference: Statistical Learning and Sparsity, Hastie, Tibshirani and Wainwright
3	April 12	Optimization Methods -- SGD, Momentum method Adaptive first order methods Batch Normalization Initialization Strategies Polyak Averaging Assignment 1 Due (13 April; 23:59)	slides slides(flat)	Suggested Reading: Chapter 8 of Goodfellow, Bengio and Courville Suggested Reading: Why Momentum really works, Gabriel Goh Suggested Reading: Large Scale Machine Learning with Stochastic Gradient Descent, Leon Bottou Suggested Reading: The Zen of Gradient Descent, Moritz Hardt Suggested Reading: On the momentum term in gradient descent learning algorithms, Ning Qian Suggested Reading: On the importance of initialization and momentum in deep learning, Sutskever et al Classic Paper: Original Momentum paper, B. T. Polyak
4	April 17	Convolutional Neural Networks Assignment 2 Out (April 16) Quiz 2	slides slides(flat)	Suggested Reading: Chapter 9 of Goodfellow, Bengio and Courville Suggested Reading: Conv Nets: A Modular Perspective, Chris Olah Suggested Reading: Understanding Convolutions, Chris Olah Classic Paper: Backpropagation Applied to Handwritten Zip Code Recognition, LeCun et al.
4	April 19	Visualizing Convolutional Networks Variants (Locally Connected Networks, Tiled CNNs, Dilated Convolutions) Motivations: Neuroscientific Motivations: Efficiency, Equivariance, Invariance, parameter tying Project Proposals Due at midnight	slides slides(flat)	Reference: Models of Visual Cortex, Poggio and Serre Reference: Stochastic Neighbor Embedding, G. E. Hinton and S. Roweis Reference: Visualizing Data using t-SNE, L. van der Maaten and G. E. Hinton Reference: How to use t-SNE effectively, Distill Reference: Visualizing and Understanding Convolutional Networks, M. D. Zeiler and R. Fergus Reference: Striving for Simplicity: The All Convolutional Net, J. T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller Reference: Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps, K. Simonyan, A. Vedaldi, A. Zisserman Reference: Understanding Neural Networks Through Deep Visualization, J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, H. Lipson Reference: Inceptionism: Going Deeper into Neural Networks , Google Research Blog Reference: Multi-Scale Context Aggregation by Dilated Convolutions, F. Yu and V. Koltun Classic Paper: Neocognitron: A Hierarchial Neural Network capabale of visual pattern recognition, K. Fukushima
5	April 24	Ultra Deep Architectures Residual Networks ResNet in ResNet Highway Networks Fractal Networks Some Analysis of Residual Learning DenseNets Similarity Learning and Siamese Networks Assignment 2 Due (April 27; 23:59)
5	April 26	CNNs on Graphs Introduction CNNs on Non-Euclidean Domains Locally Connected Networks Spectral Networks Graph Embedding: Basic Message Passing Neural Networks Quiz 3	slides slides(flat)	Suggested Reading: Spectral Networks and Locally Connected Networks on Graphs, J. Bruna, W. Zaremba, A. Szlam, Y. LeCun Suggested Reading: Deep Convolutional Networks on Graph-Structured Data, M. Henaff, J. Bruna, Y. LeCun Further Reading: Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering, M. Defferrard, X. Bresson, P. Vandergheynst Further Reading: Geometric deep learning: going beyond Euclidean data, M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, P. Vandergheynst Suggested Reading: Neural Message Passing for Quantum Chemistry, J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, G. E. Dahl Suggested Reading: Convolutional Networks on Graphs for Learning Molecular Fingerprints, D. Duvenaud, D. Maclaurin, J. Aguilera-Iparraguirre, R. G�mez-Bombarelli, T. Hirzel, A. Aspuru-Guzik, R. P. Adams Suggested Reading: Gated Graph Sequence Neural Networks, Y. Li, D. Tarlow, M. Brockschmidt, R. Zemel
6	May 01	Sequence Learning with NNs Recurrent Neural Networks Design Patterns for RNNs Intro to long term dependencies and gated architectures	slides slides(flat)	Suggested Reading: Chapter 10 of Goodfellow, Bengio and Courville Suggested Reading: The Unreasonable Effectiveness of Recurrent Neural Networks, Andrej Karpathy
6	May 03	Recap of RNNs Challenges with Long Term Dependencies Reservoir Computing (basic idea) Long Short Term Memory and GRU Attention Models: Introduction using applications in Machine Translation and Caption Generation	slides slides(flat)	Suggested Reading: Chapter 10 of Goodfellow, Bengio and Courville Suggested Reading: Understanding LSTM Networks, Chris Olah Suggested Reading: Attention and Augmented RNNs, Chris Olah and Shan Charter Suggested Reading: Visualizing and Understanding RNNs, Karpathy et al. Suggested Reading: LSTM: A Search Space Odyssey, Greff et al. Further Reading: ESN Approaches to training RNNs, Herbert Jaeger Classic Paper: Long Short-Term Memory, Hochreiter and Schmidhuber
7	May 08	Mid-term
7	May 10	Neural Networks with External Memory Memory Networks Neural Turing Machines Differentiable Neural Computers	slides slides(flat) (slides on canvas for now)	Suggested Reading: Attention and Augmented RNNs, Chris Olah and Shan Charter Suggested Reading: Neural Turing Machines, Graves et al. Suggested Reading: Graves et al.'s Nature paper that adds Temporal attention to NTM as well as memory allocation Suggested Reading: DeepMind's Blogpost on Differentiable Neural Computers
8	May 15	Mid-Term Solutions Review Introduction to Deep Unsupervised Learning Linear Projections and Linear Autoencoders Sparse Coding Autoencoders Assignment 3 Due (16 May 23:59)	slides slides(flat) (slides on canvas for now)
8	May 17	Recap of PCA, ICA, Sparse Coding Ditributed versus Localist Representations Autoencoders wrap-up (sans training) Quiz 4	slides slides(flat) (slides on canvas for now)
9	May 22	Linear Factor Models and Probabilistic PCA as motivation for Deep Factor Models Energy Based Methods Sigmoid Belief Networks and Deep Belief Networks Boltzmann Machines and Restricted Boltzmann Machines set-up	slides slides(flat)
9	May 24	Linear Factor Models, SBNs, DBNs, RBMs recap Approximate Learning in Restricted Boltzmann Machines Contrastive Divergence Boltzmann Machines Implicit density models: Intro	slides slides(flat) (slides on canvas for now)
10	May 29	Memorial day holiday: No Class
10	May 31	Implicit density models Generative adversarial networks Wasserstein GANs Bonus topic: AlphaGo Quiz 5	slides slides(flat) (slides on canvas for now)
11-12	June 6-10	Project presentations and discussions Final project report and code due 10 June 23:59

	Maximum
Assignments	4. Total contribution to grade 40%
In-Class Midterm	Total contribution to grade 20-25%
Project	Total contribution to grade: 25-30%
Quizzes + Participation	5. Total contribution to grade: 10%

Prerequisites

Syllabus

Course Work

Grading

Books and Resources

Textbooks

Tools

Logistics, Submissions etc.

Schedule

Additional Linkage

Related Courses

Notes, Surveys and Pedagogical Material