Information and Coding Theory - Autumn 2014


TTIC 31200/CMSC 37220

T Th 10:30-11:50, TTIC Room 530

Instructors: Madhur Tulsiani and Shi Li



 

This course is meant to serve as an introduction to some basic concepts in information theory and error-correcting codes, and some of their applications in computer science and statistics. We plan to cover the following topics:

  • Introduction to entropy and source coding. Some applications of entropy to counting problems.
  • Mutual information and KL-divergence. Method of types and hypothesis testing. I-projections and applications.
  • Introduction to error-correcting codes. Unique and list decoding of Reed-Solomon and Reed-Muller codes.
  • Applications of information theory to lower bounds in computational complexity and communication complexity.

The course will have 3-4 homeworks. Each student will also be expected to scribe 1-2 lectures.


There is no textbook for this course. A useful reference is ``Elements of Information Theory'' by T. M. Cover and J. A. Thomas. Also take a look at the resources section below.



Homeworks



Lecture Plan and Notes

  • 9/30: Entropy of a random variable. Prefix-free codes and Kraft's inequality.
    [Notes]
  • 10/2: Conditional and joint entropy. Subadditivity of entropy and combinatorial applications. Fundamental source coding theorem.
    [Notes]
  • 10/7: Some applications of entropy to combinatorics.
    [Notes]
  • 10/9: Mutual information and KL-divergence.
    [Notes]
  • 10/14: Proof of Pinsker's inequality. Lower bound for distinguishing coins.
    [Notes]
  • 10/16: Lower bound for multi-armed bandits.
    [Notes]
  • 10/21: Method of types and large deviations.
    [Notes]
  • 10/23: Sanov's theorem and applications. Introduction to hypothesis testing.
    [Notes]
  • 10/28: Analysis of hypothesis testing.
  • 10/30: I-projections and their properties. Parameter estimation and the Cramér-Rao bound.
    [Notes]
  • 11/4: Algebra review. Introduction to error-correcting codes.
    [Notes]
  • 11/6: The Hamming code and Hamming bound. Reed-Solomon codes and the Berlekamp-Welch decoding algorithm.
    [Notes]
  • 11/11: Analysis of the unique decoding algorithm. List decoding of Reed-Solomon codes.
    [Notes]
  • 11/13: Reed-Muller codes and local decoding.
    [Notes]
  • 11/18: Regev's information-theoretic proof of lower bound for dimension reduction in L1.
  • 11/20: Lower bound for dimension reduction in L1 (contd). Introduction to the Lovász local lemma.
  • 11/25: Moser's information-theoretic proof of the constructive Lovász local lemma. Communication complexity and the information content of protocols.
  • 12/2: Guest lecture by Brendan Juba on Universal Semantic Communication.
    This lecture will be in TTIC Room 526


Resources