Information Theory

Information theory is the mathematical framework that pins down what "information" actually means as a measurable quantity, and how much of it you can squeeze through a noisy channel or compress out of a source before something breaks. You'll work through seven problem sets and two midterms proving things from Cover & Thomas — manipulating entropy and mutual information, deriving Shannon's source and channel coding theorems, computing capacities of Gaussian and multi-user channels, and reasoning about rate-distortion tradeoffs. As a graduate course it sits upstream of nearly everything in communications, coding, and statistical learning, giving you the asymptotic tools (AEP, large deviations, Fano) that later show up whenever you need a fundamental limit rather than a clever scheme.