The fundamental theorems of information theory

Abstract

The mathematical machinery necessary for the enunciation and proof of Shannon's Coding Theorems for discrete noisy channels is developed. In Chapter I we give a systematic account of various Ergodic Theorems, including those given by Birkhoff and von Neumann for function spaces. In Chapter II we apply the Martingale Theorem of Doob to obtain a rather general version of McMillan's Convergence Theorem for the conditional information density, together with a characterization of the Kolmogorov-Sinai invariant. In Chapter III we re-cast these results into a form appropriate for use in Communication Theory, and show how they may be combined with the traditional device of Feinstein's Lemma so as to formulate a Proof of the Theorem of Shannon.