Shannon’s source-coding theorem

• Also called the main theorem. In words:

  • You can compress N independent, identically distributed (i.i.d.) random variables, each with entropy H, down to NH bits with negligible loss of information (as N??)
  • If you compress them into fewer than NH bits you will dramatically lose information

• The theorem:

  • Let X be an ensemble with H(X) = H bits. Given ? > 0 and 0 < ? < 1, there exists a positive integer No such that, for N > No,
  • The proof is in the handout

Previous slide

Next slide

Back to first slide

View graphic version