Source coding (data compression)
Question: How do we compress the outcomes XN?
- With vanishingly small probability of loss
- How do we assign the elements of X such that the number of bits we need to specify an outcome XN drops from NHo(X) to N(H + ?)
Symbol coding: Given x = a3 a2 a7 … a5
- Generate codeword ?(x) = 01 1010 00
- Want Ho(?(x)) ~ H(X)
Well-known coding examples
- Zip, gzip, compress, etc.
- The performance of these algorithms is, in general, poor when compared to the Shannon limit