How much can we compress? - Shannon’s Source Coding Theorem

On Probability and Entropy

Probability

Probability continued

Probability continued

Bayes’s theorem

Entropy

Binary entropy

Information content

Shannon information

Example: letter distribution

Entropy continued

Entropy continued

Entropy relationships

Kullback-Leibler divergence

Weighting problem

Idea

Formalize the idea

Block coding

Shannon’s source coding theorem

Typical set

AEP and source coding