Three Concepts: Information


Return assignments by the following lecture by e-mail (both Teemu and Jukka) or in writing (during the lecture).

Assignment for Lecture 2: ''Mathematical Preliminaries''

Read Chapter 2 of Cover & Thomas (to be found in the course folder), and do at least four out of the following five exercises:

  1. Show that ln is strictly concave.
    Hint: Derivatives.
  2. Show that Gibbs' inequality holds as an equality if and only if p(x) = q(x) for all x.
    Hint: When does ln(x)=x-1 hold? Alternative hint: Next exercise.
  3. Prove Gibbs' inequality using Jensen's inequality.
    Hint: Gibbs' inequality is also known as the "information inequality". Note that -ln is (strictly) convex.
  4. Exercise 2.2 in Cover & Thomas (1st edition).
    Hint: Look at Exercise 2.5 ("Entropy of functions of random variable") in Cover & Thomas (1st edition). If Y=2X or Y=cos X, in which case(s) is Y a function of X and/or X a function of Y?
  5. Exercise 2.16 (item (f) optional) in Cover & Thomas (1st edition).


 Three Concepts: Information