Three Concepts: Information

Assignments

Return assignments by the following lecture by e-mail (both Teemu and Jukka) or in writing (during the lecture).

Assignment for Lecture 2: ''Mathematical Preliminaries''

Read Chapter 2 of Cover & Thomas (to be found in the course folder), and do at least four out of the following five exercises:

1. Show that ln is strictly concave.
   Hint: Derivatives.
   
2. Show that Gibbs' inequality holds as an equality if and only if p(x) = q(x) for all x.
   Hint: When does ln(x)=x-1 hold? Alternative hint: Next exercise.
   
3. Prove Gibbs' inequality using Jensen's inequality.
   Hint: Gibbs' inequality is also known as the "information inequality". Note that -ln is (strictly) convex.
   
4. Exercise 2.2 in Cover & Thomas (1st edition).
   Hint: Look at Exercise 2.5 ("Entropy of functions of random variable") in Cover & Thomas (1st edition). If Y=2^X or Y=cos X, in which case(s) is Y a function of X and/or X a function of Y?
   
5. Exercise 2.16 (item (f) optional) in Cover & Thomas (1st edition).

Three Concepts: Information	2007