3 Concepts: Information

Course Material

Lectures

» Set 1: What is Information Theory?
    [ HTML ] [ big PDF (1.1MB) ] [ small PDF (243kB)]
» Set 2: How much can we compress? - Shannon's Source Coding Theorem
    [ HTML ] [ big PDF (3.1MB) ] [ small PDF (893kB) ]
» Set 3: Revenge of a Student - Symbol Codes
    [ HTML ] [ big PDF (871kB) ] [ small PDF (290kB) ]
» Set 4: On Minimum Description Length Modeling
    [ HTML ]
» Set 5: `Year 2020' - Topics in Information Theory for Further Studies
    [ HTML ] [ big PDF (3.6MB) ] [ small PDF (577kB) ]

(Unfortunately the quality of the HTML version is poor, thanks to Microsoft.)

Literature, further reading

• MacKay D.,
  Information Theory, Inference and Learning Algorithms,
  Draft 2.2.0, December 2000.

Some good general information sources for various parts of the course

• Bell T. C., Cleary J. G., Witten I. H.,
  Text Compression.
  Prentice Hall, Englewood Cliffs, New Jersey, 1990.
  
  
• Berlekamp E. R.,
  Algebraic Coding Theory.
  McGraw-Hill, New York, 1968.
  
  
• Blahut R. E.,
  Principles and Practice of Information Theory.
  Addison-Wesley, New York, 1987.
  
  
• Cover T. M., Thomas J. A.,
  Elements of Information Theory.
  Wiley, New York, 1991.
  
  
• Deco G., Obradovic D.,
  An Information-Theoretic Approach to Neural Computing.
  Springer-Verlag, New York, 1996.
  
  
• Frey B. J.,
  Graphical Models for Machine Learning and Digital Communication.
  MIT Press, Cambridge, Massachusetts.
  
  
• Grünwald P.,
  The Minimum Description Length Principle and Reasoning under Uncertainty
  ILLC Dissertation Series DS 1998-03.
  
  
• Lin S., Costello, Jr., D. J.,
  Error control coding: fundamentals and applications.
  Prentice-Hall, Englewood Cliffs, N.J., 1983.
  
  
• MacWilliams R. J., Sloane N. J. A.,
  The theory of error-correcting codes.
  North-Holland, Amsterdam, 1977.
  
  
• McEliece R. J.,
  The theory of information and coding: a mathematical framework for communication.
  Reading, Mass.: Addison-Wesley, 1977. Reprinted 1984 by C.U.P. in Encyclopedia of Mathematics.
  
  
• Peterson W. W., Weldon, Jr., E. J.,
  Error-Correcting Codes.
  MIT Press, 2nd Edition, Cambridge, Massachusetts, 1972.
  
  
• Rao T. R. N., Fujiwara E.,
  Error-control Coding for Computer Systems.
  Prentice-Hall, 1989.
  
  
• Williams R. N.,
  Adaptive Data Compression.
  Kluwer Academic Publishers, USA, 1991.
  
  
• Witten I. H., Neal R. M., Cleary J. G.,
  Arithmetic coding for data compression.
  Communications of the ACM, 1987, Vol. 30, No. 6, p. 520-540.
  
  

Links

» DataCompression.info
» Compression Pointers
» MIT Course on Information and Entropy
» Stanford Information Theory Class
» Entropy on the World Wide Web
» G.J. Chaitin Home Page
» David Dowe's Minimum Encoding Length Inference Page
» Bayesians Worldwide
» Quantum Information
» Information about gzip

3 Concepts: Information	2002