Supervised Machine Learning

Algoritmit ja koneoppiminen
Syventävät opinnot
We study in particular classification from the point of view of online learning and statistical learning theory. Emphasis is on provable performance guarantees. The algorithms we study include the perceptron and its variants and the support vector machine.


01.03.2011 16.00 A111
Vuosi Lukukausi Päivämäärä Periodi Kieli Vastuuhenkilö
2011 kevät 18.01-24.02. 3-3 Englanti Jyrki Kivinen


Aika Huone Luennoija Päivämäärä
Ti 10-12 C222 Jyrki Kivinen 18.01.2011-24.02.2011
To 10-12 C222 Jyrki Kivinen 18.01.2011-24.02.2011


Group: 1
Aika Huone Ohjaaja Päivämäärä Huomioitavaa
To 14-16 C222 Panu Luosto 24.01.2011—25.02.2011



The course has been graded.  See below for details.


The course Introduction to Machine Learning is not strictly required. However it includes a lot of very useful motivation, context and other background which we will here cover only very briefly.

The students are expected to have basic knowledge of linear algebra, probability theory and calculus. Some multivariate calculus will be needed, but we will briefly cover the necessary tools for those not familiar.

Although there are not many specific prerequisites from mathematics, the approach taken on the course is mainly mathematical, and familiarity with mathematical manipulations will help a lot.

Part of the homework will require some programming. Basic programming skills are essential, and familiarity with tools such as Matlab, Octave or R will be very helpful.



The course covers a selection of topics mainly related to (binary) classification. This is a large research area, and the choice of topics is somewhat based on the personal preferences of the lecturer (whose research area is computational learning theory, in particular online learning).

Emphasis will be on provable performance guarantees we can provide for learning algorithms.

Table of contents (preliminary):

  1. Introduction
    • basic concepts
    • mathematical frameworks for supervised learning: online and statistical
  2. Online learning
    • combining expert advice
    • linear classification and the perceptron algorithm
    • relative loss bounds (also known as regret bounds)
  3. Statistical learning
    • basic statistical model, connection to online learning
    • complexity measures: VC-dimension and Rademacher complexity
    • Support Vector Machine

Kurssin suorittaminen

The exam has now been graded.  Sorry for the delay,

More detailed comments about the exam will appear here soon.  For now, I'll just notice that the exam turned out to be more difficult than intended, which has been taken into account in grading, so you may have received more points than you expected.

The grade consists of homework (20%) and exam (80%).

Read also the course policies.

Please give also some feedback for the course.  You may wait until after the exam if you wish, but please do not forget!


Kirjallisuus ja materiaali

The material consists of lecture notes, exercise problems and their solutions, and possibly some additional original articles.

Lecture notes and homework material will appear here. The whole set of lecture notes (pages 1–201) is now available.

  • A new page was added after page 103 to fix  a mistake.  The page numbers from 104 are now off by one compared to earlier versions.
  • Some additional steps were added to clarify the proof on page 175.

Homework solutions need to be turned out to the course assistant Panu Luosto in advance, on paper.  The deadline is Tuesday before the exercise session at 15:00.  There will be an envelope at the door to Panu's office B233 for turning in your homework.  If you have problems with the procedure, contact directly Panu or the lecturer Jyrki Kivinen.


The course does not follow any single textbook.  Some recommended textbooks and articles on the course topic are listed under the tab Additional references.