582410 Processing of large document collections, Exercise 7

The solutions should be ready for inspection by Thursday 22.11.2001 (midnight).

1. Google is a well-known search engine for the Web. Study the article: Sergey Brin and Lawrence Page The Anatomy of a Large-Scale Hypertextual Web Search Engine , which describes the implementation of Google as it still was an academic project (~1998).

   Explain the high level architecture of Google as shown in Figure 1 (in the article):

   • Describe roughly the function of the different parts. If you don't understand the task of some part, don't mind. I recommend that you read the whole article once very fast to see what it contains, and then concentrate on the Section 4 (System Anatomy). Some helpful concrete details appear in the text later than the first mentions.

   • Try to identify which parts of the architecture (and how) are needed in the searching process. That is: What happens when a user gives a few keywords and starts a search?

2. Compare text categorization and text summarization as learning problems (~ classification problems), for instance:

   • What kind of entities do you need in the training set? What are the corresponding entities in these two applications?
   • What is to be learned? Are there some predefined classes?
   • After the learning phase, how these applications use the model etc. that was learned? That is: what happens when new documents are categorized or summarized?