General Books

There is no text book for this course. The following books cover much of the material for this course.

• Ricardo Baeza-Yates and Berthier Ribeiro-Neto, Modern Information Retrieval, Addison Wesley, 1999.
• William B. Frakes and Ricardo Baeza-Yates, Information Retrieval Data Structures and Algorithms.  Prentice Hall, 1992.
• G. Salton and M. J. McGill, Introduction to Modern Information Retrieval, McGraw-Hill, 1983.
• Van Rijsbergen, C. J., Information Retrieval, Butterworths, 1979.  http://www.dcs.gla.ac.uk/Keith/

Discussion Classes

Readings for discussion classes are to be studied in preparation for the classes on Wednesday evenings.

Discussion Class 1, August 31, 2005

In preparation for this class, explore three information retrieval systems and compare them:

• Google -- a Web search engine (http://www.google.com/).
• The Library of Congress catalog -- a very large bibliographic catalog (http://catalog.loc.gov/).
• Medline -- an indexing and abstracting service for medicine and related fields (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi).

Consider the two information discovery tasks:

• What is the medical evidence that red wine is good or bad for your health? (Use Medline and Google.)
• What in history led to the current turmoil in Palestine and the neighboring countries? (Use the Library of Congress catalog.)

Study each search service in two ways.

(a) From a technical viewpoint. Does the service search full text or surrogates? Is fielded searching offered? What Boolean operators are supported? What regular expressions? How does it handle non-Roman character sets? What is the stop list? How are results ranked? Are they sorted, if so in what order?

(b) From a usability viewpoint. What style of user interface(s) is provided? What training or help services? If there are basic and advanced user interfaces, what does each offer?

Overall, what do you consider the strengths and weaknesses of each service? When would you use them?

Discussion Class 2, September 7, 2005

Read and be prepared to discuss:

G. Salton, A. Wong and C. S. Yang, A vector space model for automatic indexing. Communications of the ACM Volume 18 , Issue 11 (November 1975) pages: 613 - 620. http://doi.acm.org/10.1145/361219.361220

This paper describes many of the concepts behind the vector space model and the SMART system.

{Note that to access this paper from the ACM Digital Library, you need to use a computer with a Cornell IP address.}

Discussion Class 3, September 14, 2005

For this class, read and be prepared to discuss the following:

K. Sparck Jones, Exhaustivity and specificity. Journal of Documentation 28, 11-21, 1972. http://www.soi.city.ac.uk/~ser/idfpapers/ksj_orig.pdf.

Letter by Stephen Robertson and reply by Karen Sparck Jones, Journal of Documentation 28, 164-165, 1972. http://www.soi.city.ac.uk/~ser/idfpapers/letters.pdf.

The first paper introduced the term weighting scheme known as inverse document frequency (IDF). Some of the terminology used in this paper will be introduced in the lectures. The letter describes a slightly different way of expressing IDF, which has become the standard form.

{Stephen Robertson has mounted these papers on his Web site with permission from the publisher.}

Discussion Class 4, September 21, 2005

Read and be prepared to discuss the following paper:

Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, Richard Harshman, "Indexing by latent semantic analysis".   Journal of the American Society for Information Science, Volume 41, Issue 6, 1990.  http://www3.interscience.wiley.com/cgi-bin/issuetoc?ID=10049584 

{Note that to access this paper from Wiley InterScience, you need to use a computer with a Cornell IP address.}

Discussion Class 5, September 28, 2005

Read and be prepared to discuss the following paper, concentrating on Sections 1 to 4, and 5.3. You do not need to study the details of the methods described in Sections 5.1 and 5.2. Section 6 is for general interest only.

E. Voorhees, D. Harman, Overview of the Eighth Text REtrieval Conference (TREC-8). http://trec.nist.gov/pubs/trec8/papers/overview_8.ps.

This is one of a sequence of publications. The full sequence of TREC publications is at http://trec.nist.gov/pubs.html.

{Note that this paper is in PostScript format. You can view it using the GhostView viewer, which is available on the Web for downloading for all standard computer systems. The PDF version of the file on the TREC Web site is damaged. Here is a PDF version that was generated from the PostScript file.}

Discussion Class 6, October 5, 2005

The purpose of this class is to explore the Jakarta Lucene search engine. It is described on the web site:

http://jakarta.apache.org/lucene/

This is a large web site and you are not expected to read everything on the site. Concentrate on the following:

• What are the underlying search mechanisms supported by Lucene? What algorithms does it use? What data structures?

• How do you load free text into Lucene? How do you load fielded text? What format options are there? How does it handle various character sets, stoplists, stemming, etc.?

• How do you incorporate Lucene queries and results into your own user interface?

• If you wanted to modify Lucene to support a novel search algorithm, how would you go about it?

Discussion Class 7, October 19, 2005

This class will be held in Phillips Hall 213. 

Read and be prepared to discuss the following paper:

Gordon Paynter, Developing practical automatic metadata assignment and evaluation tools for Internet resources. Proceedings of the Fifth ACM/IEEE Joint Conference on Digital Libraries, Denver, Colorado, June, 2005, pp. 291-300. http://doi.acm.org/10.1145/1065385.1065454.

{Note that to access this paper from the ACM Digital Library, you need to use a computer with a Cornell IP address.}

Discussion Class 8, October 26, 2005

Read and be prepared to discuss:

Caroline R. Arms and William Y. Arms, "Mixed Content and Mixed Metadata: Information Discovery in a Messy World." In Metadata in Practice, edited by Diane Hillmann and Elaine Westbrooks, ALA Editions in 2004. http://www.cs.cornell.edu/wya/papers/ALA-2003.php.

Discussion Class 9, November 2, 2005

Read and be prepared to discuss:

Sergey Brin and Lawrence Page, The Anatomy of a Large-Scale Hypertextual Web Search Engine. Seventh International World Wide Web Conference. Brisbane, Australia, 1998. http://www7.scu.edu.au/programme/fullpapers/1921/com1921.htm

Note. A second copy of this paper is available at http://www-db.stanford.edu/~backrub/google.html.

{Sections 4.1 and 4.2 are out of date. Browse through them for historic interest only.}

Discussion Class 10, November 9, 2005

Read and be prepared to discuss:

Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, The Google File System. 19th ACM Symposium on Operating Systems Principles, October 2003.
http://www.cs.rochester.edu/sosp2003/papers/p125-ghemawat.pdf.

Discussion Class 11, November 16, 2005

[no class]

Discussion Class 12, November 30, 2005

Read and be prepared to discuss:

Marti Hearst, User Interfaces and Visualization. In: Modern Information Retrieval, edited by Ricardo Baeza-Yates and Berthier Ribeiro-Neto, Addison-Wesley Longman, 1999.
http://www.sims.berkeley.edu/~hearst/irbook/chapters/chap10.html

This is a long chapter. You are encouraged to read the full chapter, but the discussion class will concentrate on Sections 10.3, 10.4, and 10.9.