CS 501 Software Engineering Spring 2003 Project Concepts Web Archiving

Client

Bill Kehoe, Cornell University Library (wrk1@cornell.edu)
Nancy McGovern, Cornell University Library (nm84@cornell.edu)

Project outline

Cornell University Library needs several tools for its work in preserving digital information.  Two projects have been suggested:

Data file version monitor

Wanted: a file-system agent that will 

• periodically roam a file system, or configurable sections of a file system. 
• determine the version number of data files in some pre-determined formats. 
• check a version registry to determine whether or not the files are in the most recent version. 
• mark those files that are not current. 
• create parseable lists of marked files for possible batch manipulation. 
• send alerts to a preservation administrator.

The version registry will also have to be created and a mechanism for update and maintenance of the registry will have to be provided.  The registry, the controller, the file systems to be searched, and any data storage/database for marked-file lists should be assumed to be on distributed machines.  The system should be portable between Unix and MS environments. We would prefer some combination of Java, Perl, MySQL, and XML because they are commonly used in the library world and would simplify maintenance.  

Website capture 

Wanted: a turnkey web-site capturing tool that will 

• allow a library to preserve remote sites by capturing all the publicly available html, gif, jpg, pdf and other files and recreating the directory/link hierarchy. 
• store the directory hierarchy with original names, so the integrity of links is preserved. 
• capture scripts and style sheets, if possible. 
• create metadata to describe the structure of the site and identify dead-end links that point to areas of the deep web that could not be captured--pages generated dynamically from database queries or password restricted areas, for example. 

The system should be portable between Unix and MS environments. We would prefer some combination of Java, Perl, MySQL, and XML because they are commonly used in the library world, are free, and would simplify maintenance.

Last changed: January 29, 2003