Final Projects

Tagging responses to clients for disaster recovery
This paper presents the results of an initial literature survey into methods of implementing tagged responses to clients that primary servers could use to recover their state. The paper details the practical problems associated with the use of asynchronous data replication methods for recovery. We describe existing academic results that are relevant to this topic, and how they will be adapted to implement our solution to the problem of recovery. Finally, we present an initial design, along with an implementation and evaluation plan for the proposed solution.

S3FS in the Wide Area
S3FS[1] is an interface implementation using FUSE and Amazon Storage Services in order to provide a reliable and secure storage location with a easy to use interface for Unix users. Although S3FS[1] presents a simple and functional implementation, there are several issues, inherent to WANs(Wide Area Network), not considered during its implementation.

This project explains some of the limitations of the current S3FS[1] implementation and addresses the performance issues with a simple solution based on existing designs and algorithms. Due to the high latency of the WANs, this project will implement an asynchronous Caching mechanism along with parallel processing of the Reads/Writes requests to S3, therefore mitigating the performance loss caused by the latency of the wide area networks.

VaporDisk - a reliable, portable windows file system
This paper introduces VaporDisk, a fast, reliable and portable Windows mixed kernel and user-space file system. This system is designed for use by the "casual computer user", i.e. someone with little or no technical knowledge and is therefore is designed to be entirely transparent to the end user. It is also designed to work with the simplest back end storage systems, such as Amazon's S3 while being extensible to later take advantage of more complex data models. VaporDisk simultaneously provides almost the same performance as a local disk, minimally impacting the user experience. This is accomplished with a write-through cache and smart revision collapsing, described in detail in the paper. Additionally, VaporDisk provides a friendly and secure web interface to allow remote access of files from an HTTP server, making data portable to any device with a web browser.

HPUFS - High Perfornace User Space Network File System
HPUFS is a system designed for high performance in Network File System by incorporating File caching and File locking. The project also involves the implementation of Automatic file synchronization tool which is used for backing up the data in the local systems to the remote Storage Server. . HPUFS is also designed to supports Low Bandwidth Networks. The paper describes the initial design and expected use. A number of programs have implemented portable network file systems. However, they have been plagued by low performance. HPUFS is implemented using asynchronous APIs (libasync) which can improve the performance.

Diamond: One File System, Multiple Devices
Diamond is a read/write higly available and partition-tolerant file system. It allows a user to edit a the file system at the same time, from many devices. Diamond consists of a set of logs, one for each device, which is stored on a trusted online backend, accessible through a simple put/get interface. This allows end-user to easily switch to another storage provider. Each device appends to its own log to modify the file system and reads from all the logs to find data. Diamond presents data through a conventionnal file system interface. End-users can mount either a read-write version of the device current file system version or read-only mount a previous version viewed by any device. This allows the end-user to access all the file versions may they be from the past or another device and to modify its current version.

SILT: S3 Integrity, Locking, and Transactions
Despite their limited put/get interface, simple hosted storage services are becoming very popular. Many companies and individuals are using them to store and backup data. Recent work has demonstrated that there are some benefits to building file systems on top of such abstractions. However, the available API lacks some features that could prove beneficiary in building distributed storage systems on top of such services. In this project I investigated extending the interface of Amazon's S3 storage service to allow for more powerful capabilities such as locking, multi-key transactions, and ensuring data integrity.

DeltaFS
DeltaFS combines a read-only network lesystem with a mechanism for storing local changes. It is intended for use on limited capacity devices with good net connectivity, such as netbooks, mobile devices, and virtual machines. The local footprint of the lesystem is proportionate to the volume of changes made; so long as most les are unmodied, the local user can be given the illusion of having access to a very large drive.

S3VN: Tools for Storing a Subversion Repository in S3
Cloud computing infrastructure services are quickly gaining ground as a preferred way of hosting online services. Popular storage services, however, are built on a model of eventual consistency that makes it difficult to build applications which require stricter guarantees. We show how to overcome these limitations by using an open source distributed lock service, and present an application that provides source code revision control using Subversion in Amazon's S3 storage environment.

Allied Agreement with Threshold Cryptography
Ally is a framework for building distributed services in a federated architecture. In a federation, nodes may occupy multiple, independent administrative domains, with complex trust relationships, connected by an asynchronous, wide-area network. We present a library that implements Byzantine fault-tolerant agreement, a critical building-block service in robust distributed systems, targeted at the federation model; threshold cryptography improves the basic agreement protocol with proactive, verifiable security and improved resilience to continuous attack.

Torrent Crawler: a tool for collecting information from BitTorrent networks
BitTorrent is a free peer-to-peer (P2P) content-sharing application with a complex and dynamic overlay strudture due to loose coupling, high churn rate, and varying responsiveness of nodes. The complexity and the dynamic nature of the overlay structure can mask the problems in the network, making errors difficult to detect and diagnosis in a timely manner. Furthermore, the heavy reliance of clients on the node local views compounds the problems such as partioning in the network or load imbalance due to biased peer selection.

In an effort to provide the network with partial global information to resolve the network problems, this project looks into introducing a tool that efficiently collects global information from BitTorrent network. The tool, called Torrent Crawler (TC) uses a number of techniques to efficiently find all participating peers of the swarm, collecting global information from the network. TC also collects the information unobtrusively to the network traffic.

Project information

Introduction

• Project proposal. The proposal is a one page description of what your project will be. It should state the problem you are solving, why it is an interesting or useful problem, what software/system you will build and what the expected results will be.
  It is due on Tuesday, March 10th.
  We will give feedback by Thursday, March 12th.
• Midterm Survey Paper.This report should include a draft of your report's abstract, introduction, related work, and design section. These sections should be in good shape and close to what they would look like in the final report. Be sure that the draft's introduction clearly states what your project's goals are, why those goals are worthwhile, and how you're going to achieve those goals. In addition to these mature sections, the report should also have an implementation and evaluation plan. Describe how you plan to implement the system (esp. the details of how it situates in the OS environment) and what experiments you will run on your final system.
  It is due by Tuesday, March 31st.
  
• Peer Review. You and everyone else will write a peer review of two or three other drafts (from other students). Your reviews will be given to you as feedback and constructive criticism on your draft paper.
  We will return our peer reviews to author by Thursday, April 2nd.
• Demo day. You will give a presentation, followed by a demonstration of your system in action to the entire class. We'll supply a laptop projector, so you should run your demo from your laptop.
  Demo day is in class on Thursday, April 30th.
• Final report. Final report due Friday, May 8th.

Expectations

• What problem are you solving?
• What is the motivation for solving that problem? Why is the problem interesting and challenging? Why would a solution be useful?
• How have you solved the problem -- how is your system designed?
• Why is your design good? What key decisions and trade-offs have you made? Is your design the simplest reasonable design?
• Does your solution fit well with the rest of the system? If your solution requires modifying every piece of hardware, software, and data in sight, it won't be credible, unless you can come up with a very good story why everything needs to be changed.
• What new ideas or techniques have you developed as part of your design? What can others learn from your work?
• How does your implemented system work?
• Can you demonstrate that your system does indeed solve the original problem? Typically you'll do this with an experimental evaluation, and present quantitative results.
• What is the relationship between your work and previous solutions to similar problems? Your report should include a Related Work section outlining the existing work that's closest to your project, and explaining how your design is different or better.
• A good report will also be well written:
  • Is the report easy to understand?
  • Is it well organized and coherent?
  • Does it use diagrams where appropriate?

Ideas

• Make CFS (cooperative file system) for Cloud Storage
  Take the file system you built in lab two and (1) store blocks instead of whole files, (2) use cloud storage as content addressable storage (i.e. the S3 key should be the hash of the value or the hash of a public key). (3) cache blocks and directories so that performance in wide-area is not aweful.
• Figure out how to use a TPM (Trusted Platform Module) to better ensure integrity in cloud storage
  Amend the S3 interface to accomodate a trusted TPM.
• Build a client scrapper for asynchronous mirroring
  Figure out how to tag responses to clients in such a way that we can reconstruct the most recently written data primary site that would have otherwise been lost in a disaster.
• Make S3FS work in the wide area.
  Conventional wisdom says S3FS won't work over the wide area network. Why? What are the operations that make the system unbearably slow? Measure existing S3FS' performance over the wide area and implement your improvements.
• A peer to peer shared file system over the wide area Internet.
  It would be neat to share your files with friends over the wide area internet over a file system interface intead of using special p2p applications. This usage scenario is different from S3FS in the sense that there is no single server (or a set of servers) to store files on. Each node should be responsible for its own file storage but also maintain a user-specific namespace that automatically incorporates peers' files. How do you make use of this usage scenario to avoid many performance killers in a cluster file system at the cost of certain semantic violations.
• Erasure coded distributed file system.
  While storage space on an individual's PC seems unnecessarily massive, the demand of storage space on a shared storage cluster tends to outpace the supply. Implement a storage saver module that works with the storage system (e.g. the distributed file system) to transparently compress and erasure-code (instead of replicating) files on disks.
• An energy-saving file system using flash memory.
  Mechanical harddisk consumes significant energy (you should take what I say with a grain of salt and do the necessary background research and preferably experiment to verify it). Could you add a layer of cache in front of the harddisk using the cheap and large flash memory (2G, $30) to allow you to spin down the disk completely most of the time? Your system might solve that "oh no, i'm working on a presentation on the airplane and my laptop battery is running out in 2 minutes" problem.
• Build a file synchronization tool.
  We all have multiple computers that store copies of our data. A file synchronizer brings these copies of files in sync with each other as they are updated. File conflicts are very annoying. How should you reduce the possibilities for conflicts as much as possible? For example, can your synchronizer be more proactive with its syncing? Additionally, traditional file synchronizers sync at the granularity of individual files. Will syncing at a finer grain be better?
• Build a more full-featured version of the distributed S3FS server lab, for example, in the spirit of Frangipani and Petal.
• Build a file system interface to another data system, e.g. cvs or web.
  For example, in cvs FS, each file is represented as a directory with different versions of the files in it.
  For example, you can access and upload your videos on Youtube via a file system interface.
• A file system utilizing multiple disks.
  It's like a userlevel RAID. Think of this usage scenario where I store recorded TVs using MythTV. Right now, I cannot make use of the space of my 2 harddisks because I cannot have a directory that spans multiple disks. Your userlevel FS should enable that. Furthermore, if you can stripe large file writes across multiple disks, you might increase the write performance also by fully utilizing the capacities of all disks.
• A cryptographic file system.
  
• A file system search tool.
  One can use a rich pool of access information available at the file system to improve the search quality. Examples of such valuable information include: how long each file has been kept open since its creation, how many users have accessed it, frequencies of access etc.

Questions or comments? email hweather@cs.cornell.edu

Policy on academic integrity