CS 6410: Advanced Systems

Who (click for slides)

Topic

Required Reading

Suggested Reading

The Impact of Architectural Trends on Operating System Performance Rosenblum et al. 15th SOSP, 1995.

Interposition Agents: Transparently Interposing User Code at the System Interface, Michael Jones. 14th SOSP, 1993, pages 80--93.

Required:  Bitcoin: A peer-to-peer electronic cash system. Nakamoto, Satoshi. Consulted 1.2012 (2008): 28.

optional: Bitcoin and Cryptocurrency Technologies: A Comprehensive Introduction. A. Narayanan, J. Bonneau, E. Felten, A. Miller, S. Goldfeder. Princeton University Press; 2016.
Read Chapters 1 and 2.

Optional:  Majority is not enough: Bitcoin mining is vulnerable. E. Ittay, and E. Sirer.  arXiv preprint arXiv:1311.0243 (2013).

The Bitcoin Backbone Protocol: Analysis and Applications.   Juan A. Garay, Aggelos Kiayias, Nikos Leonardos.  Nov. 2014 (ArXiV).

Information Propagation in the Bitcoin Network.  Christian Decker, Roger Wattenhofer.  13th nal Conference on Peer-to-Peer Computing, 2013.

required: The Design and Implementation of a Log-Structured File System, Mendel Rosenblum and Ousterhout. Proceedings of the thirteenth ACM symposium on Operating systems principles, October 1991, pages 1--15.

optional: A Fast File System for UNIX. Marshall K. McKusick, William N. Joy, Samuel J. Leffler, Robert S. Fabry.  ACM TOCS 2(3), Aug 1984, pages 181--197.

Soft Updates: A Solution to the Metadata Update problem in File Systems.  Gregory R. Ganger, Marshall Kirk McKusick, Craig A. N. Soules, Yale N. Patt. ACM TOCS 18(2). May 2000, pages 127--153.

The Zebra striped network file system,  John H. Hartman and John K. Ousterhout. Proceedings of the fourteenth ACM symposium on Operating systems principles, 1993, pages 29--43.

When to forget in the Elephant file system,  Douglas S. Santry, Michael J. Feeley, Norman C. Hutchinson, Alistair C. Veitch, Ross W. Carton, and Jacob Ofir.  Proceedings of the seventeenth ACM symposium on Operating systems principles, December 1999, pages 110--123.

Towards weakly consistent local storage systems,   Ji-Yong Shin, Mahesh Balakrishnan, Tudor Marian, Jakub Szefer, Hakim Weatherspoon.   5th ACM Symposium on Cloud Computing (SoCC), October 2016.

Goal-oriented programming, or composition using events,or threads considered harmful, Robbert Van Renesse. Proceeding of the 8th ACM SIGOPS European Workshop on Support for Composing Distributed Applications (Sintra, Portugal, 1998), pages 82--87.

Using Threads in Interactive Systems.  Hauser et al.  14th SOSP, Dec 1993.

An introduction to programming with threads, A. Birrell. Technical Report, Microsoft Research, May 2005.

Capriccio: Scalable threads for internet services, Rob von Behren et al. Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles (Bolton Landing, NY, 2003), pages 268--281.

Events can make sense, Maxwell Krohn, Eddie Kohler, and M. Frans Kaashoek. Proceedings of the 2007 USENIX Annual Technical Conference (Santa Clara, CA, June 2007), pages 87--107.

Using continuations to implement thread management and communication in operating systems, Richard P. Draves, Brian N. Bershad, Richard F. Rashid, and Randall W. Dean. Proceedings of the Thirteenth ACM Symposium on Operating Systems Principles (Pacific Grove, California, 1991), pages 122--136.

Eraser: A Dynamic Data Race Detector for Multi-Threaded Programs,  Savage, Stefan et al.  ACM Transactions on Complete Systems (TOCS), 1997.

  µ-Kernels

Operating system support for database management,  Michael Stonebraker.   Commununications of the ACM Vol 24, no. 7 (July 1981), pp. 412--418.

The Duality of Memory and Communication in the Implementation of a Multiprocessor Operating System, M. Young, A Tavanian, R. Rashid, D. Golub, and J. Eppinger. Proceedings of the Eleventh ACM Symposium on Operating Systems Principles (Austin, Texas, United States), ACM, 1987, pages 63--76.

EMERALDS: A Small-Memory Real-Time Microkernel,  Zuberi et al.  17th SOSP, 1999.

The Flux OSKit: A Substrate for OS and Language Research. Ford et al. 16th SOSP, Oct 1997.

Required:

Extensibility, Safety and Performance in the SPIN Operating System, Brian N. Bershad, Stefan Savage, Przemyslaw Pardyak, Emin Gun Sirer, Marc E. Fiuczynski, David Becker, Craig Chambers, Susan Eggers.   15th ACM symposium on Operating systems principles (SOSP), December 1995, pages 267--283.

Application performance and flexibility on Exokernel systems,  Kaashoek et al.  16th SOSP, 1997.

Microkernels meet Recursive Virtual Machines,  Bryan Ford, Mike Hibler, Jay Lepreau, Patrick Tullmann, Godmar Back, Stephen Clawson.   2nd USENIX symposium on Operating systems design and implementation (OSDI), October 1996, pages 137-151.

The Origin of the VM/370 Time-Sharing System, R. J. Creasy, In IBM Journal of Research and Development, 25(5):483-490, September 1981.

Xen and the Art of Virtualization, Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, Andrew Warfield. 19th ACM symposium on Operating systems principles (SOSP), October 2003, pp. 164--177.

Are Virtual Machine Monitors Microkernels Done Right?, Steven Hand, Andrew Warfield, Keir Fraser, Evangelos Kotsovinos, Dan Magenheimer. Proceedings of the Tenth Workshop on Hot Topics in Operating Systems (HotOS), Sante Fe, NM, June 2005.

Are Virtual Machine Monitors Microkernels Done Right?, Gernot Heiser, Volkmar Uhlig, Joshua LeVasseur. ACM SIGOPS Operating Systems Review (OSR), Volume 40, Issue 1, January 2006, pages 95--99.

Memory resource management in VMware ESX server, C. A. Waldspurger. OSDI 2002.

Virtual Machine Monitors: Current Technology and Future Trends, Mendel Rosenblum, Tal Garfinkel. Computer, vol. 38, no. 5, pp. 39-47, May 2005.

Operating System Support for Virtual Machines, S. T. King, G. W. Dunlap, and P. M. Chen. 2003 USENIX Technical Conference.

The Turtles Project: Design and Implementation of Nested Virtualization, Ben-Yehuda, M., Day, M.D., Dubitzky, Z., Factor, M., Har'El, N., Gordon, A., Liguori, A., Wasserman, O. and Yassour, B.A. In OSDI, vol. 10, pp. 423-436. 2010.

Required: Disco: Running Commodity Operating Systems on Scalable Multiprocessors, Edouard Bugnion, Scott Devine, and Mendel Rosenblum. 16th ACM symposium on Operating systems principles (SOSP), October 1997, pages 143--156.

Optional: The Multikernel: A new OS architecture for scalable multicore systems.  Andrew Baumann, Paul Barham, Pierre-Evariste Dagand, Tim Harrisy, Rebecca Isaacs,
Simon Peter , Tim Roscoe, Adrian Schüpbach, and Akhilesh Singhania . Proceedings of the Twenty-Second ACM Symposium on Operating Systems Principles (Austin, Texas, United States), ACM, 2009.

Thousand core chips: a technology perspective. S. Borkar.  In Proceedings of the 44th Annual Design Automation Conference, pages.  746–749, 2007.

Corey: An operating system for many cores. S. Boyd-Wickizer, H. Chen, R. Chen, Y. Mao, F. Kaashoek, R. Morris, A. Pesterev, L. Stein, M. Wu, Y. Dai, Y. Zhang, and Z. Zhang. In Proceedings of the 8th
USENIX Symposium on Operating Systems Design and Implementation,
pages 43–57, Dec. 2008.

Tornado: maximizing locality and concurrency in a shared memory multiprocessor operating system, Ben Gamsa, Orran Krieger, Jonathan Appavoo, and Michael Stumm. 3rd USENIX symposium on Operating systems design and implementation (OSDI), February 1999, pages 87-100.

Wiki: Trusted Platform Module

Wiki: Public Key Certificates

Wiki: X509 Standard

Software Guard Extensions (SGX)

Fabric: A Platform for Secure Distributed Computation and Storage.  Jed Liu, Michael D. George, K. Vikram, Xin Qi, Lucas Waye, and Andrew C. Myers.  In ACM Symposium on Operating Systems, 2009.  Pages 321-334.

Required:The Google file system, Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung. 19th ACM symposium on Operating systems principles (SOSP), October 2003, 29--43.

Optional:Spanner: Google's Globally Distributed Database, James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, and Dale Woodford. In Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation (OSDI'12), October 2012, 251--264.

Required:MapReduce: Simplified Data Processing on Large Clusters.Jeffrey Dean and Sanjay Ghemawat. (OSDI, 2004). December, 2004.

Optional: Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica. (NSDI'12), April, 2012.

Apache Hadoop YARN: yet another resource negotiator, Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O'Malley, Sanjay Radia, Benjamin Reed, and Eric Baldeschwieler. 2013. Apache Hadoop YARN: yet another resource negotiator.  In Proceedings of the 4th annual ACM Symposium on Cloud Computing (SoCC). Article 5, 16 pages, October 2013.

Distributed Systems:

Ordering and Consistent Cuts

Required: Time, Clocks, and the Ordering of Events in a Distributed System, Lamport. CACM 21(7). July 1978.

Optional: Distributed snapshots: Determining global states of distributed systems. Chandy, Lamport. ACM TOCS 3(1), 1985, 63-75.

Distributed Systems:

Putting theory into practice

Implementing fault-tolerant services using the state machine approach: A tutorial, Fred Schneider. ACM Computing Surveys Volume 22, Issue 4 (December 1990), 299--319.

Chain replication for high throughput and availability.  Robbert van Renesse and Fred Schneider.  OSDI 2004.

Distributed Systems:

Required: The Process Group Approach to Reliable Distributed Computing, Birman. CACM, Dec 1993, 36(12):37-53.

Optional: Bimodal Multicast, Birman et al. ACM TOCS 17(2), May 1999.

CAP Twelve Years Later (and another version here).  Eric Brewer. IEEE Computer Magazine (special issue on “The Growing Impact of the CAP Theorem”). Vol 45, Issue 2, February 2012, pp. 23-30.

Perspectives on the CAP Theorem (and another version here). Seth Gilbert and Nancy Lynch. IEEE Computer Magazine, Vol. 45, Issue 2, February 2012, pp. 30--36.

Overcoming CAP with Consistent Soft-State Replication (and another version here). Kenneth P. Birman, D. Freedman, Q. Huang and Patrick Dowell. IEEE Computer Magazine (special issue on “The Growing Impact of the CAP Theorem”). Vol. 45, Issue 2, February 2012, pp. 50-58.

isis2.codeplex.com:  Isis², A modern, open-source, group communication library for cloud computing applications.  Ken Birman, 2011.

Sinfonia: A new paradigm for building scalable distributed systems. Marcos K. Aguilera, Arif Merchant, Mehul Shah, Alistair Veitch, Christos Karamanolis. November 2009 Transactions on Computer Systems (TOCS), Volume 27 Issue 3

Distributed Systems:

Required:Paxos Made Moderately Complex.  Robbert van Renesse and Deniz Altinbuken.  ACM Computing Surveys. Vol 47, No. 3, Article 42, 36 pages, February 2015.

Required:Wiki: Paxos_Protocol

The part-time parliament, Leslie Lamport. ACM Transactions on Computer Systems (TOCS), Vol. 16, No. 2, May 1998, pp. 133–169.

Paxos Made Simple, Leslie Lamport. ACM SIGACT NEWS, Vol. 32, No. 4, December 2001.

CORFU: A Distributed Shared Log, Mahesh Balakrishnan, Dahlia Malkhi, John Davis, Vijayan Prabhakaran, Michael Wei, and Ted Wobber, ACM Transactions on Computer Systems (TOCS), ACM, 2013.

Distributed Systems:

Byzantine Agreement

Required: The Byzantine Generals Problem, Lamport et al. ACM TOPLAS 4, 1982.

Optional Easy Impossibility Proofs for Distributed Consensus Problems.  Michael J. Fischer , Nancy A. Lynch , Michael Merritt.  ACM PODC 1986.

Practical Byzantine Fault Tolerance, Miguel Castro and Barbara Liskov. USENIX Operating Systems Design and Implementation (OSDI), February 1999, pages 173--186.

Randomized Byzantine Generals, Rabin. FOCS, 1983.

Byzantine Quorum Systems, Malkhi and Reiter.

Fault-Scalable Byzantine Fault-Tolerant Services, Michael Abd-El-Malek et.al. SOSP 2005.

The Next 700 BFT Protocols. Rachid Guerraoui, Nikola Knezevic, Vivien Quéma and Marko Vukolić. In Proceedings of EuroSys, Paris, France, pp. 363-376, April 2010

Distributed Systems:

Required:Impossibility of Distributed Consensus with One Faulty Process, Fisher et al. JACM 32(2), Apr 1985.

Optional:The weakest failure detector for solving consensus, Chandra et al. J. ACM 43, 4, Jul. 1996.

Effectively nonblocking consensus procedures can execute forever-a constructive version of flp, Robert Constable. arXiv preprint arXiv:1109.3370, September 2011.

The Building Blocks of Consensus. Yee Jiun Song, Robbert van Renesse, Fred B. Schneider, Danny Dolev.  The 9th International Conference on Distributed Computing and Networking (ICDCN 08), January, 2008. LNCS, Vol. 4904, pp. 54-72.

Required:Epidemic algorithms for replicated database maintenance, Alan Demers, Dan Greene, Carl Hauser, Wes Irish, John Larson, Scott Shenker, Howard Sturgis, Dan Swinehart, Doug Terry. Appears in 6th ACM Symposium on Principles of distributed computing (PODC), August 1987, pages 1--12.

Optional:Managing update conflicts in Bayou, a weakly connected replicated storage system, Doug B. Terry, Marvin M. Theimer, Karin Petersen, Alan J. Demers, Mike J. Spreitzer, and Carl H. Hauser. In Proceedings of the 5th ACM Symposium on Operating Systems Principles (SOSP), December 1995, pages 172--182.

Astrolabe: A Robust and Scalable Technology for Distributed System Monitoring, Management, and Data Mining, Robbert Van Renesse, Kenneth P. Birman, Werner Vogels. Appears in ACM Transactions on Computer Systems (TOCS), Volume 21, Issue 2, May 2003, pages 164--206.

T-Man: Fast Gossip-based Construction of Large-Scale Overlay Topologies.  Mark Jelasity Ozalp Babaoglu.  Technical Report UBLCS-2004-7.  May 2004

Bayesian inference using data flow analysis.  Claret, Guillaume, et al. Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013.

Probabilistic programming. IGordon, A. D., Henzinger, T. A., Nori, A. V., & Rajamani, S. K.  May 2014, May. International Conference on Software Engineering (ICSE, FOSE track).

P2P Systems: DHT

Chord: A scalable peer-to-peer lookup service for internet applications, Ion Stoica, Robert Moris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan. Proceedings of the ACM SIGCOMM, August 2001, 149--160. San Diego, California, United States.

The Impact of DHT Routing Geometry on Resilience and Proximity, Krishna Gummadi , Ramakrishna Gummadi , Steve Gribble , Sylvia Ratnasamy , Scott Shenker, Ion Stoica. Appears in Proceedings of ACM SIGCOMM, August 2003, pages 381--3.

Kelips: Building an Efficient and Stable P2P DHT Through Increased Memory and Background Overhead, Indranil Gupta, Ken Birman, Prakash Linga, Al Demers and Robbert van Renesse. 2nd International Workshop on Peer-to-Peer Systems (IPTPS '03); February 20-21, 2003. Claremont Hotel, Berkeley, CA, USA.

Required: Dynamo: Amazon's Highly Available Key-Value Store, Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall, and Werner Vogels. In Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles (SOSP). October 2007, pp. 205--220.

Optional:Cassandra: a decentralized structured storage system, Avinash Lakshman and Prashant Malik, SIGOPS Oper. Syst. Rev. 44, 2 (April 2010)

Pond: the OceanStore Prototype, Sean Rhea, Patrick Eaton, Dennis Geels, Hakim Weatherspoon, Ben Zhao, and John Kubiatowicz. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies (FAST), March 2003, pages 1--14.

Storage management and caching in PAST, a large scale, persistent peer-to-peer storage utility,  Rowstron and Druschel.  18th SOSP, Oct 2001.

Wide-area cooperative storage with CFS, Frank Dabek, M. Frans Kaashoek, David Karger, Robert Morris, and Ion Stoica. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), October 2001, pages 202--215.

A Distributed Decentralized Information Storage and Retrieval System,  Ian Clark.  University of Edinburgh, 1999 (Original unpublished technical report on Freenet).

Why Gnutella can't scale, no really?,  Ritter, Feb 2001.

Congestion Avoidance and Control, Van Jacobson. Appears in Proceedings of ACM SIGCOMM, Vo1ume 18, Number 4, (August 1988).

TCP Congestion Control with a Misbehaving Receiver, Stefan Savage, Neal Cardwell, David Wetherall and Tom Anderson, Appears in ACM SIGCOMM Computer Communication Review, Volume 29 , Issue 5 (October 1999), pages 71--78.

Networked Systems:

Required: Arrakis: The Operating System is the Control Plane.  Simon Peter, Jialin Li, Irene Zhang, Dan R. K. Ports, Doug Woos, Arvind Krishnamurthy,
and Thomas Anderson,  Timothy Roscoe.  OSDI 2014.

Optional: IX: A Protected Dataplane Operating System for High Throughput and Low Latency. Adam Belay, George Prekas, Ana Klimovic, Samuel Grossman, and Christos Kozyrakis,  Edouard Bugnion. OSDI 2014.

Profiling a warehouse-scale computer. Svilen Kanev, et al.  ISCA 2015. Slides.  This paper basically says that 90% of potential compute performance in Google's main data center is lost in hardware stalls of various kinds!

High performance RDMA-based MPI implementation over InfiniBand. Liu, Jiuxing, Jiesheng Wu, and Dhabaleswar K. Panda. International Journal of Parallel Programming 32.3 (2004): 167-198.

WireDirect Enables New Benchmarks For Low Latency UDP.  Chelsio white paper.

Optional: NetCache: Balancing Key-Value Stores with Fast In-Network Caching,  Xin Jin, Xiaozhou Li, Haoyu Zhang, Robert Soulé, Jeongkeun Lee, Nate Foster, Changhoon Kim, and Ion Stoica.  26th ACM symposium on Operating systems principles (SOSP), October 2017, pages 121--136.