(2018 version.  Click here for 2019).

How the Projects Work

We want you to work on your project through the entire semester (some people even continue beyond the end of the semester, for extra credit). So starting on time is important.

Everyone should select a project during the first two weeks of the semester, and we will ask you to upload a plan at the end of that period, including (1) team members, if you work with others, (2) the specific project you will do, (3) your timetable for starting to show some hands-on experience with the key elements, (4) the split of tasks within your team.

Plan to spend 4-6 hours per person, per week, on these projects. This is in lieu of other homework, and there is no final, so the workload is actually pretty much average for a Cornell class, provided that you start on time.

Our initial hope had been to have people use Derecho for their projects, but the platform is not yet stable enough to make this a wise choice.  So we are suggesting that you consider setting up Tensor Flow instead, and working in that model.  The experience would be very valuable for landing high-paying jobs these days!  There is one Derecho project opportunity mentioned at the bottom of this page, but we will probably limit how many people can work on it.

Project Goals

For spring 2018, we are recommending that students try to build a Fog or Edge computing application, and that they consider using Tensor Flow for the "online" aspects.   We will not be teaching you to use Tensor Flow; you would need to watch the online training videos and read the documentation.  But basically, this is a Python-based language (so your code is entirely in Python), that has a distributed computing model built into it, based on a flow of events that hold data in the form of a matrix, image, video, or a numerical "tensor".  In fact they treat all such data as some kind of tensor, for uniformity, but this is because even a string can be viewed as a 1 x 1 matrix with one cell containing a string.  So if you are very aggressive about it, anything is a tensor.  Tensor Flow is about "flows" of tensor objects from data sources, like sensors, into other subsystems in a cloud.

But Tensor Flow is not obligatory.  On Microsoft Azure, a version of C# called Dandelion might of interest, especially if you work with video content.  On Amazon, you can use Tensor Flow, but there are all sorts of tier-two service options for dealing with flows of events from sensors.  You do need to figure out what you will use, on your own, because there are so many choices.  But once you decide, our TAs can help you map the problem you pick to the technology you picked.  The TAs don't know everything about everything, but they are good at this kind of system structuring.

Aim for something simple, then build on that.  For example, start by "capture a sequence of data records, and do something to them, and store the results."  Work your way up and out, taking small steps.  This way you will always have something working, even if the thing isn't the world's most exciting.  Don't wait until the last minute, swing for the outfield fence...  and strike out entirely!

 Projects can be specialized, but would generally have a structure along the following lines.

• There will be some form of sensors that the system can connect to, or that connect to the cloud and upload their data. Some projects will "fake" this in a convincing way, for example by uploading prerecorded video data rather than capturing it in real-time, to simplify things a little.  That's fine.
• The cloud will capture the uploaded sensor data and store it into some form of data structure or database -- perhaps a real database, perhaps a key-value structure, perhaps a home-build storage structure.
• Some form of data processing occurs in the cloud.
• Some form of "external control" or "integration with social networking" can be demonstrated.

For example, you could try something like this:

• The sensors might be video cameras, like the ones on your phone. A phone has limited battery capacity, so it can store data but probably wouldn't try to process it.  As noted, prerecording the video is a way to simplify the (difficult) step of capturing video continuously in real-time.
• The incoming data might be processed through a series of stages immediately upon arrival in the cloud. These could include image segmentation, tagging, etc.  You would use prebuilt tools for this, not create new solutions.
• If data seems interesting, it could be stored in a file, or in the key-value store, for additional processing.  Very dull data could be discarded, etc.
• If the system senses an emergency situation (like an object on a road, or a touchdown in a football game), it could set off some form of alarm ("watch out! There is an obstacle on the road around the next turn!" or "he scores!").   This case is intended to illustrate the quick response time aspect of such systems.  Getting fast responses at scale is the crux of what makes a fog or edge system different from classic cloud systems, which process data more or less offline, in a back-end batch layer.  That takes way too long (imagine that your football watching system yelled "he scores!" three hours late...)
• By connecting to the open-source Spark/Databricks platform, your system could do additional offline, non-realtime, machine learning based on the data received.  So we aren't throwing away all those amazing things you learn about in machine-learning classes.  Just taking the real-time actions out of the batch, non-real-time processing path.
• The system could have a console where it reports on external control actions it wishes to perform.  This would avoid the difficult task of actually showing that the system can connect back to the real world.
• Other features. For example, the cloud might give the edge devices instructions of some form, which they could "use" for a period of time.  Like a self-driving car: you could tell it where it is safe for it to drive for the next 5s, or 5m, etc.

A Few Examples

Integration of Social Media with Videos

A first example of a project along these lines would focus on social networking, like with Facebook or other companies.  This is more of an edge example than a fog computing case, but because you want snappy response, it fits the theme just outlined.

Context: the problem arises because there is a lot of interest in ways that video content could be integrated more effectively with social networking content of the kind that Facebook TAO captures and tracks (read the paper if you don't remember what TAO does, or haven't yet seen that lecture).  Right now, the architecture of the classic cloud blocks us from fully exploiting this opportunity.

So suppose that you know things about a set of people, via TAO, and now those people are in a setting with a lot of video capabilities, like a disco, or a concert, or a smart home (see the smart home example below; this could be part of the same project). And suppose that the goal is to build a scalable cloud computing service that can turn any gathering into an amazing "movie" about the event (but not in the sense of snooping on what people are talking about!).

A movie about a party or a concert might center on the dinner, or on the performance, but cut little snippets in of groups of people talking and laughing, or dancing, or doing a wave. You would want the system to tag the people, then to use the TAO network to suggest to those people that they might want to share the video with particular friends. For example, if you and your friend Jane were at the party, but your shared friend Tallia was away interviewing, you or Jane could share the video with her.

There are a number of puzzles here, and we would expect projects to focus on a narrower kind of technical story but to think it through carefully. First, you have the technology questions themselves: how to film a whole concert, or a party, or people riding on a terrain park structure. Next is the puzzle of tagging: for privacy, you definitely would want all the people in a video of this kind to agree that it is ok with them if you share it. Then beyond that, you need to figure out how to leverage the TAO social network graph, and last, how to stitch the pieces together into a social-network "enriched" story. And there are questions of scalability: how can the solution be scaled out to run on the cloud.

A scenario with fewer privacy issues would be a sports event like a professional soccer or football or baseball game. You might imagine a set of cameras, all around the stadium, that capture the big game from every possible perspective. How would you use cloud computing to enable viewers to use joysticks on their home systems to zoom around and see the event from individually personalized perspectives? Could you and your pals from before you came to Cornell watch this kind of enriched-media game together, just like in old times, but with the network linking you together? "Sammy, he never should have missed that field-goal. Look at how easy the shot was from the kickers perspective!" and then you might zoom right in to show how it looked to that kicker as he struck the ball. And Sammy might come right back "No Tom, totally wrong -- the point is that this linebacker over here was charging in and the kicker had to watch him. Look at the same shot once you keep that guy in your sights..."

Someday this will be a big business -- a super big business. Doing a project like this one could launch you on the path to entrepreneurial zillions!

Companies like Facebook, Google (YouTube), Verizon (Yahoo) and others are keenly interested in this whole area.

One concern: videos such as these (lots of them) can only be processed in limited ways by the cameras themselves. So very likely the cloud solution will need a powerful GPU cluster to crunch the data, shared by the first-tier systems that see the streams (or those machines might use their own on-board video cards as GPUs). A convincing project probably needs to at least hint at how you would leverage that kind of hardware, and why it should be feasible. But as a student at Cornell you might not easily be able to demonstrate that aspect. So think this question through before proposing to do this project. (The same concern arises about the next project, too, but it is a bit simpler because the smart cars on a smart highway know a lot about themselves, and you could focus on that case -- in which case maybe videos aren't quite as important for figuring out what in the world is going on!).

Smart Highway: A Tensor-Flow Application

As a second example, solidly in the fog computing domain, consider something we actually discussed in class: a smart highway would have video sensors and motion sensors to watch the cars on the highway, plus it might have ways for the cars themselves to upload data from on-board cameras. The processing stages would basically identify the cars and other vehicles and then compute their trajectories (their motion paths). Based on this, if a car is following the predicted path, you might record the exact path but wouldn't update the knowledge of the system. If the car has departed from predictions, you would store data into the versioned storage layer, then use the back-end Spark/Databricks tools to "relearn" the vehicle trajectories. Finally, if your system predicts that two cars may get closer than a safe limit, you could warn about this, via console messages. In reality, you would link those back to the highway and to other cars.

With this structure, here's how each step could be developed and demonstrated. Some stuff is in bold because we want you to pause and think about those points:

• You and your friends would need to create some kind of data set. You can find data sets online, or you could drive around a parking lot and generate one of your own. Break it into segments, like 1-second video snippets. To be fancy, use file names that encode the time, and later use that time information to timestamp the videos (this can override the true time, e.g. you can fake that the data was captured on Tuesday March 13 from 9:00am-9:05am or something).
• Cars could also have GPS units and could even report on their location and speed and current trajectory periodically, like a few times per second. Then you only need to use computer vision for mystery vehicles that are not smart enough to report this sort of thing. Note: there may be a way around this aspect of vision -- maybe you can basically eliminate the vision tasks entirely, by assuming a bit more help from the cars. This could be useful, because GPU computing is tricky and it isn't easy to use on-board graphics hardware even on desktop systems with heavy capabilities. Honestly, our biggest worry is that some projects might underestimate the puzzle of how to process image and video data, and end up with an impossible design just because it can be so hard to use that kind of hardware, and so slow if you can't use it.
• Next you'll need to design an upload approach. There are web packages to help, but it will be part of your job to pick one. We have had good experience with "RestFUL RPC packages", which exist for Java, C++, C#, etc.
• The server would probably need to be coded in C++. The main tasks at this level will be to replicate the knowledge model ("what the system thinks the smart highway is doing"), then use the replica on a particular server to carry out its part of the image analysis. There are existing libraries that can do segmentation and tagging; you shouldn't need to create your own. Many can even leverage the graphics processor on your computer (a form of GPU hardware).
• Things to think about: fault-tolerance for the service, and for data it captures. Speed. Zero-copy data movement.
• Use Spark/Databricks to design a system to learn which cars are on the road, to estimate vehicle trajectories, etc.
• Perhaps, assign each smart car to some sort of trajectory that it is authorized to follow, like a flight plan for an airplane. Don't assume that the car can do a lot of general purpose computing, but you can still play with various simple forms of on-car intelligence, and doing so might easy the video processing aspects of the project. Smart cars are quite idiosyncratic, and a smart highway needs to assume a least-common-denominator approach, so you can't really view the smart car as a super-intelligent robot that can drive itself to Florida while you kick back and enjoy the ride. But perhaps a smart car could follow a recommended trajectory for 100m at a time, and could be told to form a convoy with the car in front of it, etc. Simple things that you can do robustly. Don't assume that the cars can talk directly to other cars: that seems very unlikely to work in the forseeable future (short-range communication is hard to set up and prone to failure; car to cloud communication is a much more robust option). For this reason, the best way for car A to talk to car B might be via a kind of text messaging through the cloud. Think of the cars as being "controlled" by the highway, but with very limited forms of autonomy.
• Relay the learned data back to the first tier and replicate it, for use by the data capture service.

Now think about scalability: in practice, each data capture layer will be sharded to scale, with perhaps thousands of two-node shards. Will your design scale in this sense?

Note: not every project needs to cover every aspect of this problem. Feel free to slice off a subset of the puzzle and to solve just that portion, but to solve it really well. But no matter what you decide to do, it has to be a real cloud computing scenario, approached realistically, built (not done purely on paper), and validated through experiments. During your demo we want to see some evidence that the solution solves the portion of the problem you selected, scales well, performs well, is fault-tolerant, etc.

Smart highways are going to be a real thing. If you work on this topic, you might consider joining a company working on real smart highway products!

Other Project Ideas:

We won't flesh these out in equal detail, but here are some other concepts you could explore. Or dream up a project of your own (provided we approve it).

• Smart home technologies. The idea is actually similar to the smart highway, but would match more directly with the capabilities of a system like Caspar.ai, which is a product being created by a past Cornell professor, Ashutosh Saxena. Check out the videos on his web site.
  
  As the videos will show, the Caspar.ai product is a scalable system that customers talk to (like for Google Siri + Google Nest or Alexa), but also has databases of information like layouts of each apartment, names and other personal data about the people living in each unit, their preferences, etc. Companies like Google and Amazon lack that sort of knowledge of the home: when you say "Alexa, block that glare from the window" they might have no idea which window, or even what room you are in. Caspar.ai should easily be able to do this, because it knows the room layout, knows where you are siting, etc. The "actions" the system can take are things like opening and closing curtains and windows, adjusting music or lighting, tracking the pets around the apartment and telling them to stop if they chew on the furniture (or notifying you), telling you that there is a delivery guy at the door (and that he is a legitimate UPS person), etc. You can make a longer and longer list. Don't forget that scalability is a need: if you do this kind of project, you still need to use a sharded data collection approach! You can use mobile devices or even normal web pages as your input sources. Of course the more realistic the better. Inventing new features for Capar.ai would be cool too (in fact they view the system as a platform -- an operating system for the home). Ashutosh has indicated that the company can share data, to help you with your project.
  
  Note: Ashutosh would love to hire some qualified candidates, so if you do this, an actual job at Caspar.ai could be an option. But for full disclosure: Professor Birman is a member of the Caspar.ai advisory board, so this isn't a totally unbiased project proposal! In fact, it is the exact opposite: it is, without question, a very biased project idea! Still, it does seem to be a great match to cs5412.
  
  Ashutosh recommends that you email him at ashutosh@caspar.ai if you are considering working on this topic and would like to know about getting some databases or other information that might help you make your class project more realistic.

• Next generation air traffic control. This is just smart highways, but with airplanes. A system like this would start from the basics: you have controllers, radar updates, and flight plans, and controllers interact with the system by revising flight plans, which then get logged securely and reported out to any controller or subsystem with a stale copy. But then things get fancy: you can add in services that do things like watching for violations of aircraft proximity restrictions, checking for bad weather and suggesting other routings, offering ways to save fuel, delaying flights to avoid having customers miss connections, etc. In fact a system like this is similar to the smart highway, although the input data is probably easier to process. In some ways air traffic control is easier than smart highways: there aren't so many planes, they are kept very far from one-another, and every single thing in the airspace is under control of the human operators working in the air traffic control center on the ground. But this greater degree of control also makes these systems a bit more ambitious about selling services to the pilots and airlines (these days, planes pay for air traffic control services, and they might pay extra for the fancy services mentioned above).

• Campus game of humans v. zombies. Here, the idea is that players have an app on their mobile phone. Each player is a human or a zombie. A zombie tags a human to infect them, and a human can throw a bunched up sock at a zombie to temporarily stun it. That same zombie can no longer infect that particular human, although after a few minutes it can revive and attack other humans. The cloud computing system maintains a big status map, data about the players and their current states (but there can be secretly infected humans, or zombies that are miraculously cured, so the game state visible to players doesn't necessarily have to be perfectly correct and consistent relative to truth, even if the system knows the truth). This has been a popular choice for past cs5412 students.  The application is fun to build but less likely to ever matter commercially, but perhaps with significant integration into real-time devices would start to close in on this huge opportunity we've talked about in the fog/edge space.  Anyhow, not every single project will be a fog or edge example.  So this particular one is more classic.

• Help us port Derecho to libFabric and add WAN capabilities to the system. This is a very different project area. Derecho currently runs in a single data center at a time, and we would like to build a wide-area layer, in which we would use TCP (a special WAN version) between data centers, and the would extend Derecho with WAN functionality to allow us to mirror data between different locations, much like with the Smoke and Mirrors File System that Hakim Weatherspoon's team once created. We have a roadmap for this work, and it involves a series of puzzles, like getting LibFabrics to run on a WAN, testing its performance, getting Derecho itself to run on LibFabrics in that configuration, testing performance for this combination, and then adding some APIs to Derecho for exploring the capability.  In fact, the main reason Derecho is not a primary choice for our Fog projects is that without LibFabrics, Derecho needs RDMA hardware and most people have no way to get that.  But in fact for this work, we will give you access to RDMA hardware, so that you can help compare the performance with LibFabrics and without it (more precisely, with RDMA hardware or without it; Derecho would run through LibFabrics for both cases).
  
  You would need (1) superb C++ coding abilities to tackle this project, including the new C++ 14 features such as variadic templates, and the new std:: library functions. (2) Experience doing system testing. (3) Good team skills: you would work on this under supervision, as part of the team led by Weijia Song, one of our main developers. So you wouldn't really be making decisions on your own -- every step of this needs to be agreed upon. Still, it would be real and would contribute useful technology to a exciting open-source cloud/fog platform. Weijia can probably supervise a maximum of 3 or 4 students, so we won't allow unlimited numbers of people to pick this option, and the ones who do may need to interview with him first. Expect to be asked to show that you really can write code in C++!