Networking

Many of you are interested in multiplayer games. But networking is a potentially difficult topic. Fortunately, CUGL provides a very simple to use networking interface that is designed for cooperative and competitive games between local mobile devices. This interface was inspired by the the success of Family Style and is based on the code base from SweetSpace, a game designed for this course in 2020. Unlike Family Style, this interface is based on an open source standard known as WebRTC, which helps cut down on licensing and server costs.

In addition to working with the networking library, this lab is the first introduction that the programmers will have to the scene graph tools. The UI elements in this game – text fields and button – are all defined by the JSON scene graph specification. You can find this specification in assets.json under scene2s. To simplify this activity, we have completed the UI elements (and the code that instructs them what to do) for you. You only have to add the networking code.

Once again, this assignment is graded in one of two ways. If you are a 4152 student, this is graded entirely on effort (did you legitimately make an attempt), and that effort will go towards you participation grade. If you are a 5152 student, you will get a more detailed numerical score.

Table of Contents

• Project Files
• The User Interface
• Application Overview
• Instructions
• Submission

Project Files

Download: NetLab.zip

You should download the project for this assignment. This is a self-contained project just like all of the ones available from the demos page. You should pick either XCode or Visual Studio as your primary development environment. That is because these IDEs have the best debugging tools. However, if you plan to use Android, you will need Android Studio installed as well.

Running the Network Lab

For once we have an application that runs and runs the same on all platforms. Yes, you can input text into a a text field on a mobile device like Android or iOS. In that case it will pull up the virtual keyboard. The only thing you need to do is to write the networking code, which is exactly the same across all platforms.

The problem is testing the application. For that you will need two devices running the project. One device should be host and one should be a client. You can do this with two laptops but we prefer to use a laptop and a mobile device. That is because you are constantly updating the code, and you would have to synchronize this across the two platforms (though GitHub could solve this problem). With a mobile device, you could plug it into your laptop and code for both at the same time.

Note, however, that XCode will only run the application on one device (laptop or phone) at a time. To solve this problem, I first compile and run the application on my iOS device. I then stop the application and disconnect the iOS device from the laptop. The code is still installed on the iOS device, so now you can run it normally (you just will not see any debugging print statements). Now run the application on your laptop and connect the two.

You might be tempted to test this out by running more than one copy of the program on your laptop. While this will work it is not recommended for testing. Because of how WebRTC works, any such communication will be instantaneous and have absolutely no lag. To properly test networking you want at least some lag, and that means multiple devices.

The User Interface

The code for NetLab is very different from the previous labs. Look at MenuScene, which is the first scene launched after the loading screen is finished. There is no update or render method. What is going on? We are inheriting these methods from Scene2. This class is used to handle scene graphs, particularly scene graphs with UI elements automatically. The only thing we need to do is to define the scene graph and hook up all the listeners/callback functions that respond to those UI elements.

So where do we define the scene graphs? This is actually done in assets.json. If you look at that file, you will see a JSON object called scene2s. This is a collection of scene graphs. The object menu is the scene graph for MenuScene. Each of the other scenes have a scene graph defined in this file.

Because this is a data asset, it is loaded with the asset loader. You can then access the elements by name. Names are separated by underscores ( _ ). So the menu button for the host scene is referenced by the name menu_host. We intended to refactor the separation to use periods (as this is more standard), but this was not completed in time for the semester (And this is so pervasive throughout the demo code that a simple patch is impossible).

You can see us accessing these nodes in the init method. We have to use std::dynamic_pointer_cast because the asset manager does not know what type of scene graph node each asset is. Once we have the object, we can add a listener. Note that listeners are specified with lambda functions as described in the second C++ tutorial. In this project we use thisto specify our variable capture. This is typically the best approach in a class-based application as it means we capture all attributes and methods of the surrounding class and that we can use them in the lambda function.

There is one more thing. Just adding a listener to a button does not guarantee that a button works. You also have to activate it. An activated button accepts clicks while a deactivated button does not. The reason this is important is because an activated button will still receive clicks even when it is not visible. So if you switch from the menu scene to the host scene, and the menu scene buttons are still active, they will still register when you click on them.

To solve this problem we need to activate UI elements when a scene is active, and deactivate them when the scene is not. In addition, you also may need to reset the UI elements to a default state when the scene is made active. For example, if we click on the host button in menu scene, it takes us to the host scene. And if we click on the back button in the host scene, it takes us back the menu scene. If we do not reset the host button, it will still be down when we return and we will be in an infinite loop between the two scenes.

As a result, the primary methods for handling user interface elements in a scene are init and setActive. All of the scenes for this game are very similar in that regard. Look at them for inspiration on your own project.

Application Overview

The user interface is complete in the provided project. You do not need to add anything new to make it work. You can click on the buttons to go between the various scenes. You can even “play” the game, clicking on the colored buttons to change the color of the background. To start off you might want to just play around with how the scenes fit together.

The game itself is simple. You press a button and the screen turns that color. When the game is fully networked, pressing a button will change the color on your screen as well as that of any connected screen. So the only difficult part of this application is understanding the networking.

The Lobby Server

Every game must have an external server. This server acts as the “lobby” for your game. It is where players find each other, so that they can group themselves together. The server does not have any game specific code; it only groups players. The primary reason for this server is so that you identify other players to connect to. In WebRTC terminology, this is known as the signaling server.

CUGL uses a custom signaling server that provides extra player managment features beyond what normal WebRTC provides. However, because it is built on top of a standard, we were actually able to implement it in Python with websockets. Because signaling only matches players, and does not handle real time communication, this does not hurt performance. This lobby is currently running on the main GDIAC website and is configured in the file server.json.

This sever will work for this tutorial. However, you may not use this server for your final submission. You are expected to create and host your own server. Fortunately, we are creating a Docker container to make this relatively turn-key. You will get that just before Alpha Release. Until that time you may use our server.

The ICE Servers

The lobby server is used to match players. By itself, it works perfectly fine. However, it only works if the players are on the same network. As that is often the case for mobile games, that is okay the vast majority of the time.

But to make our application more robust (and pass internal Apple testing), we need to be able to connect applications on different networks. This means connecting devices behind separate home networks. In some cases, these networks may have very robust firewalls blocking communications.

WebRTC is built with all of this in mind, so it has a way around all this. This is the purpose of the ICE (Internet Connectivity Establishment) servers. These are separate servers, beyond the game lobby. Their entire purpose is to allow devices on separate networks to connect to each other.

The beauty of ICE servers is that they are a standard. You do not need custom servers for them, and (to a limited degree) there are many freely available ICE servers out there. Furthermore, you application can list as many ICE servers as you want in server.json. You can think of ICE servers as kind of like DNS servers, if you are familiar with how those works.

The important thing to understand about ICE servers are the types: STUN servers, and TURN servers. Each of them behave a little differently and have their strengths and weaknesses. To make your application as robust as possible, we recommend that you have one of each. Indeed, that is exactly what we have done in the provided code.

STUN Servers

A STUN (Session Traversal Utilities for NAT) server is the simplest form of ICE server. It observes your application from outside your home network and reports the visible addresses and ports to other applications. It works if there are no firewall settings on your network, which is actually the case for most personal home networks.

A STUN server only needs to communicate to your application when it first connects to the game lobby. The game lobby will remember the important information for you after that point. Because this, STUN servers are very cheap to support and there are many free options out there. In particular, Google hosts the following STUN servers, all at port 19302:

stun.l.google.com
stun1.l.google.com
stun2.l.google.com
stun3.l.google.com
stun4.l.google.com

However, you will notice that we have provided a custom STUN server in the server.json file. That is because of the next type of ICE server.

TURN Servers

To get around firewalls, you need a TURN (Traversal Using Relays around NAT ) server. Firewalls prevent direct communication between devices. So a TURN server acts a middleman, forwarding all communication between effective devices while penetrating the firewall. So now communication is not longer direct peer-to-peer, even though it still looks that way in your game.

You do not have to do anything special to use a TURN server. All you have to do is provide it to the server.json file. If your application needs it, it will use it. If it does not need it then (for performance reasons) it will not.

But the problem is that TURN servers are not free. Unlike STUN, they are maintaining high speed communication between your players. You have to pay for that server cost.

Fortunately, the software is free (or at least open source). So we have packaged a combo STUN/TURN server for you at address 46.101.187.102. They both use the same port; the only difference is how the software talks to it. That is why we have configured server.json the way we have.

But as with the lobby server, you may not use this server for your final submission. When we give you the Docker container for the lobby server, it will also contain the STUN/TURN server. You will run both of them on the same instance, but using different network ports. Again, you will receive instructions on this later.

The Host

As with Family Style, one player must initiate a game. That player is designated the “host”. We will talk about this designation during a later course lecture, but the important thing to know about the host is that the host controls the play session. The host determines when the game starts and when the game ends. Beyond that, the host is not particularly special. Everything a client sends is broadcast to all players, not just the host.

Because the host is in control of the play session, if the host loses connection with the lobby server, the game is over. While there is support for host migration in CUGL, it is a little complicated and this activity does not use it. On the other hand, clients can rejoin if they drop out, as the lobby server will put them back in contact with the host.

The room ID is a string. It can theoretically be a string of any length with any type of characters. However, the lobby server we have provided gives a four-character hexadecimal string. This allows for 65,536 rooms. This should be enough unless you are Family Style.

However, in testing, we discovered that the font our game uses is really bad for hexidecimal strings. The letter B looks like 8 and D looks like 0. So to make the user interface simpler, we convert this hexadecimal string to a 5 digit integer using the provided function hex2dec. This is what will be displayed on the screen. Indeed, this suggests that it is best to have an interface for your game that does rely on the player typing in strings (more on that later).

The host then waits for players to join. In our application there is a text label indicating the number of players so far. In your game you may want to convey more information, such as the names of all players. When the host believes that everyone has joined, the host starts the game.

The Client

While the host joins the punchthrough server immediately, the client must wait for a room ID. That is why this network interface is for local games. The host must communicate the room ID to the players outside of the game (typically verbally). The room ID is a password that allows the players to join that game.

In this application we have a text field. You activate the text field by touching with a finger or a mouse. You can then type text into it. On a mobile device, activating a text field will activate the virtual keyboard. The players type in the room ID and hit start. This will connect the players to the host but will not start the game. The game is only started when the host indicates that is started.

While the text field approach is simple and works for this lab, it is not the preferred approach for mobile devices. It is much easier on the player to ask them to press buttons. This is the approach that Family Style took, as seen below (from the host’s perspective).

Instructions

Through this assignment you will work on three files:

• NLHostScene.cpp
• NLClientScene.cpp
• NLGameScene.cpp

For once you should not need to modify the headers. Each of the files above has a section marked student methods. The methods in this section are either empty or incomplete. It is your job to complete them.

When you write your game, you normally factor out all network processing and controls into a NetworkController, just like all of the previous labs had an InputController. But in this lab, we have decided to separate the network code for each scene. This makes it very clear what the network controller is doing at each step of the process.

However, this separation can lead to some nasty race conditions, which we will discuss later in the instructions. But you should not worry about those. The purpose of the lab is just to get everything connected. We will worry about synchronization issues when we give a proper lecture on networking.

Task 1: The Host

The first thing to do is to set up the host. There are six methods in the student methods section.

• update, which receives the latest update from the server
• processData, which interprets the data from each update
• connect, which joins the passthrough server
• checkConnection, which checks the network status
• configureStartButton, which updates the user interface
• startGame, which starts the game

Despite being part of the student methods section, the first two are done for you. We completed update so that you had an idea for how to implement it in the other scenes. And processData is unnecessary because the clients don’t send any data to the host before the game begins. The later will likely not be true of your game, as most games like to send player names or something similar.

The key thing about update is that you must call receive every animation frame, even if you are not expecting any messages. This is how you update the connection state. While CUGL does provide a callback interface for receiving messages, it suffers from the same problem that input callbacks do - you have to do extra work to sychronize the results to the current animation frame.

Connect to the Server

The first thing to do is to connect to the server. The host scene has a _config attribute that was defined in _init. Read the documentation for NetworkConnection use the alloc method to create a smart pointer for a host connection. Assign this to _network. The call the method open to initiate the connection. You should also immediately call checkConnection().

Check the Connection Status

The function checkConnection should look at the state of the network connection. As the state of the network changes, the state of the scene should change to match. If the network status is NEGOTIATING, the scene status should be WAIT. If the network status is CONNECTED, then the scene status should either be IDLE or START, and it should be IDLE the first time connection it established. Finally, for any network status of DENIED, MISMATCHED, INVALID, FAILED or DISCONNECTED it should disconnect the network connection and set the state back to WAIT. The latter case is also the only time this function should return false.

In addition, when this host is connected for the first time, you should get the room ID. Use the function hex2dec to convert it into a number, and assign the text of the _gameid to this room value (otherwise how else are the players to know where to go). Similarly, you should update the number of players when it changes.

When you have finished this step, you can actually test your application for the first time. Start the application and go the host scene. Do you seen a room number? If so, you are connecting successfully!

Configure the Start Button

Why do we need a function to configure the buttons? The button will be deactivated until the game has a room ID (this is very much not the case now). In particular, until connection with the punchthrough server is made, the button will say “Waiting” and will be deactivated. It is only once that connection is made that it will be activated and changed to “Start Game” (which is what this method does right now).

Changing the text on a button is somewhat tricky. That is because a button does not have a text attribute. It only has two children: up (what it looks like when it is up) and down (what it looks like when it is down). And technically down is optional, as it can just darken up. But up does not have a text attribute either. It has two children: one for the text and one for the image. So changing the text requires that we set the text attribute in the child of a child. To simplify this process we have provided the method updateText.

You should not need to access _network to implement this method. Everything should be clear from the scene state and the current properties of _startgame.

It might seem weird that we are having you implement this method. In a lab that is about networking. But look at where this method is called. It is called in update, because the buttons depend on what is happening on the network and not what the player is doing. This will be quite common in your application.

Start the Game

The host can start is own game simply by setting _status to START. The NetApp class will see this change and swap the scene. But that is not good enough. You want the clients to start as well. How do you do that? By sending a message to the clients that they should start.

The broacast method in NetcodeConnection is very straight-forward. You send a vector of bytes. If you want to send another type of data, you need to first convert it to a vector of bytes using NetcodeSerializer. However, that is not necessary in this assignment. You just need to send an array of bytes.

In fact, because clients know to start when they see the first message from the host, you only need to send one byte. We sent the byte 255. But you can choose whatever you want.

Task 2: The Client

It is now time to connect the client. This time there are five methods in the student methods section, all of which are unimplemented:

• update, which receives the latest update from the server
• processData, which interprets the data from each update
• connect, which connects to the host
• checkConnection, which checks the network status
• configureStartButton, which updates the user interface

Connect to the Host

The client does not connect to the lobby server right away. The player has to enter the room number first. Fortunately, we have written the code to process the text field and get the room number from the user. And the connect method is called by the start button (or when you press enter in the text interface). So all you have to do is set up a connection.

Once again, use an alloc method to create a smart pointer for a client connection. Assign this to _network. You should also immediately call the open method as well as checkConnection().

Check the Connection Status

The function checkConnection is the same as it was for HostScene. Only the logic differs, because ClientScene has a different set of states. A client starts out as IDLE, waiting for the player to enter a room. Once the network connection has been created and the network status becomes NEGOTIATING, the scene status is now JOIN, indicating it is read to join a room. If the network status changes to CONNECTED then the scene status becomes WAIT unless the state is already START. This indicates we are waiting for the other players but ready to go at any time. Finally, for any network status of DENIED, MISMATCHED, INVALID, FAILED or DISCONNECTED it should disconnect the network connection and set the state back to IDLE. The latter case is also the only time this function should return false.

You should remember to update the number of players whenever it changes. This important to see if the network connection is working (or when it is dropped).

Update the Network State

We did not implement update for you this time. But it is pretty much the same as it was for HostScene. Do not do anything unless a network connection is established. If one is established, process the data, check the connection state, and configure the start button (we will talk about the latter one later).

Once you do this, it should be possible to test your connection. Launch an application with your working host on another device, be it laptop or mobile. Then try to connect to it with this one. If both devices note that there are two players then you can be confident in your connection.

Configure the Start Button

This one seems really weird. We understand that we no longer want the start buttons to say “Start Game” after we press it, but why is this function called in update? There are two reasons for this.

First, we want the button to say three different things, depending on the scene status. That means it needs to change whenever update changes the status. At the start, when the scene is IDLE, the button should be active and should say “Start Game” When the scene has status JOIN, we want to deactivate the start button and have it say “Connecting”. This text should change to “Waiting” when the scene status is WAIT.

The second reason for method is very subtle. When we press the start button we immediately switch to JOIN. But we cannot deactivate the start button each time. Why not? It is unsafe for a button listener to process code that deactivates it. This is the same as unsafely deleting from a list that you l oop over. Or for those of you in the introductory course, it is the same as deleting a box2d object in the collision handler. We cannot do this. So we let update handle it for us.

Process the Network Data

The last thing to do is to process messages from the network. Since we are a client, we are waiting for a message - a message to start the game! The function processData receives a byte vector from the network. So as soon as we see our first message we know to start the game. You can check that it is a correct message (one element, with a value of 255) if you want to, but this is not necessary. Once you receive the message, set the status to START. The class NetApp will take care of the rest.

Why is this a race condition? While update is only called once an animation frame, there is no guarantee that processData is called only once. When you invoke recieve, the callback function provided is executed for each message sent that frame. So there could be more than one. In this case, the host could send a start message and quickly chose a color soon after. Both of these messages would arrive to the client at the same time.

Why is this a problem? Because the code to handle a color change is in GameScene (and we have not yet written it). As there is no such code in ClientScene, this message will be dropped. Indeed, there is no way to fix this problem, so do not try. This is exactly why we recommend that you have a single network controller, instead of spreading out the code across multiple scenes like we have done here.

Task 3: The Game

You are almost there. It is now time to play the game. The class GameScene has four methods in the student methods section, all of which are unimplemented:

• update, which receives the latest update from the server
• processData, which interprets the data from each update
• checkConnection, which checks the network status
• transmitColor, which sends the color change to the other players

In the game, the players change the background color. There are multiple ways to do this, but we chose to change the clear color in the application. This is the the color that the application uses to “erases” the screen before starting again.

Update the Network State

You should know how to do this one by now. You do not need to update a start button, but the other elements are the same.

Check the Connection Status

There is a lot less to do here, since we are not trying to connect to the a machine. We are just trying to make sure that our current connection remains live. Remember to update the number of players as they change. You should not ever receive a DENIED, MISMATCHED, or INVALID at this point. But if you get FAILED or DISCONNECTED, you should disconnect the network connection and quit (so that NetApp takes us back to the menu screen).

Oh, and if you do quit, it is very important that you change the background color back to its normal color, which is the web color "#c0c0c0". See the documentation for how Color4 objects work.

Transmit a Color

When you transmit a color, you should first set the clear color. That is because you never receive your own messages, so you cannot rely on processData to do all the work. Then transmit the color at a vector of 4 bytes. As each attribute of Color4 is a byte, this should be easy. The order (r,g,b,a or a,b,g,r) does not matter as long as you are consistent.

Process the Network Data

To complete the game, you need to write code to process an incoming message. Whenever you receive a 4 element byte vector, turn it back into a color and set the clear color to that value. When you do this, you are done with the lab. Congratulations.

What is the problem? The network interface guarantees that messages sent from a single sender are sent in order. But there is no guarantee on the order of messages from two different senders. Suppose two players both press a color at exactly the same time. They both set their devices to the color they pressed and transmit this out. Since no device receives its own message, the next message they each receive is the color of other the device. So they swap colors and are now in inconsistent states.

You will not be able to fix this problem with the way this lab is organized. Do not try. We will talk about how to address this problem during lecture.

Submission

Due: Mon, Feb 19 at 11:59 PM

This assignment should not be as hard as the previous one. The code is fairly straightforward and we have done a lot of the heavy lifting for you. The only challenge is the testing setup.

When submitting the assignment do not submit the entire project. That is too large and CMS will crash under the load. Instead we want you create a zip file containing the files:

• NLHostScene.cpp
• NLClientScene.h
• NLGameScene.cpp
• readme.txt

The file readme.txt should tell us if you worked in Visual Studio or in XCode for the desktop version. You should also tell us how you tested your network connection (two laptops, two mobile devices, or one of each).

Submit this zip file as lab3code.zip to CMS