Due Date: 16 December 1998

Name of group: MIA

Team members: Jason Howes and Ted Bonkenburg

Task: Unspecified Application – SPOT – Slide Presentation Over Telephony

Editor's Note: Things broke down somewhat during our demo. Right afterwards we realized that it was due to the fact that we had set Record to record from the CD Device instead of the microphone for when we were recording out final credits presentation (we had music with it). After we set it back to what it was, things started working perfectly again as they had before the demo. We realize that you believed us that everything worked, we just feel better providing some sort of justification, because we had previously tested over and over again and it works fine. We would be perfectly happy to demo again if it is so desired.

The original plan for our project remained fairly consistent throughout, although it was slightly refined through each report. The original report had our client running as an applet spawned by a server, but we realized that the client would require access to native code. Therefore, we soon converged upon the following description:

We feel that we accomplished much beyond our original specification, while at the same time failing in some areas. We will first start with our shortcomings to get them out of the way, and then move on to the good things we accomplished.

Failure:

Due to lack of time, we were unable to add all of the features that we had in mind for the lecture client. Most of these features are actually ideas we thought of beyond the scope of our original intentions, but we would have liked to implement them. The one feature that we did not add that was in our original specification was the notion of having two methods of viewing, slides synched with audio and free viewing while audio is still playing. Instead, we allow the client to pause the audio and move around the lecture as he or she pleases before resuming. An additional victim of this style was the idea of having a "current slide" visible in the client interface and having a quick "jump to slide" option. Instead, we have the "jump-to-topic" feature, which is actually an improvement. A second feature that we wanted to achieve but didn’t have time was an extremely nice looking GUI interface for both the SPOT client and the Lecture Wizard (our lecture creation tool). Although the current GUI is adequate, we would have liked more time to make a nicer Java Swing version.

Perhaps our greatest defeat was the fact that we did not manage to use the total integration of all the parts of the telephony system. Although we had heard that almost all of the elements could be made to work together, we decided it would be safest for our demo to use only the signaling/data-exchange portion. Although some of these issues were out of our hands, our main goal for this project was to get SPOT working under the complete MIA telephony system.

Victory:

At the same time, we feel victorious because we have managed to meet or exceed almost every one of our goals. We have created a skeleton web infrastructure for our SPOT client. We have built a lecture server that can support more than one client listening to audio over telephony at the same time. We have also built a fully functional client application, and the tools to create lectures on top of that. Finally, we have run them all together, over signals/data-exchange. Additionally, the audio is very clear and fresh and the slide presentation works almost flawlessly.

The work that we did can be divided into x main areas:

The first priority of our project was to make sure that we would be able to get PowerPoint automation working, or that it was at least possible. Once we became convinced that it was possible, we knew we could accomplish our goals, since good telephony would provide an excellent audio stream source. We also knew that we had an excellent team in MIA, so we were positive the audio would work out. The PowerPoint automation proved difficult, but not impossible. Once we had that figured out, we made a set of PowerPoint automation classes that had an easy to use interface. We decided to package this up with a nice interface because we realized that we would need the automation for both the client application and the tool to generate lecture presentations.

When it became clear that no team was taking the initiative of creating a package to read and write audio to/from the sound card, we talked with management and it was decided that we would take care of this service. In order to do this we did what was described in our last report and made the JAudio package, which we put on "cvs" and advertised for other groups to use. We soon realized that JAudio was too low a layer for the conventional needs of ourselves we decided to create higher level threads based on JAudio. When it became apparent that most other groups would also need similar services, we decided to make the three threads we created as open ended as possible and spent many hours testing them (we assumed others would use them, so we tested them rigorously). We also created heavily commented JavaDoc for these threads (see interfaces provided) and put everything on "cvs". Our manager knew about them, and between ourselves and our manager we managed to let every applications group know of their existence, although most seemed disinterested (signals and one or two other groups did end up using them). They consisted of:

AudioThread - Wrapped around either an InputStream or a Connection object and wrote all incoming data (sound) to the audio card.

RecordThread - This constantly recorded from the audio device and send the naked data to either an OutputStream or a Connection object.

AudioTransferThread - This read data from a RandomAccessFile and sent to either an OutputStream, a Connection (which used Connection.waitForWrite() for flow control), or our own GConnection object using a simple form of flow control. The GConnection was a datagram connection object that was implemented by us and also made available to all of the other groups. It was used for early testing of our lecture system, and once signals and data-exchange were ready, their code dropped neatly in its place.

The lecture wizard was a utility for creating lecture presentations. It used both the RecordThread and the PowerPoint packages as well as an AWT gui. The lecture wizard shows the given PowerPoint presentation and uses the RecordThead to record audio to a ".nad" file (which stands for Naked Audio Data). As the .nad stream is written, whenever the author changes slides or unpauses on a slide other than the slide that was paused on, the current offset in the data stream is written to a ".wad" file (which stands for Where's All the Data). Besides pausing recording and resuming, the author also has the ability to enter different topic. These are stored along with slide and audio offsets in the ".wad" file. During presentation playback, the viewer can easily jump to a particular topic.

The lecture client made use of the PowerPoint automation package and both the AudioThread and the AudioTransferThread. It actually had two pluggable ways of playing back presentations, either local or remote. If local was used, an AudioTransferThread was piped into an AudioThread through streams. The audio offset in AudioTransferThread was controlled by the user clicking the forward, back, and other user interface buttons on the client. The local presentation even had a different look to the client gui, since it didn't offer the choice of what server to connect to, etc. In remote mode, the client connected to a lecture server. The AudioThread was then receiving directly on either a signals Connection object or our own GConnection object. The client application allowed the user to watch a presentation, or actively change slides, in which case the audio would be synched with the user's new slide.

We did most of the testing and debugging of the client using our own GConnection, so that when it came time to integrate with the other groups are code was already very stable. This made integration much easier and very quick.

The lecture server was designed as a multi-threaded server and can thus serve more than one client at a time. Since it doesn't need to use the audio device (it only reads from a file using AudioTransferThread) it can serve multiple clients, although the most it was ever tested on was two signal Connections at once (which seemed to work fine!). It also serves the PowerPoint presentation file if necessary and controls the changing of a client's slides during a presentation.

Although we have managed to get the PowerPoint automation working, there are still a few problems. It is not possible to present a PowerPoint slide show without fully starting PowerPoint, so the actual PowerPoint application resides on the taskbar while our lecture browsing client is running. We have control over it, but the best we can do is minimize it and attempt to keep it out of the way. We had hoped that we would be able to use PowerPoint’s functionality transparent to the user. In fact, we envisioned a GUI with PowerPoint running embedded within it. However, this is a minor problem, since it does not really affect the functionality of our client application. A second problem is due to a bug in COM automation itself that is noted on the Microsoft web site in the "Support and Knowledge Base". Once PowerPoint is started, it is impossible to shut it down from within our client. Therefore, when exiting our client, the user must explicitly close PowerPoint. Again, neither of these problems affects the operation of our client.

In addition to the above technical problems with COM automation, too much of our time in general was spent researching various technologies in order to find the best ones to use. For example, JDirect does not support recording and MCI isn’t powerful enough for use in the JAudio package. Nevertheless, time was spent in researching these directions. When the technologies were finally decided on, Java itself proved to be a barrier since much of the stuff we are using involves native calls and a large portion of our time was spent creating wrappers and other classes to deal with this.

Although we have had no major problems coordinating with others in this project, we did however come across many minor hassles that in the end may have served to improve our project. This is due to a few main reasons:

1) Because we did the JAudio package, the other groups began to rely on us as their "audio source". In an attempt to help others, we lost valuable time that we could have put towards our own project. This was only a minor inconvenience, however, because we felt that the aid we were giving to others would help MIA out as a whole, and force us to create increasingly efficient and extensible releases. In the end, we used all of it ourselves, either for testing or in our final implementations.

2) As an application, we relied on the core teams (signaling, data exchange, gateway, etc.), but we planned our timeline for this project so that we did everything but the telephony aspect first. This gave our core teams time to develop and finalize their interfaces so that when we did finally use them we were pretty sure that they would no longer change. As we were getting ready to test our application, it appeared that the other teams might not manage to create a workable telephony system. We therefore spent a great deal of time implementing our own "Connection" object, complete with flow control and buffering. With this we managed to test our application extensively before dropping in the final signaling and data-exchange code. During this testing period we were able to eliminate bugs that may have been harder to catch using someone else’s code.

We would like to note that our management team worked very hard to pull the project together. They were very involved and attempted to learn and teach the whole system so that the applications groups would be able to pull off working demos. They remained very approachable and were always willing to explain what is going on with the other groups or explain the entire system to us. Finally, they also seemed good at referring us to other members of the group who may have been more experienced in a given area, thus encouraging teamwork among MIA members.

This project was a great learning experience for us. We were forced to learn a number of new technologies in a limited amount of time, and at the same time to be proficient in them. The following is a small listing of just some of the things we have learned in tackling this intensive project:

1) We have learned many Java language details. We have now had substantial exposure to serialization, streaming, threading, class organization, synchronization issues, and using Java as efficiently as possible. We have also begun to realize just how slow Java really can be, especially when used for a technology such as internet telephony that demands high performance and consistent throughput. To combat some of the shortcomings of Java we leveraged the Java JNI architecture for calling native functions from within Java.

2) We also learned how to use CVS better, and some other technologies, such as COM, using jactivex, and mutimedia stuff (audio). One of the things we learned after much experimentation was the necessity of flow control. When serving audio data from a file, if there is no flow control then it is impossible to stream audio data across the network, and we learned this after much trial and error.

3) In addition to learning many new technologies, we also learned how to approach a very large-scale project in an equally large group of people (at least for us). Although this might not be large as far as a typical software-engineering project may go, this was perhaps the largest group project that we have ever worked on at Cornell University Computer Science.

a) We have learned the Java JNI and the Windows™ waveform API in building the JAudio package.

b) We have learned how to automate PowerPoint using Java COM. We have become more familiar with the COM technology.

c) We have learned much about the overall picture in this telephony project, and have focused on the only interfaces that we will be using, the "signaling" interface.

If we were to do this differently, we would not do the project in Java. Since our project deals with both native audio and Microsoft COM automation, it would have been much easier to use C++. We have spent too much of our time attempting to access native or vendor-specific code when this would have been less complex in C++. Additionally, the poor performance of Java really began to show when we started using the audio portions of our project. Although Java usually provides a much easier and quicker coding environment, the fact that we required much native code and needed additional performance for our application make Java a loser. This is not to say that Java isn’t a suitable tool for some of the other areas, such as signals or some other applications.

Class JAudioBlock

public JAudioBlock(byte data[]) throws JAudioException

public JAudioBlock(int desiredLength) throws JAudioException

public void waitUntilFinished() throws InterruptedException

public byte[] getData()

Class JAudioDevice.

public JAudioDevice(int deviceID) throws JAudioException

public void close() throws JAudioException

public boolean inUse(int deviceID)

public JAudioIn in;

public JAudioOut out;

Class JAudioIn

public void addBlock(JAudioBlock block) throws JAudioException

public void start() throws JAudioException

public void stop() throws JAudioException

Class JAudioOut

public void putBlock(JAudioBlock block) throws JAudioException

public void start() throws JAudioException

public void stop() throws JAudioException

Class mia.apps.gspot.shared.AudioThread

java.lang.Object

|

+----java.lang.Thread

|

+----mia.apps.gspot.shared.AudioThread

public class AudioThread

extends Thread

This class is used for sending audio data to the audio device, either

directly from a connection or from an InputStream.

The AudioThread won't try reading from the InputStream/Connection

until StartPlaying() is called, so it can be initialized in advance.

Note:The AudioThread has a granularity of sample size bytes. That

means that the last sample size bytes may never play. The suggested

value for sample size is 128 bytes.

public AudioThread(InputStream in,

int sampleSize,

int device) throws JAudioException

Creates a new AudioThread which dumps the bytes from a given

input stream to the sound card.

Parameters:

in - A standard InputStream object.

sampleSize - The size of each audio buffer (recommended

>= 128)

device - The audio device to use (device 0 is standard)

Throws: JAudioException

If AudioThread is unable to start an audio device.

public AudioThread(InputStream in,

int sampleSize) throws JAudioException

Creates a new AudioThread which dumps the bytes from a given

input stream to the sound card. This will automatically search for

an audio device to use.

Parameters:

in - A standard InputStream object.

sampleSize - The size of each audio buffer (recommended

>= 128)

device - The audio device to use (device 0 is standard)

Throws: JAudioException

If AudioThread is unable to start an audio device.

public AudioThread(Connection in,

int sampleSize,

int device) throws JAudioException

Creates a new AudioThread which dumps the bytes of the given

telephony connection to the sound card.

Parameters:

in - The connection object on which to read the data.

sampleSize - The size of each audio buffer (recommended

>= 128)

device - The audio device to use (device 0 is standard)

Throws: JAudioException

If AudioThread is unable to start an audio device.

public AudioThread(Connection in,

int sampleSize) throws JAudioException

Creates a new AudioThread which dumps the bytes of the given

telephony connection to the sound card. This will automatically

search for an audio device to use.

Parameters:

in - The connection object on which to read the data.

sampleSize - The size of each audio buffer (recommended

>= 128)

device - The audio device to use (device 0 is standard)

Throws: JAudioException

If AudioThread is unable to start an audio device.

public void run()

This constantly attempts to read a block of data from the DataIn

stream and send it to the sound card.

Note:The granularity will be based on the given sample size. i.e.,

when the thread is terminated, up to sample size bytes may never

be sent to the audio device. Try sample size of 128 bytes.

Overrides:

run in class Thread

public boolean StartPlaying()

Starts playing the data.

Returns:

true if success, false if failure.

public boolean StopPlaying()

Stops playing and cleans up the thread.

Returns:

true if success, false if failure.

public boolean SetMute(boolean muteVal)

Mutes/Unmutes the sound based on the input.

Parameters:

muteVal - Set to true to mute, false to unmute.

Returns:

true if success, false if failure.

public boolean Mute()

Mutes the playback.

Returns:

true if success, false if failure.

public boolean Unmute()

Unmutes the playback.

Returns:

true if success, false if failure.

public boolean GetMuteState()

Gets the current mute state.

Returns:

true if paused, false if not paused.

public int GetNumDevices()

Gets the number of audio devices on the system.

Returns:

The number of audio devices.

public int GetOffset()

Gets the current offset in the audio stream seen so far (in bytes).

Returns:

The current offset in bytes.

Class mia.apps.gspot.shared.RecordThread

java.lang.Object

|

+----java.lang.Thread

|

+----mia.apps.gspot.shared.RecordThread

public class RecordThread

extends Thread

This class is used for recording from the audio device and outputs either

to an OutputStream or directly to a Connection object.

public RecordThread(OutputStream output,

int sampleSize,

int device) throws JAudioException

Creates a new RecordThread which dumps the bytes to the given

OutputStream.

Parameters:

output - An OutputStream for the bytes of recorded data.

sampleSize - The size of each audio buffer (recommended

>= 100)

device - The audio device to use.

Throws: JAudioException

If RecordThread is unable to start an audio device.

public RecordThread(Connection connection,

int sampleSize,

int device) throws JAudioException

Creates a new RecordThread which dumps the bytes to the given

Connection.

Parameters:

connection - An mia Connection to send the bytes of

recorded data to.

sampleSize - The size of each audio buffer (recommended

>= 100)

device - The audio device to use.

Throws: JAudioException

If RecordThread is unable to start an audio device.

public RecordThread(Connection connection,

int sampleSize) throws JAudioException

Creates a new RecordThread which dumps the bytes to the given

Connection. This will automatically search for an audio device to

use.

Parameters:

connection - An mia Connection to send the bytes of

recorded data to.

sampleSize - The size of each audio buffer (recommended

>= 100)

Throws: JAudioException

If RecordThread is unable to start an audio device.

public RecordThread(OutputStream output,

int sampleSize) throws JAudioException

Creates a new RecordThread which dumps the bytes to the given

OutputStream. This will automatically search for an audio device

to use.

Parameters:

output - An OutputStream for the bytes of recorded data.

sampleSize - The size of each audio buffer (recommended

>= 100)

Throws: JAudioException

If RecordThread is unable to start an audio device.

public void run()

The run of thread. This constantly waits for recording of a block to

finish and then sends it to the output stream. If pause is on, then

the recorded blocks are discarded. This exits when

RecordThread.Stop() is called.

Overrides:

run in class Thread

public boolean StartRecording()

Starts recording.

Returns:

true if success, false if failure.

public boolean StopRecording()

Stops recording and cleans up the thread.

Returns:

true if success, false if failure.

public boolean SetPause(boolean pauseVal)

Pauses/Unpauses recording based on boolean input.

Parameters:

pauseVal - Set to true to pause, false to unpause.

Returns:

true if success, false if failure.

public boolean Pause()

Pauses the recording.

Returns:

true if success, false if failure.

public boolean Unpause()

Unpauses the recording.

Returns:

true if success, false if failure.

public boolean GetPausedState()

Gets the current pause state.

Returns:

true if paused, false if not paused.

public int GetNumDevices()

Gets the number of audio devices on the system.

Returns:

The number of audio devices.

public int GetOffset()

Gets the current offset in the audio stream (in bytes).

Returns:

The current offset in bytes.

Class mia.apps.gspot.shared.AudioTransferThread

java.lang.Object

|

+----java.lang.Thread

|

+----mia.apps.gspot.shared.AudioTransferThread

public class AudioTransferThread

extends Thread

This class is used for reading audio data from a .NAD file to be sent

either to an output stream or over telephony through a connection.

Note: This class includes a crude attempt at flow control, so that data is

not sent to the connection too quickly or slowly. It also provides some

buffering. Note: If EOF is reached, the thread goes into a "finished" state

until setoffset is used.

public AudioTransferThread(RandomAccessFile nadFile,

OutputStream output,

int sampleSize,

long bufferTime) throws Exception

Creates a new AudioTransferThread which dumps the bytes from

a file to the given OutputStream.

Parameters:

nadFile - The .NAD audio file which is to be read.

output - An OutputStream for the bytes of audio data.

sampleSize - The size of each audio buffer (MUST BE

DIVISIBLE BY RATE/1000 [i.e. by 8])

bufferTime - The amount of data to keep buffered on the

output stream in ms

Throws: Exception

If the sampleSize given is not divisible by the RATE/1000

public AudioTransferThread(RandomAccessFile nadFile,

Connection connection,

int sampleSize,

long bufferTime) throws Exception

Creates a new AudioTransferThread which dumps the bytes of the

given .NAD file to the given Connection.

Parameters:

nadFile - The .NAD audio file which is to be read.

connection - A signaling connection.

sampleSize - The size of each audio buffer (MUST BE

DIVISIBLE BY RATE/1000 [i.e. by 8])

bufferTime - The amount of data to keep buffered on the

output stream in ms

Throws: Exception

If the sampleSize given is not divisible by the RATE/1000

public AudioTransferThread(RandomAccessFile nadFile,

GConnection connection,

int sampleSize,

long bufferTime) throws Exception

Creates a new AudioTransferThread which dumps the bytes of the

given .NAD file to the given GConnection (object which simulates

signal's Connection)

Parameters:

nadFile - The .NAD audio file which is to be read.

connection - A GConnection object.

sampleSize - The size of each audio buffer (MUST BE

DIVISIBLE BY RATE/1000 [i.e. by 8])

bufferTime - The amount of data to keep buffered on the

output stream in ms

Throws: Exception

If the sampleSize given is not divisible by the RATE/1000

public void run()

The run of thread. This constantly reads from the inputFile and

then writes to the OutputStream/Connection. It does this at a rate

so that not too much data is buffered on the receiving side.

Overrides:

run in class Thread

public boolean StartTransferring()

Starts transferring the file.

Returns:

true if success, false if failure.

public boolean StopTransferring()

Stops transferring and cleans up the thread.

Returns:

true if success, false if failure.

public boolean SetPause(boolean pauseVal)

Pauses/Unpauses playback based on boolean input.

Parameters:

pauseVal - Set to true to pause, false to unpause.

Returns:

true if success, false if failure.

public boolean Pause()

Pauses the recording.

Returns:

true if success, false if failure. returns false if it was already

paused.

public boolean Unpause()

UnPauses the recording.

Note: The connection object requires continuous writes at a

certain rate. When using a connection, don't use pause!

Returns:

true if success, false if failure. If it wasn't paused, then this

returns false.

public boolean GetPausedState()

Gets the current pause state.

Note: The connection object requires continuous writes at a

certain rate. When using a connection, don't use pause!

Returns:

true if paused, false if not paused.

public static int GetNumDevices()

Gets the number of audio devices on the system.

Returns:

The number of audio devices.

public int GetOffset()

Gets the current offset in the audio stream (in bytes).

Returns:

The current offset in bytes.

public boolean WaitForOffset(int offset)

Waits for the current offset to be exceeded.

Note 1: The calling thread will be put to sleep and woken up

when one of three conditions is met: 1) The current offset in the file

has exceeded the given offset.

2) The function SetOffset() is called.

3) StopTransferring() is called.

Note 2: Only one thread may call WaitForOffset!!

Parameters:

offset - The offset to wait for (from the beginning of the .nad

file)

Returns:

true if condition <1> was met, or false otherwise.

public void SetOffset(int offset)

Sets the current offset in the audio stream from the beginning.

Class mia.apps.gspot.shared.GConnection

java.lang.Object

|

+----mia.apps.gspot.shared.GConnection

public class GConnection

extends Object

implements Connection

This class implements a Fake signaling data connection so that we can be

sure our stuff works, even if the whole telephony thing goes down in

flames :-) Note: If you are using the non-local version, then you must

specify the buffer size in advance. Either way, you must always use the

*same* buffer size. That is, if you attempt to read 512 bytes of data at a

time, then you must always write 512 bytes of data, etc. This is best used

with the RecordThread/AudioTransferThread on the sending side and the

AudioThread on the receiving side with the buffer sizes set the same!

public GConnection(int bufsize)

Creates a local connection --> Acts as a two-sided connection

stream.

This can be used for local testing. An

AudioTransferThread/RecordThread and an AudioThread can all

be given the single, same connection object instantiated with this

constructor.

Parameters:

bufsize - The buffer size to be used for reads and writes.

public GConnection(InetAddress remoteAddr,

int port,

int bufSize)

Creates a remote connection --> Everything is sent to the given IP

address.

This can be used as a fake data-exchange/signaling connection. A

GConnection must be established on both data endpoints, given

the address of the remote endpoint. The "bufsize" must be the

same on both endpoints.

Parameters:

InetAddress - The remote machine's address.

port - The reomte machine's port.

bufsize - The buffer size to be used for reads and writes.

Should be the same as sampleSize given to

RecordThead/AudioThread.

public int read(byte b[],

int offset,

int length) throws IOException

This is the read function to use.

This function blocks. Make sure length is the same as buffersize,

or you lose data!

Parameters:

b - The byte buffer to fill

offset - The offset at which to copy data.

length - The amount of data to copy (Should be

sampleSize)

public void write(byte b[],

int offset,

int length) throws IOException

Best function to use for writing (non-blocking).

This function writes data to the connection. It is non-blocking, so

some form of flow-control should be used. (Both RecordThread

and AudioTransferThread do some form of flow-control).

Parameters:

b - The byte buffer to write.

offset - Should always be set to 0!

length - Must be the same length as any length requested for

read!

close

public void close()

Closes the connection.

We used mainly the signaling interfaces, which also served as an interface to data exchange.

Connection.close();

In further attempts at undergoing this project, we would advise the course staff to not encourage students to go the Java route. Instead, we would encourage them to advise certain students and teams to use C or C++, especially those who wish to use COM and the lower level teams such as the Gateway and especially data exchange.

We would also advise that the course staff give away the routines that we worked on (not necessarily our code, but similar) for recording and playing sound. This proved to be a big pain that should not have been necessary, since we feel that the project should be more focused on the transfer of data and not it’s capture or playback.

Finally, we advise the course staff to make two demo dates. The first would be a preliminary demo, perhaps one month in advance, in which everything is expected to roughly work. The final demo would be a final demonstration of all that was touched up between the first and final demos. The first demo would provide the staff the ability to give direct advice to the students based on the work they had done, and it would spur the students to make a better project. Also, one of the puzzles should be dropped and the final project should be emphasized more, since much more can be learned from this project.

8. References Used: