You must work either on your own or with one partner. If you work with a partner you must first register as a group in CMS and then submit your work as a group. Adhere to the Code of Academic Integrity. For a group, “you” below refers to “your group.” You may discuss background issues and general strategies with others and seek help from the course staff, but the work that you submit must be your own. In particular, you may discuss general ideas with others but you may not work out the detailed solutions with others. It is not OK for you to see or hear another student’s code and it is certainly not OK to copy code from another person or from published/Internet sources. If you feel that you cannot complete the assignment on you own, seek help from the course staff.
Completing this project will solidify your understanding of numeric arrays, character arrays, cell arrays, and objects. You will also practice reading, parsing, and writing text files, as well as learn a bit about digital audio.
As usual, use only the functions and constructs learned so far in the course or discussed in this assignment.
For the past forty years, electronic musical instruments have used a protocol called MIDI (Musical Instrument Digital Interface) to communicate, separating the act of playing a note from the production of its sound. In this project, you will write a program to interpret MIDI data saved in a file and synthesize audio to be played back through your computer’s speakers. Your program will also be capable of transforming certain properties of the song, showing off the flexibility of MIDI compared to recorded music.
MIDI data can be seen as a sequence of events. For our purposes, each event has the following properties:
In order to synthesize sound in Matlab, you will first reexpress this information in a slightly different form. Rather than having separate events for pressing and releasing a key, it would be convenient to have a single event that specified the note’s duration. Instead of note numbers, we would like to know the frequency of the fundamental pitch (so note number 69, an “A”, would have a frequency of 440 Hz). It would also be helpful to have times expressed in seconds, rather than beats, and to normalize the loudness of a note to be between 0 and 1. Finally, notes from different instruments could be managed in separate lists, rather than interleaving them all together.
We have provided a Matlab class, Note, to store these derived properties for a single note played on an instrument. Read the code in Note.m to understand which properties are available and how to invoke the constructor.
NoteYou may be starting this project before we have discussed Objects and Classes in detail in lecture, so here are the basics you need to know for this project:
Note by constructing it: n= Note(0, 1, 440, 1) creates a new Note object, referenced by variable n, that starts at time 0, is held for 1 second, has a pitch of 440 Hz, and is played as loud as possiible (amplitude 1).Note by using a period (.) between the variable name and the property name. For example, n.frequency is how to refer to the frequency of the Note object referenced by variable n. You can both read and write properties, just like array elements.Note is a “handle class,” meaning that variables store references to Note objects, rather than storing the object data directly. This means that when you copy the contents of a variable, both variables will refer to the same Note object, so property changes assigned through one variable will affect properties read from the other (this is probably only relevant in transformNotes()). To get a new, independent Note object, call its constructor.Note, you cannot iterate over them using for n = notes (n will be a cell rather than an object handle). So iterate using indices instead (notes{k}.frequency works fine). Similarly, we recommend appending using indices rather than using concatenation (Matlab’s concatenation rules are unfortunately inconsistent when it comes to objects); for example, if notesByCh is a cell array of cell arrays of Note objects and length(notesByCh) ≥ ch, then notesByCh{ch}{end+1}= n will append the Note referenced by variable n to the list of notes for channel ch (the end keyword is a shortcut for length(notesByCh{ch}); you may use it on this project).MIDI data for this project is provided in text files that look like the following:
# Tempo [bpm] 120 # Beats Channel Event Note Velocity 0 1 note-on 64 120 1 1 note-off 64 0 2 1 note-on 62 110 3 1 note-off 62 0
The first (non-blank, non-comment) line specifies the tempo in beats per minute. All other lines represent a MIDI event, the properties of which are given in the order above and separated by spaces. Blank lines, or lines beginning with the ‘#’ character (indicating a comment), should be ignored.
Important words and numbers (“tokens”) are separated by spaces. It’d be convenient to work with lists of these tokens, rather than arrays of characters, so we can quickly jump to the ones we’re interested in. Newer versions of MATLAB provide a function strsplit() to do this, but since you might have an older version (and because it’s good practice), we’ll be writing our own.
Implement the following function as specified:
function tokens = tokenizeLine(line)
% Split a string into words separated by whitespace.
% `line` is a 1-row character array containing plain text (but no newline
% characters). Upon return, `tokens` is a 1-row cell array where each
% cell contains a non-empty character vector corresponding to a
% whitespace-delimited token from `line` (the order of cells matches the
% order of token occurrence in `line`). Multiple consecutive whitespace
% characters should be treated as a single delimiter.
%
% Example: tokenizeLine('Hello world! 1112') returns
% {'Hello', 'world!', '1112'}For this project, it is sufficient to define “whitespace” as meaning space characters (but check out the isspace() function for maximum generality).
It’s time to get into the habit of formally testing your code as soon as (or even before) you write it; we’ll have even more to say about this in the Testing section below. The project archive contains the file tokenizeLineTest.m with a “unit test” for the tokenizeLine() function. You need to write at least two additional tests of this function in the locations indicated in that file. Each test case should try to stress your function in a new way, so you catch as many potential mistakes as possible. Some stressful ideas include: leading or trailing spaces, single-character tokens, etc. Ensure that your function passes all of your tests.
A single MIDI instrument is often limited in how many notes it can play simultaneously; this is called its polyphony. Our synthesizer in this project will be monophonic – it can only make sound for one note at a time (no chords). When parsing MIDI data, if a note is played (“note-on”) while another note is still playing (no “note-off” yet for the previous note), then that second note should be ignored. This limitation only applies to a single instrument (channel); overlapping notes from different instruments should be preserved.
Write a function, parseMidiLog(), to read MIDI data in the above format and produce several sequences of Note events, one for each channel. The output is a cell array of cell arrays of Notes (as well as the tempo, which is just a double).
function [bpm, notesByCh] = parseMidiLog(filename)
% Extract tempo and notes from a file of MIDI events.
% Reads the text file specified by `filename`. The first content line
% should specify the tempo in beats-per-minute, which is returned in `bpm`.
% Subsequent content lines represent MIDI events (ordered by time) with the
% following fields:
% <time [beats]> <channel> <event type> <note> <velocity>
% Blank lines and lines starting with '#' are ignored.
% Returns `notesByCh`, a 1D cell array of nested 1D cell arrays containing Note
% objects, such that `notesByCh{ch}{k}` is the `k`th note played on the `ch`th
% channel. Each channel's instrument is monophonic, so keys pressed while
% another key is held down are ignored.To convert a MIDI note number d to a frequency, use the following formula from the “MIDI Tuning Standard” (this would be a good candidate for a subfunction):
Hint: For each channel, have a way to save note properties (except for duration) when a new “note-on” event appears for that channel, then ignore other events on that channel until a “note-off” event appears for the same note, at which point a Note object can be completed with the duration.
When playing audio on your computer, your sound card expects it in a format called “pulse code modulation” (PCM). This specifies the intensity of the sound’s pressure wave at many points in time, called “samples.” CD-quality audio provides 44,100 samples every second, but we’re going to be less ambitious and only synthesize 8,192 samples per second (this is Matlab’s default sampling frequency). In Matlab, each sample’s value is a number between -1 and 1.
To play PCM audio in Matlab, we use the sound() function and pass it a vector of sample amplitudes. For example, to play a pure tone (sine wave) of 440 Hz for 1 second, we could do the following:
t= linspace(0, 1, 8192); % Generate the time [s] of each sample
s= 0.5*sin(2*pi*440*t); % Store the value of a half-height sine wave at each sample time
sound(s) % Play the audio through your speakersTry this out to make sure your speakers are working. You can also plot the data in s (though you will need to zoom in horizontally to make out the wave).
Your next task is to convert a sequence of Note events (for a single instrument) into a vector of PCM data so that you can hear what the notes sound like—this is called “sound synthesis.” Implement the following function:
function wav = synthesize(notes, ch)
% Generate PCM samples for a sequence of Notes in 1D cell array `notes`.
% Samples (between -1 and 1) are returned in `wav` with a sample rate of
% 8192 Hz and the first sample corresponding to a note timestamp of t=0s.
% `ch` indicates the channel number, which may affect the character of the
% sound.A few remarks:
Test your function in stages by passing its results to the sound() function. First, give it a cell array consisting of a single Note, analogous to the A-440 example above. Then pass it two Notes whose timestamps are separated by a gap of silence. Finally, try playing back one channel of parsed MIDI data. Don’t worry if the audio sounds “harsh;” we’ll address that later.
With recorded music, it is very difficult to change the tempo or to transpose the key (you can change both at once by playing it faster or slower, but changing them independently is tricky). But with MIDI it is easy! Implement the following function to transform a sequence of Notes into one with a different tempo in a transposed key:
function out = transformNotes(notes, tempoMult, keyShift)
% Change tempo and key for one channel of notes.
% `notes` is a 1D cell array of Note objects, copies of which are stored in
% the 1D cell array `out` with their properties transformed according to
% the tempo multiplier `tempoMult` (>1 yields a faster tempo) and a pitch
% transposition of `keyShift` semitones (>0 yields a higher pitch). If
% `notes` is empty, then `out` is an empty cell array.A few tips and clarifications:
Notes are “handle objects,” so copying them to a different array and changing their properties WILL affect the object in the original array. To create new, “decoupled” copies of Note objects, you must call its constructor.After a user has transformed a song to their satisfaction, they should be able to save the result as MIDI data so that it can be played by other synthesizers that may not have a transformation feature like ours. To support this, implement the following function:
function writeMidiLog(notesByCh, bpm, filename)
% Convert Note objects to MIDI events and write them to a file.
% `notesByCh` is a 1D cell array (one cell per channel) containing 1D cell
% arrays of Note objects, such that `notesByCh{ch}{k}` is the `k`th note played
% on the `ch`th channel. Each Note is converted to a pair of events: one
% note-on and one note-off. `bpm` (in beats-per-minute) is the tempo to
% use when converting times to beats. Events for all channels are
% interleaved and written in timestamp order to the file specified by
% `filename`.A lot needs to happen in this function, so a few hints are in order:
Notes to a pair of MIDI events (the template file recommends a subfunction for doing this). Store all events for all channels in a single cell array, and store their timestamps in a separate numeric array.log() function computes a natural (base-e) logarithm, but you can divide by log(b) to convert the result to some other base b. Remember that note numbers and velocities must be integers, so round accordingly.You will now write an interactive program for working with MIDI data. We provide you with the basic structure in MidiPlayer.m, which consists of a while-loop that prompts the user for a command. Commands include:
To mix multiple instruments for playback, simply add up all of their sample vectors (note that some may be shorter than others). Observe that when you do this, the summed samples might become larger than 1 or smaller than -1. If they do, you should “normalize” the samples by dividing them by the largest sample magnitude. Do not normalize a song if its mixed samples do not exceed the range [-1,1].
By decomposing each command of our application into a separate function, we’re able to test small pieces of our code in isolation, making it much easier to pin down errors. This is known as unit testing, since each test only exercises a small piece, or “unit,” of our code with respect to its specifications. You should take advantage of this situation by writing unit tests for each function in this project (and when designing code from scratch in the future, you should strive to make it as unit-testable as possible).
Write your tests in a script, just like tokenizeLineTest.m, calling your functions with small inputs and comparing their outputs with known-correct answers. Use the error() function for reporting when these comparisons fail. You have seen many examples of such tests in Matlab Grader! Run your test script after every change to make sure you don’t break things once you get them working. You do not need to submit this script, but when asking for help from consultants, you should show them which of your tests is failing so they know what you are expecting to achieve.
Once you’re confident that your individual units work, it’s time for integration testing. We’ve provided several sample MIDI log files for you to try out in MidiPlayer:
Do not attempt to run MidiPlayer until all of your unit tests are passing! Consultants should ask to see your unit tests before assisting you with this script.
There are many ways to improve our synthesizer, but this project is already long enough. For additional practice, though, consider implementing one of the following optional enhancements (all involve modifying your sampleNote() subfunction):
The synthesized audio sounds harsh because the amplitude abruptly changes from and to zero when each note starts and stops. It would sound much better if there were a gradual transition. It might also sound more realistic if the amplitude gradually decayed after a note is first struck. You can achieve these effects by multiplying the samples by an “envelope” – a function with values between 0 and 1 that dampens the sound volume at different times.
A standard way to parameterize an envelope is by breaking it into four phases: attack (a fast but smooth transition from 0 to 1), decay (a slow decrease from 1), sustain (a value above 0 that the decay levels off to), and release (a transition back to 0 after the note has been released). Since sampleNote() isn’t allowed to produce samples after the end of a note, we will have to settle for a “pre-release” envelope instead.
We suggest starting with the following envelope functions, but feel free to experiment with your own; just be sure that the transitions at the start and end of a note are smooth:
1 - exp(-t/A), with A around 10ms1 - exp(-(tf - t)/R), with R around 10msexp(-t/D), with D around 750msWe recommend computing the envelope in a subfunction, given a vector of sample times. Test your subfunction by plotting envelopes for notes of different lengths.
Why did the notes2Pcm() function take the channel as an input parameter? So you could synthesize different sounds for different instruments. Modify sampleNote() so that it produces different waveforms for at least channels 1-3. Common alternatives to sine waves are square waves, triangle waves, and sawtooth waves. You can be creative here, but be sure that your waveform is periodic with a period equal to the reciprocal of the frequency of the note.
Another way to modify an instrument’s sound would be to add overtones – additional sine waves (generally of smaller amplitude) at integer multiples of the fundamental frequency of the note. You could even add a low-frequency modulation envelope (e.g. multiply by 0.8 + 0.2*sin(2*pi*3*t)) to get a tremolo effect like on an organ or a vibraphone.
Submit your files on CMS (after registering your group). They should include tokenizeLine.m, tokenizeLineTest.m, parseMidiLog.m, synthesize.m, transformNotes.m, writeMidiLog.m, and MidiPlayer.m.