The purpose of this project is to create musical blocks that output music without requiring some sort of musical talent.

Musical Blocks tracks the position of the blocks on a flat surface in a range seen by the Wiimote. The path of the blocks is then converted into a sequence of notes which is output. One can create an interactive system by having many blocks which can be used by many people simultaneously. Unlike traditional computer interfaces used to create electronic music, the musical block is a more expressive and physical/hands-on tools for creating music. In our design, the user can move blocks on any 2-D surface in a specified range and modify the musical rhythm in real time by changing the path of the block. Each block has its own unique sound associated with it and the musical rhythm would continuously play unless the block is stationary. One block is allocated to change the volume of the music played by the other blocks.

High Level Design

Project Inspiration

This project is inspired by the idea from the project called Siftables from MIT media lab. Siftables came from the idea of modifying the way we interact with computer in terms of pictures, music, or any sort of digital information. According to one of Siftables creator David Merrill, when we were children, reaching out and moving blocks, we are actually learning to think and solve problems by understanding and manipulating spatial relationship. Spatial reasoning is deeply connected to how we understand the world around us. We were inspired by the concept of Siftables and decided to create a musical block system specifically targeting young children. Without being a child musical protege, children can try to create a world of spatial interaction by manipulating and creating music. This can also serve to cultivate creativity and curiosity in children. Our decision to create a sound based on path is inspired from the collection of games called electroplankton (with a specific game interface called Tracy) on the Nintendo DS. Tracy allows the user to draw a line on the touch screen and create the sound based on the line drawn. We were fascinated by these two concepts and decided to create musical blocks. In our design we used three blocks, one to represent a plucked string sound, one to represent a violin sound, and one to control the volume.

Background Math

We used the Karplus-Strong Algorithm to compute the sound to output. The Karplus-Strong Algorithm uses physical modeling synthesis to generate a sound. A mathematical model of this algorithm is

In this model, x(n) represents a white noise which is used as an excitation for the initial part of the sound. The output sound is fed back to the delay line and averaged to create the pluck-sound. The averager is used to create the damping feature of the string. Studies of Ideal Plucked String and why Karplus Strong Algorithm works:

plucked string
The ideal plucked string has a boundary condition of 0 for all times. If the string is fixed at position L, the boundary condition at that point is y(t, L) =0. The traveling wave, when plucked, will reflect from a fixed end with an inversion (or a reflection coefficient of -1). Initially, an ideal plucked string has an initial displacement and zero initial velocity. Karplus-Strong Algorithm initializes the data and creates an initial sound which is just white noise. The physical displacement of a string is given by the superposition of left- and right-going traveling waves, the initial amplitude of each delay-line section should be half the amplitude of the initial, physical string displacement. Karplus-Strong Algorithm used an average for this reason and this results in a decaying waveform. A more realistic plucked string would need to take into account the propagation loss due to frequency. In general, damping will increase with frequency.


A physical model for the bow-String such as a violin is very complicated in terms of keeping track of bow pressure, bow velocity, bow width and bridge distance. A basic model of bow-string interaction model consist of two traveling waves represented by velocity v at the contact point coming from vin from nut and vib from the bridge and outputting two travelling waves of von and vob. Bow String interaction is represented by

Once this coupling has been solved, the new outgoing waves are calculated by

This model is descripted in the following figure

In the next figure f represents the frictional force while Hlt and Hrt are the filters that model the losses of the left-going and right-going waves propagating toward the bridge and the nut respectively.

We noticed that this model looks a little like the physical model of the Karplus-Strong Algorithm. However, a plucked-string waveform has a feature of decaying slowly over time while the violin has a feature of creating a longer sustain time to create that bowed-string sound.

Therefore, we decided to use Karplus-Strong Algorithm to create a pulled-string sound. To create the sound, we used an ADSR envelope and convoluted with Karplus-Strong Algorithm output. ADSR stands for Attack, Decay, Sustain and Release time and is used to model the timbre of an instrument.

Different family of instruments has their own ADSR profiles. The attack phase is refers to how the sound is initiated. The greater the slope of attack the harder you press down on the piano or faster you strum on a guitar. The decay phase describes how rapidly the sound dies down from the attack phase to the sustain phase. Some instruments like a drum have extremely fast decay. The sustain phase refers to how long the sound resonates from when it is played. Finally, the release phase refers to how fast the sound decays away once you release the instrument such as releasing the piano key.

The violin has a slightly slower attack than the guitar but the other phases are drastically different. Because a violin is bowed to produce sound, there is virtually no decay and a long sustain. Constant vibrational energy is delivered to the violins sound box and the sound only fades once the bowing stops.

As mentioned previously, due to some of the similarities between the physical model of the plucked-string and bowed-string, we decided to apply the Violin ADSR Profile onto the plucked-string output from the Karplus-Strong Algorithm.

Synthesis of different instruments in an instrument family can be achieved through applying ADSR concepts to the Karplus-Strong Algorithm. The output of the Karplus-Strong is convoluted with the modeled ADSR envelope to produce the final output sound.

Convolution is a mathematical operation on two functions and producing the third function that blends one function with another.

The definition of convolution is (in integral form):

Design Overview

Musical blocks allows the user to move blocks in any path to create either a plucked-string or a bowed-string sequence of notes. The user can also control the volume by changing the volume block. By pressing buttons on the Wiimote, the user can also change how many consecutive notes are stored before playing the sequence of notes.


Attach Wiimote about 48cm above the block
Place 3 blocks with the 40 cm range from the center of the Wiimote

Program/hardware Design


For the design of the project, we broke each block in the above diagram into different components in our design: Bluetooth communication, serial communication, parsing block coordinates, and algorithms for playing different sound.


For detection of the led blocks, we used the Wiimote which can sense infrared light of wavelength peaked around 950nm. Therefore, we decided to use the infrared led -- Vishay/Semiconductors TSAL6400 which outputs light peaked around 940nm.

We used the led calculator to determine what voltage the power supply need to supply and what resistor value we needed to allow at least 4 infrared led to operate at around 50mA with 1.2V across them. Based on the result the led calculator provide, it was determined that we should use 9V battery with two parallel 75 Ohm resistor. We chose to connect the 4 infrared leds in series because we want maximum current of 50mA which might get comprised when connecting 4 infrared led in parallel. The brightness of the light is proportional to the current. Therefore Wiimote needs to be high enough to give all 3 Infrared led blocks a larger field of movement. The Vishay led gives 25 degree radiant angle so the spacing of the leds need to be fairly close to each other so that Wiimote recognize the 4 infrared led on each block to consider it as one block. This stresses the importance of led brightness.

We tried to give the led circuitry more height to create a block-like interface that could be played around with. We mounted the led board and a 2x2 led square on top of the 4cm x 5.3cm plastic box which contains a 9V battery to run the leds.


In this project, we used the Wiimote as an IR sensor for led blocks that we made. The Wiimote interfaces with a laptop using an application called GlovePIE. Glove pie gets the Bluetooth data and determines the x and y coordinates of the blocks. The user can press "1" and "up" simultaneously on the Wiimote to increase how many notes they want to hear consecutively from block 1. If the user presses "1" while clicking gdownh arrow, the notes they hear consecutively from block 1 will decrease. Similarly the user can do the same thing for block two by pressing "2" instead of "1". For this project, we support up to 9 consecutive notes.

For the setup, we hang the Wiimote about 48cm above the block. The Wiimote was able to provide a range of about 40cm diameter circle on the plane. Each block is about 4cm x 5.3cm. Therefore, to create an illusion of a wide range for blocks to move freely, we decided that 4 blocks might be too cluttered and tight. Therefore, we decided on 3 blocks with one block that only moves in the y direction to control the volume.

Sound Output

We used an 8 bit DAC to output the sound.


A basic 8-bits Resistor-to-resistor digital-to-analog converter is shown in the above diagram. In general, for n bits DAC, 2n resistors are required. The input digital bits can be translated into a range of 0V to +5V. The calculation of the analog output can be described by:

Vout = Vref* Val / 2^(N)
Where Vref = 5V and Val is defined as digital value and N represent the bit amount.
For example, if the digital input = 0x11001011, Vout = 5*203/(2^8) = 3.96V.

The calculation of the output analog signal of the R2R Ladder DAC depends a lot on how well matched each resistor is to the others. Therefore, we used the same resistor pack so that the resistances only differ in 2% of the value and are considered well-matched. We decided to use a R2R ladder DAC rather than PWM for signal generation to create a cleaner signal and hopefully a better sounding audio output.


There are several aspects to the software design of the program. The different components are music, parser, Note Determination and interacting with GlovePIE using a serial communication.


To create the plucked-string sound, we used the Karplus-Strong Algorithm that Professor Land wrote in Matlab. We converted the Matlab code to C code. By generating the Matlab code, we were able to see the waveform:

Matlab Karlpus Strong

By taking the code, we were able to create a plucked string that has a sharp attack, no sustain and very slow decay. We used fixed-point multiplication so that the algorithm would not take very long in computing the audio output.

To create a pulled-string sound, we created an ADSR (Attack, Decay, Sustain, Release) envelope to convolve with the Karplus-strong algorithm. We used a fix-point multiplication because floating point multiplication takes a long time. As mentioned in the background math section, we could create a violin-like sound by sustaining the sound the plucked-string sound.

We first tried creating it in Matlab to figure out if itfs possible to create a bowed-string sound. We created the ADSR envelope that was close to the Violin ADSR profile in the Background Math section.

Matlab ADSR

We were able to create from a plucked-string audio to a pulled-string audio with waveforms of:

Matlab Plucked String


Matlab Violin

Audio output:

We used the interrupt service timer0 overflow for output of the audio. We used PortA .1 to output digital bits which uses the DAC to produce the analog output sampled at 62.5 kHz. We wanted to create something that has a positive and negative swing. Therefore, we added 128 the signal which acts as a 0 reference point.

Note Determination

To determine which note to play, we used a slope-based calculation of the path of the block. If the user chose to hear each note at a time, the user can create a two-point path. GlovePIE first sends the coordinate data of the blocks to hyperterm. When the user inputs in more data by moving a block, the parser determines the x and y coordinates of the block and calculates the slope by using the previous coordinates.

We created a frequency table that determines the frequency of the note we wish to play. We provide 13 frequencies of notes depending on the slope of the two given points on a path. We determine the notes by the below diagram:

Note Determination

We divided a circle into 13 parts and use the fact that

where (x1, y1) are the first coordinate and (x2, y2) are the second coordinate on a path and theta is the angle with the positive x axis. However, notice that the first and third quadrant calculation of tan theta will give the same value. Therefore, we need to divide the circle into half with the top half (y2>y1) and the bottom half (y2 < y1) representing different notes. Notice that since 2*pi/13 is not an even split on the circle, n=6 is split between y2 < y1 and y2 > y1. We created a delay table for the Karplus-Strong Algorithm where a larger n value shown in the diagram above will create an audio output frequency that proportional to n. For Karplus-Strong algorithm, the delay length = Fsample/Fout. We used the timer0 overflow to play the audio output. Therefore, Fsample = 62500. We create a frequency table that ranges from 500-1000Hz. The lower index will give a lower frequency than the higher index. Therefore, using n we can play a high frequency if the user creates a steeper slope. The steeper the slope, the higher the frequency the audio output will be.

Serial Communication

glove pie

We needed a way to communicate the Wiimote with the microcontroller to figure out if the block has moved or not. We found an application called GlovePIE that coordinate with the Wiimote to sense the x and y coordinate of the LED blocks. Now there is an issue of getting the data to the microcontroller. We discovered a feature in GlovePIE to emulate a keyword. The RS-232 connection is used to get the string from HyperTerminal whenever gEnterh key was pressed. Therefore, we wrote a script in GlovePIE to send in block 1fs x and y coordinates whenever block 1 moved and the same for block 2. The format of the date we chose to transmit through hyperterm was as follows:

Block 1: x1####y1####
Block 2: x2####y2####
Block3: x3####y3####

We wanted block 3 to control the volume. However, we noticed that GlovePIE is not very consistent with calling each block 1 or 2 or 3. In other words, GlovePIE sometimes switch between block 1 and block 2 or block 1 and block 3. Therefore, we decided to add some error checking so that block 2 doesnft, in the middle of playing, become the volume control block. We decided, arbitrarily, that block 3 will always be the rightmost block. In addition, to increase the volume, move the block 3 up and down to decrease the volume. As a result, block 3 x coordinate will not change much. We added extra code in GlovePIE to check all the y coordinates of each block to determine which one is the right-most block. We then make sure that when GlovePIE emulate the keyboard, it will give the right-most block coordinate in a format of "x3####y3####".

When testing GlovePIEs emulation of the keyword, we noticed that GlovePIE sends data even if the block is not moving. We debated between making the microcontroller handle this behavior or writing extra script in GlovePIE to take care of the case. We chose the ladder because we knew that music algorithm will take most of the CPU cycles. Therefore, we wrote extra code in GlovePIE to emulate the keyboard only if the block coordinates change in the first three decimal places.

We also realized that hyperterm has a limit on how many characters it can take. Therefore, we decided to slow down the emulation keyboard typing rate so that hyperterm can catch up. We played around with the manipulation of gwaith command in GlovePIE. After trial and error, we used gwait 3msh between each character transmission and gwait 10msh between each blockfs x and y coordinate. This allows hyperterm to take in data with minimum error.

We allow the user to press "1" and "up", or "1" and "down" or "2" and "up" or "2" and "down" on the Wiimote to change the amount of notes played consecutively. The formats we used to transmit this onto the hyperterm are:
WiimoteTransmit onto Hyperterm
1 and upp1u
1 and downp1d
2 and upp2u
2 and downp2d

We used the code from Lab 3 to receive characters from the hyperterm using the UART character-ready ISR. Using this, GlovePIE was able to send characters to the hyperterm and the microcontroller was able to process them for parsing.

Parser: task2()

The main routine handles the parsing of the coordinates and changes the number of notes played consecutively far each block. Due to the heavy CPU usage when playing music, we decided to use the ISR timer0 overflow that was used for audio output to time task2 which calls the parser. The timer0 overflow every 256/16MHz=16 microseconds. Therefore, 62.5 overflows would result in 1mSec. Since wefre using a count variable which is an integer, we cannot get .5 count. Therefore, we used the method in Lab 3 to alternate between 62 counts and 63 counts which, after many cycle, will even out to create a relatively decent 1mSec timer. Every 100mSec, we decided to call the parser.

Every 100mSec, we check to see if r_ready is 1, which happens when a new character has been transmitted. If it is, we check to see what values r_buffer contains. Sometimes the interaction with GlovePIE and hyperterm result in some sort of mistake format like: "x1y1##". Therefore, we error check to ensure that the index has 12 characters "x#####y#####" including the g\nh character for enter. Then we make sure to see if the value is gxh in the 0th index and y in the 6th index. If it is, we go on further to figure out which block (1, 2 or 3) the data is for. After that we take the data and figure out what note to play by calling pathToNote(c) method. For block 3, we only parse the y-coordinate to change the volume of the output audio. By moving the block 3 up, the volume will increase and moving the block 3 down will decrease the volume. Glove pie gives the y coordinate of the block as a value between 0 to 1.2. We wanted to support 5 levels of volume control. Therefore, we determine that "x3####y3####" such that "####" after y3, ranging from 0 to 240 will mean maximum volume and every interval of +240 will change the volume down further. For example, 480 to 720 will be a moderate volume while 960-1200 will mean the lowest volume. Initially, we set the volume to maximum for demo purposes.

The parser will also try to check if "p1u", "p1d", "p2u", or "p2d" is detected. We created a variable pmax1 and pmax2 which controls how many notes are played consecutively. Initially, we set the pmax1 and pmax2 to 1 for demo. The user can change them from 1 to 9 by using the keys on the Wiimote. If the user decided to change from a higher number to lower number, the parser will re-initialize so that the user will have to input the coordinates from the beginning again. For example, if the user decides to change pmax1 from 9 to 5 and 4 coordinates are already input into hyperterm, the user cannot input only one more coordinate to produce a sound. Instead, the user must input in all 5 coordinates over again. The coordinate(s) that the user inputted before changing to 5 will be discarded.

Attempted Design

We tried to create another instrument by convolving ADSR with Sine wave to create a oboe sound. We were able to implement it Matlab. However, when we tried implementing in Matlab, we had many issues with getting the timing correctly and the output audio didnft sound anything like oboe. Our implementation in Matlab uses ADSR envelope that look like:


Problems Encountered:

We were using a laptop with Vista installed for serial communication. Hyperterm didnft come with Vista so we download a trail beta version which is very unstable and tends to freeze about every 15 minutes. We also encountered problems with the microcontroller resetting itself at random time. We discovered that we can reduce the resetting problem by creating a greater delay in GlovePIE. We believe that most of the problem is associated with hyperterm processing speed.

We had many problems with getting the ADSR and Karplus-Strong Algorithm to work together. We kept hearing different parts of a note separately and playing each note took a very long time. We realized that this was probably due to using too many multiplications. Therefore, we decided to use fixed-point arithmetic to do all our calculations and reduce the number of multiplies as much as possible. However, even after changing this we still hear some parts of the note were delayed. After the help from Professor Land, we discovered that using float2fix(a) in playing the audio will take a long time. Therefore, instead of using the macro for float2fix(a), we converted all our parameters to fixed format by hand instead of writing the code for it.

Then we had an issue with creating the sound. We encountered a problem with not getting enough resolution when using fixed point arithmetic. Therefore, it was hard to get the ADSR envelope to ramp up as much as we like to. Also the ramp slope is very fast until the resolution for fixed point arithmetic gets saturated. Therefore, we decided to increase our resolution by shifting everything left by 6 or multiplying by 64. After we finish all our multiplies and calculations, we right shift the output by 6 for outputting the audio signal. This preserves most of the accuracy by avoiding the resolution problems of using 8.8 fixed point. After this fix, we were able to get a pulled-string sound although it sounds a bit like a synthesizer. It doesnft sound as great as implementing in Matlab because of some resolution loss. However we found out that when using higher frequencies, the output sounded much more like a real violin.

Results of the Design:

In the proposal, we expected to combine the two project ideas from Siftables and Tracy. We were able to fulfill that part of the expectation. We intended to create 4 blocks instead of 3 blocks. However, due to the range that Wiimote can provide, we were able to only integrate three moving blocks rather than 4. We originally thought of creating an acceleration and velocity based game where how fast you move the block determine how the music is played. However, due to the significant time delay in the serial communication, we were unable to get accurate time for the data. Therefore, we decided to change our design to create a slope-based blocks music player. Slope-based blocks involve figuring out the slope of each block data point given to the microcontroller.

Due to the simplicity and ease of implementation, we decided to add a feature that wasn't mentioned in the proposal. We figure that it would to nice to have the user inputs in multiple paths before playing the music to hear different kinds of sound they can make rather than one path at a time. We allow the user to input in at least 9 points/paths.

Due to some lag in hyperterm, we were unable to take data continuously. As mentioned previously, we had to use gwaith command in GlovePIE so that hyperterm can process the data. Chart of approximate time between each data collected:
Lines typed in hypertermtimeerror line contained

Based on these above data, we determine that we can transmit to hyperterm about 3 lines in 2 seconds. Therefore, the sample speed of the hyperterm per line is about 1.5 lines per second.

It was a major pain to get GlovePIE to work with hyperterm. GlovePIE tends to take data very fast while hyperterm is limited by the baud rate and one line to take x and y coordinates for each block. Therefore, we programmed the GlovePIE to wait a little while before sending data to hyperterm. Also when playing music, we were unable to take data because the microcontroller is using all of its CPU cycle to handle the music rather than serial communication. Therefore, we cannot collect any data whenever the music is playing. We tried to get concurrency between serial communication and music playing working by using two interrupt services as done in Lab 3. However, due to the fact that it took too long in playing the music using the interrupt service, we were unable to do concurrency like Lab 2.

Error given by GlovePIE onto the hyperterm when trying to play note(s) consecutively:
Total Linesnotes played consecutivelyline error


For safety, we held down the led boards with electric tape. We also used the standard 9V battery to power the led. The resistors are used to protect the led. In principle, there could be some danger to the eye because infrared led works around 940nm which is not in the visible spectrum. Thus, the pupil wonft close in order to protect the retina from the bright infrared led light source. We looked at the data sheet for the infrared led and notice that the radiant power is 35mW for running led at 100mA. In the lab, wefre only running 4 leds at 50mA. We found some legal restrictions regarding infrared laser, infrared emission of heat lamps, and the medial IR heat lamps which work around 500W. We couldnft find any medical record or legal restrictions or safety regarding the IR working around 70mW. The effect of infrared light is mostly heat and not related to destruction of biological cells like UV light.


We think that this will bring more excitement to the ordinary blocks that children play with. This will create more interaction for children with music. Most of all, they donft need any sort of musical talent. Also as David Merrill stated on TED Talk, learning to interact with blocks allow children learn about problem solving, and understand about spatial relationship.


Overall, we were pleased with how the project came out. We feel that hardware and software interaction should be a little faster. The audio quality of the plucked-string and pulled-string are satisfactory. We encountered many problems in the implementation but we were able to satisfactorily solve most of them.

We saw many great ECE 476 final projects using the Wiimote. However, we believe that there are limits to using a Wiimote. Wiimote becomes very unstable when trying to track multiple objects. Due to the fact that Wiimote sometimes tend to switch and get confuse about which led it considers e1f at first, we feel that it might be nice to find alternative IR tracker. The major disappointment we had with Wiimote is that we werenft able to add a fourth block to control the tempo of our audio output in our project. Because of the overload Wiimote has when tracking four sources, we left that feature out in our project.

Ethical and Legal Considerations:

We closely follow the IEEE Code of Ethics throughout our project. Our project consisted of playing sound and we used the headphones as not to annoy any of the group who are concentrating and thinking about their projects.

For Karplus-Strong Algorithm, we took the code Professor Land wrote in Matlab. Also for ADSR implementation, we used the idea from DSP lab to implement ADSR envelope using target rather than only trying to create the envelope section by section. By using their method, we were able to shorten some code because we no longer require to implement the decay part of the envelope. We also used the fixed point multiplication code provided in lab 2.

Future Considerations:

Due to the lack of time and inaccuracy of the Wiimote, we were unable to add the fourth block. The range Wiimote provides doesnft allow us to create a larger 2-D space for blocks to move.

To reach the goal of greater interactive environment, it would be nice if there are more blocks and more instruments. We believe that Wiimote will be insufficient to create that environment. If the project gives more time and is of a greater scale, a project like Siftables and Audiopad, where the blocks understand their relationship with other blocks would be plausible. If the project is done on microcontroller, the blocks would communicate with each other using near-field communication.


Parts List

Budget and Parts List Quantity Amount Source
Infrared LED Vishay/Semiconductors TSAL6400 12 $5.34 Digikey
Resistor Pack 1K 1 476 Lab Stock
Resistor Pack 2K 2 476 Lab Stock
Resistor (Low-pass filter and LED board) 5 476 Lab Stock
Atmel ATmega 644 1 $8
ECE 476 Custom PCB 1 $4
MAX232 1 Sampled Maxim-IC
Perf Board 3 $3 476 Lab Stock
9V batteries 3 $3.75 Battery Warehouse
DIP Socket 2 $1 476 Lab Stock
Header socket/plug 2 $1 476 Lab Stock
Wiimote 1 previously owned
STK500 1 $15 476 Lab Stock
40 pin socket 1 $.50 476 Lab Stock
20 pin socket 1 $.50 476 Lab Stock
Serial Connector 1 $1.65 476 Lab Stock
jumpers 3 $1.50 476 Lab Stock
White Board 1 $6 476 Lab Stock
Total: $51.24

Division of Tasks

Research on Karplus-Strong & ADSRJulie
Matlab implementation of different instrumentJulie
Soldering and assemblyJulie
Implementation of Karplus-Strong & ADSRBoth
Bluetooth and Serial Communication with Hyperterm Taige
Testing and Debugging both
Lab Report Both
Website DesignTaige


Microcontroller code
Glove Pie code


Audiopad : interactive music player
Siftables : another project that create an interaction with digital information and media in physical way
Electroplankton : interactive music video game
Video on Electroplankton:
ADSR & Karplus-Strong: creating new instrument sound by applying ADSR to Karplus-Strong
ADSR & Sinewave: creating sounds by applying ADSR to sine-wave
Computer Music in Matlab: create ADSR in Matlab
Physical Audio Signal Processing:
Digital Music with microcontroller:
ADSR envelope:
Ideal Plucked String:
Bow-String Interaction Model:

data sheets




Some ECE 476 Final Projects that we looked at:
Digital Music Synthesizer
Air Drums
Air Jam