The purpose of this project is to create musical blocks that output music without requiring some sort of musical talent.
Musical Blocks tracks the position of the blocks on a flat surface in a range seen by the Wiimote. The path of the blocks is then converted into a sequence of notes which is output. One can create an interactive system by having many blocks which can be used by many people simultaneously. Unlike traditional computer interfaces used to create electronic music, the musical block is a more expressive and physical/hands-on tools for creating music. In our design, the user can move blocks on any 2-D surface in a specified range and modify the musical rhythm in real time by changing the path of the block. Each block has its own unique sound associated with it and the musical rhythm would continuously play unless the block is stationary. One block is allocated to change the volume of the music played by the other blocks.
In this model, x(n) represents a white noise which is used as an excitation for the initial part of the sound. The output sound is fed back to the delay line and averaged to create the pluck-sound. The averager is used to create the damping feature of the string. Studies of Ideal Plucked String and why Karplus Strong Algorithm works:
We noticed that this model looks a little like the physical model of the Karplus-Strong Algorithm. However, a plucked-string waveform has a feature of decaying slowly over time while the violin has a feature of creating a longer sustain time to create that bowed-string sound.
Therefore, we decided to use Karplus-Strong Algorithm to create a pulled-string sound. To create the sound, we used an ADSR envelope and convoluted with Karplus-Strong Algorithm output. ADSR stands for Attack, Decay, Sustain and Release time and is used to model the timbre of an instrument. Different family of instruments has their own ADSR profiles. The attack phase is refers to how the sound is initiated. The greater the slope of attack the harder you press down on the piano or faster you strum on a guitar. The decay phase describes how rapidly the sound dies down from the attack phase to the sustain phase. Some instruments like a drum have extremely fast decay. The sustain phase refers to how long the sound resonates from when it is played. Finally, the release phase refers to how fast the sound decays away once you release the instrument such as releasing the piano key.As mentioned previously, due to some of the similarities between the physical model of the plucked-string and bowed-string, we decided to apply the Violin ADSR Profile onto the plucked-string output from the Karplus-Strong Algorithm.
Convolution is a mathematical operation on two functions and producing the third function that blends one function with another.
The definition of convolution is (in integral form):
For the design of the project, we broke each block in the above diagram into different components in our design: Bluetooth communication, serial communication, parsing block coordinates, and algorithms for playing different sound.
We used the led calculator We tried to give the led circuitry more height to create a block-like interface that could be played around with. We mounted the led board and a 2x2 led square on top of the 4cm x 5.3cm plastic box which contains a 9V battery to run the leds.
In this project, we used the Wiimote as an IR sensor for led blocks that we made. The Wiimote interfaces with a laptop using an application called GlovePIE. Glove pie gets the Bluetooth data and determines the x and y coordinates of the blocks. The user can press "1" and "up" simultaneously on the Wiimote to increase how many notes they want to hear consecutively from block 1. If the user presses "1" while clicking gdownh arrow, the notes they hear consecutively from block 1 will decrease. Similarly the user can do the same thing for block two by pressing "2" instead of "1". For this project, we support up to 9 consecutive notes.
For the setup, we hang the Wiimote about 48cm above the block. The Wiimote was able to provide a range of about 40cm diameter circle on the plane. Each block is about 4cm x 5.3cm. Therefore, to create an illusion of a wide range for blocks to move freely, we decided that 4 blocks might be too cluttered and tight. Therefore, we decided on 3 blocks with one block that only moves in the y direction to control the volume.
A basic 8-bits Resistor-to-resistor digital-to-analog converter is shown in the above diagram. In general, for n bits DAC, 2n resistors are required. The input digital bits can be translated into a range of 0V to +5V. The calculation of the analog output can be described by:
Vout = Vref* Val / 2^(N)
The calculation of the output analog signal of the R2R Ladder DAC depends a lot on how well matched each resistor is to the others. Therefore, we used the same resistor pack so that the resistances only differ in 2% of the value and are considered well-matched. We decided to use a R2R ladder DAC rather than PWM for signal generation to create a cleaner signal and hopefully a better sounding audio output.
To create the plucked-string sound, we used the Karplus-Strong Algorithm that Professor Land wrote in Matlab. We converted the Matlab code to C code. By generating the Matlab code, we were able to see the waveform:
By taking the code, we were able to create a plucked string that has a sharp attack, no sustain and very slow decay. We used fixed-point multiplication so that the algorithm would not take very long in computing the audio output.
To create a pulled-string sound, we created an ADSR (Attack, Decay, Sustain, Release) envelope to convolve with the Karplus-strong algorithm. We used a fix-point multiplication because floating point multiplication takes a long time. As mentioned in the background math section, we could create a violin-like sound by sustaining the sound the plucked-string sound.
We first tried creating it in Matlab to figure out if itfs possible to create a bowed-string sound. We created the ADSR envelope that was close to the Violin ADSR profile in the Background Math section.
We were able to create from a plucked-string audio to a pulled-string audio with waveforms of:
To determine which note to play, we used a slope-based calculation of the path of the block. If the user chose to hear each note at a time, the user can create a two-point path. GlovePIE first sends the coordinate data of the blocks to hyperterm. When the user inputs in more data by moving a block, the parser determines the x and y coordinates of the block and calculates the slope by using the previous coordinates.
We created a frequency table that determines the frequency of the note we wish to play.
We provide 13 frequencies of notes depending on the slope of the two given points on a path. We determine the notes by the below diagram:
We divided a circle into 13 parts and use the fact that
where (x1, y1) are the first coordinate and (x2, y2) are the second coordinate on a path and theta is the angle with the positive x axis. However, notice that the first and third quadrant calculation of tan theta will give the same value. Therefore, we need to divide the circle into half with the top half (y2>y1) and the bottom half (y2 < y1) representing different notes. Notice that since 2*pi/13 is not an even split on the circle, n=6 is split between y2 < y1 and y2 > y1. We created a delay table for the Karplus-Strong Algorithm where a larger n value shown in the diagram above will create an audio output frequency that proportional to n. For Karplus-Strong algorithm, the delay length = Fsample/Fout. We used the timer0 overflow to play the audio output. Therefore, Fsample = 62500. We create a frequency table that ranges from 500-1000Hz. The lower index will give a lower frequency than the higher index. Therefore, using n we can play a high frequency if the user creates a steeper slope. The steeper the slope, the higher the frequency the audio output will be.
We needed a way to communicate the Wiimote with the microcontroller to figure out if the block has moved or not. We found an application called GlovePIE that coordinate with the Wiimote to sense the x and y coordinate of the LED blocks. Now there is an issue of getting the data to the microcontroller. We discovered a feature in GlovePIE to emulate a keyword. The RS-232 connection is used to get the string from HyperTerminal whenever gEnterh key was pressed. Therefore, we wrote a script in GlovePIE to send in block 1fs x and y coordinates whenever block 1 moved and the same for block 2. The format of the date we chose to transmit through hyperterm was as follows:
Block 1: x1####y1####
We wanted block 3 to control the volume. However, we noticed that GlovePIE is not very consistent with calling each block 1 or 2 or 3. In other words, GlovePIE sometimes switch between block 1 and block 2 or block 1 and block 3. Therefore, we decided to add some error checking so that block 2 doesnft, in the middle of playing, become the volume control block. We decided, arbitrarily, that block 3 will always be the rightmost block. In addition, to increase the volume, move the block 3 up and down to decrease the volume. As a result, block 3 x coordinate will not change much. We added extra code in GlovePIE to check all the y coordinates of each block to determine which one is the right-most block. We then make sure that when GlovePIE emulate the keyboard, it will give the right-most block coordinate in a format of "x3####y3####".
When testing GlovePIEs emulation of the keyword, we noticed that GlovePIE sends data even if the block is not moving. We debated between making the microcontroller handle this behavior or writing extra script in GlovePIE to take care of the case. We chose the ladder because we knew that music algorithm will take most of the CPU cycles. Therefore, we wrote extra code in GlovePIE to emulate the keyboard only if the block coordinates change in the first three decimal places.
We also realized that hyperterm has a limit on how many characters it can take. Therefore, we decided to slow down the emulation keyboard typing rate so that hyperterm can catch up. We played around with the manipulation of gwaith command in GlovePIE. After trial and error, we used gwait 3msh between each character transmission and gwait 10msh between each blockfs x and y coordinate. This allows hyperterm to take in data with minimum error.
We allow the user to press "1" and "up", or "1" and "down" or "2" and "up" or "2" and "down" on the Wiimote to change the amount of notes played consecutively. The formats we used to transmit this onto the hyperterm are:
We used the code from Lab 3 to receive characters from the hyperterm using the UART character-ready ISR. Using this, GlovePIE was able to send characters to the hyperterm and the microcontroller was able to process them for parsing.
The main routine handles the parsing of the coordinates and changes the number of notes played consecutively far each block. Due to the heavy CPU usage when playing music, we decided to use the ISR timer0 overflow that was used for audio output to time task2 which calls the parser. The timer0 overflow every 256/16MHz=16 microseconds. Therefore, 62.5 overflows would result in 1mSec. Since wefre using a count variable which is an integer, we cannot get .5 count. Therefore, we used the method in Lab 3 to alternate between 62 counts and 63 counts which, after many cycle, will even out to create a relatively decent 1mSec timer. Every 100mSec, we decided to call the parser.
Every 100mSec, we check to see if r_ready is 1, which happens when a new character has been transmitted. If it is, we check to see what values r_buffer contains. Sometimes the interaction with GlovePIE and hyperterm result in some sort of mistake format like: "x1y1##". Therefore, we error check to ensure that the index has 12 characters "x#####y#####" including the g\nh character for enter. Then we make sure to see if the value is gxh in the 0th index and y in the 6th index. If it is, we go on further to figure out which block (1, 2 or 3) the data is for. After that we take the data and figure out what note to play by calling pathToNote(c) method. For block 3, we only parse the y-coordinate to change the volume of the output audio. By moving the block 3 up, the volume will increase and moving the block 3 down will decrease the volume. Glove pie gives the y coordinate of the block as a value between 0 to 1.2. We wanted to support 5 levels of volume control. Therefore, we determine that "x3####y3####" such that "####" after y3, ranging from 0 to 240 will mean maximum volume and every interval of +240 will change the volume down further. For example, 480 to 720 will be a moderate volume while 960-1200 will mean the lowest volume. Initially, we set the volume to maximum for demo purposes.
The parser will also try to check if "p1u", "p1d", "p2u", or "p2d" is detected. We created a variable pmax1 and pmax2 which controls how many notes are played consecutively. Initially, we set the pmax1 and pmax2 to 1 for demo. The user can change them from 1 to 9 by using the keys on the Wiimote. If the user decided to change from a higher number to lower number, the parser will re-initialize so that the user will have to input the coordinates from the beginning again. For example, if the user decides to change pmax1 from 9 to 5 and 4 coordinates are already input into hyperterm, the user cannot input only one more coordinate to produce a sound. Instead, the user must input in all 5 coordinates over again. The coordinate(s) that the user inputted before changing to 5 will be discarded.
We tried to create another instrument by convolving ADSR with Sine wave to create a oboe sound. We were able to implement it Matlab. However, when we tried implementing in Matlab, we had many issues with getting the timing correctly and the output audio didnft sound anything like oboe. Our implementation in Matlab uses ADSR envelope that look like:
We were using a laptop with Vista installed for serial communication. Hyperterm didnft come with Vista so we download a trail beta version which is very unstable and tends to freeze about every 15 minutes. We also encountered problems with the microcontroller resetting itself at random time. We discovered that we can reduce the resetting problem by creating a greater delay in GlovePIE. We believe that most of the problem is associated with hyperterm processing speed.
We had many problems with getting the ADSR and Karplus-Strong Algorithm to work together. We kept hearing different parts of a note separately and playing each note took a very long time. We realized that this was probably due to using too many multiplications. Therefore, we decided to use fixed-point arithmetic to do all our calculations and reduce the number of multiplies as much as possible. However, even after changing this we still hear some parts of the note were delayed. After the help from Professor Land, we discovered that using float2fix(a) in playing the audio will take a long time. Therefore, instead of using the macro for float2fix(a), we converted all our parameters to fixed format by hand instead of writing the code for it.
Then we had an issue with creating the sound. We encountered a problem with not getting enough resolution when using fixed point arithmetic. Therefore, it was hard to get the ADSR envelope to ramp up as much as we like to. Also the ramp slope is very fast until the resolution for fixed point arithmetic gets saturated. Therefore, we decided to increase our resolution by shifting everything left by 6 or multiplying by 64. After we finish all our multiplies and calculations, we right shift the output by 6 for outputting the audio signal. This preserves most of the accuracy by avoiding the resolution problems of using 8.8 fixed point. After this fix, we were able to get a pulled-string sound although it sounds a bit like a synthesizer. It doesnft sound as great as implementing in Matlab because of some resolution loss. However we found out that when using higher frequencies, the output sounded much more like a real violin.
In the proposal, we expected to combine the two project ideas from Siftables and Tracy. We were able to fulfill that part of the expectation. We intended to create 4 blocks instead of 3 blocks. However, due to the range that Wiimote can provide, we were able to only integrate three moving blocks rather than 4. We originally thought of creating an acceleration and velocity based game where how fast you move the block determine how the music is played. However, due to the significant time delay in the serial communication, we were unable to get accurate time for the data. Therefore, we decided to change our design to create a slope-based blocks music player. Slope-based blocks involve figuring out the slope of each block data point given to the microcontroller.
Due to the simplicity and ease of implementation, we decided to add a feature that wasn't mentioned in the proposal. We figure that it would to nice to have the user inputs in multiple paths before playing the music to hear different kinds of sound they can make rather than one path at a time. We allow the user to input in at least 9 points/paths.
Due to some lag in hyperterm, we were unable to take data continuously. As mentioned previously, we had to use gwaith command in GlovePIE so that hyperterm can process the data.
Chart of approximate time between each data collected:
Based on these above data, we determine that we can transmit to hyperterm about 3 lines in 2 seconds. Therefore, the sample speed of the hyperterm per line is about 1.5 lines per second.
It was a major pain to get GlovePIE to work with hyperterm. GlovePIE tends to take data very fast while hyperterm is limited by the baud rate and one line to take x and y coordinates for each block. Therefore, we programmed the GlovePIE to wait a little while before sending data to hyperterm. Also when playing music, we were unable to take data because the microcontroller is using all of its CPU cycle to handle the music rather than serial communication. Therefore, we cannot collect any data whenever the music is playing. We tried to get concurrency between serial communication and music playing working by using two interrupt services as done in Lab 3. However, due to the fact that it took too long in playing the music using the interrupt service, we were unable to do concurrency like Lab 2.
Error given by GlovePIE onto the hyperterm when trying to play note(s) consecutively:
For safety, we held down the led boards with electric tape. We also used the standard 9V battery to power the led. The resistors are used to protect the led. In principle, there could be some danger to the eye because infrared led works around 940nm which is not in the visible spectrum. Thus, the pupil wonft close in order to protect the retina from the bright infrared led light source. We looked at the data sheet for the infrared led and notice that the radiant power is 35mW for running led at 100mA. In the lab, wefre only running 4 leds at 50mA. We found some legal restrictions regarding infrared laser, infrared emission of heat lamps, and the medial IR heat lamps which work around 500W. We couldnft find any medical record or legal restrictions or safety regarding the IR working around 70mW. The effect of infrared light is mostly heat and not related to destruction of biological cells like UV light.
We think that this will bring more excitement to the ordinary blocks that children play with. This will create more interaction for children with music. Most of all, they donft need any sort of musical talent. Also as David Merrill stated on TED Talk, learning to interact with blocks allow children learn about problem solving, and understand about spatial relationship.
Overall, we were pleased with how the project came out. We feel that hardware and software interaction should be a little faster. The audio quality of the plucked-string and pulled-string are satisfactory. We encountered many problems in the implementation but we were able to satisfactorily solve most of them.
We saw many great ECE 476 final projects using the Wiimote. However, we believe that there are limits to using a Wiimote. Wiimote becomes very unstable when trying to track multiple objects. Due to the fact that Wiimote sometimes tend to switch and get confuse about which led it considers e1f at first, we feel that it might be nice to find alternative IR tracker. The major disappointment we had with Wiimote is that we werenft able to add a fourth block to control the tempo of our audio output in our project. Because of the overload Wiimote has when tracking four sources, we left that feature out in our project.
We closely follow the IEEE Code of Ethics throughout our project. Our project consisted of playing sound and we used the headphones as not to annoy any of the group who are concentrating and thinking about their projects.
For Karplus-Strong Algorithm, we took the code Professor Land wrote in Matlab. Also for ADSR implementation, we used the idea from DSP lab to implement ADSR envelope using target rather than only trying to create the envelope section by section. By using their method, we were able to shorten some code because we no longer require to implement the decay part of the envelope. We also used the fixed point multiplication code provided in lab 2.
Due to the lack of time and inaccuracy of the Wiimote, we were unable to add the fourth block. The range Wiimote provides doesnft allow us to create a larger 2-D space for blocks to move.
To reach the goal of greater interactive environment, it would be nice if there are more blocks and more instruments. We believe that Wiimote will be insufficient to create that environment. If the project gives more time and is of a greater scale, a project like Siftables and Audiopad, where the blocks understand their relationship with other blocks would be plausible. If the project is done on microcontroller, the blocks would communicate with each other using near-field communication.
Some ECE 476 Final Projects that we looked at:
Wiimote
Sound Output
We used an 8 bit DAC to output the sound.
Where Vref = 5V and Val is defined as digital value and N represent the bit amount.
For example, if the digital input = 0x11001011, Vout = 5*203/(2^8) = 3.96V.
Software
There are several aspects to the software design of the program. The different components are music, parser, Note Determination and interacting with GlovePIE using a serial communication.
Music
Audio output:
We used the interrupt service timer0 overflow for output of the audio. We used PortA .1 to output digital bits which uses the DAC to produce the analog output sampled at 62.5 kHz. We wanted to create something that has a positive and negative swing. Therefore, we added 128 the signal which acts as a 0 reference point.
Note Determination
Serial Communication
Block 2: x2####y2####
Block3: x3####y3####
Wiimote Transmit onto Hyperterm 1 and up p1u 1 and down p1d 2 and up p2u 2 and down p2d Parser: task2()
Attempted Design
Violin
Problems Encountered:
Results of the Design:
Lines typed in hyperterm time error line contained 40 40sec 10% 74 60sec 20% 39 30sec 2%
Total Lines notes played consecutively line error 27 1 4 15 1 0 20 3 2 14 5 4 11 9 2 11 9 2 Safety
Usability
Conclusion:
Ethical and Legal Considerations:
Future Considerations:
Appendix
Parts List
Budget and Parts List
Quantity
Amount
Source
Infrared LED Vishay/Semiconductors TSAL6400
12
$5.34
Digikey
Resistor Pack 1K
1
476 Lab Stock
Resistor Pack 2K
2
476 Lab Stock
Resistor (Low-pass filter and LED board)
5
476 Lab Stock
Atmel ATmega 644
1
$8
ECE 476 Custom PCB
1
$4
MAX232
1
Sampled
Maxim-IC
Perf Board
3
$3
476 Lab Stock
9V batteries
3
$3.75
Battery Warehouse
DIP Socket
2
$1
476 Lab Stock
Header socket/plug
2
$1
476 Lab Stock
Wiimote
1
previously owned
STK500
1
$15
476 Lab Stock
40 pin socket
1
$.50
476 Lab Stock
20 pin socket
1
$.50
476 Lab Stock
Serial Connector
1
$1.65
476 Lab Stock
jumpers
3
$1.50
476 Lab Stock
White Board
1
$6
476 Lab Stock
Total: $51.24 Division of Tasks
Research on Karplus-Strong & ADSR Julie Matlab implementation of different instrument Julie Soldering and assembly Julie Implementation of Karplus-Strong & ADSR Both GlovePIE Taige Bluetooth and Serial Communication with Hyperterm Taige Testing and Debugging both Lab Report Both Website Design Taige Code
Microcontroller code
Glove Pie code
References
Audiopad : interactive music player
Siftables : another project that create an interaction with digital information and media in physical way
Electroplankton : interactive music video game
Video on Electroplankton:
ADSR & Karplus-Strong: creating new instrument sound by applying ADSR to Karplus-Strong
ADSR & Sinewave: creating sounds by applying ADSR to sine-wave
Computer Music in Matlab: create ADSR in Matlab
Physical Audio Signal Processing:
Digital Music with microcontroller:
ADSR envelope:
Ideal Plucked String:
Bow-String Interaction Model:
data sheets
LED
vendors
Digikey
Digital Music Synthesizer
Air Drums
Air Jam