Virtuoso: A Touchscreen Music App

Software Design and Implementation

Overview

The most important part for the software design is the sound synthesis and the touch screen control. As described above, we devided this app into 6 modes.

Sound Synthesis

In this project, two synthesis method are used to generate the notes of different insturments. The additive synthesis and FM synthesis are adopted. Below we will introduce the two methods.

FM synthesis

We use the FM synthesis (frequency modulation synthesis) method to produce the piano sound. By using the Fast PWM function on the microcontroller we can generate a sound with particular frequency, which was already realized in previous labs. However, this method can only get a sine wave that does not match any timbre from an instrument. In order to change the timbre of the sound, we decided to use the FM synthesis method.

The FM synthesis is a form of audio synthesis where the timbre of a simple waveform is changed by frequency modulating it with a modulating that is also the audio range, resulting in a more complex waveform and a different sounding tone. It is developed by John Chowning at Stanford University in 1967-68. Bessel functions represented the mathematical principle of this method. The formula for elementary FM synthesis is given by

where (Ac, fc) specify the carrier sinusoid and (Am, fm) specify the modulator sinusoid. In order to get the real wave of the sound, the changes of Ac and Am with time should be like the following. So Ac and Am are replaced by some math expression and the output x(t) is given by 

where Ac is determined by attack_exp and tau_amp and Am is determined by freq_depth and tau_freq. So the most crucial part to synthesize a piano sound is to find some good parameters for these variables, including fc.

Though it is easy to calculate the formula above in a matlab program, we need to do some improvements to the formula to implement it in a C code. First, we replace t^(attack_exp) with (1-e^(-t/(tau_att )) ). Then we let Xn equal to e^(-t/(tau_amp)) and Yn equal to (1-e^(-t/(tau_att )) ), so

Let ∆t*k=2^(-P), so we get the code Xn+1 = Xn—(Xn>>P). It is similar to calculate for Yn. In our program for FM synthesis, we use amp_fall_main and some other variables to represent P in the timer1 compare ISR. For every 256/8000s, the program recalculates the parameters for the next Δt and then use the sine[] table and function to calculate the output x(t). If pluck equals to one, which means the key is pressed, all the variables should be set to the initial value. In addition, some variables should be right shifted for 8 bits because of the multiply calculation. Finally, x(t) is added to OCR0A to modulate the duty cycle of PWM.

Additive synthesis

In additive synthesis, we try to match the Fourier spectrum of the desired sound. Since we tried many parameters for the FM synthesis and the sound was always like a pluck instrument, we decided to use the additive synthesis method to generate the flute sound. Additive synthesis is a sound synthesis technique that creates timbre by adding sine waves together. The formula for additive synthesis is given by

where (Ai, fi) specifies each sinusoid with different frequency. According to the synthesis method, the form of Ai is much similar to Ac in the FM synthesis. So we use the similar algorithm to calculate Ai(n) and Ai(n+1) and right shift. The derivation is alike so we don’t do it again here.

In the program, we use three sine waves to create desired flute timbre. We calculate the variables amp_fall1 and amp_rise1 every 256/8000s by right shifting fall_1 8 bits. Then amp_1 is calculated as Ai. The variable inc_1 is used to set the fi and then calculated in sine[] table. So are the same for wave sine2 and sine3. Finally, we add these sine waves together to the OCR0A output with changing PWM duty.

To get the exact Fourier spectrum of the sound, we use a matlab program to analyze the .wav file of one note sound and show its frequency and amplitude distribution graph. The code is in the appendix. We use two notes to show the FFT outcome. One is from 262Hz piano and the other is from 262Hz flute. They are all generated from a simulated instrument program on the PC.

piano.wav

flute.wav

The graphs show that we need to add many sine waves together to generate a real-like sound. It is difficult to achieve on the microcontroller since those calculations will consume much of the CPU. To simplify it, we use three sine waves to generate the sine wave and the frequencies are f, f*3 and f*5. It simulate the sound well for the above program.

Sound output (DDS)

By using the DDS we can modulate the frequency of the wave. In the timer1 compare ISR, we change the value of OCR0A by a sineTable[]. We get the values in sineTable[] previously by sampling 256 points in a sine wave period. So each time the program goes into the ISR, OCR0A will change based on the sampling points. That means the output voltage value from PinB.3 will change in a sine wave form. This is the realization of DDS. For the frequency, the program goes into the ISR every 1/8000 s, so 2^16/8000*freq = 8.192*freq. From the above equation, we can convert frequency to DDS. Then the DDS is realized from the procedures described in the above paragraph.

LCD drawing

The LCD we use is a 2.8" TFT screen. It has a resolution of 320 × 240, as the following figure shows. There are 240 pixels each line and 320 pixels each column. We can use (x, y) coordinate to represent each pixel on screen.

Initialize of LCD screen

After a careful check of datasheet, we decide to use 8-bit interface. That is, the data bus is of width 8. There are some other control signals, including CS(chip selection), WR(write), RD( read), RST( reset), RS(register selection). The connection of these control and data signals to the AVR can be seen in the appendix. The ILI9325 has more than 40 16-bit control registers. To initialize the registers, it means we have to write the control bits to these registers. The following figure shows how we could write to these registers with 8-bit interface. According to the timing characteristic, we write several functions. They are:

void LCD_Write_bus( char VH, char VL) //write to bus

void LCD_Write_COM(char VH, char VL) //write command register

Void LCD_Write_DATA(char VH, char VL) // write data to register

Details of these functions are not included here. If you have interest, you could have a look at our code in appendix. Since our interface is 8-bit, it means that we have to send the data in high/low byte, so that it will send a 16-bit data. After writing the above three functions, it means that we could writing to these control registers. Details of the control bits written to each register are not shown here, they are also included in the code.

RGB565 interface

The ILI9325 chip has 172,800 bytes RAM for graphic data of 240RGBx320 dots. Here we use the 16 bit RGB interface, namely RGB565, that 5-bit for red, 6 bit for green, and 5-bit for blue. That is to say, we need to write 2 bytes(16 bits) data to each dot. In this way, we could use the funtion write above which also use a 16-bit data interface. A macro is written so that we can transfer the color we R,G,B value into RGB565, which is:

#define RGB565(r, g, b) ((r >> 3) << 11 | (g >> 2) << 5 | (b >> 3))

Drawing on LCD

After configuring the LCD, we write the function of drawing a pixel on the screen. This is doing to by the following procedure. There are two registers, 20/21 which represents the horizontal/vertical GRAM address set register. So our draw_pixel(unsigned int x, unsigned int y, unsigned int color) function, will first write the x and y value to the horizontal and vertical GRAM address set register. After doing this, we are telling the chip that we will write color to the dot (x, y) on screen. Then, after setting location, we will write the value of color to register 22, which is the GRAM data register. In this way, we can draw on pixel on the LCD with the color and location we want.

If we want to fill an rectangular area on the LCD, instead of using draw_pixel() function, we could use another way. By writing to register number 50 ~ 53, we can set the start/end position of horizontal/vertical address, that is, a rectangular area is now confined. The we can directly write values to register 22(GRAM data register), the chip will automatically increment the address vertically/horizontally, according to the control bits we set(that is, after filling a dot, the chip will automatically goto next dot, we only need to write another color value into register 22).

After having the pixel drawing and filling functions, we could extend them to other functions, a list of them a given below.

After having the pixel drawing and filling functions, we could extend them to other functions, a list of them a given below. After having these functions, it means that we could build our user interface based on them.

void Pant(char VH,char VL) // Pant the total screen to one color

void draw_line(unsigned int x1,unsigned int y1,unsigned int x2,unsigned int y2,unsigned int color)

void draw_circle(unsigned int x, unsigned int y, unsigned int r,unsigned int color)

void draw_rectangle(unsigned int x1, unsigned int y1,unsigned int x2,unsigned int y2, unsigned color)

LCD touch detecting

The Touch Panel is of the same size as the LCD Screen. It covers over the screen so that if we press a place on touch panel, we will know the (x, y) coordinate which corresponding to the screen.

The XPT2046 is a 4-wire resistive touch screen controller that incorporates a 12-bit 125 kHz sampling SAR typeA/D converter. The XPT2046 can detect the pressed screen location by performing two A/D conversions.

The following diagram shows the block diagram of the XPT2046. We will use the SPI interface provided by this module. We write two functions, WriteSPI() and ReadSPI() to communicate with the interface. The method we use to get the (x, y) coordinate is called differential reference. The theory is not explained here. If you have any interest, you can have a look at the datasheet of XPT2046 given in appendix.

There is a IRQ signal output, that only when we press the touch panel, the IRQ signal will become low. When released, it will become back to high. So we use INT0 external interrupt to detect the pressing. Every time the when detecting the falling edge of IRQ signal, we read the x, y data in the interrupt and decide the next step to do.

Mainpage

The mainpage of our program will draw the user interface, which will tell users several modes to do. The modes include xylophone, flute, piano record, piano replay, free piano jam, and piano tutorial. We start drawing the mainpage by drawing several rectangles with different colors. We also shows the string of each mode in the rectangles. One figure of how our mainpage looks like is shown below. When we are in the mode of mainpage and we press the touch screen, we will go into one mode according to the y coordinate we get. For example, if we find that 40 < y < 80, we will transfer to flute mode.

When we are in other mode, if we would like to go back to mainpage, just click on the lower left part of the screen and we will go back to mainpage and draw the mainpage again.

Xylophone mode

The xylophone is the first version of musical instrument we do. It simply consists of 8 color bar as shown in the below figure. Eight rectangles with different colors are drawn. Everytime when we touch the screen, in the ISR, we will get the (x, y) coordinate. According to (x, y) coordinate, we could know which color we are pressing. Each color corresponds to one note, from C5, D5, E5, ... to C6. To generate the waveform of each note, FM synthesis is used here.

Flute mode

The flute mode is the last musical instrument we implement. To draw the flute interface, the draw_rectangle and draw_circle functions are used. The below figure shows the how the flute is drawn. We first draw two black rectangles and on each rectangle we draw four white concentric circles. This will form 8 holes that we could draw on. A microphone is also used that when we are air blowing and at the same time covering one of the holes, a note will be generated.

We also add some visual effect when we covering the hole. That the outside circle will expand and shrink to show that the hole is covered. This is done by a timer counter, that when we cover the hole, we will reset the counter, clear the outside circle and draw a bigger circle. When the counter is bigger than a certain number, we will draw the outside circle back. This will cause a visual effect of expand and shrink.

Here, we use additive synthesis to generate flute notes.

Piano mode

To draw the piano on the screen, we first use Pant(0xff, 0xff) to pant the LCD white. Then we draw eight black lines to divide the black keys. We also draw five black rectangles to represent the black key. The following figure demonstrates the piano. In all the piano related mode, FM synthesis is used to generate the waveform we want.

Free Piano Jam

The free piano mode, like its name indicates, user can play the piano free. To detect a note, every time when we press the touch panel, we detect the (x, y) coordinate in the INT0 external interrupt and thus determine which area (x, y) belongs to. There are total 5 black key and 8 white keys on the screen. That is to say, 13 notes can be generated. In the free piano mode, in the upper right , we can change the keys we want, from low, middle, to high. The figure below shows the three states. We use green to fill the area to show current key is low(C3 to C4), red to represents middle(C4 to C5) and blue to represents high(C5 to C6). A table of frequency talbe is also given for reference. The transition between low, middle, high is a simple state machine. When the state is low, if we press "-", nothing will happen, we will be still in low. But if we press "+", that to increase key, the color will transfer from green to red, which means we have change the key to middle.

We can leave the free piano mode by tapping the lower left of the screen and go back to main page.

Record/Replay

The record mode has the same interface of free piano, except that there is no transition between low, middle, high keys. The aim for record mode is that we could record a piece of consecutive notes with precise tempo. We use two 100 element arrays to save the key we pressed and the time interval between two consecutive notes. In this way, we can record precisely save these consecutive notes we pressed. To record the key we pressed, we have a mapping between the 13 keys on screen to number 1~13, while 0 in the array means no press. So every time we enter record node, we will reset all the elements in the note array to zero. To record the tempo, we have a counter, which is auto increment in the timer1_compare interrupt. Every time we detect a new note is pressed, we save the current counter number to tempo array and reset it to zero. When we detect a overflow( more than 100 notes), we will automatically go back to main page.

If we have recorded the notes we want, we could go back to mainpage by tapping the lower left part of screen.

The replay mode has the same interface of free piano, except that there is no transition between low, middle, high keys. The goal of replay mode is that we could automatically replay the music we recorded. We will display what keys we have pressed with precise tempo. The two arrays, note array and temp array will be used here. In replay mode, when we press the screen, no music will be generated.

To demonstrate the key we pressed, we write two functions, draw_key( ) and clear_key( ), which will fill the area of key into pink, or clear the pink color to white/black. Every time a counter counts to the value of current interval between notes, it will clear the old key and draw the current key. Then the program will wait until count to next interval. When we detect that the current note array element is 0 (no key pressed), it means that we have replayed to the final note, the program will automatically go back to main page. Certainly, we can also go back to main page manually by clicking the lower west part of the screen. The figure below demonstrates the piano key that is filled pink

Tutorial mode

In the tutorial mode, we have two songs, "Mary had a little lamb" and "Twinkle Star". The notes and tempo are previously saved in arrays. This mode is similar to replay mode except that the notes will not be automatically played. Users can follow the tutorial and play the right notes himself. If user press the WRONG key, the WRONG note will be generated.

There are also two ways to go back to main page, manually or automatically when the song is over.