Gesture based MIDI glove

Background Math

Geometry of pitch and roll. The accelerometer gives values as the magnitude of force in the X, Y, and Z directions, this values themselves don’t give the orientation of the device. To calculate the pitch and roll of the device we used the formula:

We considering applying a digital filter to the accelerometer data to suppress noisy measurements, but we decided that the resolution we were looking for was coarse enough that a filter wouldn’t have a noticeable effect on the notes the microcontroller sent.

Logical Structure

There were 3 main sections to project. The first was the 2 flex resistor connected to the microcontroller into the A0 and A1 ports, which are the analog inputs to the internal ADC. We had our accelerometer/gyroscope, the LSM9DS0, which was connected to the serial bus clock line, C0, and to the bi-directional serial bus I/O data line, C1. This is so we could take data from the accelerometer/gyroscope (we’ll use IMU as an abbreviation for the rest of the report) and manipulate it from the Mega 1284. We also had a voltage buffer circuit that helped to stabilize the voltage when there were changes in current going into the IMU. Lastly was the serial to USB converter, which allowed us to attach the microcontroller to our computer.

Hardware Software Tradeoffs

For a bulk of the processing, we relied on the Atmel Mega 1284 to handle internal calculations and any information it received from our accelerometer/gyroscope (we’ll use AG as an abbreviation for the rest of the report). The amount of hardware in this design was not very extensive. We had 2 voltage divider circuits for the flex resistor, a prebuilt AG and a voltage buffer that allowed for any current being drawn by the AG to not change the voltage. We used the AG hardware to determine values such as the heading and acceleration of the glove and sent that information via I2C/ TWI to the controller. Also, by making the design more software oriented, we were able to easily change the functionality of different specific movements of the glove by modifying the code instead of possibly having to change hardware parts to achieve a different result. We decided to not use an actual MIDI port and a 31.25 kHz signal because it has been done many times before and we decided that it would be more flexible to use a serial to USB cable to allow us to focus more on the actual sound generation.

Hardware Design

Flex Resistors

The flex resistors played a crucial role in the MIDI controller. They served as our on/off switch for different functionalities of the glove including turning a note on/off and turning the pitch bend on/off. The pointer finger was used to to turn MIDI channel 1 on and off based on the voltage output from the flex resistor circuit. The middle finger was used to control our pitch bend/vibrato. This function would only work if the point finger was also held down as you can not pitch bend a note that is not playing. We got the design for the flex resistor circuit from the Sparkfun website, where we purchased the resistors, which are essentially a potentiometer.

The flex resistors have a resistance of 30 kohms when straight and 50 kohms when bent 90 degrees, according to the website. However, the ranges weren’t that concrete when we test the resistors and we got ranges anywhere from 25 kohms when straight to about 60 kohms when bent. We took this into when writing the code that determined the threshold for “on” or “off”. The best way to incorporate the flex resistor into our design was to make a voltage divider with the flex resistor and a 10 kohm resistor. The output of the voltage divider is then fed into the analog input. With this circuit, the output will be about 75% of Vcc when straight and about 83% of Vcc when bent completely.

Serial to USB

In order to actual send MIDI signals, we first needed to be able to send serial communications between the microcontroller and the computer. For this, we used a serial to USB converter that is available to us in lab. The Tx port of the converter was connected to port D1 on the microcontroller and the Rx port of the converter was connected to port D0 of the microcontroller.

Accelerometer and Gyroscope

The accelerometer/gyroscope was probably the most important hardware part of our circuit. This is the piece of equipment that gave us all of our headings, directions, direction changes, and accelerations which we used to give the MIDI controller movements actual meaning. As described below, the IMU was also connected to a voltage buffer in order to help maintain a constant 3.3 V voltage source. The Vcc of the IMU was connected to this buffer circuit.

The other pins of the IMU that were used were GND, SCL, and SDA. The SCL and SDA ports were connected to the serial clock I/O pin, C0, of the microcontroller and the serial data I/O pin, C1, respectively. I2c requires a pullup resistor so that the sender can pull the line down, rather than pulling the line up which allows for mixed logic level devices.

The accelerometer part of the device is used to aid us in finding the orientation of the user’s hand. It does this by gauging the orientation of a stationary object with relation to the Earth’s surface. When you accelerating in a particular direction, it can’t distinguish that from the acceleration due to the Earth’s gravitational pull. The accelerometer can give the user a direction of gravity and with a three-axis accelerometer like we had, we were able to use it to determine the orientation of the hand relative to the Earth’s surface. This was used to determine which pitch would be played through the synthesizer. Depending on the orientation of the hand, a different note would be played. This also allowed us to determine “starting points” for functionalities that required use of the gyroscope.

The gyroscope helped us by being able to measure the rate of rotation around a specific axis. This only gives us a value while the device is still in motion. Once it stops moving, we will get a reading of zero. This was critical in some of our key functionalities. It allowed us to make a flick of the wrist to the right about the Y axis equal to an instrument change by having a threshold rotation rate of 240 degrees/second. Anything faster than that would cause an instrument change while anything slower would be ignored. We also used it to change the key of the piece of music we were playing in. Because a change in key is just a shift of all of the notes in the key by a specific half step up or down, we made it so that when the user flicks their hand up or down about the X axis, the key would either go up or down appropriately. This makes the mapping of the notes more effective. If you only have the notes in a specific key available to be played, it maps a smaller range of notes to the “pitch space”, making it easier to hit the correct notes. Additionally, if you overshoot or undershoot the note you want to play, you will just end up playing another note in that key, which doesn’t sound nearly as bad as playing something out of key. One of the last functionalities the gyro provided us was the ability to cause a pitch bend or vibrato. If both of the flex resistors were held down and you wiggle your hand about the Y axis, it will cause a vibrato in the note you are playing.

Voltage Buffer

After talking with Bruce, we decided that a voltage buffer would appropriate in this circuit. The voltage buffer always gives a precise voltage controlled by the trimming potentiometer while allowing current to flow from the voltage source to ground. We needed this specifically because the accelerometer needed a 3.3 V source rather than just a 3.3 V signal. We found that the accelerometer was not functioning with just a trimpot even though the gyroscope and magnetometer were.

Final Product

Below is a picture of the whole project together:

Software Design

Midi Stream

The generation of the midi stream was relatively straightforward. We wrote bytes to the UART serial stream which was interpreted on the computer by the Hairless MIDI software with the uart_putchar function. To send a note we sent a status byte with the note on command and the MIDI channel we wanted to play on. We then sent a 0 with a 7 byte unsigned velocity value appended, then a 0 with a unsigned 7 bit note value. We also implemented sending a pitch bend command, which had the status and channel byte, along with two data bytes containing a signed 14 value split across two bytes. We also implemented changing the instrument which had a status byte and a 7 bit data byte.

MIDI Note and Command Generation

With the MIDI standard there are note on and note off commands. This represents a key being pushed down and released. Since our instrument worked like that either your finger was bent or it was not (pressed or released). To turn off the note you need to send the same note code that was sent for the note on. When the flex resistor registered a bent finger the ADC interrupt would set a flag telling the main task to play a note. When the play_on flag was set, the main task would get the roll and pitch from the function. With this value it would call the function which calculated the note that was to be played. It would then store this note in a global variable so it could be later turned off. When the ADC interrupt found that the middle finger was bent we set a flag adc_chan1_on. If a note was playing (we had seen the play_on flag but not the play_off flag) we would use the roll of the accelerometer to bend the pitch of the note. We would get the difference in roll between when the note was played and the current roll, and bend pitch proportional to the difference. Similarly to how a note on command is sent we also send a note off command when the finger is detected to be unbent. Rather than calculated what note to turn off from the accelerometer’s roll we use the note stored in the global we set when the note was played. Each time the main task executes we also check the gyroscope to see if a “flick” gesture is happening. We see if the rate from either the X or Y axis is above a certain threshold. The threshold we chose was not an obvious decision to make. We had to make sure that it wouldn’t be inconvenient to flick your wrist at too high of a rate and we didn’t want to make the threshold so low that it would be triggered in accidentally. We chose a value of 240 degrees per second. Since the gyroscope sample very frequently we had to make sure that each of the flicks only activated one change in the program. To do this we used a counter so that when a flick was registered we would count down to zero. A flick could only happen when this was zero. It would count down to zero in 300ms. There were four types of flicks, up, down, left, and right. A left or right flick would an instrument change command, incrementing or decrementing the instrument value. An up or down flick would change the key by incrementing or decrement the key variable up or down respectively. A key of 0 indicates a C-Major key. All of this code is commented in project.c and documented under project.c in documentation.

I2C

We used a library for the i2c from Peter Fleury. It includes functions for reading and writing bytes on the i2c bus. Each i2c device has an address which allows it to know that bytes are destined for it or that it can take control of the bus to send. Effectively the API is i2c_read(address) and i2c_write(address, byte).

IMU, Inertial Measurement Unit

The IMU used registers to communicate with the microcontroller to retrieve the values. To read from a register we need to write the address of the register we want followed by the byte we are writing to the register. To read from a register we first write the address of the register, we then request a read from the IMU and then read the byte from the i2c bus. To use the IMU we needed to set the scale (2g, 4g, etc.) of of each of the sensors, set the refresh rates, and enable each of the 9 axes. Each of these values are bits in a register. There is a separate register for each of the 9 axes which contain the current value of the respective axis. The accelerometer and magnetometer were on a separate i2c device from the gyrometer due to the limited amount of possible registers (7 bits = 2^7 = 128 possible registers). Documentation of the IMU code can be found in the imu.c and imu.h webpages

Things We Tried Which Did Not Work

We went through many iterations of the design of the controller until we found a combinations of functions that we liked that we thought was also practical for the user. Initially, we had an upward rotation about the X axis as a parameter that controlled the speed of the vibrato of the note. This seemed interesting at the time, however, became difficult to accurately play the right note with the hand turning about two different axes at the same time.