Piano Gloves

High Level Design    top

The genesis of this design came from an interest in sound synthesis as well as one of our team members being a musician. Previous 4760 projects have controlled music through glove interfaces so we attempted that ourselves on the PIC32.

The main design for this project was made up of two main components, which happened to be the hardware and software. The hardware component was made up of 10 Schmitt trigger circuits each connected to a flex sensor corresponding to a finger for a pair of hands. The software was loaded onto PIC32 and could read in signals from 10 digital IO ports and simulate a piano key press corresponding to the octave from C3 to C4, as well as the keys D4 and E4.

Background Math

The main mathematics used in this project relate to the Karplus-Strong Algorithm. This algorithm is used for physically modeling a string plucked in space. This algorithm works by loading some initial waveform (often noise, or in our case a low passed triangle wave) into a shift register of a desired length. The output is then fed through a delay line with a low pass filter, in our case, that returned a dampened average of the last two samples, then through an all-pass filter that would feedback to the shift register. This algorithm, in our design, was mostly used to model the dynamics of the individual strings. The sound of the piano came from other modifications we made in software.

Logical Structure

The main structure of our project was the user interfacing with the gloves. By flexing the fingers, the user changed the resistance of the flex sensors, effectively triggering the Schmitt triggers associated with a particular finger. This signal then propagated into a port on the PIC32. These 10 ports, each representing a finger (and therefore, a piano key) were continuously scanned in the music thread, and would set flags to simulate the key press upon a finger flex. This flag would trigger the string simulations in the ISR that wrote to the off-board DAC, which was connected to a final low pass filter and then to the final output of a female audio jack. The user could plug speakers into this output to listen to the keys they were playing.

Hardware/Software Tradeoffs

The main tradeoffs in software was the sample rate of our sound vs. length of each string vs. the number of strings to simulate simultaneously. The sample rate we ended up using for the final version was 11,025 Hz, whereas CD quality sound is 44,100 Hz. We used this frequency because higher frequencies would require the length of the shift registers representing the strings to be much longer, which would eat into the memory requirements. It would also shorten the number of cycles available between interrupts for the ISR and glove handling thread to run. If the number of cycles for the ISR to complete and return exceeded the period of the sampling frequency, the microcontroller effectively doubles the sampling frequency, as the timer interrupts are disabled inside the ISR. This gave a ceiling on how many strings (3 per key) we could simulate at once. Our final design was able to simulate 30 string (all 10 keys) simultaneously at 11,025 Hz without overflowing on the ISR.

Standards

We used a sample rate of 11,025 Hz, a common standard in digital sound recording. We also used a 3.5mm audio jack to output the sound. This output allows almost any speaker and headphone to be attached to our circuit.

Our code is under the MIT Open Source license which is as follows:

The MIT License (MIT)

Copyright (c) 2015 Sean Carroll, Natalie Moore, James Talmage

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

            

Existing Patents, Copyrights, and Trademarks

We used the protothreads library as created by Adam Dunkels (adam@sics.se) and modified by Bruce Land for threading our program.

After some research we found a similar product. The sound generated from this product is much different and closer to a xylophone rather than a piano. Its glove interface is also different in that the fingertips need to make physical contact with a surface. Our gloves only require the fingers to be flexed to a certain point.

Hardware Design    top

To determine if a note was being pressed or not, we attached a flex sensor to each finger of a pair of gloves. Two wires came out of each sensor and were then attached to a Schmitt trigger on a white board. Because the resistances of the flex sensors varied by up to 30%, we had to create a different Schmitt trigger for each sensor. Figure 2 shows our inverting Schmitt trigger design with switching voltages at around 1.2V. The output of each Schmitt trigger went into a pin on the MCU. When the finger was bent, the Schmitt trigger would output a high value and when the finger was straight, the Schmitt trigger would output a low value. Software was used to interpret this value and then play a sound. Figure 3 shows the assembled Schmitt triggers on the whiteboard. Figure 4 shows the analog input to the Schmitt trigger as well as the digital output.

We sewed each flex sensor to the underside of each finger so the flex sensors made good contact with the fingers. We decided to put them on the underside of the finger because it made a slightly larger flex as well as giving feedback where a physical key would be.

Software Design    top

The main challenges in building the software for this project were in simulating a realistic sounding piano in real time. Storing samples of a real piano was determined to be infeasible due to the memory constraints of the PIC32 package we were using. We wanted to keep the same package that we had been using, and generating the sounds ourselves seemed like a more interesting challenge than simply playing back prerecorded sounds.

There are several parts of a piano that need to be taken into account when creating a physical model of a piano’s sound. Each note has two or three corresponding strings that are struck by a wooden hammer covered in felt when the note is played. The two or three strings are tuned so that they are at slightly different frequencies. The lower notes on a piano have strings that are heavier and longer than the strings for the upper notes. When the hammer strikes the strings, they start to vibrate against the soundboard inside the piano, which then amplifies the string vibrations to make a perceptible sound.

In generating the piano sounds, we had to decide what model of a piano was most feasible while still sounding accurate. After hearing Bruce’s lectures on using the Karplus-Strong algorithm to physically model a string that had been plucked, we decided to attempt to modify this to suit our needs. Our model of a piano key was to use the Karplus-Strong algorithm, but rather than loading the shift register with random noise so as to simulate a pluck, we loaded the shift register with a triangle wave slightly offset from the end of the array, with the majority of the array still at rest. We determined through much iterative testing that this produced a sound more analogous to a hammer strike. After loading the shift register with this wave, we low-passed the shift register using parameters derived from investigative testing done in MATLAB at the start of the project (Code). From here, on each time step, we used the standard low-pass and shift delay used in Karplus Strong to calculate each value in time to be outputted to a 12-bit serial DAC, which was passed through a hardware low-pass and connected to a female audio jack so external speakers could be used. The other techniques we used in software to achieve a sound more like a piano key were: simulate three strings at once per key, two of which were offset by some amount from the fundamental string; after some delay following the initial key-press, reload the shift register with the initial shift register of the hammer strike, to simulate the second strike that occurs when you press down on a real piano key; an echo that consisted of a circular buffer implemented as an array that gave the tone the added richness that comes from the rest of the piano body.

Results    top

Since the flex resistors were only tolerant to 30% and fingers do not move independently from each other, R3 of the Schmitt trigger needed to be tuned to adjust when a finger hits the threshold. This can mostly been seen the different between the ring finger and the neighboring pinking and middle fingers. The chart below outlines the calibrated resistors we used.

Wires were not in contact with human skin since the gloves separated them from the hands as well as electrical tape insulation. The maximum voltage in the design is 3.3V as well which is regulated from the USB power source.

The Schmitt Triggers provided a robust interface to reduce analog noise from reading directly from the flex sensors. Having a digital output from the gloves also allows for the gloves to be completely modular from the software. A different glove design or even different input method would still work with our software. This becomes beneficial since every hand has a different shape and size. In our testing we found it difficult for some to move certain fingers independently from other fingers, which forced us to tune the position of the flex sensor on the glove. Having a flex sensor, which is lower on the finger, may work better for people will small hands but people with larger hand may have trouble flexing to the triggering threshold. Gloves could also be designed so that people with less dexterity in their hand could trigger a key-press.

As seen in the Fast Fourier Transforms (FFT) below, a real piano has much more complex sound. However, to the human ear there is a smaller difference if the sound. Below, we linked a few songs played on both a piano and our gloves.

Resistor	Resitance
R_flex1	32.5 kΩ
R_flex2	30.0 kΩ
R_flex3	33.6 kΩ
R_flex4	28.0 kΩ
R_flex5	27.1 kΩ
R_flex6	36.0 kΩ
R_flex7	30.0 kΩ
R_flex8	30.0 kΩ
R_flex9	40.0 kΩ
R_flex10	27.0 kΩ

What did not work    top

The initial shape of the string, when hit, made large differences in the quality of the sound. The initial algorithm started with random values and made a plucking sound rather than a mallet hitting a string, which happens with a piano. To simulate the mallet dynamics we tried a saw-tooth wave and triangle wave before settling on a triangle as the better sounding hit. We also found the placement of the initial wave to make differences to the sound quality. When the wave was placed too close to the string boundary it resulted in much higher frequency components. As seen in Figure 5 the mallets actually hit close to the middle of the string. This allows more of the string to vibrate close to the time of initial hit.

Also seen in the picture is how each key consists of three strings of slightly different lengths, which we implemented, but also how the lower notes only have two and not shown are how the lowest notes on a standard piano are actually just one much thicker string. When we alter our code to play notes in these lower octaves the sound is off because of the different style of producing sound. The lower strings would have to be modeled using a non-ideal string since it has so much mass. We found the sweet spot of our algorithm to be from C3 to E4.

Conclusions    top

Our piano gloves were successful at playing all of the white keys within a 10 note span on a piano. Because we tuned each note individually, the frequencies played were the correct frequencies of each note. We were able to play all ten keys at one time, which is equivalent to simulating 30 different strings at a time. This project cost $95.50 which was within the $100 allotted budget. We believe we have achieved a piano-like sound rather than a plucking, bowed or percussive sound.

To expand on this project, we would add the ability to play the black keys on the piano. We would also add the ability to shift octaves so that a larger variety of notes could be played. If we had more room in our budget, we would have incorporated the TFT so that the note being played could be displayed for the user. Since we followed the standard of using a 3.5mm audio jack, we are able to make further additions.

We modified the Karplus-Strong algorithm for a plucked string. The initial test code we started with was from Bruce Land. We also used the Protothreads Library which is in public domain. We did not reverse engineer a design but rather modified existing MATLAB code into C code for the PIC32 using our own ideas of what would work. No non-disclosure agreements were signed for this project.

Patent opportunities may exist for this project but would require research and action. Instead our code is open-source licensed with the MIT license. The synthesis in C of a realistic piano has the potential for a publication in hobbyist magazines and websites.

While designing, we kept in mind the IEEE Code of Ethics so that all users would be safe from harm. We soldered the connections using appropriate safety gear and we covered the soldered connections with electrical tape. If we were to make this a device intended to be sold, we would eliminate using white boards and would solder all of our components for added safety. We would also decrease the length of the wires connecting the gloves to the board and tie them together so they would be less hazardous. Our project is non-discriminatory and was designed so that those with disabilities could still use our gloves. In contrast to a normal piano, the user doesn’t have to move their fingers as far so those with limited hand mobility could still use our gloves. However, those who are unable to play a real piano would have trouble using our device, but that is due to the nature of our glove design and not because we were trying to exclude them while designing our gloves. We also took care to credit any outside sources that we used for help or for inspiration for our project. In addition, the claims made in our results section were accurate and not fabricated or exaggerated in any way.

There are no legal considerations for this project as far as we know. In our opinion, our project is different enough from existing products such as the aforementioned product.

Appendices    top

Appendix A. Source Code

Check it out on GitHub

Appendix B. Schematics

Appendix C. Cost

Part	Vendor	Cost/Unit	Quantity	Total Cost
PIC32MX250F128B	Lab	$5.00	1	$5.00
FS-L-0055-253-ST Flex Sensors	Mouser Electronics	$6.45	10	$64.50
MicroStick II	Lab	$10	1	$10
White Board	Lab	$6	2	$12
Speaker	Lab	$2	1	$2
Gloves	Wal-Mart	$2	1	$2
MCP4822 DAC	Lab	$0	1	$0
MCP6242 OpAmp	Lab	$0	10	$0
Wires, resistors	Lab	$0	as needed	$0

Total   $95.50

Appendix D. Division of Labor

Sean Carroll	Natalie Moore	James Talmage	Everyone
Implemented Karplus-Strong in C with our modifications from MATLAB	Design and building of (5) Schmitt triggers	Design and building of (5) Schmitt triggers	Developed a MATLAB prototype
Software Debugging	Sewing flex sensors on (1) glove	Sewing flex sensors on (1) glove	Final tuning and testing
Amateur Composer/Piano Tuner	Musician on Staff	Glove Attendant	Hardware Debugging

Acknowledgements    top

We would like to thank Bruce Land for a fantastic class and all the TAs for all their help!