Keyboard Gloves

High Level Design    top

Hardware

The hardware consists of 10 push buttons, one per finger/thumb, as well as one flex sensor for each finger (where "finger" excludes the thumb). We also have copper contacts between the index and middle finger of each hand. These contacts determine which column of the keyboard the index fingers are pressing. We use two 8-to-1 analog multiplexers (muxes) to reduce the number of pins needed to read in the flex sensor and push button values. We also have a non-inverting amplifier circuit that amplifies the output of the mux for the flex sensor readings, providing a larger dynamic range from which the voltage can be used to distinguish between the three possible key rows for a given finger. Voltage divider circuits were created for each of the above mentioned inputs. Diagrams for each component can be found in the Hardware section, along with more detailed descriptions of the components.

Software

The responsibility of the software is to convert the sensor data into a UART message that indicates what key is pressed. For this project, we limit possible keys to lowercase letters, a comma, a period, and a backspace. We base our keyboard on a standard QWERTY keyboard and replace the semicolon and slash with a backspace. Note that we do not support SHIFT, ALT, or CTRL keys.

In order to perform this task, the software scans each button sensor to check if any of them are pressed. If so, we select one finger (prioritized from right to left) that was pressed. We change the select lines for the analog mux to select for the finger that was pressed. We then use the PIC32’s on board ADC to read the voltage across the flex sensor, which changes as the user bends their finger. Based on this value and predefined thresholds, we figure out if that finger was pressing the top row (‘q’ through ‘p’), the middle row (‘a’ through ‘;’), or the bottom row (‘z’ through ‘/’) of the keyboard. We then package this information in a UART message that is sent to a Python script. This Python script read the serial input and converts it to a keyboard event that behaves as if a user pressed a key on an actual keyboard.

Gloves

We sewed the flex sensors onto the back of the 8 fingers (excluding thumbs). We also sewed the buttons onto the tips of the finger tips. The copper tape comes with its own adhesive, which we use to stick it to the gloves. We also sewed additional stress relief loops near the wrists that help prevent the wires to the flex sensors from being pulled apart. See the results section for pictures.

Hardware    top

Buttons

In order to detect when a user has pressed a key, we have push buttons at the end of the finger tips. Not only does this make the hardware simple (as opposed to some sort of pressure sensor), but it also gives the user tactile feedback so that they know when they have pushed a button. We also have push buttons on the thumbs to register when the user pushes the space button. We use an 8-to-1 analog mux for the 8 push buttons for the fingers and two additional microcontroller (MCU) pins for those of the thumbs. For the 8-to-1 analog mux buttons, we added a pulldown resistor shown in Figure 2. For the thumb buttons, we used the PIC32's internal pulldown resistors.

Although we initially wanted to use a digital encoder instead of an 8-to-1 mux, we did not have access to a digital encoder and did not want to wait until one arrived.

Flex Sensors

To detect which row of a keyboard the user has pushed, we have flex sensors spanning the two knuckles of each finger (closest to the finger tips), excluding the thumbs. The user’s fingers will either be extended, half bent, or bent. We use another 8-to-1 analog mux to read in one of the flex sensor readings at a time, and this value is first amplified with the op-amp circuit before being input to the MCU. Thus the flex sensors are able to distinguish between the three modes corresponding to the three possible key rows. Voltage divider circuits, shown in Figure 3, were created. We used a 30kΩ resistor to match the flex sensors' resistance of approximately 30kΩ. This maximizes the dynamic range of the output.

Index Finger Sensor

To detect whether or not the index finger is extended towards the center portion of the QWERTY keyboard (for example, for keys “f” versus “g”), we use copper tape between the index finger and the middle finger. Two additional voltage divider circuits were created, one for each hand. When the copper tape pieces touch, the two wires are shorted together, indicating the index finger is close to the middle finger. Otherwise, we know that the index finger is extended towards the center of the keyboard, so we know that one of the middle group keys is pressed. This takes 2 pins, one for each hand. The copper contacts for each hand are connected directly to the MCU pins, which have the internal pull-down resistors enabled, and as a result, no external resistors are needed.

Amplifier Circuit

To increase the dynamic range of the flex sensor reading from the analog mux, we created a tunable non-inverting amplifier circuit using an opamp, several resistors, and a trimpot. This made it possible for the software to distinguish between the different rows of keys using the amplified flex sensor input to the MCU. The circuit diagram is shown in Figure 4. We needed the trimpot in order to dynamically adjust the voltage being subtracted from the input. Because each flex sensor’s base resistor value is slightly different, the range and starting voltages for the across the different flex sensors varied. The trimpot needed to be adjusted so that none of the flex sensor values were saturated, while the resistor values were chosen to maximzes the dynamic range of the ADC.

USB Cable

We use a serial cable like in Figure 5 to communicate via USB to the computer. We use GND and VDD to power our PIC32. The data lines are connected directly to PIC32 pins.

Pinout

Table 1 is a list of each specific MCU pin assignment with respect to the above mentioned hardware inputs and outputs.

PICkit 3 Setup

Instead of using a PIC Microstick, we decided to put the PIC32 directly on the breadboard and program it with the PICkit 3, making the final product cheaper and more compact. To run the PIC32 on a breadboard, we added the following components:

• 10 kΩ resistors between PIC32’s MCLR and VDD.
• (optional) switch between PIC32’s MCLR and GND for easy resetting of the PIC32.
• 0.1 uF disk ceramic capacitor between PIC32’ VDD and ground, close to the PIC32.
• 0.1 uF disk ceramic capacitor between PIC32’s AVDD and ground, close to the PIC32.
• 10 uF tantalum capacitor between PIC32’s VCAP and ground
• Connect PIC32’s Vss to GND

The above bullets describe how to run the PIC32 on a breadboard. To program the PIC32, we need to connect it to the PICkit 3, which is then connected to a computer via a USB cable. A summary of the pin connections to the PICkit 3 are shown in Table 2. The complete setup is summarized in Figure 6.

VDD and GND can be supplied by the PICkit 3. To do so, follow the these instructions. They are copied below for convenience:

1. Go to Run > Set Project Configuration > Customize...
2. In "Categories" window click once on PICkit 3.
3. In the right side you have "Option Categories" drop down box, click on "Power".
4. Now check the box for "Power target circuit from PICkit 3".
5. Choose the "voltage level" from the drop down menu.

Otherwise, VDD and GND should we powered externally. We used the USB cable to power our PIC32 externally.

Software    top

PIC32 Code

Setup

There are several components of the PIC32 that we need to set up, and each is described below in detail.

ADC

The ADC is set up with the following code:

Code Snippet 1: How to set up the ADC.

We refer the reader to the PIC32 Peripheral Reference manual for details on choosing the parameters.

Digital In/Out

It is the responsibility of the software to control the select lines of the analog mux units. This is done by setting the appropriate port pins to be outputs. Similarly, to detect if a space bar was pushed or if the index fingers are separated, we configure ports to be digital inputs. For the digital inputs, we enable pulldown resistors, simplifying the external circuit. Code to perform the above functions is given below.

Threads

We then use the Protothreads library to schedule the following three threads in a round-robin fashion:

1. Computer Interface - Responsible for sending data over UART to a Python program
2. Glove - Responsible for converting sensor input from the glove to a character to press
3. Debug - A thread used to display data

Calibration

Although we initially had global thresholds shared across every finger, we found that each flex sensor is slightly different. Therefore, to figure out whether a finger is pressed at the top, middle, or bottom row, we needed different threshold values for different fingers. To determine these values, we implemented a calibration phase, which can be optionally turned on by setting the CALIBRATION preprocessor constant to 1.

Calibration is implemented by prompting the user to press a key, and then storing flex sensor value in an array. For each key, we request a fixed number of samples from the user. Once the user has finished providing samples for each keys, we take an average of the samples to give us the average flex sensor value for each finger. The threshold between consecutive rows is halfway in between the two average flex sensor values for that finger. For example, if the sample flex sensor value is 50 when the user presses ‘q’ is 50, and 70 when the user presses ‘a’, then the threshold between the top row and middle row for the left pinky is set to (70+50)/2 = 60.

Computer Interface Thread (UART + Python)

To send a message to the computer, we use UART communication protocol. We spawn a thread that uses direct memory access to send a message via a USB cable. We spawn a thread to avoid blocking the main program. The format of the message uses the protocol described in Table 3.

In calibration mode, the computer interface sends a character to the computer to prompt the user to press a certain key. Otherwise, it sends either a character or a spacebar message. Note that we needed a separate message format for space bars because the Python script expects space-delimited input.

The computer interface thread waits until a flag called new_key_pressed is true to look at the next key_pressed value and react accordingly. The Protothreads library is a cooperative operating system, so it is the responsibility of the Glove thread to set this new_key_pressed flag true and also update the key_pressed variable.

Glove Thread

The gloves thread is responsible for convert the sensor data into character pressed. It scans each of the finger buttons by setting the control bits of button analog mux, waiting 3 milliseconds to let the output of the mux to change, and then reading in the value to see if a button is pushed. If so, we set a variable called pressed to indicate that a button was pressed in this pass through the thread. To prevent a single button press being registered as multiple button pressed, we debounce the input based on this pressed variable.

Figure 7 has a diagram of the state machine used to debounce a button pressed. The variable pressed is the input. Each time there is a transition to the PUSHED state, the detected keypad input is valid and is handled according to the specific button that was pushed.

Once a valid key has been pressed, we know which finger has been pressed. This tells us which column of the keyboard was pressed. If the index finger was pressed, then there is ambiguity on which column is pressed. To disambiguate this result, we read whether or not the respect index finger is close to the middle finger on the same hand by reading the index finger sensor. Together, this tells us which column of the keyboard was pressed.

We change the flex sensor analog mux to select the line for the appropriate finger. Based on the flex sensor value and the thresholds, we determine which row was pressed. Once we have the row and column, we look up the character in a predefined array, store it into the key_press variable, and set the new_key_pressed variable to true.

The exception to the above execution is if we are in calibration mode. In that case, the only difference is that we store the flex sensor value into an array of samples and simply set new_key_pressed to be true.

Debug Thread

We reuse the UART communication thread to send data to the computer. This is done by setting the first character to ‘d’ in the messages. For our uses, we used this to print out the measured samples from calibration as well as the calculated threshold values.

Python Code

We use Python 2.7 and the pySerial and pywin32 libraries to convert the serial input from the PIC32 to keyboard commands. The serial module from pySerial library allows us to connect our Python program to a serial port, e.g. COM5. This is done by creating a serial object, configuring it, and then opening the port, as shown below:

Code Snippet 3: Python code to set up serial communication.

We can then use the ser.readline() method to read a new line from UART. We defined a simple message format for Python to parse the input. See Software - PIC32 > Computer Interface Thread for more detail on this protocol. If a message indicates that a key needed to be pressed, we use the win32com module from the pywin32 library to create a Windows keyboard press event as shown below:

Code Snippet 4: Python code to create a keyboard event on Windows.

By implementing this part of the code in Python, it allows for greater flexibility since one can now connect the gloves directly to a Python program. Although this implementation limits the use to windows computers, similar Python modules exist for Linux and Mac.

Initially, we were hoping to register the PIC32 as a HID/USB device so that we could plug it into any computer and have it work, without needing to run a Python script. However, with our time and budget constraints, we decided that this would not be feasible and found that the Python script was sufficient.

Typing Tests    top

The goal of our project is to allows a user to type. To test this, we used three online typing tests that are free to use:

1. Basic Typing Speed Test, which has the user type a simple paragraph.
2. Online Typing Tutor :: Learn to Touch Type, which has the user type a few simple words.
3. Typing Test, which has the user press different letters for 30 seconds.

We had the glove user (Vitchyr) perform these tests with the gloves and with a Dell KB212‑B QuietKey USB keyboard. We were limited to tests that only had lowercase letters and basic punctuation because our keyboard gloves did not implement the SHIFT key [1]. Screenshots of the test are shown below, in Figures 8, 9, and 10.

In the first test, we required the user to press backspace if they typed incorrect letters until what they had typed so far was correct. This most accurately reflects a real use case, since users typically fix all typographical mistakes in preference for writing quickly. So, the first test measures the effective words per minutes. The second and third tests ignore any incorrectly typed letters. Instead, these tests focus on the speed with which the user can press keys and type words. The results are discussed in the next section. Unfortunately, different tests we used reported slightly different metrics, but we are still able to compare the results.

The user had about 4 hour with the gloves on. However, most of the time was for development (e.g. making measurements and adjusting the buttons) and not for typing. Overall, the user typed approximately 500 words with the gloves during development. When the time came to actually run the experiments, the user calibrated the glove by sampling each key three times. After that, the user tested each calibrated key once, and the typing tests began immediately. The user was not given time to practice typing between calibration and the typing tests.

Results    top

Typing Tests Results

The results of the typing tests are summarized in Table 4.

One interesting result is that the words per minutes (WPM) of the second test is similar to the WPM of the first test, despite the focus on speed. This is probably because the user is used to correcting mistakes with backspace, and so the benefit of being able to ignore mistakes is offset by the different typing style required.

If we look at the keyboard results column, we see that the user achieved 96 characters per minute (CPM) for test 3, but 441 CPM for test 2. This discrepancy is most likely explained by the nature of the tests. Test 2 had the user type full words, so the user could see the entire word to type, as well as the next word to type. For test 3, the user was only give one letter at a time. Once the user pressed that letter, the next letter was prompted. This is much less natural than typing words, which prevents the user from using muscle memory to fluidly type. In addition, the fact that the user could look ahead in test 2. So, the faster CPM in test 2 is expected.

In contrast to the keyboard results, the CPM for the gloves did not change significantly from test 2 to test 3. This implies that the user was not able to use the muscle memory to type words faster, and that the user was not significantly slowed down by having the letters prompted one at a time. Furthermore, the accuracy dropped by 10%. Overall, this suggests that the main bottleneck in using the gloves is figuring out how to press the next key. The user was not able to get into a natural typing mode that would increase their CPM.

Both tests 1 and 2 suggests that the effective WPM of the glove is about 10 times slower than an actual keyboard. Considering that the user only calibrated each key three times and had little practice using the gloves, these results are promising. We believe if the user is able to become comfortable with the gloves, they will be able to use their muscle memory from using QWERTY keyboards to significantly increase their WPM.

Calibration Results

We were able to successfully calibrate and use the keyboard in one sitting. Table 5 gives some typical values for the sampled thresholds.

As seen in Table 5, the values differ from column to column (finger to finger) since not all flex sensors are exactly the same. As expected, columns 4 and 5, which both correspond to the left index, and columns 6 and 7, which both correspond to the right index, have similar values. Since none of the thresholds are 0 or 1023, we know that we did not saturate the output of the amplifier circuit.

Hardware Results

Figure 11 shows the input (bottom) being amplified by the op amp circuit to give the output (top). Observe that the input has a DC offset, but the output does not significantly amplify this offset. The trimpot controls the offset of the output.

Pictures

Below are pictures of the final circuit and glove. Figures 12 and 13 show the circuits. Rainbow ribbon cables were used to connect the two breadboards. Figures 14 show the glove. Push buttons can be seen on the tips of the finger and flex sensors along the back of the fingers. Additionally, copper tape contacts can be seen between the index and middle fingers. The stress relief loops can be seen more clearly in Figure 15. Figure 1 at the top of the report shows all of the components used in this project, including USB cable with the GND, Vbus, Data-, and Data+ lines exposed, and the PICkit 3 to program the PIC32.

Videos

Below are two videos. Video 1 shows a demonstration of using the keyboard gloves. Video 2 gives a technical overview.

Conclusion    top

Expectations and Future Work

We were able to create gloves that successfully emulated a QWERTY keyboard as expected. Although we have some room for improvement, we were able to implement all of the key features. If we were to repeat the design and construction of the project, we would attempt to add control keys, such as SHIFT, CTRL, and ALT, as these keys are also important to the normal function of a keyboard. Another area of improvement is the buttons on the end of the finger. They are rather small, so it is sometimes difficult to press them. More importantly, we would either increase the length of each wire in order to increase mobility and flexibility when wearing the gloves, or we would make the gloves wireless in order to even further gain mobility.

Although this version is limited to Windows, it can easily be expanded to different operating systems by updating the Python script for the appropriate operating system. A more long term but technically challenging solution to this problem would be to register the PIC32 as a Human Interface Device/USB. This is how the protocol used by actual keyboards. This modification would make our gloves behave exactly like keyboards, allowing us to plug them into practically any operating system and use them immediately. One disadvantage is that there would be less flexibility. We would have to conform to this standard rather than using an easily modifiable Python script, but it may be worth the portability.

Another limitation is that the placement of the buttons were adjusted to fit one user. This makes it more difficult for other uses to use the same gloves. We could resolve this issue by either producing multiple sizes of gloves or by using material better suited for easily adjusting to various hand shapes and sizes.

Applicable Standards

By using a keyboard emulator written in Python, we avoided registering the gloves as a Human Interface Device (HID). Instead of HID, we used UART serial communication to interface between the PIC32 and the computer. We also note that in order to use the gloves, the user is required to adhere to the standard Touch Typing method in terms of finger placement and key assignments, which we include as a relevant standard necessary for this project.

Ethical Considerations

With regards to the IEEE Code of Ethics, our team adhered to all 10 properties under Section 7 (Professional Activities). In particular, by stating the limitations of the keyboard gloves in terms of typing speed, typing accuracy, range of possible keys, as well as limitations on the hand size needed for the best typing results, we followed the commitment “to be honest and realistic in stating claims or estimates based on available data.” Furthermore, all proper credit was given to both documentation and individuals that were of help to us during the design and implementation portion of our project. In terms of preventing any possible injury, the safety of using the keyboard gloves as well as interacting with the necessary hardware was always taken into account by avoiding human contact with the voltage levels used to power various parts of the project. Additionally, contact with the gloves and the hardware was limited to the two group members working on the project, both of who were qualified to be working on this project.

With respect to the commitment to “to improve the understanding of technology; its appropriate application, and potential consequences,” our team adhered to this property by ensuring the gloves were only used for their intended purpose of being a keyboard that interfaces via USB with a computer. We do acknowledge that the gloves may imply a bias towards individuals with a hand size similar to our test user, with the unintentional effect of being better suited for male users. We stress that this particular set of gloves were constructed as a proof of concept of the keyboard glove project, and that this effect is unintentional.

Legal and IP Considerations

In terms of legal considerations, we do not foresee any issues, as all parts were either purchased on our own or taken from the lab supply of components, to which we were granted access by our instructor Bruce Land. No sample parts were used from manufacturers, and so we did not need to sign any non-disclosure agreements. We did not reverse-engineer any designs, nor did we outright use any other individual’s designs. Additionally, we did not find any existing products that exactly matched our particular keyboard glove design; however, we did see similar products as mentioned in the introduction. This project was primarily educational and we do not plan to manufacture gloves for sale. Thus, we will not violate any intellectual property patent/trademark issues. We do foresee publishing opportunities online as well as in various journals and magazine articles that focus on our particular method of implementation.

Appendices    top

A. Parts List and Costs

B. Source Code

• PIC32 code
• Python code

C. Division of Labor

Vitchyr Pong

• Designed, implemented, and tested software
• Designed, built, and tested amplifier circuit for output of flex sensor analog mux (Figure 12)
• Designed, built, and tested analog mux circuits (Figure 12)
• Assembled and tested stand-alone PIC32 circuit to use PICkit 3 (Figure 12)
• Soldered index finger sensors

Gulnar Mirza

• Handled glove design and construction (Figure 13 & 14)
• Soldering of all wires, push-buttons, and header pins/sockets
• Placement and sewing of all components on gloves
• Designed analog mux circuits for flex sensors and push-buttons
• Designed and built external voltage divider circuits for push buttons and flex sensors (Figure 11)
• Assisted with testing and debugging of hardware

D. References

This section provides links to external reference documents, code, and websites used throughout the project.

Acknowledgements    top

We would like to thank the instructor Bruce Land, who helped us throughout our project. He helped us debug various parts, order supplies, and design the amplifier circuit.

We would also like to thank our lab teaching assistant Taylor Pritchard, who helped us debug our project, filmed our demonstration video, and suggested that we use Python for serial communication.