• A calculator
• A monitor
• A stopwatch
• A rock-paper-scissors game

The project idea initially involved a robot to output a raster on sidewalk. Then it involved a portable multiplayer gaming device. Then it involved a virtual keyboard / abacus. Finally, it evolved into the current project.

One of the ECE 476 assigned readings, "Synthetic Serendipity", features a device which uses gestures and body motions to interact with a wearable computer - We immediately realized that it would be very possible create such a Human Controllable Interface (HCI) with PVDT film.

We focused on the interface aspect because computer interfaces have not changed drastically in over 30 years (since the mouse and its cousins touchpad, trackball, and thumbstick). A major obstacle to ubiquitous and portable computing is the intrusiveness of computer interfaces. Previously, the size and awkwardness of computers presented the major obstacles to portability. However, with today's technology, it seems that today's primary obstacle is the lack of an intuitive and convenient interface.

The orthodox interface for current computation is the keyboard and mouse, devices that do not lend themselves very well to portability. A significant amount of research has been made into alternatives such as voice interfaces, eye focus tracking, and one-handed keypads. We intended to experiment with a system that uses hand gestures and movements.

There are two main goals to this project.

1. To create a simple input mechanism
2. To create example applications that use the input scheme

The project high level design involves several modules:

The equation for the frequency response of the low pass filter is:
f = 1/(2*pi*RC) = 3.1 Hz

Most people will not be moving a finger more than three times per second, so the filter does not restrict operation.

The gain of a non-inverting op-amp is equal to:
A = 1 + R1/R2 = 7.1

This gain is necessary to amplify the signal from the piezoresistor for the ADC.

On the hardware, we were unable to obtain a 3-axis accelerometer (which would allow up-down motion detection) from Kionix. However, since the gesture sensing is the primary focus, a 2-axis accelerometer sampled from Analog Devices provides adequate functionality.

There is a size and weight tradeoff that is made in order to incorporate the 4x20 LCD. If a smaller (4x16 or 2x16) LCD was used, there would be some space and weight savings. In addition, the backlight is not disconnected, and contrast level is locked by hardware.

Another issue is using lycra gloves for the foundation of the gesture sensing. While the double lycra glove allows more reliable and accurate sensing, they may make wearing the device less than completely transparent.

While accurate sensing is always a goal, reliability and device issues force the sensing to and processing to rely on reading finger position as a binary state rather than a continuum of positions.

On the software side, the limited memory on the MCU prevents us from implementing any long and complicated programs. This requires a systems programming mindset focusing on specialized code and design simplicity.

The device software is ANSI C compliant as much as the Codevision compiler allows and does not infringe on any trademarks or copyright...

There are patents for gesture input systems such as the electric field sensing but none of them use piezoelectrics for sensing motions.

The device may seem similar to the Nintendo Power Glove but there are a few differences. First, the Power Glove uses optics and light intensity to gauge finger bend while this device uses piezoresistors. In addition, the Power Glove does not sense hand motions.

• Sensor Module
• Signal Processing Module
• Display Module

The heart of this project is the sensor module. There are two main sensing mechanisms inside the sensor module.

1. Piezo film strips for the fingers
2. Accelerometers for the hand

Piezoelectric materials deform when subjected to an electric field and produce an electric charge when mechanically deformed. The equivalent electrical circuit is an AC voltage source in series with a capacitor. Thus, these materials can be used as mechanical sensors by analyzing the voltage change across the film to deduce the deformation of the film. The voltage spike is in response to a changing mechanical stress and not to the absolute stress, essentially acting like a dynamic strain meter. However, by adding in a capacitor, the charge can be stored for a longer time and make it so that the charge generated is proportional to the absolute stress. These properties are the basis of the finger-sensing scheme.

Piezo films are laid across the fingers and taped down to ensure a proper response from the piezo. However, the current generated by the piezo is very small. If the piezo is just connected to a voltmeter, the voltage spike is on the order of 3 - 5 V. When the piezo is connected to other resistive elements such as the rest of our circuitry, the voltage spike drops down to 0.3 - 0.5 V so signal is amplified in the signal processing module.

A 2-axis accelerometer from Analog Devices is mounted on top of the hand to tracks left-right and forward-back hand movement. The accelerometer is taped down to prevent shifting. Unfortunately, we were unable to obtain a 3-axis accelerometer to allow for up-and-down movement.

The Analog Devices accelerometer, ADXL203CE, has a range of +-1.7 g with a sensitivity of 1 V/g. This allows the accelerometer to be used directly without amplification. In addition, the accelerometer has a resting state voltage of about 2.3 V.

This amplifier has a gain of approximately 7 and is enough to ensure that the signal ranges from 0 to +5 V. A low pass filter with a frequency response of 3 Hz filters out noise. The frequency response is fast enough since it is unlikely that someone will move a finger more than three times a second. Diodes are placed at the input to the ADC to protect the MCU.

The signals from this module feed into Port A on the MCU.

The third non-MCU hardware component is the Optrex DMC20481 4x20 character LCD. The LCD is similar to the LCD that used in class (uses the Hitachi 44780 interface), but there are two main differences. First, the LCD has a backlight (which is not used). Second, the LCD needs a -5 to 0 V input to the contrast pin with -5 V being all black (full contrast) and 0 V being no contrast.

We use a TC7660 charge pump voltage converter to convert the +5 V to -5 V. The schematic is shown below.

Software development for the project focuses on building a very basic (single- threaded) operating system to provide a foundation for small applications. It presents the applications with a small API for interfacing with the gesture sensors and performs some necessary device functions. It also provides a simple GUI for displaying a menu for launching the applications.

At the highest level, the operating system is merely a shell program which calls the application driver functions and also includes some commonly used functions and variables. While this system can be expanded to provide more functionality and complexity (e.g. multi-threading, complete hardware abstraction layer, synchronization, etc...), the project time constraints limit us to the most basic implementation.

Developers can easily write new applications for the system by following the template given in the sandbox.c code. Applications can be added to the system GUI by adding a few lines of code in config.h and os.c. From the operating system GUI, the user can select to run the application - the operating system then calls the provided application driver function which transfers the thread of execution to the application. From here on, the application has full control of the device (with some caveats). To reduce code redundancy, the operating system provides some interrupt code and functions for common functionality. A millisecond-precise timer is provided along with USART interrupt access. A read_input() function allows applications to interface with the gesture sensors, and a wait_ms() function provides a method for delaying the application execution. Global variables include an LCD buffer and variables indicating input (from the sensors).

Despite the design simplicity, the operating system still presented some issues - while applications receive full device control, there should be some way for the user to quickly return to the main GUI or restart the device. In addition, the operating system may need to regularly execute some maintenance operations. The solution for this problem is to assume the developer is cooperative and require occasional calls to a system function exec_system(). This system function provides a method for the operating system to execute the necessary operations without having device control. To provide quick restart and exit, the system function periodically reads user input and resets the watchdog timer only if a specified input is not detected. If that input is detected, the watchdog is not reset, and the device eventually "reboots."

Another issue arose from deducing the workings of the Codevision IDE - to provide modularity, the code expects developer applications to be separate *.c and *.h files. However, the Codevision IDE is very finicky about the project settings necessary to allow multiple files (for this particular project, os.c MUST be added to the project first). In addition, other issues like lack of pointers or a heap or even the dereferencing operator made coding a bit more challenging.

However, the main challenge is simplifying application integration. Currently, adding a new application requires adding the application *.c and *.h to the project then adding the necessary lines of code to the GUI code. In addition, the application needs to follow a few rules - it needs to call system_exec() occasionally. Outside of these operations, the developer and application has full freedom. We consider this an adequate level of compromise (our conclusion from writing the sample demo applications).

The device is able to read inputs from the fingers and the hand. We are able to cycle through the menu options, run different applications, and reset the system without much problem. The device does not feel excessively bulky or uncomfortable to wear for short periods of time. The software infrastructure functions correctly and allows us to add new applications to the system. Overall, we are satisfied with the project. However, as always, it would have been nice to integrate greater functionality and streamline the design.

The MCU runs at 16 Mhz. We experienced no problems with slowdown or artifacts while running any of the applications. In general, there are no time-critical programs except for input polling currently in this device so speed is not a crucial factor. Note, this device should probably not be used for hard real time systems, at least not in its current form. The code would need to be tweaked for any real-time application.

The polling speed is easily changed to accommodate people of different reflexes by simply modifying a #define in the OS. There is an absolute max polling speed of three inputs per second, but it is unlikely any human will be able to sustain such a rate with any amount of accuracy (unless they are using performance enhancers).

Finger and hand sensitivity can be easily changed for each person by modifying the thresholds for each input inside the OS. We have found that our default thresholds have been adequate for most people. Most false positive inputs were due to a twitchy finger or hand. Excessive false negatives is fixed by modifying the sensitivity. These settings must be calibrated.

Safety is a huge concern for us since we are mounting an electronic device on top of the arm. To ensure that the device is safe to wear, we have done the following things:

• Wrapped all contacts (piezos, circuit boards, etc.) in electrical or duct tape.
• Mounted the device on some insulating material such as the glove armguard so that no electronic component is in direct contact with skin.
• Clustered most of the wires in bundles to avoid a "wire maze" and reduce the probability of a wire being caught in something.
• Reduced the profile of the device by mounting them as close to the armguard or glove as possible. This is to also reduce the probability of a component being caught in something

Our device does not interfere with the operation of other devices (no RF, minimal CPU noise). There might be some slight danger to people with electronic pacemakers or other highly sensitive life-saving equipment.

If anything, our device can be integrated into other Atmel MCU based projects as an input device. Alternatively, other programs can be integrated into the device.

The device is usable by most people. Currently, the device is right-handed only, but it can become left-handed by flipping the orientation of all the different boards around and finding a left-handed glove. The large amount of #defines in the operating system allow for customization to fit most people's needs. Furthermore, we have tested the device with multiple people and the defaults were adequate for them, but the option exists for fine-tuning.

A RS232 interface was originally planned, but we simply ran out of time to fully integrate it into the OS. The hardware capability is already on the chip, but the software to integrate it to the OS is not. However, the code should be trivial for a motivated developer to produce.

We feel that the device was a success. We were able to create a usable interface and a module system for easy integration of applications. The glove functions as expected and has a wide range of customizable options. It is unfortunate that we were unable to fully integrate the RS232 connection, but that was a minor feature. It would have been nice if we would have been able to fine/create packaging for the circuit boards with a cover for the LCD, but that is also beyond the scope of our original intentions.

Our design does not use any standards nor does it reuse someone else's design or code or intellectual property. We were fortunate to not have to sign any non-disclosure agreements (NDA) to receive any of our sample parts, nor did we ask for any parts under false pretenses.

There is a possible patent opportunity in this device. After we had conceived the initial device and were researching similar devices, we could not find any piezo-based gesture input devices. Seeing as how there were patents for specific types of gesture devices and not the general category, and further seeing no applicable patents, we can conclude that it might be possible to patent this.

2) There is no possibility of a conflict of interest. We do not work for nor own stake in any of the companies mentioned. In addition, all of this work is freely available.
5) By basing this device around piezoelectrics and accelerometers, we encourage others to look into integrating these components into their projects, whether they be related to gesture sensing or not.
7) We are open to any suggestions for improvement or bug fixes. In addition, we are willing to accept contributions from anyone and will credit you.
8) We strived to make this device as customizable as possible so that almost everyone will be able to use it.
9) We attempted to make this device as reasonably safe as possible. With the exception of an overall packaging, we avoid contact with the skin by any electrical component.

• os.c, os.h, config.h
• app_std_timer.c, app_std_timer.h
• app_std_sandbox.c, app_std_sandbox.h
• app_std_monitor.c, app_std_monitor.h
• app_std_rps.c, app_std_rps.h
• app_calc.c, app_calc.h

• Part gathering
• Soldering
• Some circuitry
• Calculator application
• Glove calibration
• Webpage

• Piezo and lcd circuitry
• OS writing
• Rock Paper Scissors, Timer, and Monitor applications
• Connecting the components together

• The calculator module compiles, but has not actually been tested on the MCU.
• There is 1 pin available for PORT A, which should be linked to bit 5 in the input_data_bin variable. Currently, bits 7-6, and 4-0 are used already in that variable, leaving bit 5 available.
• PORT B and D are unused (USART would have used some of the pins, but that is not currently implemented). One thing is that the headers for PORT D are partially used up for VDD and GND. However, wires could be directly connected to the MCU pins.
• Vertical motion on the wrist can be measured by putting piezos on top or bottom of the hand-wrist joint. Unfortunately, we did not think of this earlier.

• Accelerometers can withstand a lot of heat and still work. In addition, they give really nice voltage responses (at least those from Analog Devices)
• Piezos can be iffy if not placed correctly.
• Most places give out samples, especially if you stress the fact that you are a poor college student with a tight budget. Other places do not.
• When things are not working (such as the fingers not registering), check if it is not user error first.
• (input_data_bin & 0x10 == 0x10) is not the same as ((input_data_bin & 0x10) == 0x10).
• Wires have a tendency to break. Over and over again. At the worst times. At the worst joints. Especially when they were working the day before.
• With enough solder, anything can repaired
• With enough tape (electrical or duct), anything can be protected

• Professor Land for demystifying the Atmel Mega32 with his lectures.
• The TAs for keeping the lab open for so many hours and for their assistance
• The TAs for the idea of a toe-controlled version of this input device.
• Cornell Taekwondo for help in finding the arm guards
• Paul at Renton Company, LLC, for sending the LCDs so quickly and including the charge pump
• The people who tried on the device and gave us feedback
• Blizzard Entertainment - for the exceptional unique idea (borrowed from Diablo 2: Lord of Destruction)
• Mary - for her digital camera