Skip to main content



Easy Input is a head-controlled keyboard and mouse input device for paralyzed users.

"The 10-Sec Pitch"

Easy Input is a head-controlled keyboard and mouse input device for disabled users.

The system uses accelerometers to detect the user's head tilt in order to direct mouse movement on the monitor. The clicking of the mouse is activated by the user's eye blinking through a reflective sensor. The keyboard function is implemented by allowing the user to scroll through letters with head tilt and with eye blinking as the selection mechanism.

This project was a five week design lab for ECE 476 at Cornell University. Video of the final demonstration will be available in the Results section.

Sara Xiangrong Huang (xh33)
Jesse McMullen (jdm55)
(Frank) Wai Shing Wong (ww239)

High Level Design

Inspiration | Concept | Standards

Project Inspiration

This project was motivated by the novella/film “The Diving Bell and the Butterfly” based on the life of Jean-Dominique Bauby. At the age of 42 Bauby, the then French editor the magazine ELLE, suffered a massive and spontaneous stroke while driving down a road in rural France. After several weeks in a coma, he awoke with what’s known as “locked-in syndrome”, with full muscular paralysis, and with only the control of his left eye remaining. Despite the loss of almost all of his physical ability, his sight, hearing, and especially mental function remained unchanged. He (the butterfly) was essentially trapped inside his body (the diving bell). With the aid of an extra-ordinarily patient speech therapist, he gained the ability to communicate again by blinking as she orally read a frequency sorted alphabet to him. In this way he would slowly form words and then sentences, and over the period of months, with extreme difficulty, was able to write the aforementioned account of his experience. This primitive form of communication was excruciatingly tedious to both Bauby and his “interpreter”, requiring extreme patience from both individuals as letter entry was quite slow and any mistakes were not easily corrected.

Overall Concept

We constructd an interface system that would allow a similarly paralyzed user to interact with a computer with almost full functional capability. That is, the system operates as a mouse initially, but the user has the ability to toggle in and out of a keyboard mode allowing the entry of text. This is achieved by using the control from a single eye, tracking the position of the pupil for direction, and using blinking as an input. As detection of eye motion proved too challenging, we built an accelerometer based tilt detector to determine head motion, so that, although not as applicable in this particular case, it might be use by a quadriplegic individual.

Relevant Standards

      PS/2 Standard for Keyboard and Mouse

PS/2 Connector

Front View of a female PS/2 Connector (Photo credit: Wikipedia)

Nowadays, most mice and keyboards use Universal Serial Bus (USB). However, we decided to simulate a mouse and a keyboard using PS/2 instead of USB because it is easier and seems to be more doable within one month time. Because the computers in the laboratory do not have PS/2 ports on the mother board, we will need a PS/2 to USB cable to connect to the computer. PS/2 transmits serial data at a rate of 10 to 16kHz with 1 stop bit, 1 start bit and 1 odd parity bit. There are 6 pins in a PS/2 connector. The pin specifications are as follow.

Serial Signal Waveform

Waveform of Device-to-Host Communication (Photo credit: Computer-Engineering)

      PS/2 for Keyboard

When a key is pressed, a unique make code for that key is sent to the computer. When a key is released, another unique break code for that key is sent. According to Keyboard Scan Codes: Set 2 which is used by all modern keyboards, a make code is either 1 or 2 bytes and a break code is either 2 or 3 bytes. There is a simple relationship between the make code and break code. For example, the make code for an 'A' is 0x1C and the corresponding break code is 0xF01C. Besides the key press event, the computer may send some commands to the keyboard. We also need to handle these commands such as RESET and RESEND in order to make it works.

      PS/2 for Mouse

There are many extensions on how the mouse communicates with the computer. We stuck with the most basic one because we do not need features such as more buttons and scrolling.

The standard PS/2 mouse sends movement and button information to the computer periodically using a 3-byte packet. The specifications of the packet are shown below. The mouse samples the movement at a frequency from 10Hz to 200Hz. Similar to a PS/2 keyboard, we need to handle command from the computer.

Mouse Bits

Packet Specifications (Photo credit: Computer-Engineering)

Implementation

Hardware | Software

We use an accelerometer to detect the head movement and a photo sensor to detect eye blinking. Both sensors' outputs are connected to an A/D converter on the microcontroller. Processed digital information is transmitted remotely using wireless protocol created by Meghan Desai.

transmitter diagram

Block Diagram for the Head Unit

On the other hand, a wireless receiver collects all the correct information through id filtering and then sends signals to a ground microcontroller through USART. After the signals are interpreted by the ground microcontroller, mouse or keyboard instructions are sent to the computer via PS/2-to-USB cable.

receiver diagram

Block Diagram for the Computer Interface

Hardware Details

Head Unit

Although initially we had considered monitoring eye movement for directional control, the inability to find a functional and adequately priced CMOS camera, combined with the difficulty of this task, caused us to reconsider things. Instead, we opted to detect head motion using an accelerometers. Although, the accelerometers we were using (FreeScale’s MMA6560Q) was quite sensitive, outputting 800mV/g, we found early on that this was not fine enough to accurately detect lateral motion or rotation. We were therefore forced to detect the relative tilt of the head (front-back and side to side) which caused the lateral axes to begin to read the sine of the gravitational force. The FreeScale chip was nice as both axes were RC filtered and their output could be feed directly into the first two pins of port A, and the internal ADC employed.

head unit

Acceleration Sensor Usage

For our input mechanism, we decided to use blinking as we wanted the device to be functional for non-vocal or ventilated users (blowing or sucking was another option). Our first idea, and the one we implemented, was to use a led/photodiode pair to reflect light off the eye. We found that Optek Inc. makes a OPN710 round receiver, consisting of a 935nm LED and a phototransistor mounted on the same unit. This detected a strong increase in signal upon blinking. We were worried about detecting the difference between normal and intentional blinks, but we found that for most users the intentional blinks produced a much stronger signal, and they were always much longer the ~300ms normal blink duration. The phototransistor produced fluctuations in current, so we read the voltage change off a load resistor. This voltage was amplified by a gain of 4, before going into another port A pin and read by the ADC.

Finally, a wireless RF transmitter (RCT-433-AS) was used to transmit the motion data to the “base unit”. This was wired up similarly to Meghan Desai’s sample transmitters. The output of the TX UART pin was inverted using a 2N3904 transistor and then sent into the Radiotronix chip. Special care was taken to stabilize the voltage supply to this chip through capacitors and an RF choke. Incidentally, there was a bit of RF noise that was picked up in the wires that ran from the blink sensor to the head unit, though not enough to be a problem so we did not low pass filter the blink signal.

The circuit was powered by a 9V battery which proved to be the heaviest and most cumbersome part of the whole unit.

Computer Interface - PS/2 Connection to Mouse and Keyboard

A second MCU collected data from the head unit, processed it, and directed the computer to perform appropriately. This required implementing a complimentary RCR-433-RP chip to detect our RF transmissions. As with the transmitter, the voltage supply here was tightly regulated, and the data line inverted into the RX UART pin of the mega32.

To connect our Easy Input device to the computer, we bought a USB-to-PS/2 connector from eBay. The device worked similarly to the previous project – Minority Mouse Report, but this time with both a keyboard and mouse implemented. In the PS/2 protocol, an open-connector interface is required. The DATA and CLOCK lines are both open-collector with pull up resisters to Vdd. There are two possible states in an open-collector interface, i.e. low, or high impedance. In the "low" state, a transistor pulls the line to ground level. In the "high impedance" state, the interface acts as an open circuit and doesn't drive the line low or high. Furthermore, a "pullup" resistor is connected between the bus and Vcc so the bus is pulled high if none of the devices on the bus are actively pulling it low. A general open-collector interface is shown below.

open-collector interface

Open-Collector Interface (Photo credit: Computer-Engineering)

For simplicity in our design, we decided to use a resister to replace the transistor. The choice of resisters are not important here. We use a 10k Ohms resister as the pull up resister and 330 Ohms resister between input and output pins.

ps2 connection

PS/2 Connection Setup

The entire lower unit was powered via the desktop, through the usb/ps/2 power supply. Additionally, two LEDs demonstrated that the unit was detected properly by the PC (this was a slight problem solved sampling by unplugging the port once or twice).

Software Details

Transmitter Side

The transmitter code does two things. It runs ADC to get the input from the accelerometer and the object sensor. Then it transmits the data to the receiver.

        Accelerometer

The output voltage for the accelerometer ranged from 0V to 3V which maps to 0 to 153 after ADC conversion. When the MCU starts, it first takes 100 readings from the accelerometers and takes the average of the sample data. The averages are used as a reference point for the initial head position. When the head of the user is tiled up/down or left/right, the reading from the accelerometer is subtracted from the value of reference point. The differences determined the levell of head tile. We decided to have three discrete level of head movement detection to eliminate the effect of noise and provide several speed states.

        Object Sensor/Photodiode

For the object sensor, the maximum output voltage after the amplifier is 4V. Similarly to the accelerometers, the MCU takes an average of the first 100 readings as a reference. When the user blinks, the value after ADC is checked against the reference to see if it is above the threshold.

        Wireless Transmission

We used the wireless driver from Meghan Desai's Wireless Project. After the ADC conversion, a packet of length 3 is transmitted to the receiver. The first byte is the X movement and the second byte is the Y movement. There are 7 possible values that encode seven levels of movement, i.e. one for no movement, three for positive movement and three for negative movement. The third byte is the blink variable, when blink is detected, value 255 is sent. Otherwise, the value 126 is sent.

Receiver Side

The receiver code does three things, it receives the data from the transmitter through wireless, then converts the data to meaningful control logic and sends the control signal to computer using PS/2 protocol.

        Wireless Reception

After the microcontroller finishes all the initialization steps, it enters an infinite loop that listen to the wireless channel. Whenever a packet is received, it first checks the transmission ID to see if the packet is sent by our transmitter side. In our implementation, the total length of each packet is 7 bytes. The first two bytes are used by the wireless protocol. The third byte is the transmission ID and the four byte is the length of actual data. That is always three in our case, X movement, Y movement and blink.

Packet Byte Usage
my_rx_data[0]
Used by Wireless Protocol
my_rx_data[1]
Used by Wireless Protocol
my_rx_data[2]
Transmission ID
my_rx_data[3]
Length of Data
my_rx_data[4]
X Movement
my_rx_data[5]
Y Movement
my_rx_data[6]
Eye Blink

        Mouse and Keyboard Logic

There are two modes of operation, either mouse mode or keyboard mode. When it is in mouse mode, the X and Y movements are directly mapped to the movement of mouse. The more your head tilts, the faster the mouse moves. When the user blinks, the mouse clicks. For this part to work, we want to distinguish between the unintentional blink and intentional blink. We observe that intentional blinking is significantly longer than unintentional blink. So the program has to receive 10 consecutive blink packets from the transmitter to actually send a mouse click signal to the computer.

For the keyboard mode, you can select the character you want to enter by tilting your head right or left. The English characters are arranged by their frequency to speed up the selection. The scrolling of characters is done by sending a “backspace” signal before sending the characters. The characters are scrolling at a rate of 2.5 cps. When you reach the character you want, you can blink to start a new character selection. When you tilt your head up, you can change between lower case, upper case, number and punctuation. When you tilt the head down, a “backspace” signal is sent to the computer. To switch between the mouse and keyboard mode, you need to close your eye for about 4 seconds.

mouse mode

Mouse Mode Usage Diagram

keyboard mode

Keyboard Mode Usage Diagram

Results

The head unit of our final design is taped to a cap with the photodiode mounted on a goggle. The whole design performed well and demonstrated all the functionalities we tried to implement. The Easy Input project met all the goals we set at the beginning of the 5-week period: a comprehensive, user-friendly, practical computer input interface for people with disabilities. During our trial testings, the head unit and ground unit worked together perfectly with stable wireless communication.

Although entering text in the keyboard mode takes some time, it does provide an attractive alternative for people who cannot use regular keyboards. In general, the device makes it convenient to browse websites, listen to music, watch videos and accomplish other common operations on the computer.

In the creation of Easy Input, we ensured safety in the light emission on the photodiode through careful selection. The one used in our project emits infrared light at 935nm, which is not visible to human eyes and not intense enough to cause any damage. Therefore, anyone can use this device without concern of eye damage.

The secondary safety hazard we considered was the user fatigue. Easy Input recognizes the original position when turned on as the reference origin, so when the user returns the head position to the reference origin, the mouse stops moving. Thus it provides a resting position for user in case of movie watching or article reading online. The size of the photodiode we used is very small and the weight is minimal as well. Only the photodiode is on the safety goggle so that the goggle is too heavy to wear for average users. The accelerometer is mounted on a baseball cap to avoid further weight on user's nose. The sensitivity of the device has also been caliberated to ease the pressure on the neck when tilting the head.

Our system incorporates a high degree of usability. The compact and comprehensive design makes it ready for use by the public. The mouse functionality is fairly intuitive, and the keyboard functionality can be easily adapted to with a few instructions and practice.

Conclusion

Our final product exceeded our expectations with accurate wireless communication and precise sensoring. The acceleration sensor was sensitive enough to detect small head tilts. The distinctive signal provided by the photo-sensor helped greatly filtering out regular blinking from our data collection.

In order to be a commercial device, the head unit needs to be much smaller. This could be done fairly easy, the accelerometer, mega32, and wireless chip could be arranged on a PCB about the size of a quarter. Using a button battery would also decrease the size and the weight of the device to the point where it would be quite unintrusive. Given a few more weeks and a bit larger budget we could have implemented this ourselves. The blink sensor could also be mounted onto a clip so that it could be easily used by people with glasses.

Ethical and Legal Considerations

We closely followed IEEE Code of Ethics throughout our project. Specifically we ensured that safety precautions were considered and followed throughout our final project. We ensured that we gave credit to the contributions and resources provided of others. Although discussions and criticisms over the design within our group grew heated at times, the end effect was an improved understanding of alternatives and in depth understanding. Additionally, we were realistic when we realized that we might not be able to pursue the idea of using a phone camera to detect the eye movement through image processing. This involved accurately portraying our current status and problems to our TAs and professor. Despite being asked about our project many, many times over the last five weeks, we took the time to explain our project in depth in a manner that increased the understand of technology.

Although the idea was inspired by a film, we developed the software in house and designed the electrical circuits on our own, so we don’t anticipate any intellectual property issues. We do not have any legal considerations since we did not use parts regulated by agencies such as the FCC. In the occasions where some of our work is referenced to previous achievement by others, whether from previous projects or existing technology, proper credit and annotation are given in the reference section to clarify the details.

Social Impacts

This project is designed for new ways of using computers, which are essential to our modern life. The device can be found most useful by handicapped people who can no longer control the computers using their hands. Since the device relies on user’s head and eye movements, it can be used even by patients who are paralyzed from shoulder downward. Simple head movements don’t require too much energy, and neither does eye blinking. Therefore users won’t get tired from using this device, and in contrast, they’ll get exercise on the neck, which tends to be the problem for most regular computer users.

Appendix

Schematics | Budget | Tasks | Code Listings

Schematics

Please click on the following images to see the full size schematics.

Transmitter Side Schematic

Sensors to Transmitter Communication Circuit

Receiver Side Schematic

Receiver to Computer Communication Circuit

Budget

Our budget was not to exceed the $75 limit, and we successully achieved the goal. Most of the components were available in lab. Some were ordered through various vendor sites. Some of the components were owned by us before the project, such as the safety goggle and heat shrink wrap. Other free components include sample products like the dual-axis accelerometer. Please refer to the following parts list for further cost details.

Part Name Quantity Unit Cost Total Source
 
 
Total:
$56.32
 
Atmel ATMega32 Microcontroller
2
$8.00
$16.00
ECE 476 Lab
Custom PC Board
2
$5.00
$10.00
ECE 476 Lab
Dual Axis Acceleration Sensor
1
free
$0
ECE 476 Lab
PS/2-to-USB Cable
1
$2.00
$2.00
Ebay
Reflective Object Sensor
1
$8.32
$8.32
Digikey
Safety Google
1
free
$0
Previous Owned
Baseball Cap
1
free
$0
Previous Owned
9V Battery and Holder
1
$5.00
$5.00
ECE 476 Lab
Wireless Transmitter
1
$3.50
$3.50
ECE 476 Lab
Wireless Receiver
1
$4.50
$4.50
ECE 476 Lab
Heat Shrink Wrap
1
free
$0
Previously Owned
Small Solder Board
2
$1.00
$2.00
ECE 476 Lab
Miscellaneous Components
1
$5.00
$5.00
ECE 476 Lab

Tasks

All three of us contributed equally to this project and collaborated on all components of the final design. Therefore a lot of tasks listed below are shared among the three of us.

Tasks Labor
Accelerometer Hardware
Huang
Accelerometer Software
Wong
Wireless Communication Hardware
McMullen
Wireless Communication Software
Huang, McMullen, modified from Meghan Desai's Wireless Project
PS/2-to-USB Communication Software
PS/2-to-USB Communication Hardware
Wong
Custom Prototype Boards for Microcontrollers
Huang, McMullen
Reflective Object Sensoring Circuit and Software
McMullen
Soldering and Assembly
Huang, McMullen, Wong
Testing and Debugging
Huang, McMullen, Wong
Final Project
Huang, McMullen, Wong
Website
Huang

Code Listings

The system clock is set to 16 MHz with no fuse bits. The microcontroller is loaded on an ECE 476 prototype board. Further information can be found in the ATmega32 datasheet. Please click on the following items to download the code.

References

Thank you to Bruce Land for his assistance and providing the supplies in the lab. Thank you to our TAs who kept the lab open in all possible hours of the 5 week period and their invaluable advice, especially Sam Lee who helped us greatly throughout the course. Thank you also to the LASSP Machine Shop for the use of their drill press and band saw.

Datasheets

Previous Projects

Vendor Sites