Sign Language Translation

High Level Design

Glove

The sign language translator starts with the Glove, the heart of the project. The black glove contains nine flex sensors, four contact sensors, one two dimensional x-y axis accelerometer and one one dimensional z axis accelerometer.

The flex sensors are the most critical sensors because most letters can be distinguished based on fingers’ flexes. All the fingers except the thumb have two flex sensors, one over the knuckle and the other over the lower joint. This provides two degrees of flexes for these fingers. For the thumb there is one flex sensors over the lower joint. All the flex sensors were commercially available flex sensors that we found in the lab.

Flex Sensors

The contact sensors help distinguish between a set of letters in which the flex sensors were ambiguous. Letters such as U and V have only one difference: the distance between the index and middle finger. Contact sensors determine which fingers are touching and how the fingers are oriented relative to each other.

Contact Sensors

Finally, the accelerometers are used for movement and orientation detection. Specific hand motions are the only way to detect the letters J and Z. For letters such as G and Q, the only way to distinguish between them is their orientation--while G has the palm facing sideways, Q has the palm facing downwards.

Accelerometers

Arm Circuit

The second part of the project is the Detection Unit. The output from the sensors on the Glove is connected to the Detection Unit, which is mounted on the user’s arm through the use of Velcro straps.

Detection Unit

The Detection Unit has three basic components.

Comparator Circuits:

The first of the three is the nine Schmitt Triggers, built using operational amplifiers with positive feedback. The nine flex sensors are digitized through these one bit analog to digital converters (ADC’s).

Each flex sensor has a unique threshold level for detecting flexes, necessitating individual comparator circuits with potentiometers.

Individual Comparator Circuit Layout

Twelve Comparator Circuits

Radio Transmitter:

The second part of the Detection Unit is the radio transmitter which transmits the letters signed to the Base Station. This makes the project to be wireless and easily portable. We use a Radiotronix WI.232FHSS-25-FCC-R transceiver.

Transceiver

ATMega644 Microcontroller:

The outputs from the accelerometers are inputted to the microcontroller where it is read through the microcontroller’s on-chip ADC. The comparator circuit’s sensors and contact sensors are also connected microcontroller through its various ports. The microcontrollers main purpose is to analyze the data from the sensors to check whether a letter is being signed and send transmission requests to the transmitter.

ATMega644 Microcontroller

Base Station

The Base Station receives its input through a radio receiver and outputs its results to an LCD screen, a speaker set, and a serial USB connection to a computer. It consists of two main parts:

Base Station Top View

Base Station Side View

Radio Recevier:

The radio receiver filters its incoming transmissions according to an address specified by the Detection Unit. It receives the letter signed by glove and sends the letter to the microcontroller on the base station.

ATMega644 Microcontroller

This second microcontroller sits on the base station and outputs the letter to an LCD display placed next to it. As the letters are received, the text scrolls from right to left and top to bottom, allowing the person the user is talking to to easily read the letters. It then converts the text to speech and outputs the letter through speakers. Finally, it connects to the computer through the serial port and interfaces with a COMM Port.

Scrolling Letters on LCD

Software

The software for the project uses Matlab, Java, and C to interface with the microcontrollers and sensors on the glove.

C is used to program the microcontrollers, read input from the sensors on the glove, recognize the letter, and transmit information to the base station. Matlab is used to interface with the Base Station through the serial port connection. Finally, Java is used to implement a sign language game for users. The game is similar to a typing game. Letters fall from the top of the screen and the user tries to sign the letters before they cross the line at the bottom. This game can help users learn sign language in a fun competitive environment.

Entire Program Flow

Educational Sign Language Software

Hardware/Software Tradeoffs

Wireless vs Wired Glove:

The very first design decision was to choose whether or not to make the glove wireless. A wired glove would make it difficult for the person the signer is talking to to read the translation. This could be overcome with long wires, but long wires are cumbersome. This project instead uses a wireless glove. This eliminates the discomfort from long wires. Also, it allows the translator to be used portably. A base station receives the translations and displays the letters on its LCD. The Detection Unit and base station can be entirely powered by 9V batteries. The base station’s speakers in our implementation do require an AC source, but portable speakers powered by batteries do exist. Therefore the entire system is designed to be portable, with the exception of the educational software that can only run on a computer.

This modular design could be implemented individually and would make the glove portable as well as easily usable as the person signing would be able to hand the Base Station to whomever he or she wanted to sign to.

Contact Sensors vs Hall effect Sensors:

A Hall Effect sensor varies its output based on the magnitude and direction of the magnetic field around it. By putting a sensor on one finger and a magnet on another finger, we can distinguish the relative position between two fingers. For example, letters R, U and V have the same flex but the relative position between their index and middle finger differ. A single hall effect sensor would be able to distinguish them.

Letters R, U and V

Contact sensors signify which fingers are touching each other. They do not give us any information of the relative position between fingers except for whether they are together or apart.

Contact sensors instead of Hall Effect sensors are used because Hall Effect sensors are expensive whereas contact sensors can be created using copper tape and wires. Also, in testing the contact sensors proved to be more reliable. The contact sensors provide a clean digital signal, whereas the Hall Effect sensors have to be calibrated and converted with an ADC.

Long flex sensors vs shorts multiple sensors per finger:

Many letters require different levels of flex for each finger. While some letters requires the knuckles to be bent, others had straight knuckles but other joints flexed. A single long flex sensor is more reliable than a shorter one but it did not provide enough information regarding how exactly the finger was flexed. Because loss of information creates ambiguity in the flex of the fingers, multiple short flex sensors are used per finger.

Comparator circuits vs ADC:

The Atmega 644 has internal ADC’s which could have been used for the flex sensors. However, each flex sensor only needs to be converted to one bit: flexed or not flexed. A one bit ADC is not difficult to set up: a Schmitt Trigger with a proper threshold can serve as a ADC. Additionally, there are nine flex sensors, while the Atmega only has 8 pins with internal ADCs. Finally, the accelerometers need to be converted, and they need more than one bit in the digital output. For these reasons, nine external comparator circuits convert the flex sensor inputs to digital signals. The disadvantage is that the comparator circuits take up a significant amount of space on the detection unit.

Design Standards

The transmit/receive modules operate at 915 Mhz. The FCC states in Title 47 Part 15 that devices that operate at 915 Mhz do not require a license so long as they have a field strength less than or equal to 25 microVolts per meter at a distance of 300 meters. Our transmitters are rated for 11 dBm, which is 12.6 mW. At 300 meters away from the transmitter, the field strength is calculated by equating the time averaged amplitude of the poynting vector with the power per area. Hand calculations predict a field strength of 9 microVolts per meter, which is compliant with the FCC regulations.

There is no association which sets the signs for American Sign Language, but we did search for several different sign language alphabets and compare them to get a consensus for the correct hand position for each letter. We found there was no significant variation in the hand positions.

Patents

No patents on educational software for Sign Language was found. However, there are patents on the following:

Electronic translator for assisting communications

Inventor: Greene, Jr. , et al.

Patent Number: 6,377,925

Abstract: An electronic translator translates input speech into multiple streams of data that are simultaneously delivered to the user, such as a hearing impaired individual. Preferably, the data is delivered in audible, visual and text formats. These multiple data streams are delivered to the hearing-impaired individual in a synchronized fashion, thereby creating a cognitive response. Preferably, the system of the present invention converts the input speech to a text format, and then translates the text to any of three other forms, including sign language, animation and computer generated speech. The sign language and animation translations are preferably implemented by using the medium of digital movies in which videos of a person signing words, phrase and finger spelled words, and of animations corresponding to the words, are selectively accessed from databases and displayed. Additionally the received speech is converted to computer-generated speech for input to various hearing enhancement devices used by the deaf or hearing-impaired, such as cochlear implants and hearing aids, or other output devices such as speakers, etc. The data streams are synchronized utilizing a high-speed personal computer to facilitate sufficiently fast processing that the text, video signing and audible streams can be generated simultaneously in real time. Once synchronized the data streams are presented to the subject concurrently in a method that allows the process of mental comprehension to occur. The electronic translator can also be interfaced to other communications devices, such as telephones. Preferably, the hearing-impaired person is also able to use the system's keyboard or mouse to converse or respond.

Sign language editing apparatus

Inventor: Ikeda , et al.

Patent Number: 5,990,878

AbstractA sign language editing apparatus includes a glove-type sensor for converting movement of fingers in the sign language into an electrical signal to produce time series data of sign language words, a sign language word data editing device for adding predetermined additional data to the time series data inputted from the glove-type sensor to process the time series data, a sign language word dictionary for storing the time series data of sign language words processed by the sign language word data editing device, a sign language sentence data editing device for reading out the time series data of sign language words stored in the sign language word dictionary in accordance with the predetermined characters inputted from the input unit and adding predetermined additional information to the time series data of sign language words to produce time series data of sign language sentence, a sign language animation synthesizing device inputted with the time series data of sign language sentence produced by the sign language sentence data editing device to produce sign language animation expressing movement of the sign language, and a display unit for displaying the sign language animation produced by the sign language animation synthesizing device.

Method and apparatus for translating hand gestures

Inventor: Hernandez-Rebollar

Patent Number: 7,565,295

AbstractA sign language recognition apparatus and method is provided for translating hand gestures into speech or written text. The apparatus includes a number of sensors on the hand, arm and shoulder to measure dynamic and static gestures. The sensors are connected to a microprocessor to search a library of gestures and generate output signals that can then be used to produce a synthesized voice or written text. The apparatus includes sensors such as accelerometers on the fingers and thumb and two accelerometers on the back of the hand to detect motion and orientation of the hand. Sensors are also provided on the back of the hand or wrist to detect forearm rotation, an angle sensor to detect flexing of the elbow, two sensors on the upper arm to detect arm elevation and rotation, and a sensor on the upper arm to detect arm twist. The sensors transmit the data to the microprocessor to determine the shape, position and orientation of the hand relative to the body of the user.

Hardware Design

Flex Sensors

The flex sensors are variable resistors. As the flex sensors are bent, the resistance changes in an approximately linear fashion. The flex sensors were originally 6 inches long. As discussed above, the original flex sensors did not provide enough information about the flex of a finger: there was too much ambiguity in the measured resistance. The glove uses flex sensors that are cut into two pieces, with the exception of one flex sensor cut into three. This approach provides nine flex sensors which are used for all knuckles and lower joints except for the thumb--for the thumb there is only one flex sensor over the lower joint. All the flex sensors are connected to the comparator circuits, discussed a couple sections below.

Accelerometers

There are two accelerometers, one x-y axis and one z-axis accelerometer. The x-y accelerometer (Analog Devices ADXL203) has ground and Vcc inputs, and two outputs. The outputs are, strangely enough, highly correlated, so the microcontroller only uses one of the outputs. The output is low passed as suggested by the data sheet, with a 20 nF capacitor and an internal 320kOhm resistor. Also suggested by the datasheet, there is a 0.1 microFarad capacitor between Vcc and ground. The z axis accelerometer (Freescale Semiconductor MMA1260) has ground and Vcc inputs, and one output. The output is low passed as suggested by the data sheet, with a 1kOhm resistor and a 0.1 microFarad capacitor.

Contact Sensors

The contact sensors employed use a simple design. Copper tape is taped to the top of the middle finger and the sides of the pinky, ring, and middle fingers. For these fingers, the copper tape is wired to inputs D2-D5 on the microcontroller, with pull up resistors enabled. When they are not in contact with anything, the input on the microcontroller is high. There is also copper tape on the thumb and the top of the index finger, which are wired to ground. When the pinky, ring, and middle fingers come into contact with the index finger and thumb, the corresponding inputs on the microcontroller are pulled to ground, resulting in an active low signal. The contact sensors provide a very stable signal.

Glove Circuit

There is a small solder board on the glove which contains the accelerometers and receives all the outputs from the flex sensors. For all the flex sensor outputs, there are two sets of headers from the Detection Unit that plug into two sets of pins on the Glove Circuit. The headers and pins provide a clean interface between the Glove and Detection Unit: to disconnect them, only the two sets of headers need to be unplugged.

Glove Circuit

The flex sensors are wired to the indicated inputs. The orange lines indicate copper tape.

The orange lines indicate copper tape

Comparator Circuits

The detection unit has 12 comparator circuits. There are three extra comparator circuits, which would have been used in case some comparator circuits failed (and one did fail). These comparator circuits are actually on two small solder boards that have been attached to a larger solder board. Initially it was assumed that the glove could fit the entire Detection Unit, and the comparator circuits were placed on small solder boards to fit on the glove. However, this approach would result in a very bulky glove, and instead the large Detection Unit is used.

The comparator circuits are Schmitt Triggers. The threshold is set with a potentiometer and is the negative input of the operational amplifier. The typical threshold for the flex sensor is about 2V, although it does vary somewhat across the flex sensors. There is positive feedback with a 3 MOhm resistor to reduce hysteresis. Because the Schmitt Trigger uses a voltage as an input and the flex sensors are variable resistors, there is a voltage divider as an input to the Schmitt Trigger. The flex sensors have a resistance on the order of megaohms, so we use a 1 MOhm resistor in series with the flex sensor for the voltage divider.

The flex sensors’s resistance lowers as they are flexed. This results in a reduction of the analog input voltage to the comparators, and consequently an active low signal.

Detection Unit’s Microcontroller

At first glance, the microcontroller appears to only have four inputs. This is because there are many more pins that are connected with wires under the microcontroller, providing a much cleaner look. Also, if the wires were to cross over to the other side, the color coding would get confusing because many of the colors are repeated (there are not enough colors for a unique color per input).

Wires under the Microcontroller

The accelerometers are wired to inputs A6 and A7. Port A must be used for the accelerometers because the internal ADC’s on the Mega644 are only on port A. The contact sensors are wired to inputs D2-D5. The transceiver is wired to output D1.

Detection Unit’s Transceiver

The detection unit’s transceiver (Radiotronix WI.232FHSS-25-FCC-R) is used only for transmissions of the translated letters. The receive of the transceiver is wired to the transmit of the microcontroller. All six ground contacts of the transceiver are connected to the microcontroller’s ground. Vcc is provided to the transceiver, which is actually regulated at 3.3V. Fortunately this does not cause any problems when the base station receives the transmitted packets. The transceiver operates at 2400 baud and 915 Mhz.

Detection Unit Mounting System

To provide a secure mount, protect the user’s skin from scrapes, and prevent static discharge, there is a second solder board under the original solder board which has the detection unit circuit. Using the provided screw holes, the second solder board is connected to the original solder board with several washers and screws. Velcro straps sewn to the solder board affixes the Detection Unit to the user’s arm. There is nothing on the underside of the second solder board, which prevents scrapes. Finally, the gap between the two solder boards provides sufficient protection from static buildup from the Velcro straps.

Mounting System

The transceiver also has a small solder board underneath it. There are no electrical connections from the solder board - the purpose is to prevent exposing the pins of the transceiver.

Base Station Transceiver

The base station’s transceiver (Radiotronix WI.232FHSS-25-FCC-R) is used only for receiving the translated letters. The transmit of the transceiver is wired to the receive of the microcontroller. All six ground contacts of the transceiver are connected to the microcontroller’s ground. Vcc is provided to the transceiver, which is actually regulated at 3.3V. Fortunately this does not cause any problems when the base station receives the transmitted packets. The transceiver operates at 2400 baud and 915 Mhz.

Base Station Microcontroller

The base station’s microcontroller also at first glance appears to use only a few input/output pins, but port C is connected under the microcontroller for a cleaner look. Output B.3, which is connected to the internal OCR0A, is connected to the low pass filter for the audio output. Input D.1 is wired to the transmit of the transceiver.

The placement of the transceiver caused a couple problems. Firstly, the antenna is too close to the protoboard, so a slice of it was cut. Additionally, the module for serial communication is too close to the antenna, so it is mounted upside down.

Upside-Down UART and Dremel'd Protoboard

Software Design

Detection Unit

The Detection Unit initially sets up Timer0 at 1 millisecond ticks. Timer0 decrements three global variables time1, time2 and time3. When these variables reach zero, they each execute a different task and get set back to their initial values. The three tasks are:

read_sensors()

It reads the inputs from port A, port B0-B2 and port D2-D5. These ports are connected to the sensors on the Glove. This function is executed every 100 ms.

read_analog()

The accelerometers are read in this function every ten milliseconds. The ADC alternates between ports A6 and A7 to read from both accelerometers. These accelerometers are used Z and J through the following waveform detections with the correct flexes:

J Waveform

Z Waveform

compare()

This function analyses the inputs and decides on what letter has been signed based on an algorithm based on a tree search recognition and masking tables. This algorithm is explained below in the Detection Algortihm section. After detecting a letter, the letter is debounced to ensure that it wasn’t detected by mistake through the State Machine (also described below in the Debouncing Section). Once debounced, the letter is sent across USART to the Base Station through a protocol described in the Base Station section.

Base Station

The Base Station uses port B3 as an output as it is internally conencted to OCR0A. It uses Timer0 as a Pulse Width Modulator (PWM) with a period of 256 cycles or 62,500 samples/sec. However OCR0A is done using Timer2, whose overflow flag is set and OCR0A is set 7800 per second.

The Base Station begins by initializing the LCD and the UART and USART. UART has only its transmissions enabled and receptions turned off while USART has receptions enabled and transmissions disabled.

It uses USART to receive packets from the Detection Unit. The packets received contain a sync byte, an address byte for filtering, and the payload. The paylod contains three bytes. The first byte is the letter that is being signed and the other two bytes contain compressed information about the state of the flex and contact sensors.

After receiving this information, a flag is set on the Base Station which signifies that a new letter has been transmitted. This new letter is first displayed on the LCD in a scrolling banner fashion, allowing readers to easily understand a series of letters received.

Next, the Base Station converts text to speech using the compressed header files at 8kHz sampled frequency. This is done in Timer2_OVR_vect.

Finally, the Base station connects to UART and send the received packets to the COMM PORT on the Computer by first compressing the information into a string and then iterating through the string and transmitting each bit through port D1 on the microcontroller.

Voice Synthesis

The microcontroller plays back pre-recorded samples from a free text to speech demo. The samples do not fit on the microcontroller uncompressed. At a 8khz sample rate and 16 bits per sample, with half second recordings per letter, the recordings would take 208 kilobytes, which is far more than the 64 kilobytes of flash on the Atmega 644.

The audio is compressed in Matlab before loading it is loaded onto the microcontroller. The audio is scaled to about unity gain, and the first derivative is quantized with three breakpoints. The first derivative is quantized as opposed to the samples themselves because there is not much change in between the samples, and the smaller values allow for better quantization. The three breakpoints results in 2 bits per sample.

Four of these two bit samples are packed into a char by shifting them into the correct positions. For example, the first sample is shifted into bits 7 and 6 of a char, the second sample is shifted into bits 5 and 4, the third sample is shifted into bits 3 and 2, and the last sample is placed in bits 1 and 0.

The breakpoints are found with an optimizing function in Matlab, which tries all possible breakpoints and quantization values with the range [0, 1] at a given step size to find the inputs that give the lowest mean squared error.

All 26 letter audio files are compressed in this manner.

On the microcontroller, the samples have to be unpacked. For every character in the table of samples representing the letter, four two bit samples are pulled off and outputted in succession. To set up the microcontroller to output audio, PORTB3 is set to an output. The PWM is set up with a period of 256 cycles, which is 62,500 samples/sec in fast PWM mode. However, OCR0A, the output wired to PORTB3, is updated with timer 2 updates at 7.8Khz, which is close enough to the original 8Khz recording. Timer 0 is set up at the full rate.

Serial Interface

Matlab is used to interface with the Serial Port. It constantly reads the COMM PORT connected to the Base station. Whenever it receives a value in the correct syntax, it checks the packet to extract the character signed by the user and analyzes that rest of the packet for information on what finger are flexed.

Next, it converted the flexed finger inputs to binary, and saves it to a text file which is later accessed by Java when playing the game.

Quick Sign

The educational software, Quick Sign, is implemented in Java. The purpose of the game is to allow signers to become adept at signing quickly by signing the necessary letters before the letters reach the bottom of the screen. For every letter they sign, the user earns a point and for every letter that crosses the bottom, a miss is registers. The program contains three Java files:

Letter.java:

This file contains the definition of each letter. It has three global variables: the character that it represents, the x-coordinate and the y-coordinate of the letter on the JPanel. This is quite basic with just getters and setters for all the global variables.

GamePanel.java:

GamePanel extends JPanel and overwrites its paint method. It contains an ArrayList of Letters and at every turn, it paints the letters in its respective x,y-coordinates. It also keeps a global variable for the score and the number of misses incurred by the user during the game. Finally, it also displays which letter the user is currently signing and what fingers are flexed or in contact.

Main.java:

This file contains the main method that executes that program. Initially, it sets up the JFrame and GamePanel. Then it set the JFrame visible, packs it up using GridBadLayout. Finally, it enters a while loop that gets executed every 300 milliseconds. In the while loop, it executes the following.

When the program initially starts, it creates a black Graphical User Interface (GUI) with a white line at the lower end of the JFrame. Letters are randomly generated and are assigned an x coordinate. The y coordinates are initially set to zero. The program uses two parameters to determine the difficulty of the game. The first parameter controls the probability that a new letter will be generated every 300 ms. The second parameter will determine the distance the y-coordinate will move to the white line every time step. These two parameters together decide the difficulty of the game. As signers become more and more adapt with the glove, the faster they will be able to sign and will be able to play the game at a harder level.

Also in the while loop, letters that reach the bottom as removed. Also, using FileReader and BufferedReader, the game constantly checks the file written to by Matlab’s analysis and prints out the letter signed by the user of the glove as well as the orintation of the fingers so that the user can adjust for any discrepancies in signing. It also allows us to constantly check and detect any hardware failure.

Below is a screen shot of the game where the score is displayed on the top left corner, misses on the top right and the letter signed and the sensor outputs, in binary to the microcontroller on the Detection Unit.

Game Screen Shot

Transmision and Reception

The transmit/receive implementation makes use the USART ports D0 and D1 onboard the Atmega644. The microcontroller uses a baud rate of 2400, no parity, an 8-bit data size, and asynchronous operation. Note that there is only 1 USART on the board: USART0. Thus all relevant registers have a 0 appended to the registry name.

To set baud rate, the microcontroller sets the registers UBRR0H and UBRR0L. The microcontroller uses UBRR0 as a count-down timer to space transmit events. Note the register cannot be directly primed with the baud rate: the system clock speed must be accounted for (16 Mhz). The microcontroller uses the following formula to set UBRR0: ((F_CPU/(BAUDRATE*16UL))-1). The control register UCSR0C stores the rest of the flags. To see how the transceiver module hardware is configurationed, refer to the hardware details section.

To set up the transceiver, the 2 8-bit registers UBRR0L and UBRRLH are set with the computed value for UBRR0. The UCSR0B is set to either transmit or receive, depending on the function. Finally the UCSR0C register is set to reflect the desired settings: asynchronous, 8-bit, 1-stop bit, no party transmission.

To initiate a transmission, the microcontroller utilizes the transmit/receive buffer registers UDR0. It first checks if the buffer is empty and hangs until it is. Then it loads the buffer with 8-bit data signal and the data is sent through Port D1, which outputs to the transmitter module. The SendByte function, which takes an 8-bit string as input, implements these steps. SendPacket calls a sequence of several SendByte functions: a predefined synchronization signal, an address signal to filter out others using the same RF channel, and the payload itself.

To receive, the microcontroller is interrupted with a USART0 RX interupt whenever a transmission has been completed. Within the vector, the microcontroller uses a state machine with a switch-case statement: if a synch signal is received, it s for address. If address is correct, it waits for data. In the last step the microcontroller reads buffer register UDR0 and outputs it to the LCD.

Analog-Digital Conversion

The wrist-mounted Atmega644 senses acceleration input through its ADC. There are 2 accelerometers: 1 for x-y acceleration detection in the plane of the board , and 1 for z-acceleration perpendicular to it. Initialization of the ADC is handled through hardware register ADCSRA. Bit ADEN enables the ADC and bit ADSC begins a new one every time it is set. ADC period is divided of 128, to operate at 1/128 of our system clock frequency of 16 Mhz. The ADMUX register is set to select an input and set the reference signal VREF.

To manage 2 separate inputs Atmega644’s multiplexer was used, channel 7 for the z-accelerometer and channel 6 for the x-y accelerometer. This can be done by setting the relevant bits in ADMUX. Also within ADMUX, VREF is set to 5 volts. The ADC compares the incoming signal to VREF, converts it into a discrete 8-bit value, and outputs it to register ADCH. Our converted signal value Vin can be computed through the formula: Vin = ADCH * Vref / 256.0.

Data Analysis: Look-up Table, Masking Table, Tolerance Table and Hierarchical Detection

Analysis of which letter is being signed is done through the concatenation of two algorithms.

The microcontroller reads the inputs as digital values. The first algorithm uses tables for each of the letters. The tables contain expected bits for each letter from all the sensors. Every 100 msec, the program goes through the entire table and searches for the set of bits that matches best with the acquired input. When the microcontroller matches an input signal to a letter many ambiguities exist necessitating error analysis and setting tolerances and masks for each letter. A masking table is also used to ignore sensors that are activated randomly during hand movement. Unstable readings from the flex sensors require a lot of ambiguity detection and various tolerance levels, where a tolerance level is a number that signifies the number of sensors that that are not the same as our expected entry in the table.

The second method is a tree search which contains hierarchical detection. The first level relies on contact sensors, which provides the most stable readings and the 26 letters are divided into 6 groups. Further levels of detections pick up the most reliable sensor for each individual letter and narrows letters down until there is only one letter left. Letters ‘G’ and ‘Q’ are distinguished through the z acceleration, which reveals the tilt of the palm plane. Letters ‘K’ and ‘P’ are distinguished in the same way. Hierarchical detection assumes that the user is signing one of the 26 letters and thus makes the conclusion as soon as there is only one possibility left, without going through all the sensors. Thus in reality there is chance that the user is signing incorrectly but still can have a certain letter recognized.

Compared with the first method, hierarchical detection is more stable when it comes to recognizing letters, since sensors are used selectively and letters are recognized on their individual cases. By using both these algorithms together, we are able to come up with a recognition algorithm that is better than both of these methods individually.

Debouncing

Debounce code is added to prevent letters from being recognized during the gesture transition. Three states are used: STATE_WAIT, STATE_DETECTED, STATE_DETECTED_ONCE and STATE_CONFIRMED. If the letter changes, state transitions from STATE_WAIT to STATE_DETECTED. If the same letter is detected again, state transitions to STATE_DETECTED_ONCE, otherwise the STATE falls back to STATE_WAIT. Next time, if the same letter is detected once more, the letter is recognized the it is transmitted to the base station through the radio. No debouncing is used after letter ‘J’ or ‘Z’ is detected, since they are recognized by the acceleration and will not remain to have the same signature acceleration.

Results

The sign language translator works in real time.

All 26 letters can be successfully recognized by detecting the fingers’ positions. Hierarchical detection is fast and stable for most of the letters, but confusion exists among the following groups of letters:

• 'U' and 'E'
• 'V' and 'W'
• 'G', 'Q', and 'L'

The ambiguity exists mainly because of two reasons. The flex sensors are too sensitive, so they are sometimes activated when other parts of the hand are moving. Also, some flex sensors are not sewed on the glove in their ideal position, and thus cannot be activated for certain letters. By signing carefully and lengthening the debounce time, the program is able to recognize these ambiguous letters with at most 1 or 2 misinterpretations. The fastest rate we can achieve is about 1 to 2 letters per second.

The solder board on the Detection Unit is isolated from the user’s arm with the use of the second solder board, leaving no chance of the user being shocked from that circuit. There is Vcc on the copper tape for the contact sensors on the glove, but the internal pull up resistors are always enabled so the risk of hurting the user or damaging other electrical components is low. All the wires on the glove have insulation so there is no risk of shocking the user in this manner.

The Detection Unit does transmit on an unregulated band, but it sends a sync and address byte before transmitting any data. The address byte was randomly generated, and so long as it is not the same as anybody else’s address, and others also filter based on the address, there should not be any interference with others’ designs.

Anybody with the ability to see can pick up the glove and learn how to use it. It does take some learning, about fifteen minutes, to manipulate the glove to correctly translate the letters that are easily confused. Once this initial learning period is passed, however, the results are fairly accurate.

While there are patents on sign language translators based on gloves, there is no patent on sign language education software. There is also a publication opportunity for the education software.

No nondisclosure agreements were signed for this project.

Conclusions

The transceivers used in this translator conforms with FCC regulations, as mentioned previously. It also translates all the letters in the American Sign Language alphabet.

Bruce Land freely provided the code for text to speech synthesis. The voice used is released for private use only, and the voice is not in any videos published to the web. The sample code provided freely in the Atmega644 datasheet for transmitting over USART for the transceivers was used as a starting point for the transmission code. Otherwise, no code was reused.

Entire Program Flow

Parts list

Ethics

Safety is one of the highest priorities for this translator. The translator is carefully to not expose the user to any current from the circuit. While there is Vcc on the copper tape on the fingers of the glove, the internal pull up resistors will prevent damage.

During the development of the translator, we took safety precautions were taken when soldering, using the Dremel tool, and using the hot glue gun. We wore safety goggles when soldering and using the Dremel tool, and made sure to not let the hot glue gun or soldering iron touch anything other than solder or the object being glued.

A sign language translator has already been patented, and there are no claims of originality for this translator.

To educate others in how this translator works, this highly detailed documentation was written. A person educated in electrical engineering and familiar with the AVR architecture can recreate this translator with minimal difficulty. By developing this translator we hope that commercial sign language translators will one day reach the market.

The velcro strap is long enough to fit essentially all people’s arms. We provided help when asked for it. We lent the USB serial modules on occasion, and provided the transmitter code to Jun Ma and David Bjanes.

Appendices

Commented Code Listing

• base_station.c
• Compressed Audio:
• a.h
• b.h
• c.h
• d.h
• e.h
• f.h
• g.h
• h.h
• i.h
• j.h
• k.h
• l.h
• m.h
• n.h
• o.h
• p.h
• q.h
• r.h
• s.h
• t.h
• u.h
• v.h
• w.h
• x.h
• y.h
• z.h
• transmitter.c
• tableA.h
• tableB.h
• tableD.h
• tolerance.h
• lcd_lib.c
• lcd_lib.h
• uart.c - Modified to be compatible with simultaneous use of the transceiver
• uart.h - Modified to be compatible with simultaneous use of the transceiver
• downsample.m
• Find4Code476.m
• findOpt.m
• Speech_encode_GCC.m

Schematics

Schematic for the Detection Unit

Schematic for the Base Station

References

This section provides links to external reference documents, code, and websites used throughout the project.

Acknowledgements

Thanks to Bruce for providing the text to speech code, along with all his help throughout the semester. Thanks to all the TAs for all their help. Thanks to Mister Glove for providing helpful information on the transceiver. Sorry Bruce for cutting off part of your name on the protoboard for the base station.

Division of Labor

Ranjay Krishna

• Created sign language education software in java
• Created Matlab interface between java and Serial Comm ports
• Soldered Protoboards and flex sensors
• Wrote the framework for Detection Unit software in C
• Wrote algorithm for letter detection using Masking and tolerance tables in C
• Connected Detection Unit Together
• Wrote scrolling algorithm for LCD display
• Wrote algorithm for Receiver to display to LCD, convert text to speech and send to UART simultaneously
• Wrote the Documentation
• Set up Website

Seonwoo Lee

• Designed and built comparator circuits
• Designed and built mounting system for Detection Unit
• Compressed AT&T voice for voice synthesis
• Recorded own voice for voice synthesis, although it was not used
• Implemented battery mounting systems
• Created base for the Base Station
• Wrote transceiver code
• Wrote code to simultaneously receive from the transceiver and transmit over UART>
• Wrote the Documentation

Si Ping Wang

• Designed and Soldered Glove
• Wrote algorithm for letter detection with tree based search
• Knitted the entire Glove together with slots for flex sensors
• Wrote transceiver code
• Wrote interface with accelerometer
• Wrote code to simultaneously receive from the transceiver and transmit over UART
• Wrote the Documentation

Jonathan Wang

• Fixed ambiguity between register names for USART
• Purchased Hall Effect sensors