April 15, 2003

The Sound of One-Hand Signing

GW Engineering Doctoral Candidate Jose Hernandez-Rebollar Creates an Electronic Translation Device for American Sign Language

By Matthew Lindsay

American Sign Language (ASL) is one of the most used languages in the United States. Yet communications between the ASL-fluent and those unfamiliar with the deaf community’s chosen parlance are a constant dilemma. It’s especially problematic in emergency situations because no electronic translation methods exist for ASL. It’s a problem that GW doctoral candidate Jose Hernandez-Rebollar has devoted three years of his life to solving — how to construct a device that translates the sometimes complex movements of ASL into written and spoken form.

To that end Hernandez-Rebollar, who studies electrical and computer engineering at the School of Engineering and Applied Science (SEAS), devised the AcceleGlove. His ASL translation device uses accelerometers, a microcontroller, and algorithms he wrote especially to translate ASL into the written and spoken word in two basic steps.

First, the AcceleGlove is placed on the hand and strapped to the arm, allowing the accelerometers on the glove to act as sensors that generate signals from the movement, orientation, and positioning of the hand and the fingers in relation to the body. These signals are analyzed by a microcontroller to find the position of the fingers and hand trajectory.

The second step of the process is to take the recorded position and trajectory and find what they mean. To do so, the position of the fingers and the trajectory of the hand are run through Hernandez-Rebollar’s algorithms to detect the gesture and classify the gesture in a certain category, to find the correct word associated with the hand movement. This entire process takes milliseconds from the time the sign is made, to recognition of the sign, and the computerized voice saying the corresponding word.

Hernandez-Rebollar realized that while there are dictionaries to translate almost every spoken language, there is no electronic means of translating ASL. According to Thomas E. Allen of the Gallaudet Research Institute, the accepted range for numbers of speakers of ASL in the United States is between 500,000 to two million people, leading some to refer to ASL as the fourth most-used language in the United States.

“I started thinking about an electronic translation method for ASL before I came to GW,” says Hernandez-Rebollar. “The language has been around for almost 200 years, yet unlike most other languages we do not have electronic translation for ASL.”

A native of Mexico, Hernandez-Rebollar came to the United States and GW in 1998 on a Fulbright scholarship after completing his undergraduate and master’s work at University of Puebla, and later working at the National Institute for Astrophysics, Optics, and Electronics (INAOE). At INAOE, Hernandez-Rebollar was involved in building the largest millimeter telescope in the world. His specific area of interest involved the control of the telescope’s antenna. INAOE encouraged Hernandez-Rebollar to go overseas to obtain an advanced degree and training, which led him to apply for a Fulbright scholarship.

The Fulbright provided Hernandez-Rebollar funding for two years of coursework and one year of work on his dissertation. In 2000, he presented Nicholas Kyriakopoulos, professor of electrical and computer engineering at GW and Hernandez-Rebollar’s dissertation director, with the preliminary idea of creating a method of electronic translation of ASL for his doctoral research. Kyriakopoulos cautioned him about the difficulties of undertaking the project because Hernandez-Rebollar had no experience in that area.

“I thought it would be very hard and that the topic was mostly covered by others anyways,” shrugs Kyriakopoulos. “People who undertake experimental dissertations are generally risking more, there is less opportunity to change direction midstream.”

Undeterred by Kyriakopoulos’ warnings, Hernandez-Rebollar developed the primary circuits for the AcceleGlove while his dissertation director was on sabbatical during the spring 2000. Over the summer, he developed the first algorithm, or program, to identify ASL gestures. When Kyriakopoulos returned and saw the work Hernandez-Rebollar had already put into the project, he gave Hernandez-Rebollar his blessing to go forward with the AcceleGlove project. Now, Hernandez-Rebollar is on course to earn his PhD in May or the following semester.

For Hernandez-Rebollar, this research is the combination of two of his passions: helping others and inventing new devices by experimenting with electronics. In high school, he was actually more interested in being a doctor or a lawyer. However, as time passed, he realized the benefits of an engineering career. “In electrical engineering if I blow up something, I can buy another one and it’s no problem,” Hernandez-Rebollar says. “It also doesn’t hurt to get paid to do something you enjoy,” he chuckles.

When Robert Lindeman, assistant professor of computer science, realized Hernandez-Rebollar’s ability to build and test electronic products, Lindeman hired him as a research assistant to help with his work in virtual reality.

“Professor Lindeman really came to the rescue when my Fulbright scholarship was running out,” says Hernandez-Rebollar.

Without his research assistantship Hernandez-Rebollar could not have afforded to stay in the United States and in all likelihood would have been unable to continue work on the AcceleGlove.

It is still a question as to whether the deaf community would embrace a device like the AcceleGlove. In Hernandez-Rebollar’s research and testing of his glove he found two distinct feelings in the deaf community. Some feel that being deaf is not a handicap, it is simply another way of life and you should not use artificial means to overcome loss of hearing.

Meanwhile, others feel that if new technologies can help improve your life or make you feel better about yourself, why not make use of what is available?

However, the AcceleGlove is not a new technology that would simply be useful for deaf people in emergency situations. The ability to communicate through hand gestures could also be used to teach ASL, along with being modified for use in virtual reality, military settings, and in different forms of sign language. “The thing that makes Jose’s research so interesting is that it is applicable to so many different areas,” says Lindeman.

Of course, at this point the AcceleGlove is not viable as a commercial product and there is room for improvement and growth. The AcceleGlove currently only recognizes 173 words, although new signs may be added to the lexicon and recognized without Hernandez-Rebollar having to change the code. The algorithms used to recognize hand and finger movements can always be improved to become more reliable in their detection of words.

Hernandez-Rebollar is proud to say that the AcceleGlove correctly translates “easy” words more than 95 percent of the time, but what he calls “hard” words have only a 60–70 percent success rate. If he can extend the research, development and testing to gloves to both hands, Hernandez- Rebollar would be able to work with an even larger dictionary of words in ASL.

Although there is more work to be done with the AcceleGlove, Hernandez-Rebollar is not sure if he will have the necessary financial support to continue his research after his dissertation. He is not sure if other pursuits, such as one of the small side projects he is always working on, will eventually pull him away from the AcceleGlove project.

If the project continues it will keep Hernandez-Rebollar busy; a language is not something easy to catalogue and translate. When James Murray set out to create the Oxford English Dictionary, he did not know it was a project that would consume 40 years of his life, and that he would not live to see it completed. Does Hernandez-Rebollar want the AcceleGlove and make it his lifelong project? Right now he looks at everything as a blank slate. Fortunately for Hernandez-Rebollar, that is when he does his best work.

 

Send feedback to: bygeorge@gwu.edu