With more than 400,000,000 hearing-impaired individuals worldwide [1], it is evident that hearing loss is a major issue in society. The prevalence of hearing loss makes it obvious that it is necessary for an effective and efficient means of communication with the deaf community. I created a reliable translator from American Sign Language to English to help improve communication between individuals who cannot understand American Sign Language but speak English and individuals who are hearing-impaired.

What it does

My project allows its users to wear a glove, and gesture words, letters, or numbers in American Sign Language. My application translates the gesture into English text. The project allows an individual who knows no sign language at all to communicate with the deaf. Furthermore, it can be used by the deaf community as a "sign-to-text" feature, which would be analogous to our "voice-to-text" features.

How I built it

In summary, SignArm was coded with Base Shield IDE, C++, MATLAB, and a lot of hardware. Its corresponding web application was created using Javascript. The app works by using a binary system with the aid of multiple accelerometers and motion sensors. The motion sensors are programmed to detect a change in electrical resistance through the method of triangulation in polar coordinates. Subsequently, a one-hundred million long binary sequence is recorded, in which each "0" represents a cross-sectional area that is straight, whereas a "1" corresponds to a cross-sectional area that is curved or bent.

After that, each term in the binary sequence is compared to key sequences in a Amazon's web database, provided by the National Deaf Association, using a MATLAB script. If the recorded sequence matches at least 80% of a key binary sequence, the result is outputted.

Challenges I ran into

I had some trouble minimizing the computational complexity of my database searching algorithm. With millions of entries to search through, I was having to wait close to a minute for each response. However, through research, I learned a lot about asynchronous programming, and I ended up switching to Amazon Web Services for my entries. Now, there is no noticeable delay in output.

Accomplishments that I'm proud of

This was my first time using Base Shield. I learned a lot about wiring them in the process. Furthermore, this project was extremely math-intensive. I am proud that I managed to get the project to work in time without error.

What I learned

This was my first time using Amazon Web Services. I learned a lot about its capabilities. I also learned how to incorporate several different programming languages into one single application. Lastly, I learned a lot about statistics (e.g. Monte Carlo Simulations to test our results against actual data)

What's next for SignArm

Unfortunately, I was a one-person team, so I was not able to add all of the features that I wanted in the time I had. In the future, I would like to make the glove wireless via Bluetooth. Additionally, there is currently a ~3% margin of error (approximated using 10,000,000 MATLAB simulations). I would like to minimize this as much as possible. Lastly, a feature that reads the text aloud after translating it could be beneficial.


Share this project: