Sign Language Interpreter

Why we chose this project

The purpose of this project is to bridge the communication gap between the deaf community and non-ASL speakers, without the use of cameras and computer vision, but rather with wearable hardware. This project would be beneficial as a teaching resource for those learning ASL and people who are unable to speak and rely on ASL for communication.

What it does

The Interpreter reads data from an accelerometer attached to the user's thumb. The sensor reads the thumb's acceleration in all 3-axes. The data is then sent to the ESP software where its movement is compared against training sets of different gestures. If the software recognizes a gesture, it displays its label, when and for how long the gesture appears.

How we built it

An accelerometer is attached to an Arduino UNO which sends the data to the ESP software. The algorithm is trained to recognize specific motions we label and add as training sets. We are able to record our gestures and categorize them under a specific training set. Upon comparing incoming data to collected training sets, the algorithm can determine what gesture was performed and display what the gesture was and for how long it was performed.

Challenges we ran into

ASL requires the use of all 30 finger joints and wrists to effectively communicate. This means that we would need at least 32 accelerometers and some method to compare their positions. Hardware limitations meant we only had access to one accelerometer, meaning we would not be able to use real ASL. This would limit the amount of training sets we could create and the range of motion we could use. To resolve this issue, we trained the software on simple gestures that would only require one accelerometer. This heavily limits the number of gestures that could be recorded, however, it still serves as a valid proof of concept.

Accomplishments that we're proud of

Creating distinct data sets to train the software took an extremely long time. Differentiating noisy data from accurate data from indistinct data severely limited the efficiency of manually training the algorithm. It was extremely satisfying however to be able to accurately differentiate gestures from random movement reasonably frequently with a relatively small amount of of data.

What we learned

Through this project, we learned about the operations of accelerometers, gyroscopes and the basics of machine learning.

What's next for The Sign Language Interpreter

Moving forward, using more sensors would enable the project to recognize more complex movements. and build more comprehensive data sets. This would also mean upgrading from the ESP software to collect more data sets and more efficiently retrieve them. In order for the project to reach practical usage, it would also need to be able to send recognized gestures from the software to a display.

Built With

• arduino
• esp