Signlator

Inspiration

Deaf people use sign language to communicate with each other. But because the majority of us don't understand sign language, so they still have to type in their cellphones when communicating with people with hearing disabilities, which is inconvenient. The advance of technology has empowered people with disabilities and facilitate communications, but there are few works which address this problem. This inspired us to propose a solution both for easier communication with deaf people and for people to learn sign language.

What it does

In one way, the program we built can pick up the gestures and return the corresponding meaning of it.

How we built it

We used Leap Motion as the sensor to pick up the gestures and performed feature selection to get the data we needed. We did Logistic Regression on the data we collect and train the model. After training the model, we perform multiple validation testing to try and get the best model with the highest accuracy.

Challenges we ran into

Leap Motion is not sensitive enough to differentiate some similar gestures
It took us a while to learn the API of Leap Motion
Time didn't permit us to collect an even larger amount of training data
Time didn't permit us the fine tuning of the neural model thus we had to use simpler models (Logistic Regression).

Accomplishments that we're proud of

Build a dictionary for some letters in the alphabet in sign language
Output a model containing variables needed to predict a motion ## What we learned
We gained an understanding of how Leap Motion works and how we can build a program that works with the sensor.
We learned to build a data pipeline
Use logistic regression to predict the data model ## What's next for Signlator
Upgrade the training method (instead of using Logistic Regression, we can use a neural network or recurrent neural network to build a model).
Choose a more accurate motion sensor
Implement other program features such as the ability to teach users on how sign language. Because we have a model that can understand sign language, we can use the program to train people who don't know sign language (think of Duolingo but for sign languages).
Better user interface.