Inspiration
Did you know that 0.1% of the Swiss population is deaf. Even though Switzerland is a well developed country, those people can still have trouble communicating with other, which can lead to marginalization. As a hearing person, we are not aware of the importance of communication in our everyday life. Imagine having trouble communicating with your doctor while describing your symptoms! This is why we wanted to tackle this problem.
What it does
The app uses AI to translate the sign language gesture into text, and then into speech that allows any interlocutor to easily understand what the user tries to say. Finally the spoken answer of the interlocutor is converted to text that the user can read.
How we built it
We decided to make an android app for portability reasons. We integrated some AI at multiple stage of the creation : we used machine learning for the sign language recognition (we had to create our own data set to train our model), and for all our speech-to-text/text-to-speech translations we used the language recognition framework of Apple.
We constructed our app as follows: we have two main features that represents both way of communication. On one hand, we have sign language to speech : we trained a model to recognize the ASL and output the corresponding letter on a computer and integrated it into our app using coreML. Multiple letters forms a word that can be red using a button. On the other hand, we can translate speech to ASL characters : we first have the spoken answer converted into text, which itself is converted in an animation containing the different alphabet characters.
Challenges we ran into
At first, we wanted our app to recognize sign language. However, after a few researches, we discovered that this is still an open question. Due to the lack of data set, the best AI we found has a 60% accuracy (and impossible for us to reproduce). Hence, we decided to go for alphabet characters. Here, the challenge was to have a decent recognition. As we did not found a correct data set, we had to create our own.
Accomplishments that we're proud of
We are really proud to make an app that can have a real impact. While designing the app, we had a certain problem in mind, and we are convinced that our solution is on the right track to improve the life of some disable individuals.
What we learned
As none of us knew how to sign, we all had to learn some rudiment of alphabet to test our app. We also familiarize ourselves with Apple Machine Learning frameworks such as CoreML we had to integrate a fully trained neural network onto our app,
What's next for Undeaf
We started with a small step (translation of alphabet characters) but we still think big. Our real goal is to be able to translate sign gestures of a deaf person in his everyday life.
Log in or sign up for Devpost to join the conversation.