TalkToTheHand

Inspiration

A deaf person's struggles in Stats class inspired his classmate to create a tool that can help all deaf people communicate more effectively with others and vice versa.

What it does

Using the Myo armband, a deaf person can use sign-language and transmit his/her to an Android app that will voice and transcript his/her signs. Those who aren't deaf can also communicate, as their speech translates to text that the deaf can read.

How I built it

We trained a Myo armband with machine learning. Using the KNN (K-nearest neighbor) algorithm, our code can classify gestures and transmit the correct message. Our Azure Speech Cognitive Service then transmits the message into audible speech. A person can then speak and have his/her message transcribed for the deaf to read.

Challenges I ran into

We tried to run Speech to Text service in Azure, but our student authentication was denied. With only a little time left, we decided to pivot to focus on submitting on time.

Accomplishments that I'm proud of

Using KNN. Using Azure Cognitive Services.

What I learned

Sampling. KNN. Machine learning. Azure Speech Cognitive Service. Postman API calls. Android Studio

What's next for TalkToTheHand

We plan to integrate a filter on the raw data provided by the myo armband to increase our accuracy. We plan on expanding our dictionary of gestures to include facial and head movements too.