As students who have been interested in medicine at one point or another during our college years, we wanted to create a hack that could be beneficial to those with health ailments or disabilities. We were inspired by a mentor to address the communication gap that many mute or deaf people experience by creating an app to help convert sign language to visible words or phrases for almost anyone to understand.
How it works
Our Android app works with the Myo armband, which utilizes accelerational, rotational, gyroscopic, and EMG data to determine several complex gestures that may be used in American Sign Language. First, the user must train the app to recognize sign language gestures. The training data is then used by the app to recognize the gestures unique to the user. The gesture is converted into a word and the word is read out loud via TextToSpeeech. Although this would require some cooperation and work from the user, we believe that it is a step closer towards an easier future.
Challenges We ran into
One of our biggest difficulties was simply the Myo armband itself. It was buggy and difficult to keep a connect. It's own built-in gestures were not reliably recognized. The Myo's issues carried further by making machine learning of gestures extremely difficult. But perhaps more challenging was dealing with Weka, a machine learning software that we used. Its classes and functions were simply unintuitive and much of our time was spent simply understanding and making Weka work the way we wanted to.
Accomplishments that I'm proud of
We are proud that despite the most frustrating of problems (Hi Weka), we did not give up. We didn't take any naps or breaks and we continued to push through because what we were building was not just cool but also had real potential to make a difference.
What I learned
We learned that sometimes its more effective when two or more people work on the same piece of code than it is if everyone individually works on their part of the project. Collaboration was definitely key in solving some of the issues we've had. Also, almost nothing works like you want it to!
What's next for MyoSign
Sleep, then back to coding!