Audiosyne

Inspiration

We wanted to do something with a Myo, as it was hardware that we never worked with before. Since it gave us data about one's arms, such as position, orientation, and engagement of muscles, we thought that a good use-case of something like this would be to interpret sign language.

What it does

It recognizes sign language that the user does and translates them real-time into speech.

How we built it

We built it using sensor data from Myos, using them to detect custom gestures and using text-to-speech technology in order to output the result as speech. We built two applications using this: one is a C++ application that utilizes a hackathon project that already implements custom gesture training and recognition (link) while the other is a web app that has the gesture recognition built in-house.

For the web app, we utilize the Google Predictions API to match the gesture to the phrase. We record snapshots of one's arm's orientation and EEG data of a complete gesture every 0.25 seconds for up to two seconds and send that array of snapshots to the API which returns the closest gesture based off the custom training model.

Challenges we ran into

We ran into challenges in standardizing a user's action so that we can match actions of variable length as well as slowed down or sped up actions. We tried using Fourier transforms to do this, but in the end it proved too challenging for the time we were given and so we resorted to recording the action for two seconds and timing when we start and stop the action.