Gesture Dictionary

• 

  Gesture Dictionary

Inspiration

We wanted to create an alternative and augmentative communication device for individuals with communicate through gestural systems and to provide individuals who communicate primarily through non-ASL gesture schemes (usually due to a form of disability) with a framework to document and translate their own gestural system to provide better interaction with their community and peers. As of now, the best AAC 'device' that people who communicate like this have is the idea of folding a piece of paper in half and writing the gesture on one half and the translation on the other side. We, Beginner's Luck, decided that we could do better.

What it does

Our project uses Myo Armband, an integrated sensing device that has EMG, accelerometer, gyroscopic, and orientation data to record a gesture, and then store it with a translation as far as what this gesture means to the individual who performed it. When this gesture is then performed while in the translate mode, the application will translate their gestures for other individuals in their environment to be able to more easily communicate with these individuals.

How I built it

We built an iOS application and used the Myo SDK to use the raw data to record a gesture. Then we sent this data through a database to store it as a gesture that can be translated. When in translate mode we use a pattern recognition schema to translate the gesture.

Challenges I ran into

Data analytics, database issues, newbies to all of the technologies used.

Accomplishments that I'm proud of

The UI of the app, and the overall idea of the positive impact that this application could have with proper implementation.

What I learned

How to work with swift, objective c, Parse (database), Myo SDK, data processing and analytics

What's next for Beginners Luck

Translating the gestures. Fixing the database error that we were getting. Lots of other features to make the app as accessible and user friendly as possible. This could turn into a tool that enables someone to communicate like they never have before. It's a project