Michael's cousin, Chris, has had autism since infancy. Chris lives a good life, but he obviously has many difficulties, especially when communicating. Individuals with autism find it extremely difficult to find the right words to say for a given situation - they may completely understand what's going on, but forget the right words to express themselves. This is called "Expressive Aphasia." Our hack offers a solution to these stressful moments, by giving the user "HINTs" on what to say

What it does

The app listens for a voice (a 'friend') speaking to the user. The friend's voice is recognized, and we generate an appropriate response for that particular user (i.e. what that user would actually say in response) based off off a Keras/Tensorflow model. This response is presented to the user as a viable recommendation. The user's actual response is recognized and saved to the model as training. This pattern is repeated for as long as the user leaves the app on, so as to facilitate a conversation.

How I built it

Our hack uses google glass (android), and flask (python) on the backend. We're using keras and tensorflow to build out the ML model.

Challenges I ran into

Michael had never made a native Android app until MHacks (react-native, anyone?). That was a lot to take in. Vineet had to wait 15 hours to train our model (we used eBooks like Alice in Wonderland).

Accomplishments that I'm proud of

This is Michael's first ever ML hack! I'm sure Vineet is proud of something too but he's sound asleep.

What I learned

Michael learned more Android dev than he expected to :). As well as some ML techniques.

What's next for HINT

With a cheaper (and not discontinued :P ) headset, and some tweaking, this is a viable app. It could help thousands of people who have autism live more fulfilling, enjoyable lives. We need ALOT more training data if we want our model to be accurate. I think the UI could be made more simpler if we could record both people and use a classifier to sort out who's who. That way, the user won't have to be prompted for a recording twice.

Chris has said on people with Autism that "They're smart and they want friends." It's hard to make friends if you can't find the words to express yourself, trapped in your own mind. HINT frees you, it knows what you need to say when you need to say it. HINT gives a voice to those who can't find the words. HINT is the key to finally being understood.

Built With

Share this project: