Autism is the fastest growing developmental disorder worldwide – preventing 3 million individuals worldwide from reaching their full potential and making the most of their lives. Children with autism often lack crucial communication and social skills, such as recognizing emotions and facial expressions in order to empathize with those around them.

The current gold-standard for emotion recognition therapy is applied behavioral analysis (ABA), which uses positive reinforcement techniques such as cartoon flashcards to teach children to recognize different emotions. However, ABA therapy is often a boring process for autistic children, and the cartoonish nature of the flashcards doesn't fully capture the complexity of human emotion communicated through real facial expressions, tone of voice, and body language.

What it does

Our solution is KidsEmote – a fun, interactive mobile app that leverages augmented reality and deep learning to help autistic children understand emotions from facial expressions. Children hold up the phone to another person's face – whether its their parents, siblings, or therapists – and cutting-edge deep learning algorithms identify the face's emotion as one of joy, sorrow, happiness, or surprise. Then, four friendly augmented reality emojis pop up as choices for the child to choose from. Selecting the emoji correctly matching the real-world face creates a shower of stars and apples in AR, and a score counter helps gamify the process to encourage children to keep on playing to get better at recognizing emotions.

The interactive nature of KidsEmote helps makes therapy seem like nothing more than play, increasing the rate at which they improve their social abilities. Furthermore, compared to cartoon faces, the real facial expressions that children with autism recognize in KidsEmote are exactly the same as the expressions they'll face in real life – giving them greater security and confidence to engage with others in social contexts.

How we built it

KidsEmote is built on top of iOS in Swift, and all augmented reality objects were generated through ARKit, which provided easy to use physics and object manipulation capabilities. The deep learning emotion classification on the backend was conducted through the Google Cloud Vision API, and 3D models were generated through Blender and also downloaded from Sketchfab and Turbosquid.

Challenges we ran into

Since it was our first time working with ARKit and mobile development, learning the ins and outs of Swift as well as created augmented reality objects was truly and eye-opening experience. Also, since the backend API calls to the Vision API call were asynchronous, we had to carefully plan and track the flow of inputs (i.e. taps) and outputs for our app. Also, finding suitable 3D models for our app also required much work – most online models that we found were quite costly, and as a result we ultimately generated our own 3D face expression emoji models with Blender.

Accomplishments that we're proud of

Building a fully functional app, working with Swift and ARKit for the first time, successfully integrating the Vision API into our mobile backend, and using Blender for the first time!

What we learned

ARKit, Swift, physics for augmented reality, and using 3D modeling software. We also learned how to tailor the user experience of our software specifically to our audience to make it as usable and intuitive as possible. For instance, we focused on minimizing the amount of text and making sure all taps would function as expected inside our app.

What's next for KidsEmote

KidsEmote represents a complete digital paradigm shift in the way autistic children are treated. While much progress has been made in the past 36 hours, KidsEmote opens up so many more ways to equip children with autism with the necessary interpersonal skills to thrive in social situations. For instance, KidsEmote can be easily extended to help autistic children recognize between different emotions from the tone of one's voice, and understand another's mood based on their body gesture. Integration between all these various modalities only yields more avenues for exploration further down the line. In the future, we also plan on incorporating video streaming abilities into KidsEmote to enable autistic children from all over the world to play with each other and meet new friends. This would greatly facilitate social interaction on an unprecedented scale between children with autism since they might not have the opportunity to do so in otherwise in traditional social contexts. Lastly, therapists can also instruct parents to KidsEmote as an at-home tool to track the progress of their children – helping parents become part of the process and truly understand how their kids are improving first-hand.

Built With

Share this project: