We wanted to build an app using the power of machine learning to enable autistic children to practice their social skills by exploring emotions with or without supervision from parents or therapists.

What it does

It accepts a list of user provided images (say by a parent or therapist) and identifies any faces and emotions in them using the Microsoft Emotion API. It then quizzes the child on the most significant emotion present in each face identified and gives them a score for their performance.

How we built it

The UI was build using React and the build process handled by create-react-app. It was hosted statically on Firebase since the app was entirely frontend. The requests to the Emotion API were performed using the browser native fetch method.

Challenges we ran into

We experienced difficulties with subtle timing issues with the React component lifecycle. In addition, although the Emotion API was mostly reliable, it is in beta and we did experience a few undesirable and incorrect results on specific images.

Accomplishments that we're proud of

We were very pleased with our eventual success in hunting down hard to trace bugs in our React component management.

In terms of the project, we were very pleased that, as a team of two (after our third member fell ill a few hours into the event) we were able to conceive an idea that was both technically feasible in the 24 hour timeframe, and also genuinely socially beneficial.

What we learned

We learned a lot about the details of the React component lifecycle. It was also our first real exposure to the emotion recognition API.

What's next for Learn. React. Understand.

We would like to develop the curated collection of images available so that the app can be effectively used even while entirely offline. In addition we want to improve the styling on mobile, making the app a true progressive web app.

We would also like to add a leaderboard to encourage (friendly) competition. Finally, we are interested in the potential of the technology for helping autistic children experiment and engage with their own emotional responses.

Built With

Share this project: