The ability to identify and respond to emotional cues from other people is integral in forming relationships. However, for individuals with autism spectrum disorders, it can be very challenging to navigate the feelings of others. Resources already attempt to address this, but as with many mental health services, the global demand is much greater than supply, putting a strain on both the healthcare system and on parents; that's where our app comes in. Requiring nothing more than an Internet connection, I Know That Feel is a free resource that aims to address these issues.
What it does
Our web application allows users to learn to identify and mimic emotional responses through playing facial emotion recognition games. The website is divided into 2 sections: one where children may play the games, and one for parents, where they may monitor their children's scores over time. Its easy to read format and reliance on visuals makes it easy to use for children who may be overwhelmed by visual/auditory stimuli or have difficulties reading.
How we built it
We created a web application hosted using Azure, using Flask for Python for our backend. Our application relies on the Flickr API to get photos of people with different emotions, along with the Indico machine learning API to detect facial expression from captured photos. We save information about each user's gameplay in a SQLite database which we can then query to return graphical analysis of user stats.
Challenges we ran into
One of the main challenges for us was getting started with Flask, which none of us had ever used before; we quickly realized that isn't as well-documented as we'd imagined. Furthermore, one of the key components of our application, the Indico facial emotion recognition API, doesn't always work as accurately as we'd hoped; thus, we would love to improve upon our emotion recognition algorithm to make it more precise.
Accomplishments that we're proud of
We are proud of pulling together to create the app, given the little experience that we had when we started. We are grateful to have been able to design a polished UI that supports both children and parental usage; we feel that the application has significant potential in becoming a valuable resource to benefit young children with developmental disabilities.
What we learned
We learned a lot about how Python, in particular how the Flask framework works, as well as about storing and querying for data through SQLite. We also learned much about the variety of API's that are available for analyzing human sentiment and making human interactions easier.
One of our original goals was to incorporate text-to-speech and speech-to-text interaction during the games, which would allow us to target a younger audience with more flexibility; after learning about APis such as Nuance and Microsoft's Project Oxford, we hope to be able to expand our application to utilize such abilities. We definitely hope that the application has the opportunity to make a social impact in improving healthcare treatment for children with autism spectrum disorders.