Inspiration
Making friends and maintaining relationships can be hard. This challenge is further complicated if you are a youth living with autism. Autism spectrum disorder (ASD) is characterized by difficulties with social skills, speech, and nonverbal communication. People with autism often struggle to identify emotion and react appropriately. Such struggles can make it difficult to form friends, even though, like all youth, youth with autism yearn for connection. The obstacle these youth face in identifying emotion was our inspiration for this project. By facilitating emotive identification through technology, we hope to be able to support the widespread emotional development and well-being of youth with autism.
What it does
On regular intervals, Emote will capture an image of your screen and send it to the Microsoft Azure Cognitive Face API to analyze the expressions on any face present. The top three closest matches in emotion will be displayed on a screen, with the size of the emotion name corresponding to the likelihood of the match.
How we built it
We used mss, a screenshot library, to grab an image of the user’s screen on regular intervals, we implemented the Azure Cognitive Services Face API (with the Python module cognitive_face) to get emotion data from what's on the user's screen, and we used Tkinter to create a custom accessible GUI for our application.
Challenges we ran into
Our biggest challenge was a result of using tkinter to format our screens. Specifically, we had difficulties making our string variable update labels without reconstructing the screen. The solution was simple: we realized the variable needed to explicitly be defined globally.
Accomplishments that we're proud of
Our biggest accomplishment was thinking of an ambitious product and developing it in 36 hours! Many of our team members had never created a product geared to a specific audience, so having to worry about the user interface and accessibility was new to us. Additionally, since our product is directed toward an audience with a disability that none of our group members have, we made sure to do thorough research in order to avoid misguided design errors. Overall, we are proud of the product that we developed and cannot wait to share it with youth with autism.
What we learned
During project development, we applied many theoretical skills from class to create a real project with social purpose. In the process, we learned to use an API and used the Python library Tkinter to create engaging graphics. Through various design interactions, we also gained a greater understanding of the challenges faced by people living with autism and autism spectrum disorder itself.
What's next for Emote: A Tool for Youth with Autism
Our next plan for Emote is to add a feature that uses the text-to-speech and natural language processing APIs to support sentiment analysis. It will assist autistic youth to interpret emotions from text messages, emails, and more.
Log in or sign up for Devpost to join the conversation.