-
-
This is our demo on a web app. Every second, a video frame is sent to the backend and analyzed by Hume's Expression Management tool.
-
This is what our tool will be utilized in online calls. Red squares follow faces on the screen and note the emotion detected on the face
Inspiration
All three members of the team have had personal experiences with people on the autism spectrum, and seen firsthand their struggles with deciphering emotions. With the recent coronavirus pandemic, the use of online conference programs such as Zoom have exacerbated the issue of reading emotional cues. This is not only a problem for the 75+ million people struggling with ASD worldwide, but also for people with alexithymia, a condition that affects individuals’ ability to understand emotion (which makes up 10% of the world population). Our team decided to solve this problem through the development of Evatone, a tool that provides assistance in emotion identification on video conferencing platforms.
What it does
Our tool is emotion detection software specifically incorporated for video conferencing. It allows users to see what major expressions they elicit as they speak and identify major emotions expressed by the other participants in the video conference. This appears as labels on video participants’ faces, allowing people with Autism and alexithymia to quickly gauge the emotions of others in the meeting room.
How we built it
We created a live webcam feed using Typescript (part of our front end) that takes in video input. At set intervals, we send a frame from the video as a JPG image to the Python backend using Flask REST APIs. In our backend, we used the Hume Expression Measurement streaming API, along with a web socket to maintain an open connection, to analyze the facial expression of the frames in real-time, and detect the emotion. We parsed the output of Hume’s list of emotions detected to include only the emotion with the highest score (as this represented the dominant emotion), and then sent this data back to the front end (Typescript) to display it on our live webcam feed. Using cv2’s face detection model paired with div elements part of our html, we then overlay a red box over the face detected in the webcam with the dominant emotion label.
Challenges we ran into
One challenge we ran into was ensuring that the live webcam feed was running the entire time our application was running. Originally, we built out both the live webcam feed and the backend with Python, using Python’s cv2 library to capture the frames. However, we found that the cv2 webcam frames conflicted with how we were sending frames to Hume’s API. Thus, we transitioned to using Typescript for the webcam and Python for the backend, which allowed us to run the webcam feed the whole time while simultaneously sending frames to our model.
Accomplishments that we're proud of
We are proud of being able to connect the front end and back end of our code, seamlessly presenting our backend output and data in a more presentable way and with a much better user interface, using front-end code. We are also proud that we were able to learn how to use Hume’s API. On Saturday, we spent hours at Hume’s table debugging our code and learning how web sockets work to allow real-time continuous detection of emotion in facial expressions. We’re really proud that, while we ran into challenges with using Hume’s streaming API, we pushed through, asked questions, and got our final output!
What we learned
We learned how to work with APIs, as we navigated the Hume API to incorporate it into our code for facial emotion detection. We learned about web sockets and how they allow for a continuous connection. We also learned how to code in TypeScript and how to use Flask as a framework for connecting HTML/CSS/JavaScript with our backend Python code using POST and GET.
What's next for Evatone
We envision Evatone to be incorporated into online meeting platforms like Zoom, Google Meet, and Microsoft Teams. Furthermore, we see Evatone increasing accessibility for people with autism across the online sphere. We have thought about making a Chrome extension that can scan for any face currently on screen, whether that belongs to a Zoom call or a YouTube video, and similarly detect the emotion of the face, helping people with autism navigate online social interactions with ease.
Log in or sign up for Devpost to join the conversation.