-
-
Happy detection
-
Surprise detection
-
Zoomer app
-
Graph of emotions in a time frame in Zoomer
Inspiration
We were inspired by the current challenges with teaching online during the Covid-19 pandemic. This is the first time teaching virtually for many professors and teachers, and without the regular in-person interactions with students, they may have a hard time knowing if their teaching style suits their students.
What it does
Zoomer recognizes and detects the facial emotions of students during online classes and lets the teacher know of their moods so that they can adjust their teaching style accordingly. Moreover, Zoomer stores the emotions at different times throughout the class and plots the emotions in a graph so that the teacher can see when students were feeling a particular time.
How we built it
The sentiment analysis model used for the app was trained from scratch with Keras using a convolutional neural network and the Facial Expression Recognition (FER) Challenge dataset (2013), which consists over 35800 labelled images of facial expressions in grayscale with a resolution of 48x48. The data contains a wide range of human subjects across age, sex and ethnicity as well as some non-human subjects such as cartoons and images that doesn’t contain any face but is either blank image or contain some text. The images are produced in wild settings and have diverse lighting since they are obtained from web scraping. As a result, the resulting model performed well and is more robust to changes in environment such as camera quality and ambient lighting. Data augmentation such as random horizontal flipping, translation and zooming was applied to the dataset for even greater robustness. Robustness was an important consideration as the model would be deployed on a wide range of environments with wildly different camera qualities and lighting conditions. Finally, class weights are computed during training to account for class imbalance in the dataset.
We created the user interface for the prototype using Tkinter. In addition, we used OpenCV to access the video feed from the webcam and MTCNN to allow multiple face detection in real time. Detected faces are cropped and sent to the model after some preprocessing and the results overlaid on top of the video using OpenCV. Other libraries such as time, pandas and matplotlib were used to visualise change of aggregate emotion scores within a time frame.
Challenges we ran into
The model we initially used had bugs which required some painstaking fixes. We were also challenged by coming up with how we wanted the UI to look and what other components to add to the app that users might appreciate, such as a graph of positive/neutral/negative emotions at different times during the online class and an "Emotions" panel to view general moods for reference.
Accomplishments/What we learned
We're proud that it finally works after a lot of persistence! Not all of us have had much experience with AI related projects, so getting into a bit of sentiment analysis and dealing with pretrained models was a nice introduction for some of us. Not all of us knew about Tkinter too so some of us were exposed to another GUI package-- it's generally good to know of other options. Overall, we got to learn a little from each other because of our various past experiences.
Log in or sign up for Devpost to join the conversation.