Inspiration
We know that Covid has passed but virtual meetings are here to stay. We all know that during zoom/teams meetings, since we are not in a room face-to-face with each other, it is hard for us to understand our audience and the atmosphere in the “room”. For example, it is hard for a professor to look at all the students during a class. With our app, the professor can make the class more lively, more active and more responsive by observing the emotion of the classroom. If the emotion in the classroom is surprised, for instance, the professor might choose to pause and take more questions or go over the topic again.
Make dull meetings fun with Emobot!
What it does
Emobot uses a Machine learning model for facial and emotion recognition. The program takes periodic screenshots of your screen, and uses the first machine learning model to detect faces and crops them out. The second machine learning model uses those saved faces and predicts the emotion they most closely represents. It then returns an emometer to show the current mood of the
How we built it
We built in a number of different steps and divided into 2 main sections. Half the team worked on the webapp while the other half worked on training the model, image processing and classifying.
For the Webapp Our goal was to have the entire program run as a web application hosted on the internet so that anyone, anywhere could use it for free. We developed the front end using HTML, CSS, and Bootstrap. For the back end, we used Flask and Jinja to take in requests from our clients and run our python program in the back end. For the Machine Learning Model We trained the emotion detection model from scratch using the AlexNet Convolutional Neural Network and fed it preprocessed grayscale 227x227 images of 7 different emotions in google colab. The libraries that we used included tensor flow, Matplotlib, Keras and Numpy. After training our model we saved the weights and imported them to our project. In order to use the model to predict images we first used pyscreenshot to take screenshots and loaded images using cv2. Using cv2 we classified faces within the screenshots and saved them. Those faces were then preprocessed, converted into arrays and then predicted using the model. The emotions for all the pictures were saved in an array and then represented in a bar graph showing the current mood in the picture.
Challenges we ran into
Challenges for the machine learning model A lot of the libraries had issues when importing for some reason such as Tensorflow, Scipy and Keras. Our model also took some time to train and it was unable to save due to insufficient space in google drive which resulted in us having to retrain the model. We also had trouble importing large files to github (our ML Model was 700mbs). Challenges for the webapp The main challenge that we ran into was efficiently connecting the web app to the machine learning model. The machine learning model is too large, causing delays due to latency, and so we were unable to reach our final goal. However, through these struggles, we learned a lot of key concepts in back-end web development.
Accomplishments that we're proud of
We ended up with a fairly accurate machine learning model that can predict emotions from a picture with multiple faces. We also managed to create a very user friendly UI.
What we learned
We learned how to train a convolutional neural network and properly process our images for the model. We also learned about front-end web development and how to make a webpage a responsive.
What's next for emobot
For our next steps we want to deploy our model to a webapp that can predict emotions in realtime.
Log in or sign up for Devpost to join the conversation.