sentiface: A Tool to Analyze Facial Sentiment in Videos

sentiface allows the user to analyze people's sentiments in a video. Using the Microsoft Azure platform and machine learning, sentiface identifies and gathers information from people's faces in the video. It then charts each person's sentiments in a data visualization display for the user to analyze.

Use Case

sentiface is designed to help users without any programming skills to harness the power of machine learning to generate data from visual sources, automating the tedious task of emotion labeling. One possible application of this user-friendly platform is for constituents to be able to see, in great detail, how speeches such as a State of the Union address emotionally affect all that are in the audience, including our elected congresspeople. Since the Azure Face API can extract predicted age, gender, and emotion data, we can graph distributions conditioned any of these variables, such as the distribution of emotions given gender is male. In addition, since we have timestamp information, we can show how an individual person's emotions progress over the course of the video (the multiple face detections of the person are identified using the DBSCAN clustering algorithm described further on.)

Tech

sentiface was build using Python, React, scikit-learn, and the Microsoft Azure Face API. The Face API handles face detection, emotion classification, as well as lots of other information about each person in the video. It greatly sped up our design process as we didn't need to design the machine learning components from scratch. Once the video has been analyzed, we calculate face embeddings using OpenFace, which essentially reduce the faces down to a semantically meaningful 128-dimensional space, and pass those into the DBSCAN clustering algorithm to match faces to distinct persons in the video. Using this approach, we don't have to know ahead of time who the algorithm should be looking for in the video.

License

This project is licensed under the MIT license in the LICENSE.txt file.

Share this project:

Updates