- Atrik Das - 456
- Han Simeng - 444
- Lakshyajeet Dwivedee - 430
- Rohan Gautam - 454
S.A.D.F.A.C.E. stands for Sentiment Analysis Data For A Computer-vision Extension and has been designed for YouTube. In this data driven age, it is imperative for content creators as well as companies to get more metrics on how audiences perceive their content. S.A.D.F.A.C.E. does just that through live recording and analysis of viewer emotions.
YouTube is definitely not at its prime now as it once used to be. Videos can fail to target the right audience or flat out get demonetized because they happened to break even a single one of YouTube’s stringent content policies. This leads to unhappy creators that stop taking creative risks in order to avoid having their only source of living being snatched away from them. S.A.D.F.A.C.E. can help them tailor their content to their audience and adapt to changing opinions.
Some possible applications of this project are:
- Helping video creators get insight on their audience and make better content.
- Help companies better market their products in advertising by analyzing consumer reactions.
- Help in psychoanalysis of patients by detecting abnormal reactions.
What it does
Our project is a Chrome Extension which is built to detect the user's emotions as the user watches a YouTube Video. The percentage of each emotion is captured for each second of the video and stored in the database by the backend. This emotion data is then shown in easy-to-understand real-time graphs on a dashboard for anybody to analyze, especially the video creators. You can also search for any YouTube video on this dashboard to see it's emotion data as and when the viewer sees the video. Emotions shown: Angry, Disgust, Scared, Happy, Sad, Surprised, Neutral
How we built it
- Python to do the emotion recognition utilizing two pre-trained models(haar-cascade files).
- A python backend to set up a flask server, to interact with the mongo-db database, and to interact with the chrome extension.
- Chrome Developer Dashboard was used to publicly list the extension on the Chrome Web Store.
- Various Chrome APIs were called such as ActiveTab and VideoCapture to give the extension the required features.
- The front end is built using React and Material UI.
- Axios is used to listen for video searches and query the MongoDB database hosted on MLab.
- Chart.js is used to display the data retrieved from the database.
Challenges we ran into
- We were unable to use “fetch()” to make AJAX requests to the database as it always returned an undefined value. Axios was used as a more robust alternative.
- Due to may async requests, the front-end was very prone to errors which could cripple the system. Thus, it was necessary to add error handling for situations such as when the entered video doesn’t exist in the database.
- We were unable to print the current timestamps from a YouTube video to the extension popup because the DOM loaded slower than the code was processed even though they were being logged into the console.
- Figuring out which requests to send to what environment so that a particular API worked in the chrome extension was a complicated process.
Accomplishments that we're proud of
- Getting live data from a MongoDB database and showing it in an easy-to-understand, sleek User Dashboard using React.
- Made a chrome extension that interacts with the backend with flask.
- Making the backend able to interact smoothly with the database, frontend and the chrome extension.
What we learned
- How to make asynchronous requests to remote MongoDB databases using axios
- How to display data in a succinct and comprehensible format using Chart.js
- How to manage a vast amount of data through states and props in react via component lifecycle methods.
- How to send requests between the various environments of a chrome extension.
- How to query information from YouTube videos into the extension.
What's next for S.A.D.F.A.C.E.
- Partner with sites like YouTube to add a reward program to incentivise viewers to record their emotions.
- Enabling crowdsourcing of data instead of taking the data from only one user per video as it is now.
- Build our extension so that it can run 100% online (for instance, webcam works offline for now).
- Synchronize the reactions video perfectly with the actual video and allow pausing of the video (we have it working with Selenium but not with the actual extension).