reac

Remember the last time you were disappointed by a clickbait thumbnails? Reac solves just that issue by using facial expression pattern recognition to help determine if a video is click worthy or not.

Comment

Inspiration

Binge watching cat videos while laying on my bed with a bag of chips of my tummy happens to be one of my favorite past times. However, being click baited isn't. Hence my team and I decided to make a rating system for videos that is honest and hard to be faked by bots.

What it does

"reac" captures the facial expressions of the audience at every second of the video. The five type of expressions it can detect are happy, sad, angry, neutral, and surprise. This data can be graphed and used by both the future audience to determine whether a video is click worthy, by seeing the overall reactions to it, be it happy or sad. This data can also be used to help content creators improve on their content as it provides them honest feedback.

How we built it

It is built using python and flask frontend. Using the fer library, we were able to determine the facial expressions and the magnitude of them as well.

Challenges we ran into

Unfortunately, due to a lack of skill set and inexperienced with machine learning, we were unable to complete the project to its fullest. We did manage to output the graph to an applet but not the html file. We were not able to fully utilise tensorflow with the live feed due to hardware restrictions such as the lack of an nvdia GPU.

Accomplishments that we're proud of

We were able to collect and display a live webcam feed on the web app We were able to track the face on the live feed using opencv We were able to process the tracked faces and return a set of data containing the facial expressions We were able to graph the set of data and display it on an applet

What we learned

How to implement machine learning api's in python How to use tensorflow How to convert videos to images and then to byte arrays

What's next for reac

Porting the graph from the applet to the web app Combing multiple sets of data from different viewers to find an average expression of the video Testing with different hardware to get closer to real time facial recognition

Due to the large size of the project from the tensorflow libraries, we were unable to upload it on git.

Share this project:

Updates