Our project was inspired by the Microsoft's Project Oxford Emotions API. We were very excited about using this technology, and after discussing some ideas with the Microsoft team we began implementing it to collect data on popular movie trailers.
What it does
Comma is a web application that can determine the emotions shown in a video, such as a trailer for a movie, allowing for movies to be ranked based on their emotions. This allows for further analysis, where videos can be ranked and categorized based on the emotions expressed in the trailer.
How we built it
We used OpenCV to process the videos into frames. We then dropped half of the frames to reduce the calls needed. To further reduce the amount of calls needed we used OpenCV to stitch frames into a 2 x 5 array to batch them into one request. We then submitted this batched request to Microsoft's Project Oxford Emotion's API which gave us the confidence index of each face. We then took all the faces that it found and averaged these values together to get the average emotions in the video.
Challenges we ran into
Project Oxford only allows 10,000 requests per month, with a limit of 20 requests per minute, which would normally be a problem if we made a single request for each frame in the trailer. With each trailer consisting of upwards of 4,000 frames, this would quickly run through our API limits and prevent other movies from being analyzed. In order to get around this issue, we processed a limited selection of frames and made batch requests, allowing us to reduce the number of requests made by 40x.
The API returns a neutral rating a vast majority of the time. We believe that this is partially due to the api having a preference for rating neutral as well as the fact that most of the time we have more neutral faces rather than exagerated emotions.
Accomplishments that we are proud of
In a short amount of time our team was able to utilize an new API to get biometric data, create a database to store the data, and design a webpage to demonstrate the technology. Everyone from our team is leaving having learned something new.
What we learned
Our team varied in software and Hackathon experience. Some of the technologies that were new to the team included NumPy, OpenCV, Git, Flask, and SQLite. No one on the team had prior experience using the Emotions API and we enjoyed using this unique technology to create tangible data.