Ever wonder where that video clip came from? Probably some show or movie you've never watched. Well with RU Recognized, you can do a reverse video search to find out what show or movie it's from.


We live in a world rife with movie and tv show references, and not being able to identify these references is a sign of ignorance in our society. More importantly, the feeling of not being able to remember what movie or show that one really funny clip was from can get really frustrating. We wanted to enale every single human on this planet to be able to seek out and enjoy video based content easily but also efficiently. So, we decided to make Shazam, but for video clips!

What it does

RU Recognized takes a user submitted video and uses state of the art algorithms to find the best match for that clip. Once a likely movie or tv show is found, the user is notified and can happily consume the much desired content!

How we built it

We took on a 3 pronged approach to tackle this herculean task:

  1. Using AWS Rekognition's celebrity detection capabilities, potential celebs are spotted in the user submitted video. These identifications have a harsh confidence value cut off to ensure only the best matching algorithm.

  2. We scrape the video using AWS' Optical Character Recognition (OCR) capabilities to find any identifying text that could help in identification.

  3. Google Cloud's Speech to Text API allows us to extract the audio into readable plaintext. This info is threaded through Google Cloud Custom Search to find a large unstructured datadump.

To parse and exract useful information from this amourphous data, we also maintained a self-curated, specialized, custom made dataset made from various data banks, including Kaggle's actor info, as well as IMDB's incredibly expansive database.

Furthermore, due to the uncertain nature of the recognition API's, we used clever tricks such as cross referencing celebrities seen together, and only detecting those that had IMDB links.

Correlating the information extracted from the video with the known variables stored in our database, we are able to make an educated guess at origins of the submitted clip.

Challenges we ran into

Challenges are an obstacle that our team is used to, and they only serve to make us stronger. That being said, some of the (very frustrating) challenges we ran into while trying to make RU Recognized a good product were:

  1. As with a lot of new AI/ML algorithms on the cloud, we struggles alot with getting our accuracy rates up for identified celebrity faces. Since AWS Rekognition is trained on images of celebrities from everyday life, being able to identify a heavily costumed/made-up actor is a massive challenge.

  2. Cross-connecting across various cloud platforms such as AWS and GCP lead to some really specific and hard to debug authorization problems.

  3. We faced a lot of obscure problems when trying to use AWS to automatically detect the celebrities in the video, without manually breaking it up into frames. This proved to be an obstacle we weren't able to surmount, and we decided to sample the frames at a constant rate and detect people frame by frame.

  4. Dataset cleaning took hours upon hours of work and dedicated picking apart. IMDB datasets were too large to parse completely and ended up costing us hours of our time, so we decided to make our own datasets from this and other datasets.

Accomplishments that we're proud of

Getting the frame by frame analysis to (somewhat) accurately churn out celebrities and being able to connect a ton of clever identification mechanisms was a very rewarding experience. We were effectively able to create an algorithm that uses 3 to 4 different approaches to, in a way, 'peer review' each option, and eliminate incorrect ones.

What I learned

  • Data cleaning is ver very very cumbersome and time intensive
  • Not all AI/ML algorithms are magically accurate

What's next for RU Recognized

Hopefully integrate all this work into an app, that is user friendly and way more accurate, with the entire IMDB database to reference.

Share this project: