The inspiration behind Looper comes behind trying to make the film watching experience more interactive and a keen interest in the rising accuracy and power of machine learning object detection algorithms.
What it does
Looper captures frames from a running film or video on your computer and processes the image to detect objects within the scene. After identifying the objects and bounding them with boxes, Looper uses reverse image lookup and facial recognition to identify actors. Then, the image and web links to actor info are sent to the iPhone, where the user can click on the actors and get information. The iPhone can also be used as a controller to pause or play playback.
How we built it
We wrote an iOS application that uses PubNub to communicate in real-time with a node.js server. Using a modified machine learning library and image processing scripts, the server processes the movie scene and sends back information such as box coordinates and web links to be displayed and used on the iOS application.
Challenges we ran into
The biggest challenge we ran into was processing the image to crop the actors and conducting a reverse image search to determine the actor's name and IMBD link.
Accomplishments that we're proud of
We're most proud of creating a seamless experience between the phone and the computer to capture the scenes and display the information on a phone without disrupting the movie watching experience. The process may seem slow at first, but can be sped up exponentially if a machine with a GPU were used.
What's next for Looper
Looper is already able to detect the object type such as cars and shoes, but it cannot identify the specific types of cards and computer. We used a facial recognition model to identify actors, and with time and energy, we can build similarly accurate models for cars and shoes. Once we have these, we can use a pipeline of models to identify the specific cars and computer in scenes and provide information to the customer. Of course, this opens up a whole new realm of advertising for the industry, as companies can curate special interactive content for audiences.