So much information on the internet such as news articles or YouTube videos are gated by domain-specific knowledge which can make it difficult to follow. While users can always Google search for well-known figures, it can be exhausting to constantly be searching for reliable, unbiased information. This is especially true for users trying to get into politics, sports, or other communities with a high barrier of domain-knowledge.

We wanted to use computer vision to save our users Google searches by showing relevant information regarding political figures or celebrities as users browse the internet. Our extension scans the page, identifies well-known figures, and provides relevant information in real-time.

What it does

PlainSight is a Chrome extension that scans images on the current page, identifies who is in the picture, and summarizes them in real-time. This all happens at the click of a button.

How we built it

PlainSight is implemented as a Google Chrome extension that captures an image of the current page and sends this to a Google Cloud Function. This function uses OpenCV to identify the faces in the image and their relative location. Then it sends these individual faces to a Google Reverse Image Search API which labels these faces with the corresponding name. The Google Cloud Function takes these names and queries the Wikipedia API to get summarized information regarding the well-known figure. This is then presented to the user through the Chrome Extension.

Challenges we ran into

We had problems with speed. Initially we were doing the computer vision analysis through a JavaScript library on the clientside which is running TensorFlow. We ended up moving this to our backend Google Cloud Function which runs the computer vision algorithm in Python.

We also had problems selecting the API. We tried to use the Clarifai API since it has an ML model for celebrity detection. This only worked for celebrities as opposed to politicians, so we decided to use the Google Reverse Image Search API instead.

What we learned

OpenCV, computer vision, developing Chrome Extensions, working with Google Cloud.

What's next for PlainSight

We submitted the extension for approval on the Chrome Extension store. We would also like to replace the Google Reverse Search API with Amazon Rekognition since Amazon does a better job at giving specific name classifications as opposed to classifications such as "man" or "senator."

Built With

Share this project: