There has been an explosion of visual data in the past decade. Presently, is difficult to keep track of the many photos we take and share across various platforms. Consider Google Photos. Finding a particular photo is often challenging and sometimes unfeasible.

Wouldn't it be nice if you could just describe the image you are looking for, and an AI agent would find it from your photos? Rapid advancements in deep learning have the potential to revolutionize how we search and organize our data. With FilterAI, we do just that – use deep learning for simple, fast, efficient content-based image retrieval.

What it does

When a user is looking for a particular photo, he types in a simple description into the search bar. We use state-of-the-art deep learning models to filter images that whose descriptions match his query and display them in a clean, minimalistic UI.

How I built it

We used NeuralTalk2 a state-of–the–art image captioning model to generate captions for photos. We built our webapp in Django, using Materialize for the front end and a metric based on word2vec, to compute description similarity.

Challenges I ran into

We had to finetune our caption to query similarity metric, and ran into some issues while setting up Torch to generate captions.

Accomplishments that I'm proud of

Today, image search is largely based on manually collected image tags and metadata, which is difficult to scale. The spate of advancements in computer vision made possible by deep learning presents an opportunity to do better and efficient content-based image retrieval. We're mostly happy to have built a novel application from research code.

Built With

Share this project: