Conjure

A filter on mountain ranges, and questions about the weather in one of the images.
One of the clusters from k-means clustering on our image description embeddings.

Inspiration

This summer, we realized that the US military requires AR/VR solutions for visualizing real-time and synthetic data, targets, and mission objectives along with data visualization tools for effective decision making and actionable insight from large data sets. Therefore, we decided to embark on our journey to building this tool by using HackMIT to jumpstart our progress. For the sake of the 24 hour time period, we chose to focus solely on being able to get actionable insights from large image datasets. However, as we were building, we soon realized that the tool we were building would be perfect for navigating our large photo libraries. Oftentimes, we’ll be hit with a strong pang of nostalgia remembering “that one night in San Diego where I captured the fireworks going off on the side of the highway” or simply want to pull up “all instances of my friend Vanessa wearing a full length red dress.” And thus came to life Conjure, a dual use tool that both allows you to retrieve vital insights from your data and also pull up those snapshots of the past on demand.

What it does

Conjure allows users to upload a dataset of images, or their entire camera roll, that we can then filter through using natural-language prompts, such as, "locate images of nature" or “pull up my pictures from pride parade 2023.” Users can also opt for a categorization of their data and automatic classification into automatic labels. Functionality to ask questions about a specific image, like, "What is the boy in this image wearing?" is also built in.

How we built it

To build our project, we had to start off by labeling our data. To do so, we chose to use a combination of a VLM (Vision Language Model) to generate text descriptions of each image in our dataset and Optical Character Recognition (OCR) to retrieve any image in the text. After that, to retrieve insights from our data, we used an embedding model from Hugging Face to represent the text description of each image as a vector. Using cosine similarity and K means clustering we were able to categorize and recognize patterns in the dataset. We were able to then implement the ability to interact with the data through both chatbox queries and ASR (automatic speech recognition). Chatbox functionality was built through the use of an LLM that ran on the text descriptions generated by the initial LLM. Lastly, frontend was done on flask and node.js to tie everything together seamlessly.

Challenges we ran into

The biggest impediment of progress was the difficulty in deploying our code on large scale computing power. We knew that we wouldn’t be able to run these models on our local machines so we opted to use the Modal credits we received at HackMIT. However, as much as we have grown to appreciate its power now, the learning curve was very steep at first and we found ourselves struggling to get our code base running on cloud computing.

Accomplishments that we're proud of

Being able to fork the git repo for Llava, a VLM that was used in our pipeline, and rearranging the code to tailor to our use case. This required a lot of careful understanding of what each part of the code was doing and a high level of intuition in a short period of time.
More generally, we had such a large list of features that we wanted to have implemented and simply the fact that we were able to get the associated models, backend, and frontend for each of them working in 24 hours really showed us just how much we are capable of.
Although we had some intuition that we would be able to categorize images to some degree, we were in awe by the level of accuracy and sheer beauty of the output of our clustering algorithm.
Being able to optimize the LLM to run on such a huge dataset and honestly just get to work took us by surprise. The accuracy of the returned output to our queries was better than we could have anticipated and having such a strong LLM backbone allowed us to expand and create more features for the user.

What we learned

For most of our team, this was our first time building a full stack project so being able to see the life cycle of a project from ideation to backend development and to finally seeing everything work together seamlessly on a front end was incredibly rewarding.

Furthermore, given the time frame, we learned to perform a cost benefit analysis whenever we bumped into a deadend and effectively determine whether it was time to let go and find an alternative or stick with it and troubleshoot. We also learned that splitting up and trying multiple things in parallel instead of waiting for something to not work before trying an alternative was optimal. This was especially useful because many of the features in our pipeline were dependent on stages earlier in the process, so being able to get as much as the linear process working was huge to the progress and breadth of work we could accomplish.

What's next for Conjure

A couple of smaller picture features that we would like to add on are adding on face recognition capabilities using OpenCV so we have better image captioning and a wider breadth of possible queries. We want to be able to answer specific queries such as “pull up images of Evan and I at the beach last summer.” Furthermore, we’d also like to implement labeling for the categories that our k-means algorithm divided the dataset into and look into the capabilities of generating new images using stable diffusion based on what is already in the category.

In regards to the bigger picture, we now have a tool that is able to derive actionable insights from large image datasets so we hope to go back to our original problem statement and fully build out an AR/VR platform for visualizing both synthetic and real time data. In recent years, the military has began to rely more and more on drone data/imagery for mission planning and training purposes, so we hope to build them a comprehensive solution for all their data needs.