Magnif.io

Generating images based on user prompt and feedback
Shop feature in Magnifio using generated image

Inspiration

An image is worth a thousand words. We’ve all experienced the frustration of seeing what we want to buy on social media or external sources, but not being able to find it despite knowing exactly what it looks like. We are aiming to revolutionize search in e-commerce through image-based, customizable search for clothing e-commerce that is personalized specifically to your needs and preferences.

What it does

Magnif.io enhances your shopping experience by allowing you to describe exactly what you want to buy, visually. Here how it works: The user provides a prompt for what they’re looking for (e.g. black pants). Our platform then provides an initial image of the prompt to the user. The user can refine the image by providing feedback to the system, adjusting the image as necessary (e.g. “make the black pants in my size, and athletic”). Once the user is satisfied with how the image looks, they can “shop” the image on our platform - our platform will find purchases on the web that suit the exact specifications crafted in the image (matching size, style, color, etc.), along with any additional natural language filters such as rating, delivery date, and price range. Beyond its enhanced searching capabilities, Magnif.io integrates personal context and history to further optimize the user experience, such as your purchase history, existing discounts available, and more.

How we built it

Our system comprises of 4 crucial components: i.) Image similarity search ii.) Data Aggregation iii.) Image Generation iv.) Product Filtering We use puppeteer and selenium to aggregate the data of various ecommerce sites and generate image embeddings using CLIP. We store image embeddings and their respective metadata in Pinecone. We use the metadata of similar images to create a Langchain and OpenAI based product filtering agent. We use Stable Diffusion for Generating images and updating images for desired changes. For frontend, we are using Next.js + Tailwind, and for backend/storage we are using AWS, with S3 bucket for file storage, with hosting on EC2 instance.

Challenges we ran into

Editing the images based on user requests was especially challenging. Identifying the proper masking and defining the hyperparameters that optimized the performance of the Stability Diffusion model, as well as inferencing the filters and comparing them with the data we collected on the products, was also a hard step, and maintaining accuracy given our limited data was a key point of struggle. Beyond that, the integration of all of the separate technologies, and making sure they all worked in harmony took lots of iteration and testing.

Accomplishments that we're proud of

Integrating all of these distinct technologies and crafting them into an outstanding, streamlined end product was an arduous challenge. We had to pull together many separate moving parts, with a large number of distinct technologies. Ultimately, our final product was capable of covering an extremely broad range of functionality and utility, outperforming existing solutions in over 70 percent of the cases we tested.

What we learned

As we iterated on our idea, one of the main challenges we had was how to differentiate from existing solutions, such as OpenAI plugins, Google Lens, etc. Although we had so many tools at hand, we were struggling to provide a unique technology worthy of building. Ultimately, we discovered that, by combining technologies, and providing useful functionalities through these synergies, we could create novel solutions that solved real problems.

What's next for Magnif.io

We believe there is much room to further enhance this product and make a fully robust AI Smart shopping assistant in the future iterations. We believe the next step for Magnif.io is to truly elevate the personalization aspect, allowing Magnif.io to account for contextual information such as personal information, budget, and purchase history. These personalization features could easily be applied and implemented, given that we have already collected the relevant data for products in our database, and would enhance the experience to truly make the process of online shopping streamlined and simplified for consumers.