Inspiration
Hello, my name is Faruk Alam and I am a final year bachelor student from Bangladesh. So, I was exploring the vision language models basically the open source model like LLaVA: Large Language and Vision Assistant, Llama-3.2-11B-Vision, Qwen2-VL, CogVLM etc.
So I was thinking about how I can apply this kind of vision language model in different industries. After some kind of research, I got the idea that in the e-commerce field, I use this kind of model to improve the search quality of an e-commerce website.
In general, when we visit any e-commerce website like Amazon, eBay we can't search for products by uploading images. Let's say when I visit any social media like my Facebook or Instagram I have seen a shirt or dress that I liked. Now I want to search for a similar product on the e-commerce website. But manually checking any product would take more time. Instead, we can use any vision chatbot that will understand exactly which type of product I am looking for. So that is the simple solution we will simply upload a photo to the vision chatbot and it will find a similar product for me. So that's the project I have built
Description
This project enables product search on an e-commerce website using both text and image inputs. By leveraging LLaVA to generate image descriptions and OpenAI embeddings to encode these descriptions, the system stores product embeddings in a CSV file. Users can search for products by uploading images, which are processed into descriptions and matched against the stored embeddings. The interface is built using Gradio, allowing users to easily interact through text or image-based queries.
Technology Stack
Gradio: Frontend for creating a simple user interface LLaVA (Language and Vision Assistant): To generate descriptions from uploaded images OpenAI Embeddings: For querying the best product matches
Log in or sign up for Devpost to join the conversation.