Visual AI Chatbots for E-commerce stores

Inspiration

I was always fascinated by the capabilities of models in computer vision. In particular I found VQA models like CLIP intriguing, as you could get information from pictures and use it for a multitude of projects.

What it does

Sightbots are a way for online stores to improve the user experience and boost sales by allowing their costumers to find exactly what they are looking. You can simply describe what you want or send a picture of the prodct and you will be shown the closest fit. TikTok-Shop will also be implementing visual search, which further proves that this way of searching is the future.

How we built it

I built this project using flask and deployed it on pythonanywhere. The languages used are python, js, html, css and I used a csv dataset I found online of a Shopify store. There is also a preprocessing stage, where the images of the products are fed to a VQA model namely BLIP2 to gather info about the products like the color, fabric, style, etc. programmatically.

Challenges we ran into

The amount of code to be written turned out to be way more than expected so countless nights went with very little sleep...

Accomplishments that we're proud of

Regardless of the results of the competition I have no doubt that these chatbots can be implemented in many online stores and talks have already began to implement it with a large chain of apparel companies that makes 6.5 million dollars per year. Furthermore, I got approached by a marketing agency with more than 1.7 million followers across all social media channels to advertise the concept.

What we learned

I personally learnt a lot about hard work more than I did about programming related concepts, as I had already worked with flask before. It taught me to stick with the goal and continue pushing until the vision is complete. No quitting.

What's next for Visual AI Chatbots for E-commerce stores

Having launched very recently as a business yet so many online stores wanting the chatbots, it is safe to assume I will continue down the business path and I am also currently raising funds.

Note

The API key is used to run the BLIP2 model at a faster time from Replicate due to a lack of GPU and it is not shown in the project GitHub repo or the pythonanywhere code to avoid overuse.