AI Shopping Assistant

This is an AI Assistant that will help consumers healthy diet, find related products, get educated about them and order them right from the chat itself.

Comment

Inspiration

We are a very small early stage startup having an app for Farm to Fork model. I want to connect retail consumers to farmers. We have a long way to go. Right now we are supporting Organic and Hydroponic veggies and fruits. Our experience is people are not yet enough aware the benefits of chemical free food and a very positive impact of the cultivation method on the environment. We want people to ask diverse questions as much as possible that will address their queries. Traditional FAQs and some structured information is not enough. So we decided to use AI foundational models and combine them with the products and orders data in the app. The feature should also be useful in generating more sales.

What it does

This functionality allows users to ask questions about recipes, health benefits, product availability, agriculture and general information about our company, processes etc. The feature identifies the products available related to the query and GenAI response. In the chat window itself, it displays list of products, that user can instantly add in the shopping cart. For example user queries - Yesterday I had spicy and oily food. Can you suggest a good detox drink? Most likely Agent will provide the recipe of related detox drink and app will show the list of products like Beet, Carrot, Lemon etc etc.

How we built it

I built set of lambda those are integrated with Amazon Bedrock. I used S3 based RAG dump. I used AWS APIGateway to build APIs for websocket based chat. API authentication with AWS Cognito with Lambda authorizer. Used DynamoDB for logging the chat logs. Front we built in Flutter. Development and deployment tooling AWS Amplify, Copilot VS Code plugin.

Challenges we ran into

Initially, we started with OpenSearch based vector db via Bedrock knowledgebase. But we realized that cost of serverless vector db is not justifiable for the amount of data we want to vectorize. Right now we have about 1000 products only. Our products, availability and processes may be specific to city or city area. Our current app uses the city or zip specific catalogues. So eventually the solution should be able to handle 100s of small vector dumps. S3 based dumps will be simpler to manage and may not need a complex application logic, process and access control etc to manage all vectors in one database. So then we wrote all logic in lambda to pickup raw data file from S3, use Titan embeddings to create vector dump of it and store again on S3. The chat lambda, load the vector dump in real time and use the private methods to perform cosine similarity search. This pattern saved some monthly bill till the time we scale.

Another challenge we had about allowing users to ask / interact about only relevant topic. So we built a pattern to first identify the user intent, classify the query and then deal with that. For example, if user asks do you sell cauli flower? This query will be classified as a product query. So we limited the use of this feature for 6 to 8 categories only.

Accomplishments that we're proud of

We are able to recommend and show the related products based on user interactions. This is a unique feature and a small step towards an AI Agent that will showcase the products, its benefits, other information, take the orders and payments and will act as a relationship manager for the customer. In the target state, customers can just talk to the virtual agent like a vide conversation as they today speak with shopkeeper. We are proud that we are able to make use of serverless stack with AI to help customers in minimal cost.

What we learned

This is my first GenAI project. I learned Bedrock, embedding models, tried various other foundational models including Anthropic. Understood the pros and cons of using them. Finally used Nova Micro model which is working out enough for now. Lots of other learnings. Initially thought was to use vector database. Once we knew its cost benefit in short run, lambda and S3 based vector db was a huge win. It enables small startups to keep costs under control.

What's next for Shopping Assistant

A long way to go towards a full fledged AI Agent (video chat) who just works as a friendly sales person who speaks in local language with the customer. In short term, we want to fine tune the responses or even select the more useful foundational model, formatting of the output etc. In mid term extend the functionality to ask the Assistant to add the products in the cart, submit order, verbally tell the delivery address etc.

Built With

Share this project:

Updates

Submission history