💡 Inspiration
Customer support is often repetitive. Customers ask about delayed deliveries, refunds, payments, returns, account problems, and other common issues every day. At the same time, support agents have to search through previous conversations to understand how similar problems were handled.
I wanted to explore whether an AI agent could help automate the first layer of customer support without simply generating generic chatbot responses.
The key idea behind this project was:
Instead of asking an LLM to invent a response, can we make it learn from how the company actually responded to similar customers in the past?
This led me to build an evidence-grounded AI customer support agent for AmazonHelp using historical customer-support conversations from the Customer Support on Twitter dataset.
🛠️ What I Built
The system takes a new customer message and performs three main tasks:
Intent Classification — identifies what the customer needs, such as a delivery issue, refund issue, payment issue, product problem, or account issue. Historical Case Retrieval — searches a knowledge base of previous AmazonHelp conversations and retrieves the most relevant cases. Response + Routing — uses the retrieved historical responses to draft a reply and decides whether the request should be auto-handled or escalated to a human.
The final output looks conceptually like:
Customer Message ↓ Intent Classification ↓ Retrieve Similar Historical Cases ↓ Evidence + Customer Message ↓ LLM Response Generation ↓ Auto-handle / Escalate
For retrieval, I reconstructed approximately 83,888 usable AmazonHelp support cases from the original dataset. Each case contains the customer's problem together with the corresponding AmazonHelp response.
I used Sentence Transformers to create embeddings and FAISS for similarity search. The top 5 relevant historical cases are then provided to Gemini as evidence for generating the response.
The project also includes a simple Streamlit interface where a user can enter a customer message and see the predicted intent, drafted response, decision, reasoning, and retrieved historical evidence.
🔧 How I Built It
One of the most important decisions I made was not to treat every tweet as an independent document.
The dataset contains relationships between tweets through response IDs, so I used those relationships to reconstruct customer-support conversations.
Instead of storing:
Tweet 1 Tweet 2 Tweet 3 ...
as unrelated documents, I created support cases containing:
Customer problem: ...
AmazonHelp response: ...
This makes retrieval much more useful because the system can retrieve not only what the customer asked, but also how AmazonHelp historically resolved a similar problem.
I then created a fixed taxonomy of 12 support intents and used Gemini for intent classification and response generation.
For safety, certain situations such as potential fraud or unauthorized activity are explicitly routed toward human investigation instead of being automatically handled.
📚 What I Learned
This project taught me that building an AI application is not just about choosing an LLM.
I learned how important the data pipeline is. Reconstructing conversations from millions of tweets was necessary before retrieval could become useful.
I also learned the difference between RAG and model training. I am not training Gemini on Amazon's data. Instead, historical support cases are retrieved at inference time and supplied as evidence.
Another important lesson was that retrieval quality directly affects generation quality. If the retrieved historical cases are irrelevant, even a powerful LLM can produce an unhelpful response.
I also learned that evaluation is much harder than getting a demo to work. A system can produce convincing responses while still making incorrect classifications or unsafe automation decisions.
Challenges
- Reconstructing conversations
The original dataset contains millions of tweets and does not simply provide ready-made customer-support cases. I had to use the tweet response relationships to reconstruct AmazonHelp conversations.
- Designing the intent taxonomy
The dataset contains many different types of customer requests. I had to group them into a smaller set of practical support intents while trying to keep the categories distinct enough for classification.
- Making responses evidence-grounded
A generic LLM can easily generate plausible-sounding information. I therefore designed the generation step around retrieved historical AmazonHelp responses and instructed the model not to invent policies, refunds, guarantees, or completed actions.
- Deciding when AI should stop
One of the biggest challenges was deciding when the system should not automatically answer.
Some requests, especially security/fraud-related or case-specific issues, should be investigated by a human rather than receiving an automated answer.
- Evaluating the system honestly
My initial evaluation showed an intent classification accuracy of approximately 64% on my evaluation set. This made me realize that a polished demo does not necessarily mean the underlying system is production-ready.
That result became an important part of the project rather than something to hide: it showed me where the system still needs improvement.
What I Would Improve Next
If I continued developing the project, I would focus on:
Creating a larger, independently human-labelled evaluation set Preventing evaluation examples from appearing in the retrieval knowledge base Improving intent classification for overlapping categories Adding stronger confidence thresholds for automated handling Improving escalation detection for ambiguous and high-risk cases Adding a more rigorous LLM-as-a-judge evaluation for response quality Expanding the system to support multi-turn conversations
The goal would not simply be to make the AI answer more questions automatically. The goal would be to make it more reliable about knowing when it can safely answer and when it should involve a human.
Why This Project Matters
This project explores a practical approach to customer-support automation where historical human resolutions become the source of knowledge for the AI agent.
Rather than building another chatbot that generates answers from general language-model knowledge, I wanted to build a system that asks:
“How was a similar customer problem handled before?”
and uses that evidence to make its response.
That combination of intent classification + historical-case retrieval + evidence-grounded generation + escalation decisions is what makes this project more than a simple LLM wrapper.