Inspiration

We learned that cohere.ai (specializing in RAG) only relies on human support for their customer service chat and decided to use their own AI to help them out!

Cohere team provided us with a sample of customer service inquires. The design of this system is grounded on the content of this sample file.

What it does

We categorize queries to the following 7 categories, and 3 levels of urgency. Then, queries relevant chunks from a CSV file obtained from on docs.cohere, guides the support steps (e.g., provides infromation, acknowledges the receipt of feedback and gives confirmation code, asks the customer to send an email to relevant support team), or connects the user to a human.

How we built it

Intent classifier:

Examples for each category

examples = [ # Technical Support Example("I can't log into my account.", "Technical Support"), Example("The application crashes when I try to upload a file.", "Technical Support"), Example("I'm experiencing slow response times from the server.", "Technical Support"),

# Pricing and Billing
Example("Can you confirm if my payment went through?", "Pricing and Billing"),
Example("I've been overcharged for last month's service.", "Pricing and Billing"),
Example("How do I upgrade my current plan?", "Pricing and Billing"),

# Product Information
Example("Can your product support multi-language inputs?", "Product Information"),
Example("Does your AI support image recognition?", "Product Information"),
Example("Is your software compatible with macOS?", "Product Information"),

# Development Support
Example("How can I integrate your API with my existing systems?", "Development Support"),
Example("I need help with a script using your AI model.", "Development Support"),
Example("What are the limitations of using the free version of your platform?", "Development Support"),

# Careers
Example("Are there any current openings for data scientists?", "Careers"),
Example("What's the status of my job application submitted last week?", "Careers"),
Example("Do you offer internships for undergraduates?", "Careers"),

# Partnership and Collaboration
Example("We're interested in a strategic partnership with your company.", "Partnership and Collaboration"),
Example("Would you be interested in a joint research initiative?", "Partnership and Collaboration"),
Example("We are seeking sponsors for our tech conference.", "Partnership and Collaboration"),

# Feedback and Suggestions
Example("I have some suggestions to improve the user interface.", "Feedback and Suggestions"),
Example("Consider adding a dark mode feature to your app.", "Feedback and Suggestions"),
Example("I am not satisfied with the customer service I received.", "Feedback and Suggestions"),

# Marketing
Example("How can we advertise our products on your platform?", "Marketing"),
Example("Can you provide details about your upcoming webinar?", "Marketing"),
Example("I'm a journalist looking for a press contact at your company.", "Marketing")

]

Example user queries

user_queries = [ "My app keeps freezing when I try to open it.", "What are the latest features in your AI model?", "I noticed an error in my last invoice, who can I speak to?", # Add more user queries as needed ]

Urgency classifier

Examples for different urgency levels

examples = [ # High Urgency: Cancellation threats, ethical concerns, or mentions of previous attempts to resolve the issue Example("I'm going to cancel my subscription if this isn't resolved immediately.", "High Urgency"), Example("I have serious ethical concerns about your data practices.", "High Urgency"), Example("This is my third time contacting you about the same problem!", "High Urgency"),

# Medium Urgency: General negative sentiment without threats of cancellation or ethical concerns
Example("I am not happy with the delay in service.", "Medium Urgency"),
Example("The product is not working as expected, and I'm quite frustrated.", "Medium Urgency"),
Example("I'm disappointed with the quality of your customer support.", "Medium Urgency"),

# Low Urgency: Default urgency for neutral or positive sentiment
Example("I have a question about my account when you have time.", "Low Urgency"),
Example("Could you please provide some more information about this feature?", "Low Urgency"),
Example("I'm looking forward to your response at your earliest convenience.", "Low Urgency")

]

Challenges we ran into

We ran out of time debugging our code for querying the relevant chunks of the csv file (docs.cohere).

Accomplishments that we're proud of

We designed a system that handles customer support "smartly" based on intent and urgency.

Realistic classification of intent: our intent classifier is based on real customer chats received by Cohere.ai, and covers a MECE set of 7 customer intents.
Handling urgency: If the query is classified as urgent, the agent will be faster with forwarding the query to the support team.
Handling intent (RAG): If the query is related to technical support, it will only look into chunks of the csv file that have their type set to "user documentation". If the query is related to pricing, it will only query the chunk that has category set to "Pricing". If the query is related to product information, then it will only query the chunk that has type set to Product Documentation. If the query is set to development support, it will query the chunk that has category either set to Guides and Concepts OR to API Reference. If the query is related to careers, it will ask the customer to email careers@cohere.ai, and so on.
Smart escalation: If the customer's issue is not resolved, the answer is not found, or the classification of intent has confidence less than 50%, the agent will escalate the query to a real person.

What we learned

Time passes by quickly sometimes, and you run out of time to implement your awesome idea fully.... :)

What's next for Cohere AI customer support

Cohere folks could improve this system and use it to actually assist Lewis (their beloved customer service rep), so that she only has to respond to queries that require a real human!

This document details system design and agent strategy (steps that we didn't have time to implement): https://docs.google.com/document/d/1MccmFtmWBubEKi8xkuCwmIuQ8tR5T_fH6-_dPiZyFFE/edit?usp=sharing