NARC - Next Action Recommendation for Customer service

Inspiration

When evaluating our companies' internal challenges, we found that a significant issue lies in the efficiency of our sales and customer service teams. These teams receive a constant influx of requests from various channels—such as email, phone, and internal communications—making it difficult to track and respond to every inquiry, which often leads to some requests being missed and, consequently, a less-than-ideal customer service experience.

What it does

Our goal is to enhance—and ultimately automate—the customer service next-action workflow. When an event occurs, our system suggests the appropriate next steps for the customer service representative to take. If the system's recommendation misses the mark, a human can provide feedback. The system then analyzes this feedback to identify and learn any business rules it may have missed or misunderstood. Once it confirms that incorporating these new rules improves its performance, the system will apply them to future cases. This solution could then be applied across a variety of industries and workflows, provided the tools to take proper actions (CRM APIs, etc) exist

How we built it

Our agent is built on Pydantic AI, which significantly simplified the process of creating a custom solution with minimal boilerplate. Currently, the system operates as a Databricks workflow with events simulated as records in Unity.Catalog, though our goal is to enable real-time event processing in the future. At present, the next action recommendation is executed through the workflow, while business rule tuning is managed via on-demand notebooks.

Challenges we ran into

Getting / generating realistic data to test and evaluate our system's performance with. Right now, we are generating the data using ChatGPT to simulate a "realistic" customer service environment without using proprietary data. Also its often difficult to nail down exactly what the structure of a complex object will be coming from the agent.

Accomplishments that we're proud of

We were able to rapidly spin up an agent and quickly get a simple MVP working in a workflow. Code first agentic framework seems easy to integrate into "real" tool APIs if to take this to pilot.

What we learned

Simple frameworks like Pydantic AI provide the flexibility we need. The git integration with Databricks is very convenient when collaborating (creating branches and commiting without the need for git commands).

What's next for NARC - Next Action Recommendation for Customer service

There are plenty of exciting extensions we could make to NARC:

Evaluate on more complex/realistic data
Occasionally compress business rules / remove duplicate rules
Flag conflicting business rules
Using RAG to retrieve the most relevant rules
Once enough data is gathered, during the rule learning phase, weights could adjusted for the LLM model using RLHF. This seems like it would actually be a fairly simple lift and could be done with an additional day of time.