Inspiration
Customer support looks simple from the outside, but a good support agent has to make several decisions correctly: understand the customer's problem, search internal documentation, follow refund policies, communicate clearly, and know when a situation should be escalated to a human.
We were inspired by the idea of treating customer support as an agent environment rather than just a chatbot task. Instead of measuring whether an AI can generate a convincing response, we wanted to evaluate whether an agent can take the right sequence of actions in a realistic, stateful environment.
This led us to build Customer Support Resolution OpenEnv, a production-oriented reinforcement learning and agent evaluation environment designed around realistic customer support scenarios.
What it does
Customer Support Resolution OpenEnv simulates customer support tickets where an AI agent must resolve issues through structured actions.
The agent can:
- Reply to the customer
- Search an internal knowledge base
- Issue refunds
- Escalate tickets to human support
- Close resolved tickets
The environment maintains a complete state containing the ticket, customer information, conversation history, hidden knowledge-base articles, refund status, and step count.
We created three levels of difficulty:
Easy: Password reset requests requiring the correct response and ticket closure.
Medium: A defective product complaint from a premium customer where the agent must search the knowledge base, verify the refund policy, issue exactly a $150 refund, and close the ticket.
Hard: A frustrated customer who has been double-billed for a subscription. The agent must handle the customer's emotional state, provide an appropriate refund, and determine whether escalation is necessary.
The environment also uses a dense reward system that evaluates both correctness and efficiency. Agents are penalized for unnecessary actions and rewarded for useful knowledge-base searches, correct intermediate actions, and successful task completion.
How we built it
We built the environment around a structured state/action/observation architecture compatible with the OpenEnv approach.
The core environment uses Python and Pydantic to define structured actions and state transitions.
The agent interacts with five primary actions:
Reply(content)
SearchKB(content)
IssueRefund(amount)
Escalate()
CloseTicket()
Each action changes the environment state and generates feedback that becomes part of the next observation.
The observation provided to the agent contains:
- The current support ticket
- Customer context
- Conversation history
- Feedback from the previous action
The environment also contains a deterministic grader that evaluates the final outcome based on factors such as correctness, policy compliance, unnecessary tool usage, and efficiency.
We added Docker support so the environment can be deployed consistently, and designed it to run as a FastAPI-based environment that can be hosted on a Hugging Face Docker Space.
Challenges we ran into
One of the biggest challenges was designing an environment that was simple enough to understand but realistic enough to expose agent failures.
A basic chatbot benchmark can often reward an agent simply for generating a good-looking answer. In customer support, however, the final response is only one part of the problem.
For example, an agent might:
- Issue a refund without checking the policy
- Search the knowledge base repeatedly
- Refund the wrong amount
- Close a ticket prematurely
- Continue interacting after the problem is already resolved
- Fail to escalate a high-risk situation
We therefore had to carefully design both the action space and reward function.
Another challenge was preventing agents from exploiting the reward system. We introduced penalties for redundant or invalid actions and capped the reward from repeated knowledge-base searches to discourage tool-use loops.
Balancing correctness, efficiency, and safety was one of the most interesting parts of building the environment.
Accomplishments that we're proud of
We are proud that Customer Support Resolution OpenEnv goes beyond a traditional chatbot demo and provides a stateful environment for evaluating agent behavior.
Some highlights include:
- Structured agent action space
- Stateful multi-step interactions
- Hidden knowledge-base environment
- Policy-aware refund actions
- Human escalation mechanism
- Dense reward shaping
- Deterministic task grading
- Docker deployment
- Hugging Face Spaces compatibility
- Multiple difficulty levels for agent evaluation
The environment can be used not only to demonstrate an AI agent, but also to investigate where and why an agent fails.
What we learned
This project taught us that building an agent is very different from building a chatbot.
The hardest part isn't simply connecting an LLM to a set of tools. The real challenge is designing the environment around the agent so that its decisions can be measured objectively.
We learned about:
- Reinforcement learning environment design
- State and action modeling
- Tool-using agents
- Reward shaping
- Deterministic evaluation
- Agent failure modes
- Knowledge-base tool use
- Policy-based decision making
- Containerized AI environments
- OpenEnv-compatible architecture
Most importantly, we learned that good agent evaluation needs to measure the entire decision process, not just the final text response.
What's next for Customer Support OpenEnv
We want to expand the environment into a larger benchmark for customer-support agents.
Future improvements could include:
- More complex multi-turn support scenarios
- Larger and more realistic knowledge bases
- Multi-agent customer support workflows
- More sophisticated customer personas
- Fraud and abuse detection scenarios
- Subscription cancellation and retention tasks
- Multi-step order management
- Dynamic policies and changing business rules
- Better agent trajectory visualization
- Leaderboards for comparing different LLM agents
- Integration with additional open-source and frontier models
Our long-term goal is to make Customer Support Resolution OpenEnv a useful benchmark for measuring how reliably AI agents can operate in realistic customer-support environments, rather than simply measuring how well they can generate text.
Log in or sign up for Devpost to join the conversation.