Inspiration
Support teams solve the same operational problems repeatedly, but the knowledge contained in those resolved tickets often remains trapped in conversations.
We wanted to explore a different approach: instead of using historical tickets only to retrieve information, could they be used to create reusable skills for AI agents?
The key insight was that successful and failed tickets provide different kinds of knowledge. Successful tickets show the procedure. Failed tickets show where that procedure should stop.
This became the foundation of SkillMint: turning support history into agent skills with both procedures and safety boundaries.
What it does
SkillMint is an assay pipeline that converts support-ticket history into reusable, machine-readable skills for AI agents.
It:
- Discovers recurring request types from a ticket backlog instead of relying on predefined categories.
- Separates well-handled and poorly handled tickets within each request type.
- Extracts procedures from successful resolutions.
- Mines guardrails and stop rules from failed, reopened, or escalated cases.
- Generates a structured agent skill containing procedures, boundaries, and required tools.
- Uses an adversarial critic to identify weaknesses and unsafe assumptions.
- Evaluates the finished skill on held-out tickets that were not used during skill generation.
Our bundled demonstration contains 240 synthetic service-desk tickets across 8 recurring request types. The current New Employee Onboarding skill is evaluated on 8 held-out tickets, achieving 3/8 passes with 0 wrongly auto-resolved cases. These results are directional because the current corpus is synthetic.
How we built it
We built SkillMint as a seven-stage pipeline:
Sort the Intake → Assay Each Group → Read the Procedure → Mine the Guardrails → Strike the Skill → Send It to the Critic → Trial of the Pyx
The discovery stage uses TF-IDF and dimensionality reduction to identify recurring request patterns in the ticket backlog.
The best-handled tickets are analyzed to recover common procedures, while poorly handled tickets are analyzed to identify conditions where autonomous action should stop.
These outputs are combined into a machine-readable skill specification with declared tools.
The generated skill is then reviewed by an adversarial critic before being evaluated against a separate set of held-out tickets.
The project currently supports a bundled synthetic corpus for reproducible experimentation and can also work with Freshservice conversations when the required API configuration is provided.
The complete pipeline supports Anthropic, Groq, and OpenRouter as model providers.
Challenges we ran into
Learning from imperfect historical data
Support-ticket history contains both good and bad practices. Simply feeding every conversation into a model could cause the resulting skill to reproduce inefficient or unsafe behavior.
We therefore needed to distinguish procedure evidence from failure evidence.
Turning failures into guardrails
A failed ticket rarely states exactly what rule an AI agent should follow. We had to infer the condition that made the normal workflow unsafe and convert that evidence into an explicit stop rule.
Preventing evaluation leakage
Testing a generated skill on the same tickets used to create it would make the results difficult to trust.
We therefore separated synthesis data from evaluation data and held tickets back until the final evaluation stage.
Defining what "good automation" means
We realized that maximizing automation is not always the right objective for an operational agent.
Incorrectly completing an action that requires human intervention can be much worse than escalating unnecessarily. This led us to track wrongly auto-resolved and over-cautious cases separately.
Accomplishments that we're proud of
We are proud that SkillMint goes beyond simply summarizing support conversations.
The system can take an unstructured ticket backlog and produce a structured agent skill containing:
- A discovered request type
- A reusable procedure
- Evidence-backed stop rules
- Required tools
- Adversarial critic findings
- Held-out evaluation results
In the current demonstration, SkillMint discovers 8 recurring request types from 240 tickets, recovers 8 procedure traces and 8 stop rules, and identifies 6 critic findings before shipping.
We are especially proud of making evaluation part of the skill-generation pipeline rather than treating it as an optional final step.
A generated skill has to face tickets it has never seen before.
What we learned
Failure data is not noise
Our biggest lesson was that unsuccessful interactions can be some of the most valuable evidence in an operational dataset.
Successful tickets explain the normal workflow. Failures reveal its boundaries.
Knowledge and skills are different
Giving an agent access to company information does not automatically give it the ability to act reliably.
A useful agent skill needs procedures, conditions, tools, verification, and explicit boundaries around autonomous action.
Independent evaluation matters
A skill should not be considered reliable simply because it performs well on the examples that generated it.
Holding data back provides a more meaningful test of whether the skill generalizes.
Safe agents need to know when not to act
We learned that over-escalation and unsafe automation are not equivalent failures.
For many operational tasks, knowing when to stop and ask a human is an essential part of the skill itself.
What's next for SkillMint
The current implementation uses a controlled synthetic corpus so that the pipeline can be reproduced and evaluated consistently.
Our next step is to evaluate SkillMint on larger and more diverse real-world support datasets.
We also want to expand the skill registry and make the generated skills easier to consume across different agent runtimes and company systems.
Longer term, we want to extend the same approach beyond support tickets to other sources of organizational knowledge.
The core principle will remain the same:
Don't just give an agent more knowledge. Turn that knowledge into skills with evidence, boundaries, and tests.
Built With
- anthropic
- freshservice
- groq
- javascript
- mcp
- node.js
- openrouter
- pca
- python
- react
- tf-idf
- vercel
Log in or sign up for Devpost to join the conversation.