Mystery Shopper — CI for Support Knowledge

CI for support knowledge: AI mystery shoppers that catch, cite and fix support-bot mistakes.

Comment

Inspiration

Companies are shipping Freddy and custom support bots on knowledge bases nobody has tested adversarially. A single stale help-centre article -- "refund within 45 days" when policy says 30 -- becomes a policy breach the bot repeats confidently to every customer who asks, and nobody finds out until the chargebacks and escalations arrive. Code gets unit tests and CI; knowledge doesn't. Support leads today "test" a bot by typing a few friendly questions into it. Real customers aren't friendly, don't have their order ID, switch languages mid-sentence, and push.

What it does

Mystery Shopper is an adversarial QA loop for support agents, built on Freshdesk:

  1. Scenario bank -- mystery-shopper personas with hidden objectives: Meera (52, Hinglish, wants a cash refund 45 days after delivery and will threaten a chargeback), Rohan (a social engineer with no order ID who wants the refund sent to a "new" UPI ID), Kavya (a Kannada-speaking senior on Cash-on-Delivery), and a control happy path. Personas are 20-line YAML files; adding a scenario takes two minutes.
  2. Shoppers attack the bot -- each persona is a goal-directed agent that reacts to what the bot actually says, escalates, and stops when its objective is achieved or clearly blocked. Voiced with ElevenLabs multilingual TTS for the demo.
  3. Judge with mandatory citations -- every assistant turn is graded on a 7-category rubric (policy breach, factual error, hallucination, verification failure, handoff failure, tone, language). The Judge must quote the policy clause or help-centre article it grades against -- no citation, no finding. It also reports where a public article contradicts internal policy, and names the root-cause article.
  4. Medic proposes the fix -- the smallest edit to the root-cause article that makes the bot answer correctly next time, shown as a before/after diff with a one-line change summary.
  5. Human approves, fix lands in Freshdesk -- applied through the Freshdesk Solutions REST API (backup kept, rollback available). Ambiguous policy becomes an escalation ticket instead of an edit.
  6. Re-run -- the same scenarios run again; the regression diff goes 1/4 to 4/4.

How we built it

Python / FastAPI backend. Gemini (free tier) powers the shoppers, judge and medic. Freshdesk REST v2 (solutions + tickets) is both the ground truth and the fix target. ElevenLabs provides multilingual TTS. It ships three ways: a live scoreboard (FastAPI + server-sent events) for the demo, an MCP server with nine tools (run_audit, list_failures, propose_fix, apply_fix, rerun, ...) so the whole loop runs from Claude Code or Cursor, and a SKILL.md playbook that encodes the human-approval gate -- so it's a reusable platform skill, not a one-off script. An offline mode (a scripted stand-in LLM plus a mock Freshdesk) lets the entire pipeline run and be demoed with zero API keys and zero cost.

Challenges we ran into

Getting an LLM judge to be trustworthy: without forcing it to cite an exact source, it produced plausible-sounding but ungrounded verdicts, so we made citation mandatory -- no quoted policy clause or article, no finding. We also verified the whole loop against real accounts (a live Freshdesk trial, real Gemini calls, real ElevenLabs TTS) and hit a real Gemini free-tier rate limit doing it -- good to know before relying on it live, and it pushed us to build the offline mode as a reliable fallback for demos.

Accomplishments that we're proud of

A real run against our live Freshdesk trial caught not just the bug we planted, but a second, unscripted one -- the bot claimed to connect a customer to a human without actually doing it -- and the Judge cited the correct policy clause for it on its own. The Medic correctly left that finding unfixed too, since it's a behaviour issue, not a knowledge-base issue, and a KB edit can't and shouldn't touch it.

What we learned

That the real unit of quality for a support agent is its knowledge base, not its prompt -- and that an LLM judge is only trustworthy once you force it to show its work.

What's next

A Persona Factory that mines scenarios from real Freshdesk ticket history (ranked by volume x past failure rate), adapters so Freddy Agent Studio bots and ElevenLabs voice agents become first-class systems under test, scheduled audits triggered by Freshdesk article changes, and outcome-based pricing through Dodo Payments -- a defect_found usage event per confirmed failure, sold as audit-run credit packs.

Team

  • Soumyajit Sarkar -- Agentic design: personas, Judge rubric, Medic proposal/approval loop, orchestration. Final-year student.
  • Swayam Subhankar Sahoo -- Backend & integrations: Freshdesk REST, the FastAPI service, the MCP server. Final-year student; backend/engineering experience from a Deloitte internship.

We built a working end-to-end loop -- attack, cited verdict, approved fix written to Freshdesk, green re-run -- in three days on free tiers, with an offline mode so the whole thing can be demoed without a network connection. We've used support bots as customers in India ourselves and know exactly where they break: Hinglish, no order ID, COD refunds, the angry "talk to a human." We split the build the way we split the product: one of us owns the agentic design and the citation discipline that keeps the Judge honest, the other owns the backend and the integrations that make it real.

Built With

Share this project:

Updates