Susan - Call Issue Triage

Issue triage and bucketing for AI voice agents.

Comment

Test number: 1-559-550-4867

Inspiration

Flossy is building Fiona, an AI voice receptionist for dental offices. Fiona is deployed in dental offices across the United States and fields hundreds of inbound calls every day for offices with various scheduling rules, configurations, and behavioral expectations. Listening to every one of these calls was humanly impossible. Many calls are longer than 5 minutes and conversational issues can be subtle. Flossy needed a way to evaluate all incoming calls using a standard rubric so that they could systematically prioritize bug fixes and new product features.

What it does

Susan is an AI agent that evaluates every new call using a standard scoring system aware of individual office configurations, calendar availability, and open tickets.

Susan does all of the following:

  • Assigns each call a quality score
  • Identifies conversational and functional problems
  • Compares realtime transcripts against "true" transcripts
  • Buckets calls with similar issues
  • Performs nightly bucket merges to keep the problem list manageable
  • Reports everything in a Slack channel where the team can investigate further
  • Answers questions about calls and call issues
  • Generates a daily summary with the previous day's average score and bucket statistics
  • Logs call issue buckets to a Slack List
  • Presents daily and weekly call quality overviews with resolution and severity data

Susan's primary work space is the call-analysis Slack channel where she posts call analyses, new issue buckets, recurring issue notifications, and resolved status updates. Susan is also a Slack App/Agent that the team can ask questions to. She has access to the Slack MCP server as well as a custom MCP server for searching our database for calls, locations, and issues.

How we built it

Susan runs as a set of Firebase Functions sitting between our call pipeline and Slack:

  • Ingestion. When a call finishes and the analyzer writes its verdict, an ingestCall function claims it (atomically, so retries are idempotent), derives issue fields from the verdict, and canonicalizes the problem description into high-level tags.
  • Semantic bucketing. Each issue is embedded with Gemini (gemini-embedding-001, 768-dim vectors stored in Firestore). New issues run a vector search against existing buckets, and an AI judge makes the final call: new bucket, append to an existing one, or reopen a resolved one. Reopens keep the full history, a recurring problem shows up as one issue with a paper trail, not five duplicates.
  • Slack as the front end. Susan is a Bolt app in HTTP mode. Every bucket is a Slack thread; appends land in the thread instead of spamming the channel. Buckets also mirror into a Slack List (with Status/Severity select columns) and into an App Home dashboard built entirely from Block Kit; monospace bar meters (█▓░) for open counts, severity, and 7-day call volume, since code blocks are the only place Slack guarantees alignment. Everything on the Home tab renders within Slack's 3-second window, so permalinks are constructed rather than fetched and Firestore reads are field-masked.
  • MCP on both sides. Susan is an MCP server exposing 12 tools (query issues, resolve, digest stats, etc.) that Agent Builder and other clients can call and simultaneously an MCP client of Slack's hosted MCP server, using native Slack search to ground her answers. She sits on both ends of the protocol at once.
  • Nightly jobs. A scheduled digest posts the previous day's average score and bucket stats, and the mergeSweep keeps the bucket list manageable.

Challenges we ran into

One challenge we ran into early was that Susan was creating too many issue buckets. Slight variance in the issues, mostly caused by different conversations leading up to the issue, would lead Susan to create entirely new buckets for each call. This was no better than our original spreadsheet solution since we had to evaluate each issue independently. To fix this, we added additional summary steps before embedding call issues in the "ingestCall" Firebase function. Rather than pipe in every issue description in it's entirety, we forced our analyzer agent to add higher level tags to call issues and then used those as the starting point for the embedding step. This change improved the situation but we still had more buckets than we hoped for. Our next solution was to run a nightly "mergeSweep" function to iteratively look at new buckets created that day and see if they relate to existing buckets. If yes, the issues would be merged.

As more of the team began to use Susan, we discovered additional gaps in her capabilities. One example was that users would ask how a specific location was performing over a time period. In the beginning, Susan didn't have a way to filter calls by location (or look up locations at all). This lead us to add a new location search tool to the Susan MCP server so that Susan could answer questions more systematically. When a user asks a question about a specific location, typically by using that location's internal name (ex. "Smile Suite"), Susan can perform a fuzzy location lookup, then query that location's calls, and then summarize the latest issues.

Accomplishments that we're proud of

  • The dedup pipeline actually converges. Embed → vector search → AI judge → new/append/reopen, plus the nightly merge sweep, turned an unmanageable stream of one-off complaints into a short ranked list a human can act on.
  • The reopen lifecycle. When a "fixed" problem comes back, Susan reopens the original issue instead of filing a new one. The team sees that it recurred, which turned out to be the single most useful signal for prioritization.
  • A real dashboard inside Slack's constraints. No CSS, no fonts, no charts. The App Home "Triage Command" board (at-a-glance stats, severity meter, 7-day flagged-vs-clean volume, Growing This Week velocity ranking) is pure Block Kit and updates live when you resolve an issue.
  • Being an MCP server and client simultaneously. Susan both exposes her triage data as tools and consumes Slack's hosted MCP for grounded answers.
  • Susan replaced our spreadsheet. Nobody has opened it since.

What we learned

  • Embeddings alone don't dedup real-world issues. The same bug wrapped in different conversations embeds differently. The fix was layered: summarize to high-level tags before embedding, judge candidates with a reasoning model instead of a raw similarity threshold, and sweep nightly for what still slips through.
  • Slack's 3-second rule shapes your architecture. With processBeforeResponse, every handler has to finish before Slack gives up so we learned to precompute, construct permalinks instead of calling chat.getPermalink per issue, and mask Firestore reads down to the fields a view needs.
  • Constraints breed design. Block Kit has no charts, so the dashboard became monospace ASCII meters and everyone loves them.
  • Slack Lists are new terrain. UI-created select columns have opaque option IDs with no schema-read API; we ended up provisioning lists from the bot itself so the schema matches the data.

What's next for Susan - Call Issue Triage

  • Close the loop to the fix. Link issue buckets to the actual code/config change that resolved them, so when Susan reopens an issue she can say what fix regressed.
  • Cross-office intelligence. The same failure often appears across many dental offices and surfacing "this issue affects 12 offices" makes issues easier to ticket and debug.
  • From triage to prevention. Use resolved-issue history as an evaluation suite for the voice agent: every fixed bucket becomes a regression test that new agent versions must pass before rollout.
  • Richer Agent Builder integrations on top of Susan's MCP tools. We want to let other internal agents (deploy bots, on-call assistants) query call quality directly.

Built With

Share this project:

Updates