Support Ops Room

Agents triage, research, and draft support replies; a deterministic guardrail gate runs before the model and can't be overruled by one. Every hop is traced. Nothing sends without a human.

Comment

INSPIRATION

I kept coming back to the same observation about support work: most tickets have answers that already exist. The refund amount is in the invoice table. The renewal date is in the subscription record. The fix is in a knowledge-base article. The tiring part is not deciding what to say, it is gathering the context to say it.

So the obvious move is to let an LLM do it. Except you cannot just point a model at a support inbox. It will confidently promise an 1,800-dollar refund, cancel a plan because the customer asked nicely, or echo a customer's SSN back to them in the reply. Those are not hallucinations you can apologise for later. They are commitments.

That reframed the project for me. The interesting engineering problem is not "can a model draft a support reply" (it can, that part is close to solved). It is how do you put a model inside a real workflow without letting it do something you cannot take back. Support Ops Room is my answer: agents draft, deterministic rules gate, and a human signs off, with the entire decision path visible before anyone approves it.

📺 Two-minute demo video

WHAT IT DOES

Inbound tickets flow through a multi-agent pipeline:

$$ \text{triage} \rightarrow \text{router} \rightarrow \text{lane specialist} \rightarrow \text{supervisor} + \text{guardrails} \rightarrow \text{human approval} $$

A triage agent classifies category, urgency, sentiment, and entities. A deterministic router maps that to a lane (billing, technical, account, orders) and raises concrete escalation signals. The lane specialist runs a bounded tool-calling loop against the real database (account_lookup, invoice_lookup, subscription_lookup, kb_search, refund_calc) and drafts a reply with citations. Then a deterministic gate and a supervisor model both review it, and a human auditor approves, edits, or escalates.

Every hop streams into a live orchestration trace with its model, latency, token count, and expandable tool calls showing validated input and output JSON. It is not a log. It is the product.

The central design decision, and the one I would defend hardest:

The deterministic guardrail gate runs first, and a model cannot soften it.

Anything a rule can decide, a rule decides, with no model call and no prompt injection surface:

• Refund above the 100-dollar support ceiling → escalate • Cancellation or downgrade intent → escalate, owner confirmation required • Enterprise account, plus any of the above → escalate to a manager • PII in the outbound draft (SSN, Luhn-checked card) → redact, then needs_review

The supervisor model is then asked only what rules genuinely cannot judge: tone, completeness, overpromising. With verdicts ordered by severity,

$$ \texttt{auto_send} \prec \texttt{needs_review} \prec \texttt{escalate} $$

the final verdict is

$$ v_{\text{final}} = \max_{\prec}\bigl(v_{\text{gate}},\; v_{\text{supervisor}}\bigr) $$

so the gate always wins. My favourite demo moment is exactly this, and it opens the video: a customer asks for a refund of

$$ \$1{,}800 > \$100 = \text{the support ceiling} $$

the supervisor model returns auto_send, and the run escalates anyway, because hard rules outrank soft judgement. The safety-critical path is pure functions, which means I can unit test it exhaustively, and did: 21 tests for the gate alone.

HOW I BUILT IT

Next.js 15 App Router with TypeScript, Tailwind v4, Drizzle ORM over Neon Postgres 17, and Groq for all three model roles. Deployed on Vercel.

Groq's latency is not a cost optimisation here, it is a UX feature. Hops resolve fast enough that the live trace and the concurrency view feel instant rather than like watching a progress bar.

• Triage runs llama-3.1-8b-instant. Cheap, roughly 560 tok/s, and this is high-volume classification. • Specialists run openai/gpt-oss-120b for strong tool use, one instance per lane. • Supervisor runs the same openai/gpt-oss-120b at reasoning_effort: "high".

The piece of architecture I am happiest with is the trace event stream. The orchestrator emits typed events (run_started, agent_started, tool_called, tool_finished, agent_finished, guardrail_verdict, run_finished, error) as first-class data, not logging. One stream, four consumers:

• The live UI, as streamed NDJSON rendered while the run happens • The traces table, one persisted row per hop • Metrics, aggregating latency, tokens, and cost The central design decision, and the one I would defend hardest:

The deterministic guardrail gate runs first, and a model cannot soften it.

Anything a rule can decide, a rule decides, with no model call and no prompt injection surface:

• Refund above the 100-dollar support ceiling → escalate • Cancellation or downgrade intent → escalate, owner confirmation required • Enterprise account, plus any of the above → escalate to a manager • PII in the outbound draft (SSN, Luhn-checked card) → redact, then needs_review

The supervisor model is then asked only what rules genuinely cannot judge: tone, completeness, overpromising. With verdicts ordered by severity,

$$ \texttt{auto_send} \prec \texttt{needs_review} \prec \texttt{escalate} $$

the final verdict is

$$ v_{\text{final}} = \max_{\prec}\bigl(v_{\text{gate}},\; v_{\text{supervisor}}\bigr) $$

so the gate always wins. My favourite demo moment is exactly this, and it opens the video: a customer asks for a refund of

$$ \$1{,}800 > \$100 = \text{the support ceiling} $$

the supervisor model returns auto_send, and the run escalates anyway, because hard rules outrank soft judgement. The safety-critical path is pure functions, which means I can unit test it exhaustively, and did: 21 tests for the gate alone.

HOW I BUILT IT

Next.js 15 App Router with TypeScript, Tailwind v4, Drizzle ORM over Neon Postgres 17, and Groq for all three model roles. Deployed on Vercel.

Groq's latency is not a cost optimisation here, it is a UX feature. Hops resolve fast enough that the live trace and the concurrency view feel instant rather than like watching a progress bar.

• Triage runs llama-3.1-8b-instant. Cheap, roughly 560 tok/s, and this is high-volume classification. • Specialists run openai/gpt-oss-120b for strong tool use, one instance per lane. • Supervisor runs the same openai/gpt-oss-120b at reasoning_effort: "high".

The piece of architecture I am happiest with is the trace event stream. The orchestrator emits typed events (run_started, agent_started, tool_called, tool_finished, agent_finished, guardrail_verdict, run_finished, error) as first-class data, not logging. One stream, four consumers:

• The live UI, as streamed NDJSON rendered while the run happens • The traces table, one persisted row per hop • Metrics, aggregating latency, tokens, and cost • Replay, re-animating a finished run from those stored rows

There is one reducer (applyEvent) and one projection (stateFromTraces), so a live run and a Replay render through the identical code path. Replay was therefore nearly free to build, costs nothing to run, and doubles as the thing my end-to-end tests drive, so the suite never spends a token.

I also refused to trust model output anywhere it crosses a boundary. Tool arguments are JSON.parsed and zod-validated before execution, so a bad shape returns an error result the agent can react to rather than throwing. Structured output goes through a chatJSON() helper doing JSON mode, parse, validate, and a bounded re-ask that feeds the validation error back. Enum fields use case-normalising z.preprocess with .catch() fallbacks, so a stray value degrades safely instead of killing a run.

Then I wired it to a real HubSpot portal, because "you invented the tickets" is a fair objection. Tickets sync live from the CRM, and an approved reply posts back onto the record as a note with an audit footer naming the pipeline, the verdict, and the human who signed off.

CHALLENGES I RAN INTO

The rate limit that looked like a logic bug. /simulate fires several tickets through the real pipeline at once, and 5 of 6 runs failed. I went hunting in the orchestrator for a concurrency bug that did not exist. The actual cause was Groq's free tier capping gpt-oss-120b at 8,000 tokens per minute:

$$ 6 \text{ tickets} \times (\text{specialist} + \text{supervisor}) \gg 8\text{k TPM} $$

so most of them 429'd. Three fixes: a groqChat wrapper that honours the retry delay embedded in the 429 body ("try again in 5.0025s") and the retry-after header, else exponential backoff with jitter; a bounded worker pool instead of Promise.all; and a lighter supervisor for batch runs. The same 6 tickets now complete 6/6 with identical verdicts. The lesson that stuck: a bounded pool degrades to slower, Promise.all degrades to half of them failed.

A crashed run that silently poisoned my metrics. The dashboard said 18 tickets processed but only 16 drafts existed. One gap was by design, since a ticket with no identifiable account escalates before drafting. The other was a real bug: runPipeline had no try/catch, and since it sets status: "triaging" up front but writes the real lane and priority only after the supervisor, any throw in between stranded the ticket in triaging forever, with orphan trace rows and no record of why. Worse, the crashed run's partial half-second timing was being averaged into my latency figures, so a crash made the pipeline look faster and cheaper. Now a failure is persisted as a real trace hop, the ticket escalates to a human instead of vanishing, and failed runs are excluded from averages and counted separately.

HubSpot association type ids. Every note POST failed with a 400, "one or more associations are not valid". I had copied associationTypeId: 214 from a generic docs example; 214 is note-to-deal, not note-to-ticket. The fix was to stop hardcoding numeric ids entirely and use the v4 default-association endpoint, which resolves the correct type per object pair. A second CRM trap right after: company associations are many-to-one and portals accumulate stale links, so my demo ticket resolved to the dev portal's leftover "HubSpot" company and escalated as unidentifiable. The client now dedupes by record id and returns all linked names, and the sync layer picks the first that maps to a real local account instead of trusting HubSpot's ordering.

Small models have off-vocabulary days. The 8B triage model returned sentiment "negative", which was not in my enum, and it killed a run after retries. That exact string is now a test case, and schemas degrade rather than throw.

WHAT I LEARNED

• Put the hard constraints in code, not in the prompt. Every rule I moved out of "please do not promise large refunds" and into a pure function became something I could test exhaustively, and stopped being something a model could be talked out of. Prompts are for judgement, code is for policy.

• Observability designed in beats observability added on. Because trace events were typed data from day 2, the live UI, persistence, the metrics dashboard, and Replay all fell out of one stream. Had I started with console.log, three of those four would have been separate builds.

• Tests that cannot fail are not tests. I mutation-checked mine. Flipping the escalate severity, making the refund ceiling comparison inclusive, and replacing the severity-max with the supervisor's verdict alone each break the suite. Until I ran those checks I did not actually know whether my safety tests were load-bearing.

• Faking exactly two boundaries goes further than mocking everything. My integration suite fakes only groqChat (the network) and the DB client, and runs everything else for real: parsing, validation, the tool-calling loop, the gate, PII redaction, verdict combining, persistence. 101 Vitest tests run offline in 779 ms with no API key.

• Agentic cost intuition was way off. I assumed a five-hop, tool-calling, supervised pipeline would be expensive. The real measured aggregate over 19 tickets was 76,724 tokens:

$$ c_{\text{ticket}} = \frac{\$0.03}{19} \approx \$0.0017 $$

The economics were never the constraint. Trust was.

WHAT'S NEXT

Real embeddings for kb_search. The embedding column is already there waiting on an endpoint, and today it is weighted token overlap with titles scoring double body matches. Beyond that: tighter specialist prompts so drafts stop leaking greeting placeholders, and a light mode for people who do not live in a dark terminal.

131 tests passing. About a fifth of a cent per ticket. And nothing reaches a customer without a person.

Built With

Share this project:

Updates