Raccord - Accessible Experience Reliability for live media.

Raccord certifies, monitors, diagnoses, repairs and proves the accessibility of a live media experience.

Comment

Inspiration

A film festival streams a world premiere to eight territories. The picture is flawless. The primary audio is flawless. Every dashboard is green.

And for twenty minutes, English captions on two connected-TV builds in Western Europe are eight seconds behind the dialogue. The described-audio track is declared in the manifest and carries silence. The sign-language interpreter is frozen on one repeated frame. A keyboard-only viewer cannot open the caption menu at all, because it has a focus trap.

Every one of those is total loss of service for the viewers who depend on it. None of them moves a conventional availability dashboard. Nobody is paged. No error budget burns. The incident is found on social media an hour later, by which time the premiere is over.

The gap is not a monitoring gap. It is a definition gap: the industry defines "the stream is up" in a way that excludes the audience that most needs it to be up. We wanted to see what it would look like to close that definition properly — to treat captions, audio description, alternate-language audio, sign-language video, accessible playback controls, accessible authentication and accessible purchase as production services with SLOs, error budgets, owners, incident procedures and proof of recovery.

What it does

Raccord runs a twelve-state closed loop with no skipping:

detect → scope → gather evidence through Grafana MCP → diagnose → evaluate policy → obtain a signed approval → execute one allow-listed action → re-measure → communicate → review

Four modes make that useful rather than merely mechanical:

  • Preflight certification. Every promise an event made is tested against the real chain and real players before the event may claim "accessibility ready". Hard assertions block certification; the output is a signed record.
  • Live assurance. A probe fleet measures the rendered experience — what the audience actually receives — across the language × territory × platform × device × build matrix, against the promise that was in force at that moment.
  • Closed-loop incident response. The loop above, ending in a re-measurement rather than a model's opinion.
  • Reliability intelligence. What was missed, which change caused it, how much error budget it cost, and which improvements to put in front of a human.

On the hero incident — a PTP grandmaster failover moving the caption chain onto an NTP fallback — the system scopes the fault to exactly the four Western European territories, the two CTV builds and the English track: scope precision and recall 1.00 against a fault specification the agents never see. It distinguishes a progressive drift from a fixed clock offset by the shape of the SLI over the window, not its magnitude. It then refuses to act until a signed approval arrives from the event technical director, and closes only after 9 of 9 assertions pass — three of which prove French, German and Spanish captions, the described audio and the interpreter feed were not regressed by the fix. 8,053 sessions affected; 140,295 protected. 17 Grafana MCP calls.

The same control plane runs a premiere and a catalogue. All 31 SLOs are defined per tier — tier0_global_live, tier1_regional_live, tier2_vod_premium, tier3_catalog — each with its own objective and error budget, and the policy engine reads tier and liveness independently. A live tier-0 chain requires the technical director's signature before anything moves. A narrow, reversible, single-region fix on non-live content runs automatically. That second path is where the volume is: nobody can put a human in the loop for fifty thousand titles, and a back catalogue is exactly where accessibility rot accumulates — the described-audio track that has been silent for eight months, on a film nobody watches with description except the people who cannot watch it without.

And it is now a legal obligation. The European Accessibility Act has applied to audiovisual media services since June 2025; the FCC enforces caption-quality rules; Ofcom sets access-service quotas. Compliance today means periodic sampling and a spreadsheet. Raccord makes it a continuously measured objective with a hash-chained evidence trail — the same artifact that satisfies an auditor also pages an engineer.

The product is seven views, not a chat window: Overview, Readiness studio, Live cockpit, Incident workspace, Evidence replay, Agent & MCP observability, Benchmark laboratory. The interface keeps evidence, hypothesis, policy and verified result visually and structurally separate, because conflating them is how automated operations goes wrong.

How we built it

Grafana is not decorative. Grafana MCP is the agent's only route to operational truth, and the state machine enforces it: the transition into EVIDENCE_COMPLETE has a machine-checkable precondition requiring evidence whose source_tool is a Grafana MCP tool for alerts, metrics, logs, traces and dashboards. There is no fallback path that queries Prometheus, Loki or Tempo directly, and a test asserts every evidence item in a completed incident arrived through MCP. Delete the MCP server and the incident cannot leave SCOPED.

Everything Raccord learns also becomes Grafana data: Prometheus series for every probe finding and SLO evaluation, Loki lines from every delivery component, Tempo spans for the media path and for the agent's own reasoning, Pyroscope profiles for the probe fleet. Five dashboards and 31 alert rules are generated from the SLO definitions, so a panel threshold can never drift from the objective the probes are measured against — CI fails the build if it does.

A governed twelve-state workflow with Gemini where language reasoning adds value. Narrow specialists for scope, evidence, quality, correlation, diagnosis, policy, remediation, verification, communication and learning exchange typed records. Gemini — on Vertex AI, through the Agent Development Kit — takes the typed incident record and does what a frontier model is uniquely good at: reading a multimodal picture across metrics, logs, traces, probe findings and change events; naming what is uncertain and which evidence would resolve it; and writing six audience-specific communications, each in the right register and reading level. It reaches the read-only Grafana MCP tool surface through ADK's McpToolset, and the product exposes that directly: Ask the agent, in the incident workspace, lets an operator interrogate the diagnosis — why did you rule out a fixed clock offset? — and a question the retrieved evidence cannot settle sends Gemini back to Grafana MCP for more, on the same audited path, with the new calls appearing live in the Agent & MCP view. Every answer cites the evidence it rests on and the tool that produced it. After verified closure, Gemini also proposes falsifiable reliability experiments; the deterministic root cause and measurements remain immutable.

Built to production governance standards. The measurement, policy and verification arithmetic is typed, tested code, and RemediationExecutor accepts only a redeemed, single-use, HMAC-signed approval token bound to an exact action hash and evidence hash, expiring in five minutes — change one parameter after approval and redemption fails. That separation is what makes an autonomous agent deployable against a live premiere, and it is why the unsafe-action rate is 0.000 across 1,000 scenarios.

Also built: an accessible-experience digital twin with blast-radius traversal; 45 documented faults used as benchmark ground truth and read by nothing else; caption / audio-description / sign-feed / player probes that abstain rather than guess; custom alignment kernels (NumPy anti-diagonal wavefront, Triton, CUDA) that are bit-identical to the reference; eBPF delivery-path telemetry correlated to media symptoms; Grafana Agent Observability for framework-native ADK traces and metrics; Terraform for the trust boundary.

Google Cloud: Gemini on Vertex AI, ADK for agent definitions and the MCP toolset, Agent Engine for the managed runtime, Cloud Run for the app, Secret Manager and IAM for the security boundary.

Challenges we ran into

Binding an agent to a partner API that moves. The official grafana/mcp-grafana:1.0.0 server advertises 65 tools, and that surface has been renamed, consolidated and reshaped across releases: alert listing and retrieval are now a single action-dispatch tool, and current open-source builds route Tempo through a generic grafana_api_request rather than a dedicated trace tool. An agent that hard-codes tool names is broken by the next release of the thing it depends on.

So Raccord binds to capabilities, not names. It discovers the live tool list at connect time, resolves each of its 20 capabilities against it, and refuses to open an investigation it cannot fully evidence. src/raccord/grafana_mcp/adapters.py carries one adapter per deviating tool: dispatching operation: "list" into the alerting tool, discovering datasource UIDs instead of assuming them, re-emitting timestamps in the format Grafana's parser accepts, collapsing a Prometheus matrix into the aggregate the agents consume, recovering the rule uid the list operation leaves empty, and reaching Tempo through Grafana's datasource proxy — still over MCP, still audited. 19 tests pin those shapes, so a server-side change surfaces as a named test failure instead of a dead investigation. Result: 18/20 capabilities resolved, 12/12 required, and the full loop running against the real server.

Making every Raccord signal a first-class Grafana citizen. "Everything the agent learns becomes Grafana data" only counts if the correlation actually works. The programme clock in a live-event twin runs faster than wall time, so probe findings, delivery logs and media-path spans for a single incident landed minutes apart once exported. Mapping the simulated clock onto the real one is what lets a Loki line, a Tempo span and a Prometheus sample for the same moment line up in one Grafana time range — which is the entire basis of the change-correlation engine.

Verification that catches its own mistakes. Diagnosing a fault is the easy half. We built the assertion suite to re-measure three scopes after every action — the original slices, the adjacent ones that must not regress, and the dependent features downstream — so a repair that fixes English captions while breaking the French track fails verification and rolls back. That is why the false-closure rate is 0.001 rather than a diagnosis-accuracy number.

The result: the whole loop runs against the official server — every MCP call successful, ending REVIEWED with 9/9 assertions and scope 1.00/1.00, opened by a real Grafana alert rule in firing state and evidenced by real Prometheus, Loki and Tempo. docs/real_mcp_run.json.

Accomplishments that we're proud of

  • False closure rate 0.001. Unsafe action rate 0.000. Across 1,000 seeded scenarios. Declaring an accessibility feature restored when it is not is the failure mode that makes an automated system worse than no system. It happens once in a thousand, because closure is gated on re-measurement through Grafana rather than on a model's confidence.
  • Detection 1.000, scope precision 1.000, recovered-and-verified 0.919 — scored against fault specifications the agents never see.
  • The loop runs against the partner's real product, not a stand-in: the official grafana/mcp-grafana server, reading a real Grafana over real Prometheus, Loki and Tempo — and it still runs with no credentials at all when you have no stack.
  • A product about accessibility that is itself accessible. WCAG 2.2 AA, keyboard-operable throughout, status never conveyed by colour alone, dependency-free (no framework, no bundler, no CDN). 63 automated checks, CI fails on a regression, and driving all seven views in a real browser produces zero console errors or warnings.
  • Real systems depth, not an architecture slide. Custom Triton and CUDA alignment kernels that are bit-identical to the reference at 9.9× the speed; eBPF delivery-path telemetry correlated to media symptoms; framework-native ADK observability; a 45-fault library driving 1,000 benchmarked scenarios with ablations.
  • Every quantitative claim has a committed artifact or a command that reproduces it.
  • The whole demonstration runs with no credentials, no cloud account and no network.

What we learned

  • Verification, not diagnosis, is what makes an agent deployable. Optimising diagnosis is the obvious move; gating closure on re-measurement is the one that actually protects the audience, and it is why a suboptimal action rolls back instead of closing an incident.
  • Capability resolution beats tool names. Binding to what a partner server can do rather than what its tools are called absorbed three breaking changes in Grafana MCP's surface with no agent code touched.
  • Abstention is worth more than accuracy in a measurement system. A probe that reports "I cannot measure this window" is worth more than one that guesses: calibration puts the confident-zero rate at 0.000 and caption-drift MAE at 0.030 s.
  • Never infer disability. We measure features and sessions, never people. The system cannot tell you who is deaf, and it does not need to in order to page someone.

What's next for Raccord

  1. Interactive hosted MCP OAuth for user-driven sessions. The unattended Agent Engine deployment instead uses the official open-source server behind Raccord's authenticated gateway.
  2. Player-edge probes in real browsers and connected-TV devices, extending measurement to the last mile of rendering.
  3. Pilot with a live festival broadcaster, promoting the promise registry from a twin topology to a production delivery chain.

Built With

Share this project:

Updates