Transmission

Inspiration

On 26 August 2017, Mayweather fought McGregor and the pay-per-view stream could not carry the people who had bought it. The main event was delayed by nearly an hour. 275,937 purchases had gone through it.

In the days after, roughly 90,000 full refunds went out — and in the words of the court filings, the defendants "did not publicly announce that refund program". About $9 million was paid before the suit. Park v. Zuffa settled in July 2018 with a class of more than 208,000 people and uncapped relief of up to $99.99 each. It is the only approved US class-action settlement over a live-stream failure in the federal record.

Showtime's public line, two days after the fight:

"We have received a very limited number of complaints overall."

Eleven months later the court put the class at 208,000 people, and 90,000 refunds had already gone out quietly. Nobody inside that building knew how many people the night had actually cost them.

The obvious answer is "get better telemetry", and it is wrong

Netflix has the best live telemetry in the industry. Here is how their own engineers described running it, writing in April 2026:

"The engineers who designed [the first live pipeline] monitored dashboards on laptops, coordinated over Slack, and troubleshot in real time while millions of members watched."

"Standard monitoring tools incur propagation delays of minutes. However, during a live stream, a signal degradation that goes undetected for three minutes can affect millions of members before any mitigation begins."

"For Tyson/Paul, Netflix had over 300 people online… Coordinating that many people over Slack and Zoom during an active event with 64 million concurrent streams was unmanageable."

"…the command center and the Broadcast Operations Center had been operating largely in a fractured parallel."

The fix they shipped for that last one was human communications, not a data join. And it is not one company: DAZN today runs two 24×7 operations centres for the same live match on completely disjoint toolsets — Conviva, New Relic and Coralogix on the streaming side; DataMiner, Grafana and Probe Analyser on the broadcast side. Two live job postings, August 2026. Nothing is shared between those columns except a person.

The best argument against this project

It comes from Akamai's Will Law, talking about Tyson–Paul:

"Everyone says, 'Oh my God, there were a million people who had rebuffing issues'… I see it completely different. 59 million people didn't have rebuffing issues!"

He is right, and that is exactly the problem. A million out of sixty million is 1.7%. It does not move a fleet average, it does not cross a threshold, and it does not wake anybody up. It is also a million people, every one of them watching a stream break while a dashboard three feet away says the night is going fine.

That is what I built for: not a better dashboard, but the thing that finds the population an average cannot see, while there is still time to do something about it.

Every figure above is sourced with a URL in docs/sources.md, along with what I deliberately do not claim — no ad make-good figure, no churn number, and no count of affected viewers for any event, because none of those exist publicly.

What it does

A studio premieres a film to a hundred and ninety countries at the same moment. Two million people, one shot, and no rollback — whatever is wrong at minute seventy-eight is wrong in front of everyone, and stays wrong until somebody changes where the traffic goes.

Tonight the fleet rebuffer ratio moved from 1.4% to 2.5%. The page threshold is 3.0%. Nothing fired. Underneath that average, one region on one CDN, on the two top renditions, is at 19.1% — 122,000 people watching a premiere break up.

There are three ways to read that telemetry and two of them are wrong:

Reading	What it names	Why it fails
the fleet average	nothing at all	2M people at 1.4% absorb 122k at 19%
worst absolute ratio	fastly / apac / 480p	that cell has been above 5% every night of its life
ratio against its own baseline	cloudfront / eu-central / 1080p + 2160p	the only cells that changed — 15× on themselves

The third reading is the product. The console has five views: Air is the dashboard the operator already has, ending in the arithmetic of the average written out. Slices is all 72 delivery cells with one toggle — worst ratio, or against its own baseline — and flipping it lights up a different cell. That toggle is the entire argument. Signal is the log side: 502s by origin shield, the deploy timeline, and the errors from the onset onward. Investigate is the agent, its PromQL and LogQL, and the card. Log is every investigation run.

How I built it

Google Cloud: Gemini on Vertex AI through Google's Agent Development Kit, deployed on Cloud Run with both Grafana tokens in Secret Manager. Partner: Grafana Cloud (Mimir and Loki) reached through the official grafana/mcp-grafana MCP server, spawned as a child process over stdio.

Four agents that always run, in order, because the ordering is what makes it reliable:

• measure — states the fleet position and that nothing paged. It may not look for a cause; guessing at one here biases every query after it.
• isolate — cuts all 72 cells two ways, worst ratio and ratio against each cell's own value pinned to one hour into the show, then reports which cells the second cut singles out, how many viewers are in them, and which chronically-bad cells it is dismissing and why.
• corroborate — three Loki queries on one shield: the deploy, the first errors, and a control proving the shield still serves its other renditions. That control is what rules out "fail the region over".
• call — no tools at all, so it can only write from what the first three established and cannot quietly acquire a figure nothing queried.

Then a fifth agent reads the card back against the queries. It has no Grafana access on purpose: its job is not to find a better answer, it is to decide whether this one is entailed.

Challenges I ran into

Prose could not hold the analyst, three times. Told in the brief and again in its own step that a chronically bad cell is not tonight's incident, it still opened with the worst absolute number. The fix was never a better sentence — it was structure. Four steps that cannot be reordered, a final step with no tools, and a checker that cannot see the database.

The checker earned its place immediately. It struck out "the issue started within the last quarter of an hour", because no query had established the onset. It changed "is causing" to "is present in logs". And when the analyst reported 139,965 affected viewers — the whole of CloudFront in eu-central, when only two of four renditions were broken — it rewrote the line to say the figure is the region total and an upper bound. Both the draft and the correction are shown.

Mimir will not accept a sample older than two hours. Measured, not assumed: 118 minutes lands, 120 does not. So the show is a hundred minutes ending now — and because it therefore ages out of its own window, the console restages it whenever what it finds is too old to be worth showing. A judge opening the link days later gets a live show, not an empty dashboard.

An instant query will not return a sample more than five minutes old. Every panel emptied itself five minutes after seeding, which looked like an outage of its own. Queries are pinned to the show clock now, which is also more honest: this is a moment in a broadcast, not the current state of the world.

The deploy line carried shield="global". An agent scoping logs to the failing shield — the obvious thing to do — silently missed it and reported that nothing had been deployed. A wrong answer produced by a correct method. The line carries no shield label now.

Grafana Cloud's hosted MCP endpoint cannot be used by a server-side agent. It authorises through an interactive browser flow with no service-account or machine-token option, so a container has no way to hold it. The open-source server with a service account token does the same job.

Prometheus remote write, with no dependencies. python-snappy is a C extension and protobuf is a code generator for a schema of four messages. Both are hand-written here — snappy's block format permits a stream that is nothing but literals, a legal compressor that never compresses, which is exactly right for a seeder that runs once. There is a test that decompresses it.

Accomplishments that I'm proud of

The toggle on Slices. Same 72 cells, same numbers, one button. Ranked by how bad they are, every chronically poor cell in Asia-Pacific lights up, as it does every night. Ranked against each cell's own earlier self, only two cells light up. One click, and the difference between a metric and a finding is on screen.

And what the agent refuses to do. Asked to fix it, the obvious lever is rolling back the change that caused it — which would cap two million healthy viewers at 720p to help a hundred thousand. It proposes stripping two rungs from one region's manifest instead, and says so in RULED OUT.

What I learned

That reliability in an agent is a property of its shape, not of its prompt. Every failure I hit was fixed by removing a choice: a step that cannot look for a cause, a step with no tools, a checker with no database.

And that a good dashboard is not the problem. The operator's Grafana here is honest and well built. It simply cannot show a hundred thousand people inside an average taken over two million, and no amount of dashboard craft changes that.

What's next for Transmission

Watch continuously and raise the card unprompted, instead of waiting to be asked. Learn each cell's baseline from its own history rather than from a fixed instant in one show. And close the loop: propose the manifest change as a diff against the real ladder, so the lever is something you approve rather than something you go and type.

Built With

• artifact-registry
• cloud-iam
• cloud-run
• grafana/mcp-grafana
• python3.13
• vertex-ai