Apron

Apron is an irregular operations desk for live sporting events. Utilizing a crew of 8 AI agents, it watches the game, the flights and the union rules to get every crew member to their next call.

Comment

I love live sports. I love travel. And I love the intersection of the two required to bring an event into people's living rooms each day and night. That intersection is complex and, today, mostly handled through spreadsheet and text messages.

Apron is an irregular operations desk for live sports broadcast. When a game runs long, twenty-two crew with flights to eleven cities break at once — different airlines, different next calls, different contracts. One coordinator finds out at 1 AM and works it by phone.

I've built a crew of 8 AI agents, each owning one domain essential for getting a chaotic break back into a manageable plan. SPOTTER, TRAFFIC, STEWARD, ADVANCE, WRANGLER, FIXER, RUNNER, and CUSTOMS...you'll meet them all in a deep dive into this solution.

Built with Gemini 2.5 Flash on Vertex AI, Cloud Run, Firestore, and Grafana Cloud for metrics, a Loki decision trail, Agent Observability and the MCP server. Telemetry is the operational record, not a dashboard bolted on after. pnpm demo runs it cold, no credentials for a sense of the experience and depth of the solution.

Inspiration

A live sports broadcast is a pop-up workplace that gets built and torn down in a day. An NFL playoff game broadcast staffs over two hundred people. The Super Bowl runs past seven hundred. And ninety-six of the hundred most-watched programs on American television last year were live sports — up from eighty-seven the year before. This is the part of the industry that is growing, and almost none of the AI pointed at entertainment is pointed here.

Every one of those people has somewhere to be tomorrow.

The plan for that is built weeks out, against a scheduled end time and a set of union rest rules, and it is correct right up until the game refuses to end. Extra innings. A weather hold. Double overtime. Wrap slips two hours, and now twenty-two highly technical crew members with flights to eleven different cities are all individually broken — different airlines, different connections, different next calls, different contracts.

One person, usually a production coordinator, finds this out at one in the morning and works it by phone.

What made me want to build it is that the disruption is not exotic. It is the baseline. More than one in five US flights arrives late, and by the DOT's own measurement the single largest cause of delay is another late flight — larger than air carrier delay, larger than weather. Cascade is the normal condition. The question was never whether someone gets disrupted on a given show night. It's how many, which positions, and whether anyone catches it while there is still time to act.

What it does

Apron is an irregular operations desk for live production. A crew of eight AI agents, each owning one domain, working one question: how do we get everyone to their next call?

The agents split into three kinds, and the split is the architecture rather than a label:

Domain agents own a model of one part of the world.

SPOTTER watches the game — score, inning, clock, venue weather — and projects when it actually ends. Scores come from ESPN, weather from Weather.gov, and end-time prediction from Gemini with a confidence score that travels with it. TRAFFIC watches the air network. Not just the crew's booked flights: hub topology, aircraft rotations, and weather as a propagating condition. A storm in the northeast that grounds an inbound aircraft is TRAFFIC's problem before it is anyone's itinerary. Live flight status comes from AviationStack; curated schedule data covers 28 airports with direct and connecting routings. ADVANCE determines where each crew member must be next, from confirmed crew sheets or inferred from booked travel. STEWARD holds the union agreements and contracts and makes them queryable, including the rule that travel is work, not rest. CUSTOMS covers carnets, work permits and border clearance. It is provisioned per show and is dark on a domestic night. Weather is the clearest illustration of why these are separate. It is one source consumed by two agents asking completely different questions of it: SPOTTER wants to know whether play is suspended at one venue; TRAFFIC wants to know whether a system a thousand miles away takes out an aircraft nobody on the crew has ever boarded. Same input, different world model.

Channel agents hold the credentials that reach outside the system. WRANGLER is the only component permitted to contact a human crew member — via SMS through Twilio — and it is built to elicit a constraint rather than a commitment — "MCI by Monday, 1 PM" — because a tired person saying "sure, whatever works" is not a schedulable fact. Gemini parses the natural-language response into a structured constraint. RUNNER is the only component that reaches the Travel Management Company.

FIXER is the coordinator. It owns the conflict between domains, which is the thing no single domain can see.

That distinction is the whole product. A single violated constraint usually has a local fix, and the agent that owns it resolves it without anyone being paged: a delay that stays inside the buffer already booked, a projection whose confidence narrows as outs accumulate, a missing next call that WRANGLER simply asks for. Those alerts fire and clear on their own.

Conflicting constraints are different. The game ran long, and the flight can't move, and rest is mandatory, and the next call is fixed. That is a tradeoff, and a tradeoff is exactly the thing a machine should not decide alone. So FIXER builds the options and opens the decision for a person.

Agents hold. People ticket.

How we built it

Gemini 2.5 Flash via Vertex AI runs the reasoning for every agent. The architectural line we care about most is deterministic versus probabilistic. Scores, current weather and flight status are deterministic — they are systems we call, and they return facts. How long this game will run, and whether that connection holds, are judgment — those are prompted, and every answer carries a confidence that travels with it. Mixing the two without labeling which is which is how an operations tool loses a coordinator's trust in week three.

STEWARD is the clearest case for Gemini specifically: collective bargaining agreements are long, inconsistent, and full of exemptions. Making them queryable in natural language — and getting back a citation to the clause, not just an answer — is what turns a PDF into an operational input. WRANGLER uses it differently: a crew member texts back "yeah I need to be in KC by one on Monday for the football thing" and Gemini extracts a structured constraint — destination, deadline, and that the production name was volunteered (which changes its disclosure level).

Grafana Cloud is the operational telemetry layer, running three pipelines in parallel:

Prometheus metrics push via remote write (Influx line protocol) on every agent event. Five metrics track the operational state in real time: apron_agent_event (one sample per agent action, tagged by agent and event type), apron_crew_at_risk (how many crew are in jeopardy), apron_call_times_exposed (how many next calls are breached), apron_show_state_change (escalation: clear → watch → risk → down), and apron_agent_process_duration_ms (LLM latency per agent dispatch). The buffer flushes every 30 seconds to stay under Grafana Cloud's rate limits. Loki structured logs record every agent decision with labels (agent, event_type, game_id, show_id), JSON-parsed at query time. This is not a debug log — it is the epistemic history trail: when the projected final shifted, when a constraint was confirmed, when a provenance level was upgraded from inferred to crew-confirmed. The closeout report (GET /api/closeout/:gameId) queries this trail after a game ends and builds a chronological operational narrative — timeline, decisions, compliance summary, and an operational grade. Agent Observability via the @grafana/agento11y SDK (v0.13.0) instruments every Gemini generation with normalized prompt and completion text, token usage (input/output/total), latency, model version, and per-agent tagging. This is how we prove the agent decomposition — each agent's generation profile is visibly distinct in the Agent O11y dashboard. An importable dashboard ships at integrations/grafana/dashboards/apron-ops-overview.json: show status (crew-at-risk, call-times-exposed, state changes), agent activity (events/min, events-by-type), Gemini LLM performance (latency, P95 per agent), and a Loki log panel. Each pipeline degrades independently — a missing key disables that service with a log line and the rest continues.

Every agent output carries a provenance envelope — which agent, which source, deterministic or prompted, what confidence, and what disclosure level. The board renders against that envelope, which is why the same screen shows a production coordinator only what is needed to clear the night, and shows the desk that booked it the full picture. Same data, different disclosure predicate.

Firestore (Native mode) persists accounts, entities, crew, games, and assignments. Cloud Run hosts the production server. The server is native node:http with WebSocket — no Express, no framework.

Deployed on Cloud Run at https://apron-203460075246.us-central1.run.app. Public repo under AGPL-3.0 at https://github.com/markmcspadden/apron.

Challenges we ran into

Fixture mode vs. live mode timing. The server's buildLiveScenario() replaces the fixture with real game data on every board connection. When no games are active, the scenario has zero steps — the board loads empty. Testing the full Grafana pipeline end-to-end required either catching a live game window or falling back to fixture dispatch, and the two paths don't share a timing model. We built the tape fixture partly to guarantee a repeatable three-minute run that exercises all three Grafana pipelines on demand. (But I will say...watching real games come across is so fun...even if it's just watching predicted end times!)

The provenance stack. Getting the two-seat disclosure model right — production seat sees "External call · withheld" while TMC desk sees the constraint — required threading provenance through every layer: the fixture data, the crew state updates, the WebSocket protocol, and the board rendering. A fact's disclosure level is not a UI toggle; it is a property of the fact itself, set by the agent that produced it, and the board is the last place it is checked — not the first.

Accomplishments that we're proud of

The three Grafana pipelines are not dashboards bolted on after the product works. They are the operational record. The Loki trail is what the closeout report queries. The Prometheus metrics are what a production desk would watch during a live show. The Agent O11y traces are how you audit whether the agent decomposition is earning its complexity. Telemetry is a product feature, not a monitoring afterthought.

The human gate is visible, not claimed. You can watch alerts fire and clear on their own as SPOTTER's projection narrows, watch the one that escalates when STEWARD finds a rest violation, and watch the system stop and present options for a person to decide. Nothing auto-closes. Nothing auto-books.

Nobody's call time leaks. Twenty-two people routed, and the production seat never sees another crew member's private schedule. The four crew with external calls show "External call · withheld" — and the system still solves for them, because WRANGLER brokered the constraint without the commitment.

The fixture runs cold. pnpm demo needs zero credentials, zero network access, zero configuration. Every integration degrades gracefully. A hackathon judge can clone the repo, run one command, and see the full scenario play through the board in real time — including the seat switch that proves the disclosure model works.

What we learned

Agents are defined by the question they own, not the data they read. We started by drawing boxes around data sources, which produces a roster that collapses the moment anyone pushes on it. Weather broke that thinking open: one source, two agents, two genuinely different domain models. Redrawn around domains, every agent had a reason to exist, and two of them turned out not to be domain agents at all — they were credential boundaries, which is a different and equally good reason.

"Multi-agent" is a claim that invites an audit, and it should. Most of what gets called multi-agent is one model called eight times with one API key in one context. We ran our own roster against four tests and found that two of the tests were bad — "distinct data source" is nearly free to pass, and "fails independently" disqualifies exactly the components a product depends on. Blast radius and independent evaluability were the only two that mapped to real costs.

Observability has to be first-class or it's fiction. Bolting Grafana on after the agents work would have produced a dashboard that shows you the system is running. Building it into the agent event model — so that the Loki trail is the same data the closeout report queries, and the Prometheus metrics are the same gauges the board renders — means the telemetry proves the product works, not just that the server is up.

What's next for Apron

Close the loop. Today Apron stops when the plan reaches the TMC. It does not yet learn that a booking was confirmed. That is the next thing, and it is deliberately not faked in this build.

Provision CUSTOMS. Border clearance is a real domain sitting dark. A Toronto or London show turns it on, and that is the honest test of whether per-show provisioning is architecture or decoration.

Real TMC integration. The current handoff runs against a mock. Entertainment is the fastest-growing division at one major agency network, projected at 15% year over year for the rest of 2026 — the demand for this is already sitting inside the TMCs.

More sports. The mechanics are not baseball-specific. Anything that can run long — overtime, penalties, a rain delay — breaks the same plan the same way.

Built With

Share this project:

Updates

Submission history