The Gallery: Twenty Cars, One Camera, No Human

An agent that directs the Formula 1 world feed. Gemini chooses which fight the world sees, and catches 42% of the overtakes against 3% for following the leader.

Comment

Inspiration

A Formula 1 race has twenty cars and one world feed. Every camera angle exists, but a human director picks which one goes out, and that choice is irreversible - hold on the leader while a fight for eighth resolves, and the pass simply never happened as far as the audience is concerned. There is no second take on a live race.

That is a scheduling problem with a deadline, and it seemed like the right shape for an agent: too many candidate stories to watch at once, a decision required every few seconds, and a judgement at the end that isn't reducible to a number.

What it does

The Gallery directs the feed. It replays a real 2023 race from FastF1 telemetry, scores all 19 adjacent pairings ten times a second, and cuts to whichever fight is worth showing.

The split matters. Scoring is deterministic - gap, closing rate, DRS, tyre delta, weighted and combined in plain Python with no model anywhere near it. Gemini is called once per decision, for the one thing arithmetic can't settle: of these candidate battles, which is the story?

Every cut is written to a Grafana dashboard as an annotation on the race timeline, tagged with the tier that produced it.

How we built it

Deterministic layer. A tension score per pairing: 0.52 × proximity + 0.28 × momentum + 0.12 × DRS + 0.08 × tyre, scaled by track position. The momentum term was worthless at first because I'd normalised it against a guessed closing rate. Measuring an actual replay gave p50 0.004 s/s, p90 0.012, p99 0.018, so the constant became 0.015 and the term started contributing.

Agent layer. An ADK LlmAgent and Runner on Vertex AI running gemini-2.5-flash. Two changes took it from unusable to real-time: thinking_budget=0 cut latency from 3.1–7.5 s to 0.7–0.9 s, and inlining recent airtime into the prompt removed a tool round trip: 2.15 to 1.16 model calls per decision, 2129 ms to 1521 ms. The call is dispatched as a task and applied when it resolves, so a slow response never stalls the feed.

Grafana, both directions. The app provisions its own dashboard on startup, installs a battle-imminent alert rule, and pushes metrics by Prometheus remote_write : hand-encoded protobuf and snappy, because Grafana Cloud can't scrape a Cloud Run instance. It also reads back: Grafana's own MCP server runs in the container, filtered to three tools and attached to the agent, so the director can query the annotations its own cuts produced.

Serving. FastAPI, a 10 Hz WebSocket, Three.js for the 3D circuit, and Canvas for the 2D map. Deployed to Cloud Run with Cloud Build and Secret Manager.

Challenges we ran into

Most of the hard bugs were in the data, not the agent.

The race order was wrong. I was deriving lap count from arc-length wrapping around a centreline. At Spa that put Albon leading a race Verstappen won. Geometry is the wrong instrument which the fix was to interpolate lap number from official timing data and never look at position for ranking again.

Cars were doing 2764 km/h. FastF1 position coordinates are in decimetres, not metres.

The centreline came out garbage. I'd built it from a raw prefix of telemetry rather than a clean lap, which had one driver "leading" Monza by 141 seconds. Using pick_fastest().get_pos_data() fixed it. Then smoothing erased Monza's chicanes, so the resample has to happen before the smooth.

The feed went silent after eight minutes while the health check stayed green. The race is finite and the replay simply ended. Found it by leaving six clients connected and waiting.

Cloud Build couldn't reach the F1 API, so the deployed build silently served a synthetic race instead of real telemetry. Shipping a locally-warmed cache fixed it — after discovering .gcloudignore was falling back to .gitignore and excluding that very cache.

It was burning $24 a day at idle, against a 20-day deadline and a 17-day runway. The director now sleeps when no one is subscribed to the WebSocket.

That last fix caused the bug I found the night before submitting: the overtake counter kept incrementing while the director was asleep, so after an unattended night the dashboard read "1% - 407 of 61,408 passes" on a system that scores 42% when someone is watching. Gating the counter on the same flag that gates the director put it right.

Accomplishments that we're proud of

It's measured, not asserted. A capture-rate harness across four races and 764 position changes: 41.9% at Monza, 50.5% at Spa, 49.1% at Silverstone, 42.1% at Barcelona against 0–3.1% for a camera that simply follows the leader. An oracle knowing every pass in advance reaches ~90%, because one camera can't be in two places. So it catches roughly half of what is physically catchable, blind to the future, for about $1.14 a race.

It doesn't lie about itself. Cuts render cyan when Gemini chose and amber when the system fell back to deterministic on a timeout or quota limit. A fallback is never dressed up as a decision. The director's prompt also forbids inventing facts, so it was caught claiming a driver "has been quick all weekend" with no evidence for it.

I published a failed experiment. I tried to measure agent cuts against deterministic ones and the run only completed 3 real model calls out of 14 cuts. Rather than quietly drop it, the result ships labelled as not evidence, with the reason, in both the code and the README.

What we learned

Measure before you optimise, and delete what doesn't earn its place. I built a semantic response cache that never hit once at any frequency, and a guided demo overlay that made the product worse. Both are gone; the cache's negative result is documented in the module docstring so the next person doesn't rebuild it.

The other lesson is that an LLM is a bad choice for most of this problem. Nineteen pairings at 10 Hz is 190 evaluations a second, using a model there would be slower, costlier, and less accurate than arithmetic. Working out exactly where the model earns its keep, and refusing to use it anywhere else, was most of the design.

What's next for The Gallery: Twenty Cars, One Camera, No Human

Live timing instead of replay, which is mostly a data-source swap. Multiple concurrent feeds so a viewer can pick "follow the championship fight" against "follow the midfield". And a real evaluation of agent versus deterministic cut quality — the one I attempted didn't produce valid data, and it's the question I most want answered.

Built With

Share this project:

Updates