THE GATE

Drop a day's footage in. It finds what coverage you're missing, works out if there's time to get it, and tells you what each answer costs.

Comment

Inspiration

The most expensive avoidable mistake in film production is wrapping a setup you cannot get back.

Before a crew moves the camera, someone says "check the gate." It is the last moment the shot still exists. If a covering angle is missing, or a take is unusable, nobody finds out on set — they find out weeks later in the edit, when the location has been struck, the actor is on another job, and the set would cost tens of thousands a day to rebuild. A pickup day follows.

At that same moment the 1st AD is carrying a completely separate problem in their head: sunset, meal penalties every six hours, twelve-hour turnaround, overtime at 1.5× and 2×. A day of principal photography runs $50k–$500k.

These are the same problem, and nobody treats them that way. Being short of a shot only matters against the time left to get it. Time only matters against what is still missing. That trade-off gets decided in five seconds, from memory, by a tired person, hundreds of times a day.

What it does

You drop a whole day's footage in — one file, straight off the camera card. Nobody types a scene number, a take number, or a character name.

It says whether the crew can move on.

NO-GO · 44% of the day · 10 shots short · $69,020 at risk · 5h 30m left · 10 of 10 you can still get today

Everything behind that line is inspectable, and everything is editable:

  • It sorts the day out for you — cuts the file into individual takes, logs each one (shot size, movement, eyeline, focus, who is in frame), and groups them into scenes by where they were actually shot.
  • It builds the coverage matrix — four framing bands per character (wide, medium, close, over-the-shoulder), required against captured, with the recovery cost of every gap.
  • The AD overrules it, and it sticks — an actor is face down on the floor for a whole scene, so a wide of her is a shot nobody would cut to. Turn that requirement off and the exposure drops on screen. The system proposes. A person decides.
  • It will not invent a requirement — a scene that is an insert of a doorknob has nobody in it, so nobody's coverage is short. It reports "no goal set" rather than manufacturing a rule to be short of, until somebody says what the scene needs.
  • Crew size is the multiplier — the same footage and the same rules, on a two-person crew or a full studio unit, is a completely different amount of money. Change it and every figure moves.
  • It checks against the world the film is set in — change the period from present day to the 1950s and it re-watches everything: a modern illuminated EXIT sign, a Tyvek hazmat suit with a plastic zipper, acoustic drop ceiling tiles. Nobody listed those. Game of Thrones missed a coffee cup, and the whole internet found it.

How we built it

Nine agents over a deterministic core. The rule throughout: computation for facts, agents for judgment.

Footage comes in

  • Editor — decides where each take starts and ends (Gemini)
  • Scripty — logs every take: size, movement, who is in it (Gemini)
  • QC — rules whether a take can be used at all (Gemini)
  • Breakdown — decides which takes were shot in the same place (Gemini)
  • Casting — decides who is in each take (ClickHouse vector search)
  • Continuity — decides whether the angles will cut together (Gemini)

The call

  • Scout — watches the world for what could change the day (Parallel)
  • The Book — chooses what to ask of four years of this crew's data (ClickHouse via MCP)
  • The Clock — runs the rest of the day ten thousand times (numpy)
  • Steward — names the union rule that bites, and what it costs (computed)
  • 1st AD — prices every missing shot, now against later (computed)
  • The Gate — speaks the call, and never invents a number (Gemini)

Splitting a day into takes. ffmpeg's scene-detection filter proposes candidate cuts cheaply, retrying at a lower threshold when the resulting shots come out improbably long. Gemini then watches the file with audio and decides the real boundaries — it hears "action", "cut", and the clap. Letting it listen was the single highest-value change in the project.

Recognising the cast. Gemini finds every person in frame. Vertex AI multimodal embeddings turn the crop into a vector. ClickHouse vector search shortlists who it could be. Gemini decides between them — and "unsure" is a valid answer, so a blurred crop never invents a cast member.

Coverage. Four framing bands per character. Required minus captured. Pure Python.

Time. A 10,000-trial Monte Carlo in numpy, sampling each remaining setup's duration from this crew's own history, held in ClickHouse as quantilesTDigest and conditioned on shot size, interior/exterior and day/night.

Money. Each missing shot priced as a share of a pickup day — a close costs more than a wide, because you have to recall the actor. Then union rules: overtime, meal penalties, turnaround.

NO-GO is a rule, not an opinion. Anything still missing is a NO-GO, and a GO is never returned unless the compliance check has actually run.

Challenges we ran into

Almost every "the model is unreliable" symptom turned out to be the model being asked an impossible question.

  • It was watching footage in silence. Takes are separated by a person saying "cut" — stripping the audio made take boundaries genuinely undecidable.
  • It was asked to name a location from a frame showing a clapperboard.
  • It was asked to match an actor in a gas mask by face.
  • It was given only "this person, or a new person" as answers, with no way to say it could not tell — so every uncertain crop minted a new cast member.
  • It was handed a seven-value shot-size list and told to use "standard film terms", without a single one of them being defined.

Fixing the question beat every attempt to fix the answer.

Two infrastructure traps. ClickHouse cannot UPDATE a key column, so moving a take between scenes is a delete and re-insert. And ADK builds its own genai client from the environment, which silently diverges from one configured in code — that one only surfaced in production.

The one that stung. A tidy-up pass silently truncated every prompt in the system. Nothing raised. The code still imported, still ran, and quietly got worse — the editor lost its list of what counts as a cut and started calling the clapperboard its own shot. There is now a test that reads every prompt in the repo and fails if one ends mid-sentence.

What we learned

Sanity-check every structure against how the domain actually works. Partitioning by shoot day looked fine at 30 days and broke at 800. A thousand consecutive shoot days is not one film — a feature is 30–60 days. It only makes sense as a studio library of eighteen productions over four years, which is what the data models, and what justifies ClickHouse at all.

The honest answer is usually the better product. Reporting "no goal set" is more useful than reporting 100% coverage of nothing.

What's next

  • ClickHouse as the agent's memory, not its prompt. A context window is finite; four years of this studio's takes is not. Instead of stuffing history into the prompt, the agent asks for exactly the rows it needs at the moment it decides — millisecond lookups over millions of setups. The model stays small and the memory stays complete.

  • Semantic recall across the whole library. One vector per second of footage, searched in place. "Find me an angle like this one" answered against every take the studio has ever shot — fast enough to happen inside a single agent turn, so a missing insert might turn out to already exist on a production from two years ago.

  • Sub-agents that delegate. Today the nine run a fixed pipeline. With ADK's multi-agent orchestration the Gate would call specialists on demand — send Continuity back over one scene, ask the Historian a follow-up — rather than every agent running every time.

  • Vertex AI Agent Engine and the Live API. Move the crew onto Google's managed agent runtime, and let the 1st AD ask "can we move on?" out loud, without taking their eyes off the set.

Beyond that: Parallel Monitor webhooks so rain at 2pm reruns the day unprompted, continuity geometry for 180° line crossings, and ingest straight from the DIT's folder so nobody drops a file at all.

Built With

Share this project:

Updates

Submission history