ScenePilot

An AI 1st assistant director that plans, but also reschedules a rained-out shoot day, and rejects the options a real permit, statute or sunset will not allow

Comment

Inspiration

A shooting day is one of the most expensive and least forgiving logistical objects in any industry. A hundred people, a permit that opens at one o'clock, an actor who is contracted for eleven more hours, a sun that sets when it sets. And then it rains at 13:00, and a 1st assistant director has about four minutes to answer: what's still shootable, what does moving it cost, and is any of it actually legal?

The honest answer today is a whiteboard, a phone, and thirty years of experience. We wanted to know whether an agent could stand in that gap — not by writing a nicer schedule, but by being able to say no, and say why, with a citation.

That question turned into the thing we actually care about:

A cited statute, pulled off the open web at runtime, becomes a hard constraint that rejects a schedule.

Not a suggestion. Not a warning banner. A rejection, traced back to the sentence in the municipal rules that caused it.

What it does

ScenePilot is a production control room. It ingests a screenplay (Fountain or Final Draft), breaks it down with Gemini into 32 element categories, builds a Day-Out-Of-Days cast matrix, and lays the scenes onto a stripboard against a solar ephemeris and a pluggable union rule pack.

Then reality moves, and the interesting part starts.

Report rain over Shoot Day 4 in Mumbai and ScenePilot verifies the report against the outside world before it touches anything — Parallel Search queries the India Meteorological Department, and every source it used stays inspectable. A deterministic engine then evaluates the orderings and rejects the ones that don't survive contact with reality:

  • Hold the schedule — rejected: Scene 48 is an exterior with sixty background artists sitting in the rain for 150 minutes, and Scene 42, the rooftop motorcycle jump that follows it, carries a wet surface as an explicit stop condition.
  • Shoot the market street in the morning — rejected: the police permit doesn't open until 13:00.

What survives is priced before anyone commits: ₹67,500 and 60 minutes of overtime for the recommended option — ₹60,000 of carry-over plus one hour at that day's own ₹7,500 crew rate — and the part nobody wants to hear. Days 5 and 6 are night units and the market permit only exists on Day 4, so the deferred scene costs a ₹3,50,000 dedicated pickup day.

Two nights later, on the night unit, the differentiator lands. A Parallel Task dossier on the rooftop location returns Mumbai's real noise curfew — 22:00–06:00 — carrying the statute it was read from, its reasoning, and a calibrated confidence. High confidence plus a citation binds it as HARD. The producer accepts it, and a recovery option turns red where it stands, with the statute's own sentence marked inside the citation it was parsed from.

Nothing is applied without a producer. When one approves, the board re-lays itself, the call sheet reissues white → blue, and the whole decision — who decided what, on which evidence, at what time — is on the production log.

How we built it

Gemini proposes. Code decides. This is the architectural spine, and it's a deliberate constraint rather than a limitation.

Google ADK's Gemini agents handle everything semantic: scene understanding, generating research questions, judging whether evidence is sufficient, proposing recovery manoeuvres, and writing the rationale a human actually reads. Deterministic Python handles everything consequential: interval arithmetic, availability matrices, permit windows, penalty math, ephemeris, cost, and ChangeSet application. No schedule ScenePilot shows you was decided by a language model.

Both orchestrators execute as google.adk.workflow.Workflow graphs — not hand-rolled control flow, and not the deprecated SequentialAgent/LoopAgent primitives. Planning runs breakdown → research_plan → research → evidence ⇄ follow_up → plan, where the follow-up loop is a genuine routed cycle, bounded and conditional on the analyst having asked for specific follow-up queries. Rescue is terminal at awaiting_approval — there is deliberately no node downstream of the producer. GET /api/agent-graph serialises the running graph objects and the UI draws the pipeline from that response, so the diagram in the app cannot drift from the pipeline that ran.

All six Parallel APIs are called at runtime, each doing the job it's actually good at:

API What it grounds
Search Verifying a reported disruption before the schedule is touched
Extract Reading the page behind a claim instead of trusting a snippet
Task Location dossiers and hourly weather, per-field citations, reasoning and confidence
FindAll Real replacement vendors when a booked resource falls through
Memory What this production already learned, scoped per project
Monitor Watching a location's facts for change between planning and shooting

Every call writes a run row — SearchRun, ExtractRun, TaskRun, FindAllRun — holding the exact request, the queries as sent, the sources returned and the usage Parallel reported. Nothing expensive fires implicitly: Task, FindAll and Memory sit behind flags, reachable only from a named button that shows cost and latency before the click.

Even the demo video is built this way — it's rendered from a plan committed to the repo, driving the hosted app with Playwright, so a re-record is a command rather than an afternoon.

Challenges we ran into

Every one of these is a real defect we shipped and then had to find. The pattern behind them turned out to be the most useful thing we learned.

A hardcoded value contradicting a computed one — three times, and the third was inside the engine. ShootDay.golden_hour_dusk carried a class default of 17:45–19:15. The ephemeris said 18:25–19:08 for the same day, and the page printed both. That would have been cosmetic, except the validator read the default — so options were being accepted and rejected against a sun that did not exist. The same shape turned up twice more: a labor pack whose fallback constants were FWICE's numbers while the API reported DGA_SAG, a North-American guild pack named over a Mumbai shoot; and a Day-Out-Of-Days that priced every performer's hold day at a flat ₹25,000, which made a lead's idle day and a stunt double's cost the same money.

A recording key that drifted daily. Fixing the golden hour put a date-derived value into the hash keying our record/replay fixtures, so every rescue recording died — and a fresh paid re-record would have gone stale within 24 hours. The fix was to normalise solar readings by label rather than by value, proven in both directions: the same question a week apart keys identically, and eight genuinely different mutations still key differently.

A map that was invisible on every real database while every test passed. We added real Mumbai coordinates to the seeded locations. Tests build the project fresh, so they all passed. But _ensure_seed only creates a project when it is missing — on any deployment with a persistent database the rows were already there, without the new fields, and the company-move map rendered nothing. There was no error anywhere. That produced seed/migrate.py, a deliberately narrow forward-migration seam that backfills a field only where the stored value is still the model's class default, and refuses loudly for fields it cannot make that judgement about.

A downstream solver that answered its own feasibility question. The multi-day ripple solver checked the hard wrap and nothing else, so it placed a deferred EXT. MARKET STREET — DAY on a night unit at 22:00–24:30, at a location whose permit exists on one other day, and printed it to the producer in green as FEASIBLE. Its cast-availability check rejected only a zero-length booking row, so it could not fail in practice. The fix was to delete the private checks and call validate_schedule — the one engine every other schedule is ranked by.

An audit trail written and never read. Twenty-eight sites across the API and the workflows log an event at the moment they act, and the endpoint returning all of them had existed for weeks. Nothing in the UI called it. The single most important thing a producer needs from a system that rewrites their schedule — who decided what, on what evidence, when — was complete in the database and on no screen. Worse, the run-level feed that did render events had no entry for the decision kind, so a producer accepting a cited statute as a hard constraint displayed in the same grey as a database migration note.

A documented command that spent money. Our README told contributors to run pytest. The repo-root .env sets SCENEPILOT_MODE=live and SCENEPILOT_RECORD=1, so the documented command made real paid API calls and wrote fresh files over our committed fixtures. conftest.py now forces replay and no-record and asserts it, so the documented command is the safe one.

What we learned

A plausible number is worse than a blank. Four panels we built looked convincing and were fiction: a doppler radar that showed rain for any disruption type, including equipment failures; an on-set HUD whose golden hour contradicted the app's own ephemeris a hundred pixels below it; a convoy map inventing addresses the seed did not contain; and a Parallel console that was hardcoded end to end, printing gemini-2.5-flash beneath a header showing the model we actually run. We deleted all four and rebuilt three on real state. The rule we ended up with — every value on screen traces to an API response, the seed, or a deterministic computation over those; a row that cannot be sourced is deleted, not filled — is the single most useful constraint we adopted. The insurance packet is the clearest case: five fields a real claim needs and we genuinely do not hold (policy number, insurer, insured daily production cost, deductible, notice deadline) render as named blanks with a reason. A document shaped like evidence, with invented numbers in it, is worse than no document.

Where a value lives determines whether it can lie. Almost every bug above is one value existing in two places. The repair was always the same: make one of them the source and have everything read it. One active_pack(project). One city_ephemeris. One colour mapping shared by both boards. One department_for shared by the coordination engine and the call sheet's radio plan. The activity vocabulary is now sent by the API rather than mirrored in the client, for exactly this reason — a client-side copy is a copy that will drift.

Tests that build fresh state cannot see migration bugs. This is the lesson we would most like to carry to the next project. We wrote a great many tests during this build, and none of the new ones would have caught the invisible map, because they all construct a project in memory. The class of bug that only exists on a database somebody wrote months ago needs its own kind of test — and we now have them, run against a copy of a real stale database.

Deterministic refusal is a feature, not a limitation. The instinct with an LLM in the loop is to let it decide. What makes ScenePilot trustworthy is the opposite: Gemini proposes orderings and writes the rationale a human reads, and code decides what is shootable, with the violated constraint named in minutes against a real permit window or a cited statute. The most persuasive screen in the whole product is the one showing options the engine rejected — and it is persuasive precisely because no language model could have talked it out of them.

What's next for ScenePilot

  • Real productions, real paper. The parsers, call sheets, daily production reports, movement orders and sides packets are already shaped like the documents a unit carries. The next step is ingesting a real breakdown from a real production office.
  • International rule packs. DGA/SAG-AFTRA and FWICE/CINTAA ship today; BECTU and MEAA are the same shape.
  • Constraint monitoring as a standing service. Snapshot monitors already re-check a location's dossier on a schedule and report only the fields that moved. Pointed at a whole slate rather than one production, that becomes an early-warning system for permits, curfews and airspace rules changing under a schedule that was locked months ago.

Built With

Share this project:

Updates