Shotstead

Don't prompt the shot. Stage it. Block once, change the camera, and preserve continuity with proven Stage-to-Take lineage.

Comment

Inspiration

Generative video is getting better at creating individual shots. But filmmaking is not a sequence of unrelated prompts. A director needs the next shot to still belong to the same world: the same performer, props, blocking, motion, and production state, with only the intended changes.

When every angle is regenerated from scratch, continuity becomes fragile and it becomes difficult to answer a basic production question: exactly which world state produced this Take?

Shotstead starts from a different premise:

Don't prompt the shot. Stage it.

Instead of making the prompt the primary object, Shotstead makes a persistent, structured Stage the object that filmmakers and agents work around.

What it does

Shotstead is a persistent virtual stage for agentic filmmaking.

A Stage contains the room, performers, props, motion, and camera state. The same Stage is projected into three synchronized views:

  • a 2D Stage for blocking;
  • a 3D Director View for understanding the whole scene;
  • a Viewfinder for seeing what the camera sees.

A filmmaker can capture Take A from committed Stage v1, move only the camera, and commit a camera-only child Stage v2. Shotstead then proves that the performer, prop, environment, and motion are unchanged before capturing Take B from v2.

The result is not just two clips. It is explicit lineage:

Take A -> Stage v1 -> camera-only Stage v2 -> Take B

Every Take remains tied to the exact committed Stage version that produced it.

Shotstead also has an agentic execution path. Gemini on Vertex AI coordinates bounded Google ADK tools to read Stage state, validate it, and propose a camera change. Canonical Stage mutation is deliberately excluded from the model tool surface: an explicit human approval is required before the application can commit Stage v2.

Execution evidence is recorded as append-only production history and projected to ClickHouse Cloud. Gemini can then query that production ledger through the official read-only mcp-clickhouse==0.4.1 runtime and answer provenance questions such as which committed Stage version produced the latest successful Preview Take and whether it had warnings.

ClickHouse is the production ledger, not a second source of Stage truth.

How we built it

Shotstead is written in TypeScript with React and Three.js.

The core is a structured StageState contract and a versioned Stage Repository. Stage versions preserve parent lineage and reject stale writes. Camera-only diffs are checked explicitly so continuity can be proven rather than assumed.

The browser runtime renders the same Stage into the 2D Stage, 3D Director View, and Viewfinder. Preview Takes are captured from the real browser canvas using MediaRecorder.

For the agent layer, Shotstead uses Gemini on Vertex AI with Google Agent Development Kit tools. The model receives a bounded tool surface for reading, validating, and proposing changes, while canonical mutation remains application-controlled behind explicit human approval.

For the ClickHouse partner integration, runtime execution evidence is projected into an append-only ClickHouse Cloud production ledger. The official mcp-clickhouse==0.4.1 server is launched as a read-only MCP surface, and Gemini queries it through the Google ADK MCP toolset.

The public application is packaged with Docker and deployed on Google Cloud Run. The repository also includes deterministic unit tests, Chromium end-to-end tests, Docker smoke tests, and opt-in live integration gates for Vertex AI, the browser Runtime, ClickHouse Cloud, and MCP.

Challenges we ran into

The hardest problem was not rendering a 3D room. It was defining clear authority boundaries.

The Stage Repository must remain the source of current world truth. The browser Runtime should execute a Stage, not silently mutate it. ClickHouse should preserve production history, not become another competing state store. Gemini should coordinate tools without gaining the ability to approve its own canonical changes.

Another challenge was making continuity measurable. Rather than claiming that two shots came from the same world, Shotstead computes the Stage diff and verifies that a child version changes only the camera while the environment, performer, prop, and motion remain unchanged.

We also wanted the agentic workflow to use real execution paths rather than mocked integrations. That meant validating real Vertex AI calls, explicit human approval, real Chromium rehearsal and MediaRecorder capture, real ClickHouse Cloud writes, and a real Gemini provenance query through official MCP, while keeping the normal public CI completely credential-free.

Accomplishments that we're proud of

Shotstead now demonstrates a complete same-world, new-camera workflow:

one persistent Stage -> Take A -> camera-only Stage v2 -> continuity proof -> Take B

The browser can capture real Preview Takes and bind them to committed Stage versions.

A real Gemini model on Vertex AI can inspect the Stage and propose a bounded camera-only change through Google ADK without receiving canonical commit authority.

After explicit human approval, the application can commit the child Stage, execute it in the real browser Runtime, and record append-only production evidence.

That evidence can be projected to ClickHouse Cloud, where Gemini can query production provenance through the official read-only mcp-clickhouse runtime.

Most importantly, the technical architecture supports a filmmaking idea that can be understood visually: same stage, new camera, proven lineage.

What we learned

Agentic filmmaking becomes easier to reason about when agents operate on durable production state instead of reconstructing context from prompts.

A useful filmmaking agent does not need unlimited authority. Narrow tools, explicit state, human approval, immutable lineage, and queryable production history can make an agent more useful precisely because its boundaries are visible.

We also learned that provenance can be part of the creative workflow rather than an after-the-fact log. Knowing which Stage produced a Take makes experimentation safer: filmmakers can change one thing, shoot again, and still know exactly what remained the same.

What's next for Shotstead

The next step is to deepen the Stage rather than replace it.

Shotstead can grow toward richer camera and lens controls, lighting, multiple performers and props, multi-camera coverage, timeline-aware blocking, reusable Stage templates, and interchange with established production tools.

Generative video systems can become downstream execution providers for committed Stage versions instead of being asked to reconstruct the world from scratch for every shot.

The longer-term goal is a production environment where humans and agents can explore more creative variations while preserving the one thing filmmaking repeatedly depends on:

a world that still exists when the next shot begins.

Built With

Share this project:

Updates

Submission history