Auteur — The Film Bible Agent

Auteur gives AI filmmakers a persistent Film Bible, grounded by Parallel research, that keeps every generated shot coherent with Gemini Vision drift detection and automatic correction.

Comment

Inspiration

AI video models can already produce one beautiful shot. What they struggle with is remembering that shot when generating the next one: a character's face changes, a jacket switches color, a room rearranges itself.

We kept arriving at the same conclusion: consistency, not quality, is the bottleneck in AI cinema. That's a software-architecture problem, not simply a model-capability problem, since model-level consistency isn't the same as project-level continuity across a whole production.

For our prototype, generation quality was already sufficient. What was missing was a persistent layer that remembers what a film has established and holds every new shot accountable to it. That's the idea Auteur is built around: a Film Bible, a living, structured record of a project's characters, world, style, and continuity that governs every generation instead of sitting inside a prompt as a suggestion.

What It Does

Auteur turns a film concept into a finished, continuity-aware short film through a six-stage agentic pipeline:

  1. Research. A Research Agent uses function calling to decide what to search for, calls the Parallel Search API at runtime as an ADK function tool, evaluates the returned results, and issues follow-up searches when the evidence is insufficient. Every tool call is logged as an agent_tool_call event in the audit trail.
  2. Bible. Research results, together with the story's characters, setting, and tone, are compiled into a versioned Film Bible: the structured source of truth for the production.
  3. Generation. The pipeline generates shots with Veo, using the Film Bible to supply persistent character, wardrobe, world, and style context, while voiceover and score are generated alongside the shots.
  4. Consistency. A vision-based Consistency Check Agent evaluates each generated shot against the project's character reference across face identity, age appearance, facial hair, wardrobe, and a holistic overall score.
  5. Regeneration. When drift exceeds the operating threshold, Auteur feeds the evaluator's diagnosis back into regeneration as targeted corrective guidance, with no human re-prompting required.
  6. Assembly. Approved shots, voiceover, and score are synchronized and assembled into the final film.

In one captured samurai-themed production, the Research Agent made nine tool calls across two research rounds, choosing every query itself, including a follow-up search once it judged the initial results insufficient.

What makes this a loop rather than a pipeline is that the evaluator's output changes what the system does next. Consistency is enforced by the architecture, not requested by the prompt.

The Consistency Loop

Film Bible
    ↓
every generation
    ↓
evaluate
    ↓
drift detected?
    ↓
correct → regenerate

How We Built It

Auteur runs on a Google Cloud-native architecture. Google ADK orchestrates the agents, Gemini handles reasoning, synthesis, and visual evaluation, Veo generates video, Firestore persists project state, Cloud Storage stores rendered artifacts, and Cloud Run hosts the deployed backend. Parallel provides the world-grounding layer of the system, and it's also our hackathon partner.

Parallel is intrinsic to the pipeline, not bolted on. The Research Agent doesn't just call Parallel: it uses Gemini's native function calling to decide what to search for. The parallel_search function is registered as an ADK tool on the Research Agent, which receives the logline, determines what factual information is needed (era, location, fashion, music, architecture), and calls the tool with LLM-generated queries. After receiving results, the LLM evaluates whether the evidence is sufficient or issues follow-up searches:

LLM decides what to research
    ↓
calls parallel_search tool
    ↓
Parallel API
    ↓
results
    ↓
LLM evaluates
    ↓
may search again → Film Bible

This makes Parallel part of Auteur's intelligence architecture rather than a decorative integration: the LLM controls the research trajectory, not Python. If Parallel is unavailable, the Research Agent logs the failure, returns empty refs, and the Director synthesizes the Bible from the logline alone, keeping the partner integration explicit rather than hidden behind a silent fallback.

Auteur's persistence goes beyond the current prompt. A six-layer model covers project state, immutable Film Bible versions, cached research results (24-hour TTL), generated artifacts, generation history, and drift history. The Film Bible itself is versioned in Firestore, and every generation records which Bible version it consumed, making continuity decisions attributable across the life of a project.

The deployed pipeline runs on gemini-3.1-pro-preview for agent reasoning, synthesis, and function calling, veo-3.1-fast-generate-001 for video generation, gemini-3-pro-image for reference imagery, gemini-3.1-flash-tts-preview for narration audio, and lyria-002 for score generation. The ADK agent registry lives in backend/agents/adk_registry.py, with all three agents (Director, Research, Consistency Check) instantiated as ADK Agent objects, and the Research Agent has parallel_search registered as a function tool.

Our consistency scores are model-based evaluation signals, not a formally validated benchmark. The Consistency Check Agent applies a fixed rubric across face identity, age appearance, facial hair, and wardrobe, plus a separate holistic overall score, and we treat these as operational signals for detecting drift rather than objective perceptual or identity measurements. In one demonstrated closed-loop regeneration, the overall score moved from 0.85 to 0.90 after the system incorporated the evaluator's drift diagnosis. What matters isn't the number, it's that the first evaluation changed the context used for the next generation.

Challenges We Ran Into

Making Parallel genuinely agentic. Our first implementation hardcoded the search queries in Python: three static strings passed to Parallel Search. That was technically a runtime integration, but the LLM had no say in what to research. We refactored to use Gemini's native function calling instead, giving the LLM the parallel_search tool as an ADK function so it could decide what to search for based on the logline, call the Parallel Search API, evaluate the returned results, and issue follow-up searches on its own. This required handling multiple function calls in a single Gemini response (Gemini can return six or more function calls at once) and feeding all responses back in a single message to avoid a 400 error.

Building a real audio pipeline. Our first end-to-end render came back completely silent. Three separate defects compounded: generated audio wasn't being persisted correctly, video-only Veo clips had no audio stream for stream-copy assembly, and the frontend player had a stray muted attribute. We rebuilt the assembly pipeline to create per-shot audio segments, mix voiceover with the score, align them to the exact video duration, and mux everything into the final MP4.

Veo reference-image constraints. The Veo tier we initially tested didn't support the reference_images mechanism our character-consistency workflow needed. We switched to veo-3.1-fast-generate-001 and validated the ASSET reference workflow across a four-shot test production.

Lyria reliability. Some Bible-derived score prompts caused Lyria to return content-filter errors. We added retry handling with progressively safer fallback prompts so a single rejected music request wouldn't terminate an entire production.

Firestore compatibility. The Firestore client version available in our deployment environment didn't expose the async interface we'd originally designed around. We switched to the supported synchronous client and simplified several queries to avoid unnecessary composite indexes.

Accomplishments That We're Proud Of

The Film Bible as a primitive. It isn't a feature bolted onto the pipeline, it's the architectural source of truth for every generation. Every shot and regeneration is associated with a specific Bible version, letting the production's creative state persist beyond any individual prompt or session.

A real agentic loop. The evaluator's output changes what the system does next. Drift detection isn't a report a filmmaker reads afterward, it becomes corrective context for regeneration, which then triggers another evaluation: generate, evaluate, decide, regenerate.

Consistency that can improve. A demonstrated regeneration moved the overall consistency signal from 0.85 to 0.90 after the system incorporated the evaluator's drift diagnosis. The improvement matters because it happened through the agentic feedback loop rather than manual re-prompting.

A genuinely agentic partner integration. The Research Agent uses function calling to decide what to search for, calls Parallel as an ADK tool, evaluates the results, and issues its own follow-up searches. In one captured samurai run, it made nine tool calls across two research rounds and chose every query itself, including follow-ups on "Sengoku period architecture shoji tatami castles" and "Sengoku period lighting andon chochin" once it judged the initial results insufficient. Every tool call is logged as an agent_tool_call event.

Deep persistent memory. Auteur persists project state, immutable Film Bible versions, research, rendered artifacts, and drift history in a structure designed for continuity across an entire production, not just a single session.

Parallel woven into the pipeline. Parallel is the world-grounding stage of the intelligence pipeline. Its runtime results become sourced inputs to the Film Bible, which then shapes downstream generation, so the partner contributes directly to the system's creative grounding rather than sitting on the side as an optional add-on.

What We Learned

For our prototype, generation quality was sufficient. Cross-shot consistency became the dominant production bottleneck instead, and we spent more engineering effort preserving what the film had already established than improving any individual shot. That changed how we approached the problem: the answer wasn't a larger prompt, it was persistent state. Once characters, wardrobe, world details, style, voice, and story beats became structured production state, continuity turned into something the system could store, inspect, evaluate, and correct.

We also learned that an agentic workflow becomes substantially more useful once its outputs feed back into its next action. Generating and evaluating independently is useful, but generating, evaluating, correcting, and regenerating is a system. And we learned that making a partner integration genuinely agentic, where the LLM decides how to use the partner's API rather than Python deciding for it, is the difference between a tool call and an agent.

What's Next for Auteur

Longer-form films. The current pipeline is designed and tested around short films. The next major test is whether the same persistent-state architecture can maintain continuity across longer, multi-scene productions: the natural progression from a short film to ten scenes to a feature-length production.

A public Film Bible marketplace. Film Bibles are structured, reusable production assets. We want creators to be able to share and fork Bibles across projects and teams rather than rebuilding continuity from zero.

Deeper Parallel integration. We want to extend the function-calling research loop beyond initial Bible construction, so the agent can issue fresh Parallel searches mid-production whenever a new creative decision needs additional grounding.

Independent evaluation. Adding an evaluator outside the Google model ecosystem would cross-check consistency judgments and provide a more independent validation signal.

AI filmmaking doesn't only need better generators. It needs memory, grounding, and control. Auteur provides that missing layer: Parallel grounds the world, the Film Bible remembers the production, Gemini orchestrates and evaluates, Veo creates the shots, and the agent corrects what drifts.

One Film Bible. Every shot belongs to the same film.

Built With

Share this project:

Updates

Submission history