Lumora

every creator is a studio. Give a crew of specialist AI agents one prompt and they write, cast, storyboard, generate, score, and self-critique a finished film — drama, ad, animation, or explainer

Comment

Lumora — an autonomous AI film studio

Give it one logline, pick a format and a look, and a crew of specialist agents makes the film while you watch. Not a timeline. A studio.

Inspiration

I'm the guy who stays up too late watching AI-generated films — especially the apocalyptic ones. Drowned cities, last-transmission monologues, a lone figure walking through ash. I'd watch them and think: a model imagined this entire world from a sentence. The raw creative power was obviously there.

But then I'd switch hats. As a developer I constantly need to make things that aren't movies at all — pitch videos for a project, explainers for a feature, tutorials for a tool I just shipped. And every single time I hit the same wall: the tooling. Adobe Premiere, After Effects, CapCut — timelines, keyframes, render queues, manual cuts. I don't want to edit a video frame by frame; I want to describe one and have it exist. The model could already dream the film. The tools still made me assemble it by hand, clip by clip, like it was 2005.

That's the gap Lumora attacks: the imagination is solved; the assembly line isn't. So I built the assembly line.

What it does

You give Lumora one logline, pick a format (drama, ad, animation, explainer, music video) and a look — and an autonomous crew of specialist agents produces the finished film while you watch from a "director's room" instead of a timeline.

Under the hood, a crew of role-specialized agents — Showrunner, Writer, Casting Director, DP, and a Critic — hands the project down a 10-stage pipeline from concept to delivered video. Along the way Lumora:

  • Writes a story, not a synopsis — the Writer is driven by a film-school structure (a forced reversal + an earned button) and an adversarial review pass.
  • Keeps the cast on-model — one locked character descriptor anchors every shot, so faces stay consistent from the first frame to the last.
  • Self-checks before it spends — a vision Critic QCs every keyframe before paying for video, and every clip after, keeping the best take.
  • Lets you co-direct in plain English — notes like "make the ending hopeful" or "upbeat music" regenerate only the affected stage; everything upstream stays cached and free.

How we built it

A logline flows through a durable, Postgres-backed state machine across 10 stages:

$$ \text{CONCEPT} \to \text{SCREENPLAY} \to \text{CASTING} \to \text{STORYBOARD} \to \text{SHOT PLAN} \to \text{GENERATION} \to \text{AUDIO} \to \text{ASSEMBLY} \to \text{DELIVERABLES} $$

Every stage is crash-resumable and backed by a content-addressed step cache. Each step's output is keyed by a hash of its inputs:

$$ h \;=\; \mathrm{SHA256}\big(\, \text{stage} \,\Vert\, \text{inputs} \,\Vert\, \text{params} \,\big) $$

so resume, re-render, and remix never re-spend tokens — a cache hit returns instantly and for free.

The cost governor. Track 2 capped the token budget, so I made the budget the architecture. Every model call passes through one gateway that pre-checks a reserve and writes a cost-ledger row. Routing chooses the cheapest model that clears a quality bar \(q_{\min}\):

$$ m^\star \;=\; \arg\min_{m \,\in\, \mathcal{M}} \; c(m) \quad \text{s.t.} \quad q(m) \ge q_{\min}, \;\; \textstyle\sum_i c_i \le B $$

Max-tier models are reserved for the writing; routes downgrade gracefully under pressure. The analytics panel then reports the metric that actually matters — cost per finished second:

$$ C_{\text{sec}} \;=\; \frac{\sum_i c_i}{T_{\text{final}}} $$

The self-critique loop. A Qwen vision Critic gates generation at two checkpoints: it QCs every keyframe before I pay for video, and every clip after. Failures retry with critique-informed prompts; the system keeps the best take and never deadlocks:

$$ \text{take}^\star \;=\; \arg\max_{k \le K} \; \mathrm{score}_{\text{critic}}(\text{take}_k) $$

The consistency moat. One locked character descriptor is simultaneously the image prompt, the R2V video reference, and the Critic's identity anchor — so a cast stays on-model from the first shot to the last.

Shot routing sends each shot to the right model — Wan 2.7 R2V for character dialogue, HappyHorse for establishing shots, first-last-frame interpolation for continuity bridges. Shots generate in parallel, so wall-clock is the slowest single shot, not the sum:

$$ T_{\text{wall}} \;=\; \max_i t_i \;\;\ll\;\; \sum_i t_i $$

The stack: Python · FastAPI · a hand-rolled durable state machine on Alibaba Cloud ECS; Qwen3-Max/Plus, Qwen vision, Qwen-Image, Wan 2.7 R2V, HappyHorse, CosyVoice via Qwen Cloud / Model Studio; OSS + FFmpeg for media; a Next.js "director's room" on Vercel. The whole pipeline also runs end-to-end offline with no API key via a record/replay layer that synthesizes placeholder media and writes real cost rows — then flips to live with one flag.

Challenges we ran into

  • Video tasks are hostile to durability. Async generation jobs fail, and their result URLs expire in 24 hours. The answer was immediate OSS persistence the moment a clip lands, plus the resumable state machine so a mid-pipeline crash costs nothing.
  • Ordering bugs in audio + lip-sync. Multi-character lip-sync and "voice-before-generation" sequencing had subtle ordering bugs that only surfaced under parallelism. Careful per-shot session isolation fixed the races.
  • Narrative quality was the hardest non-engineering problem. Getting a model to write a story — not a synopsis — took a full multi-agent design workshop and an adversarial film-school review folded into a single prompt. Pixels were easy; plot was the grind.
  • Making "edit it" free. I wanted co-direction in plain English ("make the ending hopeful," "upbeat music") to regenerate only the affected stage. Rather than build a bespoke diffing engine, I leaned on the content cache — the note changes one stage's hash, everything upstream stays cached and free.

Accomplishments that we're proud of

  • Cost per finished second as a first-class metric. Every model call is a budgeted transaction through one gateway, so the system reports exactly what a finished second of film costs — and stays under budget by routing, not by panicking.
  • Free regeneration and co-direction. Content-addressed caching turned "regenerate" from a scary cost event into a free operation, and minimal-regeneration co-direction fell straight out of it as an unplanned feature.
  • A self-critique loop that never deadlocks. The Critic gates spend at two checkpoints, retries with critique-informed prompts, and always keeps the best take instead of stalling.
  • A consistency moat. A single locked character descriptor keeps a cast on-model from the first shot to the last across image, video, and QC.
  • Parallel generation. Shots render concurrently, so wall-clock is the slowest single shot rather than the sum of all of them.
  • Crash-resumable by design. A durable, Postgres-backed state machine means a mid-pipeline failure costs nothing — the run picks up exactly where it left off.
  • Runs fully offline. A record/replay layer drives the entire pipeline with no API key and real cost rows, then flips to live with one flag — which made development and demos painless.

What we learned

  • A prompt chain is not a crew. My first instinct was one big prompt that does everything. It produced mush. The leap was treating each role — Showrunner, Writer, Casting Director, DP, Critic — as a separate agent with its own structured output contract and a shared budget. Specialization beat generality, hard.
  • Narrative quality is a prompt-engineering problem, not a model problem. The single biggest quality jump didn't come from a bigger model — it came from baking a film-school structure (a forced reversal + an earned button) and an adversarial review pass into the Writer's prompt. The model could always write well; it needed to be told what good looks like.
  • Cost discipline is an architecture, not a setting. Once I modeled every model call as a budgeted transaction, the whole system got simpler. Budget pressure became a routing decision instead of a panic.
  • The cheapest token is the one you don't re-spend. Content-addressed caching turned "regenerate" from a scary cost event into a free operation — and a feature I didn't plan (minimal-regeneration co-direction) fell straight out of it.

What's next for Lumora

  • Longer-form films. Push past short clips toward multi-scene arcs with act-level structure and persistent through-lines.
  • More formats and looks. Expand the format/look library and let users save and share their own style presets.
  • Real-time co-direction. Make the director's room interactive — steer scenes as they generate, with shot-level manual overrides on top of plain-English notes.
  • Richer audio. Add scored, mood-aware music and broaden multi-language voice work through CosyVoice.
  • A collaborative studio. Open the director's room to teams so multiple people can co-direct a single production.
  • Even tighter routing. Add more model backends and sharpen the quality/cost routing so every finished second gets cheaper without losing the narrative bar.

Built With

Share this project:

Updates