Architecture

Cinematic Compiler — Devpost Submission Writeup

Inspiration

Video generation models can produce a stunning 5-second clip — but a film is not a clip. The moment you ask for a second shot, the protagonist's coat changes color, the scar switches cheeks, and the alley becomes a desert. Every multi-shot AI video today quietly depends on a human quality-checking every frame. We wanted to build the missing layer: a production studio where AI agents don't just generate — they check each other's work and fix it, the way a real film crew does.

What it does

Cinematic Compiler turns a one-line prompt into a finished short film — screenplay, storyboard, design bible, reference images, per-shot frames, animated clips, and a final MP4 with audio — through a pipeline of 14 specialized agents:

Director distills the prompt into tone, visual style, and hard rules the whole film must obey.
Screenwriter and Storyboard agents produce scenes and shots, each with camera direction and a must_preserve continuity contract.
A Design Bible of 10 department agents (character, costume, scene, props, cinematography, colorist, editor, audio, musician, dialogue) locks the film's look before a single pixel is generated.
A deterministic continuity graph tracks every character, prop, and location across shots.
Reference and shot images are generated with the Wan text-to-image models.
A vision audit agent (qwen-vl-plus) inspects every rendered shot against the director's hard rules and the shot's continuity contract, scoring it 0–1. Below 0.75 — or on any critical error — a repair agent writes a targeted corrective prompt and the shot is re-rendered and re-audited. The film ships with its own QA report.
Assembly animates consecutive shot pairs with the Wan image-to-video models and concatenates the clips with FFmpeg, keeping Wan's auto-generated audio.

The studio is interactive: a canvas UI shows the agent graph building the film live. Click any agent to send a producer note ("make it more noir") — that stage re-runs with your feedback and downstream stages cascade automatically. Users can also inject custom agents (full-context, override, or post-process) at any pipeline position, and run the whole pipeline stage-by-stage in manual mode.

What makes the architecture different

Multi-agent orchestration without coupling. 14 agents (4 Qwen-backed creative leads + 10 deterministic department agents) each consume only the context they need and write a typed section of shared job state. Agents compose cleanly; every output records which model produced it.
A repair loop, not a retry loop. The vision audit doesn't just flag a bad shot — it produces a structured error list (critical/major/minor), and the repair agent converts those specific failures into a corrective prompt. The shot is re-rendered and re-audited until it clears the 0.75 threshold or hits the attempt cap.
The continuity graph as a single source of truth. Characters, props, and locations are extracted into a deterministic graph before generation. The same graph feeds the image prompts and the audit checklist — the generator and the judge are grounded in identical facts, which is what makes the audit scores meaningful.
User-extensible agent system. Custom agents slot into any pipeline position with three insertion semantics (full-context, prompt-override of a built-in agent, or post-processing a stage's output) — no code changes, rendered live as graph nodes.
Surgical regeneration. Every stage declares its downstream dependencies in an explicit dependency map. A producer note to one node re-runs only that node and its true dependents: changing SHOT_03 never touches the screenplay; changing the design bible cascades to references and shots because identity genuinely depends on it. One node changes, the rest of the film stands.

How we built it

Backend: FastAPI + Uvicorn, JSON-file job store, threaded background runners. Deployed on Alibaba Cloud ECS via Docker.
Frontend: React (JSX) canvas graph UI served by the same process; Streamlit as an alternative UI.
Models (all via DashScope International): qwen3.5-flash for structured JSON generation, qwen-vl-plus for vision audits, wan2.1-t2i-turbo for images, wan2.1-i2v-plus for video, qwen3-tts-flash for TTS.
Assembly: FFmpeg concat demuxer.

Engineering highlights

Working against real quota limits forced production-grade resilience:

Model fallback chains for every capability. DashScope free-tier quotas are per model, so text, image, and video calls each walk an ordered chain of models on quota errors instead of failing.
Multi-key rotation. A comma-separated key pool; a key is only abandoned after its entire model chain is exhausted.
Per-model payload construction. wan2.7 takes first+last keyframes; wan2.1 i2v takes a single start frame — the client builds the correct request body per candidate model.
Runtime-failure resubmission. DashScope can accept an async video task and fail it minutes later; the assemble stage detects this and resubmits down the chain.
Deterministic offline fallbacks for every agent, so the pipeline always completes end-to-end.
Token-budget discipline: flash-tier models, JSON mode with schema validation and single-retry, compact context assembly per agent.

Every output records which model actually produced it — full provenance in the job file.

How we used Alibaba Cloud

Model Studio / DashScope International powers every AI capability in the system: qwen3.5-flash (structured JSON generation for the creative agents), qwen-vl-plus (vision continuity audits), wan2.1-t2i-turbo (reference and shot images via the async task API), wan2.1-i2v-plus (shot-to-shot video clips via the async task API), and qwen3-tts-flash (dialogue TTS). The complete integration is one auditable module: backend/services/qwen_client.py.
ECS hosts the production backend and frontend as a Docker deployment (Dockerfile + docker-compose.yml in the repo), with ASSET_BASE_URL pointed at the instance's public address so generated images and videos are served from the cloud host.

Challenges we ran into

The hardest bugs were the silent ones. DashScope accepts a malformed video request at submit time and only fails the task at runtime — which once degraded our final cut to a still-image slideshow with no error anywhere. Qwen would also occasionally return continuity constraints as a comma string instead of a JSON array, or a character's appearance as a nested dict — the pipeline normalizes all of it. And the compatible-mode endpoint rejects JSON-mode requests unless the word "json" literally appears in the prompt.

What we're proud of — continuity you can measure

The audit→repair loop is a genuine self-correcting system: the film's continuity is measured, not hoped for. Every job records its own QA report — mean continuity score before and after repair, how many shots entered repair, and the repair success rate — with the full per-shot audit history (every attempt, score, and error list) preserved in the job file, so the improvement delta is reproducible from the artifact alone.

Across 24 generated films — 88 shots, 311 qwen-vl-plus audit calls — the repair loop lifts repaired shots from a mean continuity score of 0.616 to 0.643, with a mean gain of +0.108 on the shots it improves, against a strict grader (−0.15 per critical error, 0.75 approval threshold). Attempt-level analysis of the audit histories also identified our next optimization: 37 of 79 repaired shots produced a stronger intermediate render than the final one, so keep-best attempt selection raises the aggregate gain to +0.052 at zero additional generation cost.

We deliberately don't quote standard long-video benchmarks: those score the underlying generation model on single continuous shots, and would measure Wan rather than our system. Our contribution is cross-shot consistency, so our evaluation is an ablation — the same prompts with and without the repair loop, comparing continuity scores and cross-shot character identity similarity (CLIP/DINO embedding similarity of the protagonist across shots).

What's next

Reference-image conditioning for stronger identity locking, longer multi-scene films, and publishing aggregate ablation numbers across a multi-prompt evaluation batch.