Storyboard Studio Director

Nemotron writes the film as code; a deterministic engine renders it and measures it. Type a story in any language, get an acting, voiced, scored short film. No image generator.

Comment

▶ Demo video: https://youtu.be/aNHZgw2Jo80 · Try it: https://studio-production-049c.up.railway.app (passcode in the Judge Access file) · Showcase and replays, no passcode: https://studio-production-049c.up.railway.app/showcase · Code (MIT): https://github.com/vansyson1308/imageforagent

Inspiration

Daily storytelling creators, teachers and small NGOs need short animated films in their own language every day. Image generators give inconsistent characters, unclear rights and output you can't edit. So we asked: what if the model didn't paint pixels at all, but wrote the film as code for a deterministic engine, and then checked the rendered result the way a director would?

In v2 the question got a concrete user. The owner runs a Japanese YouTube channel, 「ひだまり人生劇場」 (Hidamari): narrated stories for senior viewers, hosted by Haru-san in a Showa-era kissaten. A daily show needs the same host in the same set every day, in the channel's own look, narrated by the channel's own voice track.

What it does

Type a story in English, Vietnamese or Japanese and press Make my film. A crew of NVIDIA Nemotron models on Nebius Token Factory makes it:

  • Ultra (the Director) plans the shots with real coverage and writes a cast & set bible in strict JSON.
  • Super designs the characters as engine kits and writes every frame as SVG code.
  • The engine renders each frame and measures it (subject size, framing, empty or flat sets, faces covered, figures stacked, near-duplicate shots) and sends any failure back as a measured fix.
  • The critic: a vision model describes what it sees and Nano scores the shot against the plan. A revision replaces a frame only when it scores higher.

Characters act (poses, blinks, lip sync, walks). The engine adds voices, dissolves and an original synth score. You get an MP4 plus the full trace: every model call with its role, model, tokens, cost and latency.

A shot that still misses is redrawn on its own, on the same plan and cast, while every other shot stays byte-identical. The public showcase holds three v2 films (English, Vietnamese, Japanese 9:16). Each passed the measured gates and a by-eye checklist signed by the owner and QC, and each can be replayed step by step from its stored trace without spending a token.

How we built it

  • One rule: the model writes, the engine measures. Every model output passes the same validators a person's would (zod schemas → an SVG sanitizer → the engine's construct and motion schemas) before it can touch the film. Then the render is measured, and numbers are never left to a model's arithmetic.
  • Engine: Next.js 16 + TypeScript, Prisma 7 on SQLite, sharp/librsvg for rendering, ffmpeg for the cut. The construct engine (hand-written 3D math, CSG, exact depth sort) and the motion engine (time as a pure function of the spec, a no-slip walk rig, IK, lip sync) are pure and deterministic, so a remaster re-renders the same drawing at another resolution or frame rate with no model call.
  • Crew: one OpenAI-compatible client for every role on Token Factory; strict json_schema replies; thinking switched off per call where latency matters; the live /v1/models catalog resolves each role and is reported by the public health endpoint.
  • Runs stream to the browser over SSE inside the request (no job queue). Every model call is a recorded step, and the showcase replays those steps on a virtual clock, including every run a film took.
  • Kits instead of free drawing where identity matters: people, animals and sets are small JSON specs the engine builds, so a character is pixel-identical in every shot by construction.
  • Publishing is gated: measured gates first, then a by-eye checklist (owner + QC) per film. The gates, the checklists and all 95 decisions are in the repo: https://github.com/vansyson1308/imageforagent/blob/main/docs/hackathon/DECISIONS.md
  • Tests: 833 passing vitest tests, including a sanitizer bypass-vector suite, byte-identical double compiles of the construct examples and the Khronos glTF validator.

Results (real runs)

Bench v2: 30 runs on one frozen commit (394a797), 10 fixed prompts, three configs. Same prompts and judge prompt as v1. The judges were blind to the config, and neither is part of the crew. Full report: https://github.com/vansyson1308/imageforagent/blob/main/docs/hackathon/EVAL_RESULTS.md

  • Against v1, on the v1 judge (gemma-3-27b-it): the full crew scored 6.22 / 10 (n = 9, SD 0.54), up from v1's 5.49. Paired on the same 9 prompts: +0.77, better on 6, worse on 2, tied on 1.
  • Against Nemotron Super alone (paired, gemma): better on 7 of 9 prompts (mean +0.68). Caveat: Super alone planned a single shot in 8 of its 10 runs, so this is not like for like.
  • A second, stricter judge (Kimi-K3), reported in full: crew 4.38 (SD 0.79), Super alone 4.24. Paired: better on 4, worse on 3, tied on 2. On this judge the gap to Super alone is not clear.
  • Critic uplift, judged independently: +0.5 over 13 revised shots. ✅ Target met.
  • Missed targets ❌: judge mean 6.22 against ≥ 6.5; paired wins 7/9 against ≥ 8/10; $0.57 per finished minute against ≤ $0.40 ($0.18 per finished film); the Tavily research config did not improve scores (gemma 5.65) and failed 3 of 10 runs (one cast-symbol render error, fixed since; two hit the 840 s per-run cap).
  • Failures: 1 of 10 for the crew (the same render error), 0 of 10 for Super alone.
  • Every published film then passed a check by eye, signed by the owner and QC.

Challenges we ran into

  • Composition, not code. Super's SVG passed the sanitizer almost every time. The problems were framing, subject size and night shots lit bright. Prose rules didn't fix these. Measured feedback with the exact numbers did, and so did new measured gates: edge crop, night readability, faces covered and figures stacked inside each other.
  • The hero object has to read. A kite or a letter that looks like a blob ruins the story. The hero object is now checked on its own by a vision model, by its visible features, and the engine draws the object's states (torn, patched, opened) on the approved drawing.
  • Time of day. The critic called golden dusk "day" and dawn "night". Time is now judged in buckets, and a finished run can be re-judged from its stored verdicts with no new model call.
  • Finishing a film without starting over. A redo run redraws only the named shots on the take's plan and cast. QC by eye still found things the gates missed, such as a vanished walker and a floating envelope. Each became an engine fix with a test.
  • Measuring honestly. When a merge deployed a new commit mid-bench, we rolled the server back so all 30 rows came from the same code.

Accomplishments that we're proud of

  • Three published v2 films in English, Vietnamese and Japanese (9:16), each through every measured gate and a signed by-eye checklist, each replayable from its stored trace.
  • A 30-run bench on a frozen commit with two blind judges that reports its missed targets as missed.
  • 833 tests, including a sanitizer bypass-vector suite.

What we learned

  • Numbers belong in the engine, not in the prompt.
  • A second judge can change the story: Kimi-K3 shows no clear gap where gemma shows 7 wins in 9.
  • A film is finished when a person signs it, not when the critic's score clears a line.

What's next for Storyboard Studio Director

  • Cost: get under the $0.40 per finished minute we missed.
  • Vision: move the looking step to a Nemotron VLM once one is served.
  • Hidamari Shorts: make the episodes from the owner's narration WAVs as a series that reuses the same host pixel for pixel.

Real users

The first real user is the owner's Japanese YouTube channel 「ひだまり人生劇場」 (Hidamari). Its first package, 「出せなかった手紙」 (the letter that was never sent), became the Japanese showcase film 「ひだまり人生劇場・春の手紙」:

  • Format: 9:16, 7 shots, 47.9 s with credits.
  • Narration: fixed-narration mode. The owner's AivisSpeech pipeline rendered every line before the run, and the Director timed each shot to those WAVs without rewriting a line.
  • What it took: 6 runs (1 take + 5 redo runs), $0.275 of model calls and 13:58 of model time.
  • What a person had to fix: QC's redo notes on shots 4–7 (adult scale, the envelope in hand, a dusk sky with no moon), plus the critic and recognition overrides signed by the owner and QC.
  • Status: on our showcase since 2026-10-14, not yet published on the channel, so there are no audience numbers to report. Two more packages have the owner's narration ready; no film has been made from them yet.

Credits

Voice-over: Piper en_US-john-medium (LibriVox, public domain). Vietnamese narration: Piper vi_VN-vais1000-medium (VAIS-1000 dataset, CC BY 4.0; fine-tuned from Piper's en_US-lessac voice, whose training data is under the Blizzard 2013 licence), used for this non-commercial demo. Japanese narration of the published film: 音声合成：AivisSpeech / morioki（ボイス提供：もりおき、モデル制作：yuki、ACML 1.0）. Music: the engine's own synth.

Built With

Share this project:

Updates

Submission history