FilmSwarm

An AI agent swarm that turns a one-sentence film concept into a complete pre-production package — character designs, world art, mood boards, shot lists, and a production bible — in minutes.

Comment

Inspiration

Every great film starts the same way — someone says "What if..." and then spends months turning that spark into a production-ready vision. Pre-production is where films are won or lost, but it's also where most indie creators get stuck. You need a writer, a concept artist, a cinematographer, a casting director — an entire creative team — before a single frame is shot.

What if an AI swarm could be that team?

FilmSwarm was born from a simple question: can a group of specialized AI agents collaborate like a real film crew — debating creative choices in real-time — and deliver a complete pre-production package from nothing but a one-sentence concept?

## What it does

You type a single sentence — "A noir heist in a floating city where memories are sold" — and FilmSwarm's Director agent reads it, assembles the right team, and launches a 10-step production pipeline:

  1. ConceptAnalyst breaks down genre, tone, themes, and visual style
  2. StoryArchitect builds a 3-act structure with character arcs and locations
  3. ScriptWriter drafts a full shooting script with scene breakdowns
  4. CastingDirector creates detailed character profiles and generates portrait art for each cast member
  5. WorldBuilders (one per location, spawned dynamically) create establishing shots and environment designs
  6. MoodCurator assembles a color palette, visual style guide, and mood board images
  7. Cinematographer designs the shot list with camera angles, lens choices, and lighting — plus storyboard reference frames
  8. StoryboardArtist illustrates key cinematic moments as full storyboard frames
  9. ProductionCompiler + QualityReviewer assemble and QA the final pre-production package
  10. VideoDirector creates a shot-by-shot video production plan with Veo 3.1 prompts

Throughout the pipeline, agents hold Roundtable deliberations — visible in real-time — where they debate creative direction, push back on each other's ideas, and reach consensus before moving forward. You watch a Director argue with a Cinematographer about shot composition while a MoodCurator defends their color palette choices.

The agent count scales dynamically with story complexity. A concept with 5 characters and 4 locations spawns 24+ agents. A simpler concept might use 14.

The result: 13+ generated concept art images with consistent character designs, a full script, cast profiles, location designs, mood boards, a shot list, storyboard frames, a video production plan, and a complete creative vision — from a single sentence.

## How we built it

Agent Orchestration (Google ADK + TypeScript): The Director is an async orchestrator that dynamically spawns specialized LlmAgent subclasses via ADK's InMemoryRunner. Each agent gets its own session with shared mutable state — outputs from Step 3 (character portraits) automatically flow into Step 5 (world building) as reference images. The ParallelAgent primitive runs WorldBuilders and MoodCurator concurrently.

Reference Image Consistency Pipeline: This is FilmSwarm's secret weapon. Character portraits generated in Step 4 are stored in Google Cloud Storage. Every downstream image-generating agent (WorldBuilder, Cinematographer, StoryboardArtist, MoodCurator) automatically receives up to 6 reference images as base64 inlineData parts — fetched from GCS in parallel and injected before the text prompt. This means every image in the final package shares the same character designs, costumes, and visual identity.

Multi-Agent Roundtable Deliberation: After each pipeline step, a Roundtable convenes relevant agents (e.g., Director + Cinematographer + MoodCurator for the visual coherence review). Each panelist gets a perspective directive and argues their position across multiple rounds. A synthesis step produces a decisive outcome with rationale and action items. 10 Roundtables run across the pipeline — that's 30+ deliberation calls producing real creative debate.

Real-time Streaming: The pipeline emits 23+ event types over Server-Sent Events — agent spawns, thinking updates, roundtable messages, image generations, token usage, step completions. The frontend renders these in real-time with automatic reconnect and full event replay on SSE reconnection.

Generative Audio Engine: All sound in FilmSwarm is synthesized live in the browser using the Web Audio API — zero audio files. 16 unique SFX (agent spawn blips, success chords, roundtable gavels, thinking pulses) plus a generative theme tune with a chord loop (C → G → Am → F), sine pad, square-wave arpeggio, triangle bass, and hi-hat ticks. Every sound is procedurally generated from oscillators and noise nodes.

Pixel Robot UI: Each agent type has a unique pixel-art robot avatar rendered as CSS box-shadows — no images. Robots animate differently based on state: idle sway, energetic talking bounce, deliberation wobble. Speech bubbles show real-time agent output. The CRT scanline overlay, neon glow effects, and vignette are pure CSS.

Infrastructure: Docker → Cloud Build → Artifact Registry → Cloud Run (server, 900s timeout). Firebase Hosting (client) with rewrites routing /api/** to Cloud Run. Firestore for project state and event persistence. Google Cloud Storage for all generated images.

## Challenges we ran into

  • Visual consistency across 13+ images: Getting a StoryboardArtist's frame to match a CastingDirector's portrait required building a custom reference image injection pipeline. We fetch up to 6 images from GCS as base64, inject them as inlineData parts before every generation call, and include explicit instructions to maintain character appearance. The difference between "with refs" and "without refs" is night and day.

  • Agent timeout management: Gemini 3.1 Pro's extended thinking phase can take 2+ minutes per call. With 10 steps, 10 roundtables, and multiple image generations, a complex pipeline makes 50+ API calls. We added per-call timeouts with graceful fallbacks — if a roundtable panelist times out, they're skipped and the deliberation continues rather than killing the entire pipeline.

  • Dynamic agent scaling: The number of CharacterDesigner and WorldBuilder agents varies per story — a story with 5 characters spawns 5 separate CharacterDesigner agents, each running in parallel. Merging their outputs back into shared state without conflicts required careful session management through ADK's ParallelAgent.

  • Roundtable prompt engineering: Making multi-agent deliberation produce genuinely useful creative feedback (not just "I agree with everything") required giving each panelist a strong perspective directive and explicit instructions to push back and disagree. The best roundtable outputs come from creative tension.

  • SSE reliability on Cloud Run: Long-running pipelines (15-25 minutes) frequently hit CDN timeouts and connection drops. We implemented full event replay on reconnect — every event is persisted to Firestore with sequence numbers, so reconnecting clients get the complete history before receiving live events.

## Accomplishments that we're proud of

  • A complete, working creative AI swarm that produces real, usable pre-production output — not a toy demo. The generated scripts have actual scene structure, the character designs are visually consistent, and the roundtable debates produce genuinely interesting creative decisions.
  • Visual consistency across all generated images through automatic reference image injection — characters look the same whether they appear in a portrait, a location shot, or a storyboard frame.
  • Real-time Roundtable deliberations where you can watch a Director argue with a Cinematographer about shot composition while a StoryArchitect defends narrative pacing — visible as it happens with speech bubbles on pixel-art robots.
  • Public gallery where anyone can browse completed productions — click a card to see the full cast, locations, mood boards, shot list, storyboard frames, and generated images.
  • Fully generative audio — theme music and 16 SFX synthesized from pure Web Audio oscillators and noise nodes. Zero audio files in the entire project.
  • Everything runs on a free Gemini API key — no paid tiers required.

## What we learned

  • Google ADK is production-ready for multi-agent systems. The LlmAgent + InMemoryRunner + ParallelAgent abstractions made it straightforward to build a 24-agent pipeline with shared state, dynamic spawning, and parallel execution.
  • Gemini 3.1 Pro's extended thinking is powerful but slow. The model produces exceptional creative output, but the thinking phase can dominate pipeline time. Tuning maxOutputTokens and adding timeout protection were essential for reliability.
  • Gemini's native image generation is a creative game-changer. Being able to pass reference images alongside text prompts and get back visually consistent concept art — without any external image API — dramatically simplified the architecture.
  • Multi-agent deliberation produces better output than single-agent generation. The Roundtable pattern consistently improves creative quality. Agents catch inconsistencies, suggest improvements, and push each other toward better decisions.
  • Real-time streaming transforms the experience. Watching agents spawn, think, argue, and create is fundamentally more engaging than waiting for a loading spinner. The UX is the product.

## What's next for FilmSwarm

  • Video generation pipeline — Veo 3.1 integration is fully built (shot planning, first-frame extraction, character reference injection). Expanding clip coverage and quality tiers.
  • Export to industry formats — PDF lookbooks, Final Draft scripts, shot list spreadsheets
  • Interactive Roundtables — let users join deliberations as Executive Producer, steering creative direction in real-time
  • Reference image uploads — drop your own mood boards, character sketches, or brand assets to guide the swarm's creative direction
  • Pipeline resume — save and resume interrupted pipeline runs instead of starting over

Built With

Share this project:

Updates

Submission history