Agentic Cinema Production System

Inspiration

The modern film production workflow is broken. Creative teams spend weeks coordinating between writers, directors, sound designers, editors, and VFX artists, juggling spreadsheets, emails, and Slack messages. Meanwhile, AI models like Seedance 2.0 produce stunning video but remain isolated tools that require manual stitching.

We asked: What if a film could coordinate itself?

ProductionOS transforms the chaotic media production pipeline into an orchestrated, observable, and verifiable agentic workflow. Inspired by how film production coordinators manage complex shoots, we built an AI system where agents act as specialized crew members, each with clear responsibilities, communicating through a central coordinator.

What It Does

ProductionOS is an **AI Production Coordinator.

The Golden Path

Creative Brief
     ↓
Director Agent → Production Plan (scenes, characters, locations)
     ↓
Script Agent → Structured scenes with dialogue
     ↓
Asset Agent → Required assets identified
     ↓
Audio Agent → Dialogue, music, sound effects
     ↓
Video Agent → Scene assembly
     ↓
ZingrMAB → Audio-video synchronization
     ↓
Continuity Agent → Validation
     ↓
Render Engine → Final video

Key Features

Feature	Description
Director Agent	Understands creative briefs, decomposes stories into scenes
Script Agent	Converts raw scripts into structured scenes with dialogue
Asset Agent	Determines required assets, detects missing ones
Audio Agent	Manages dialogue, music, sound effects (reuses ZingrMAB)
Video Agent	Generates/assembles video segments
ZingrMAB	Synchronizes external audio with generated video
Continuity Agent	Validates scene consistency (LLM + deterministic)
Production Coordinator	State machine orchestration
Render Engine	Deterministic FFmpeg rendering

The Demo

A 30-second sci-fi teaser about AI awakening — fully produced from a creative brief through agentic coordination.

How We Built It

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Production System                       │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐ │
│  │            User Interface (Next.js + Tailwind)              │ │
│  │  Dashboard | Production View | Agent Status | Sync Monitor │ │
│  └────────────────────────┬───────────────────────────────────┘ │
│                           │                                     │
│  ┌────────────────────────▼───────────────────────────────────┐ │
│  │              API Gateway (Node.js + Fastify)                │ │
│  │  • Production CRUD                                          │ │
│  │  • Job queue management                                     │ │
│  │  • WebSocket for real-time updates                          │ │
│  └────────────────────────┬───────────────────────────────────┘ │
│                           │                                     │
│  ┌────────────────────────▼───────────────────────────────────┐ │
│  │            BullMQ Queue (Redis)                             │ │
│  │  • Async job processing                                     │ │
│  │  • Retry with exponential backoff                           │ │
│  │  • Job status tracking                                      │ │
│  └────────────────────────┬───────────────────────────────────┘ │
│                           │                                     │
│  ┌────────────────────────▼───────────────────────────────────┐ │
│  │              Worker Layer (Node.js)                          │ │
│  │  • Production workers                                       │ │
│  │  • Asset workers                                            │ │
│  │  • Audio workers                                            │ │
│  │  • Render workers                                           │ │
│  └────────────────────────┬───────────────────────────────────┘ │
│                           │                                     │
│  ┌────────────────────────▼───────────────────────────────────┐ │
│  │         Agent Services (Python + FastAPI)                   │ │
│  │  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐          │ │
│  │  │Director │ │ Script  │ │  Asset  │ │  Audio │          │ │
│  │  │ Agent   │ │ Agent   │ │  Agent  │ │  Agent │          │ │
│  │  └─────────┘ └─────────┘ └─────────┘ └─────────┘          │ │
│  │  ┌─────────┐ ┌─────────┐ ┌─────────┐                       │ │
│  │  │  Video  │ │Continuity│ │Coordinator│                    │ │
│  │  │  Agent  │ │  Agent  │ │  (State)  │                    │ │
│  │  └─────────┘ └─────────┘ └─────────┘                       │ │
│  └────────────────────────┬───────────────────────────────────┘ │
│                           │                                     │
│  ┌────────────────────────▼───────────────────────────────────┐ │
│  │           ZingrMAB (Python)                                 │ │
│  │  • Audio-video sync                                         │ │
│  │  • Wav2Lip lip sync                                         │ │
│  │  • WSOLA time-stretching                                    │ │
│  │  • SyncNet confidence scoring                               │ │
│  └─────────────────────────────────────────────────────────────┘ │
│                                                                  │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │              Storage Layer (S3/MinIO)                        │ │
│  │  • Production manifests                                      │ │
│  │  • Asset storage                                             │ │
│  │  • Render outputs                                            │ │
│  └─────────────────────────────────────────────────────────────┘ │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Technology Stack

Layer	Technology	Purpose
Frontend	Next.js 14, Tailwind CSS, shadcn/ui	Production dashboard
API	Node.js, Fastify, TypeScript	REST endpoints, WebSocket
Queue	BullMQ, Redis	Async job processing
Workers	Node.js, TypeScript	Job execution
Agents	Python, FastAPI, Gemini API	LLM-powered reasoning
Sync	Python, FastAPI, librosa, OpenCV	ZingrMAB audio sync
Storage	MinIO / S3	Asset and result storage
Orchestration	Python, state machine	Production coordination
Container	Docker, Docker Compose	Deployment

Key Technical Decisions

AI decides. Software executes. Evidence proves.
Gemini for reasoning, deterministic services for execution — The LLM handles interpretation, planning, and creative decisions. The application handles state, validation, retries, and media processing.
ZingrMAB integration — Reused existing audio-video sync infrastructure instead of rebuilding.
State machine coordinator — Productions follow explicit states (CREATED → PLANNED → ASSETS_PENDING → ... → COMPLETE).
Observability first — Every agent action generates events, every job produces artifacts.
Deterministic rendering — FFmpeg commands are generated from validated structured data, not LLM output.

Challenges We Ran Into

FFmpeg & OpenCV Dependencies

Problem: Python containers crashed with libGL.so.1: cannot open shared object file and ffmpeg: command not found.

Solution: Added ffmpeg, libgl1, and libglib2.0-0 to the Dockerfile. Used cv2.VideoWriter_fourcc(*'mp4v') instead of 'avc1' for reliable encoding.

RUN apt-get update && apt-get install -y \
    ffmpeg \
    libgl1 \
    libglib2.0-0

Container Networking

Problem: Worker couldn't reach alignment service — ECONNREFUSED alignment:8000.

Solution: Use service names as hostnames in Docker Compose. http://alignment:8000 not http://localhost:8000.

File Paths Across Containers

Problem: Worker downloaded files to /tmp/, Python service couldn't find them.

Solution: Shared Docker volume mounted at /data for both services.

volumes:
  shared-data:
    driver: local

services:
  worker:
    volumes:
      - shared-data:/data
  alignment:
    volumes:
      - shared-data:/data

🔥 BullMQ Redis Connection

Problem: Jobs stuck in "waiting" state.

Solution: Use identical Redis config across all services — host: 'redis' not localhost.

Gemini API Rate Limiting

Problem: Multiple agents hitting Gemini API simultaneously caused rate limit errors.

Solution: Added retry with exponential backoff and circuit breaker pattern.

State Machine Complexity

Problem: Productions could get stuck in intermediate states.

Solution: Added timeout detection, automatic recovery, and human intervention prompts for unrecoverable states.

Audio Duration Mismatch

Problem: Video and audio durations didn't match, causing frozen frames or cut-off audio.

Solution: Added duration validation and automatic trimming/padding before rendering.

Accomplishments We're Proud Of

End-to-End Golden Path

A complete production from creative brief to final rendered video, executed through agentic coordination:

Creative Brief → Director Agent → 3-scene plan
Script Agent → Structured scenes with dialogue
Asset Agent → Asset requirements
Audio Agent → Audio processing
Video Agent → Scene assembly
ZingrMAB → Audio-video sync (86% confidence)
Continuity Agent → Validation (PASS)
Render Engine → Final MP4

Real-Time Agent Dashboard

A beautiful, dark-themed interface showing:

Production cards with sync confidence
ZingrMAB status bar with real-time sync jobs
Activity feed with agent events
Agent status panel showing ZingrMAB as a specialized agent

ZingrMAB Integration

Seamless integration of audio-video sync into the production pipeline:

Audio sync jobs appear in the dashboard
Sync confidence scores on production cards
Real-time progress updates via WebSocket
Dedicated "Audio Sync" navigation

Production State Machine

A robust state machine that ensures productions progress through defined stages:

CREATED → PLANNED → ASSETS_PENDING → ASSETS_READY → AUDIO_PENDING → AUDIO_READY → VIDEO_PENDING → VIDEO_READY → SYNC_PENDING → SYNC_READY → VALIDATING → VALIDATED → RENDERING → COMPLETE
Automatic retry on recoverable failures
Resumable productions (don't reprocess completed scenes)

Deterministic vs LLM Separation

LLM handles: Interpretation, planning, creative decisions
Application handles: State, execution, validation, retries, files, media processing
Evidence proves: Sync confidence scores, duration checks, asset presence

Production Package

A complete, reproducible production package containing:

Production manifest
Timeline
All assets
Validation reports
Render manifest
README with reproduction instructions

What We Learned

Agent Systems Need Structure

LLMs are powerful but unreliable for execution. The most important architectural decision was separating reasoning from execution — agents plan, deterministic services execute.

State Machines Are Underrated

A production coordinator needs explicit states. Without a state machine, productions get stuck in ambiguous "processing" states. With one, you always know where you are and what to do next.

Observability Is Non-Negotiable

Every agent action, every job, every state transition must be logged and visible. The dashboard isn't a nice-to-have — it's the only way to understand what the system is doing.

Integration Over Building

We reused ZingrMAB instead of building audio sync from scratch. This saved weeks of development and gave us production-tested synchronization.

Failures Are Data

Every failure teaches you something. We built failure recovery into the system — productions are resumable, jobs retry with exponential backoff, and errors generate artifacts for inspection.

Dark UI Matters

For creative professionals, a dark theme with teal accents communicates professionalism and aligns with creative tools like DaVinci Resolve and Premiere Pro.

What's Next for ProductionOS

V2 — Enhanced Capabilities

Feature	Description	Status
Multi-speaker sync	Handle dialogue with multiple speakers	In Progress
Real-time preview	Streaming preview during production	In Progress
Emotion-aware sync	Match audio emotion to video expression	Planned
Neural audio stretching	Higher quality time-stretching	Planned
Custom agent training	Fine-tune agents on specific styles	Planned

V3 — Production Scale

Feature	Description	Status
Multi-GPU distribution	Scale GPU processing	Planned
Cloud deployment	GCP Cloud Run / AWS ECS	Planned
Team collaboration	Multi-user production	Planned
Version history	Production versioning	Planned
API for external tools	Integration with existing workflows	Planned

Commercialization

Free tier: 5 productions/month, community support
Pro tier: Unlimited productions, priority support, team features
Enterprise: On-premise deployment, custom agents, SLA

Research Directions

Agentic scene composition — Agents that understand cinematography principles
Style transfer between scenes — Maintain visual consistency
Automated storyboarding — Generate storyboards from scripts
Voice cloning integration — ElevenLabs/RVC for character voices

Repository Structure

productionos/
├── apps/
│   ├── api/          # Fastify API Gateway
│   ├── worker/       # BullMQ Worker
│   ├── webhook/      # Webhook Dispatcher
│   └── cli/          # CLI Tool
├── services/
│   ├── alignment/    # ZingrMAB Alignment Service
│   ├── agents/       # Agent Services (Director, Script, Asset, Audio, Video, Continuity, Coordinator)
│   └── render/       # Render Engine
├── packages/
│   ├── production-types/
│   ├── agent-sdk/
│   ├── shared-types/
│   ├── config/
│   └── utils/
├── frontend/         # Next.js UI
├── infra/
│   ├── docker/
│   ├── kubernetes/
│   ├── terraform/
│   └── monitoring/
├── fixtures/         # Test data
├── tests/            # Unit, integration, E2E tests
└── docs/             # Architecture, API, deployment docs

ProductionOS - Agentic Cinema Production System

AI decides. Software executes. Evidence proves.

Built With

bullmq
fastapi
fastify
ioredis
librosa
next.js
opencv
python
tailwindcss
torch