CineFlow AI

CineFlow : An intelligent, agent-driven framework for automated video editing, QC, and distribution.

Comment

Inspiration

The primary catalyst for CineFlow was the friction between non-deterministic AI outputs and highly deterministic physical systems. Most agentic pipelines generate text, code, or static images, leaving the heaviest lifting - video orchestration - undone. We wanted to build an architecture where autonomous agents don't just draft scripts or suggest edits, but actively bind those decisions to physical media and drive FFmpeg rendering directly. The goal was to establish an end-to-end orchestration pipeline that bridges semantic intent with a validated .mp4 artifact.

What it does

CineFlow is an automated, multi-agent video production pipeline.

  • Asset Resolution: It queries a MediaIntelligenceDB to bind physical paths to assets based on metadata matches.
  • Creative Orchestration: A Director Agent translates a semantic script into an array of concrete EditDecision objects, dynamically pulling B-Roll based on context.
  • Revision Loop: If a Compliance Agent flags a selected asset for copyright or derivative restrictions, the Director explicitly removes or replaces the offending clip_id prior to rendering.
  • Physical Rendering & Validation: The Render Agent executes physical video rendering via FFmpeg and uses ffprobe to validate the output file size, stream existence, and readability.

How we built it

The system is built in Python, centered around a state-machine topology defined in core/orchestrator.py. The pipeline isolates concerns across specialized nodes: director_agent.py, render_agent.py, qc_agent.py, and distribution_agent.py.

Instead of relying on API abstractions for the final output, we implemented direct subprocess.run(..., check=True) calls to pass compiled instructions directly to the FFmpeg binary. We instrumented the entire pipeline with OpenTelemetry (core/observability.py) to trace execution time across the agent decision graph and the physical render layer.

Challenges we ran into

  • State Drift & Versioning: Early on, the orchestrator suffered from state drift. The manifest.version and project.current_version were double-incrementing during the revision loop, causing the pipeline to look for assets that belonged to a future, unrendered state.
  • Telemetry Blocking: Network timeouts to the OpenTelemetry collector would stall the main thread, halting the entire render process. This was patched by pushing the exporter to an asynchronous background task.
  • Semantic-to-Physical Binding: Preventing the Director from hallucinating assets was difficult. Early iterations simply returned hardcoded string requests like "traffic in jakarta". We had to implement strict metadata constraints so the agent could only select from verifiable, physical assets currently on disk.

Accomplishments that we're proud of

We successfully moved the project from a simulation prototype to a functional system. The initial pipeline mocked FFmpeg renders with 16-byte text files and used static integers (e.g., 98, 96) for quality control.

We eradicated the mocks. The pipeline now executes a genuine creative revision loop, dynamically alters EditDecision payloads, outputs actual video files, and relies on strict ffprobe validation to prove the video artifact exists and is playable.

What we learned

The abstraction leaks immediately when agents interact with physical files. You cannot trust an LLM to accurately predict the byte-size, codec, or duration of a resulting render.

To solve this, we learned that dynamic validation must be grounded in physical measurement, not LLM estimation. The Quality Score $Q$ of a render is computed from physical metrics: resolution integrity $R$, compliance adherence $C$, and edit density $E$:

$$ Q = \alpha \cdot f(R) + \beta \cdot C + \gamma \cdot \log(E) $$

Similarly, replacing hardcoded assets required implementing semantic matching in search_broll. The similarity $S(q, d)$ between the Director's query $q$ and the asset metadata $d$ dictates the selection: $$ S(q, d) = \frac{\vec{q} \cdot \vec{d}}{|\vec{q}| |\vec{d}|} $$ Bridging this gap - forcing stochastic agents to obey mathematical and physical file constraints - was the critical engineering takeaway.

What's next for CineFlow AI

  • Remote URI Fetching: The current implementation assumes source .mp4 assets are already staged in a local directory. The next phase will implement dynamic downloading of assets from remote URIs before the render phase executes.
  • Distribution Mode: Fully implementing the REAL_MODE toggle in agents/distribution_agent.py to support actual API uploads to target platforms, rather than just returning simulated URLs.
  • Advanced Audio Sync: Upgrading the EditDecision data structure to handle separate audio tracks and waveform-based sync alignment.

Built With

Share this project:

Updates

Submission history