MediaCraft AI

An autonomous video engineer powered by Gemini on Replit that builds precision FFmpeg pipelines and self-heals rendering errors.

Comment

Inspiration

The inspiration for MediaCraft AI stemmed from firsthand friction with brittle media pipelines.

On a personal level, dealing with home entertainment devices and legacy displays requiring strict H.264 profiles meant wasting hours running cumbersome desktop tools just to make media playable. Finding and syncing accurate subtitle files was equally frustrating. Across the wider content creation space, video editors face this exact technical bottleneck at scale: manually writing cryptic FFmpeg commands, calculating aspect-ratio reframes for multi-platform delivery (like 9:16 vertical reels), and dealing with silent render crashes.

MediaCraft AI replaces these fragile, error-prone workflows by turning natural-language intent into deterministic, verified, and self-healing FFmpeg pipelines.

What It Does

MediaCraft AI is an autonomous video engineering studio hosted on Replit, designed to transform plain-English creative directions and production recipes into verified, production-ready media workflows.

Core Capabilities & Production Presets

  • Upfront ffprobe Inspection Gatekeeper: Inspects uploaded media containers down to the metal before execution—evaluating codecs (e.g., standard H.264 vs. compute-heavy 60fps HEVC), frame counts, pixel area, and audio layouts to calculate compute boundaries and prevent container thread saturation.
  • Targeted Production Presets:
    • Captions & Hook: Transcribes spoken audio, identifies high-engagement opening hooks, and compiles kinetic, word-by-word subtitle filtergraphs.
    • Cinematic Cut: Analyzes scene transitions and audio peaks to automatically cut dead air and tighten pacing.
    • Platform Refit: Re-architects landscape video into vertical 9:16 reels using intelligent scaling and background blurs without cropping key subjects.
  • Autonomous Execution: Executes media operations via spawn("ffmpeg", args)—handling tasks such as 9:16 vertical re-framing, audio extraction, subtitle burning, and web-ready H.264 compression.
  • Self-Healing Error Recovery: Captures raw FFmpeg stderr on runtime failure, routing the error signature back to Gemini for an autonomous, bounded two-attempt repair loop.
  • Studio Telemetry & Monitoring: Streams real-time processing diagnostics to the dashboard, persists job analytics to ClickHouse Cloud, and sends operational annotations to Grafana.

How We Built It

MediaCraft AI was built as a full-stack, agent-assisted media processing studio combining modern web interfaces with sandboxed, resilient backend infrastructure.

Architecture & Tech Stack

  • Frontend: Built with React and Vite, delivering a dark-themed studio workspace, responsive file drop zones, live processing event streaming, and a Ctrl+K command palette.
  • Backend API: An Express / Node.js 24 server managing media lifecycles, multipart uploads via Multer, temporary server-owned scratchpads, and execution sandboxes.
  • Media Inspection Gate: Every uploaded asset is pre-inspected with ffprobe to validate format containers, positive durations, codec specifications, and stream configurations prior to planning.
  • AI Planning & Generation: Powered by Google Gemini 2.5 Flash using the official @google/genai SDK. Gemini converts plain-language instructions and media metadata into safe, structured JSON FFmpeg argument arrays.
  • Technical Grounding (Parallel Search): Integrates the Parallel Search API (POST https://api.parallel.ai/v1/search) to query real-time technical context on obscure codecs, filtergraph behaviors, and container specs, injecting bounded search excerpts directly into Gemini's planning prompt.
  • Sandboxed Execution: Discards insecure shell execution in favor of Node.js spawn("ffmpeg", args) with server-owned input/output placeholder replacement, ensuring strict prevention of command-injection attacks.
  • Self-Healing Loop: When FFmpeg exits with an error code, raw stderr diagnostics are intercepted and fed back to Gemini for an autonomous, bounded two-attempt argument repair cycle.
  • Analytics Engine (ClickHouse Cloud): Automatically logs granular job execution schemas (duration, format, video/audio codecs, attempt counts, status) to ClickHouse over HTTPS via JSONEachRow, backed by a 500-job local buffer for uninterrupted session analytics.
  • Observability & Health (Grafana): Emits real-time operational annotations and diagnostic telemetry (active jobs, failure rates, self-heal attempts, last FFmpeg stderr) to Grafana dashboards.

Challenges We Ran Into

  • CPU Compute Constraints & Heavy Codec Workloads: Running deep filtergraphs and frame-by-frame subtitle rasterization (libass) on high-bitrate media (such as 1440p 60fps HEVC) inside shared container environments naturally saturates single-thread CPU cycles. Transcoding dense pixel arrays without dedicated GPU hardware acceleration exposed latency bottlenecks. We solved this by engineering our upfront ffprobe inspection gatekeeper to analyze stream complexity beforehand and sandboxed child processes to maintain buffer drainage and prevent deadlocks.
  • Multi-Service Architecture & Replit Routing: Triangulating and orchestrating a full-stack media workspace on Replit—connecting the React/Vite frontend to an Express API backend, setting up artifact reverse proxies, and ensuring native FFmpeg/Nix binaries execute seamlessly without path collision.
  • Deterministic LLM Output for System Binaries: Large Language Models frequently inject markdown code blocks, conversational explanations, or shell-escaped quotes when generating terminal commands. We strictly enforced structured JSON schema output and validated every argument against an allowlist before passing arrays to Node.js spawn.
  • Container & Filtergraph Conflicts in FFmpeg: Complex video scaling, aspect-ratio cropping, and audio stream routing often produce obscure FFmpeg stderr exits. Designing an autonomous prompt loop that feeds raw error logs back to Gemini without entering infinite recursive render cycles required bounded two-attempt constraints and technical grounding via Parallel Search.

Accomplishments That We're Proud Of

  • Autonomous Self-Healing Rendering Pipeline: Built an agent loop that captures raw non-zero FFmpeg error streams and autonomously repairs syntax and codec flags without requiring manual terminal troubleshooting.
  • Zero Shell-Injection Security: Implemented a secure parameter pipeline where FFmpeg runs via child process spawning with strict server-owned placeholder substitution (__INPUT__ and __OUTPUT__), eliminating command-injection risks.
  • Enterprise-Grade Observability: Connected full-pipeline observability from scratch—streaming live server-sent events to the UI while persisting structured performance telemetry to ClickHouse Cloud and emitting diagnostic annotations to Grafana.

What We Learned

  • End-to-End Replit & MCP Integration: Mastered the end-to-end Replit development lifecycle—deploying production-grade multi-service architectures, managing environment sandboxes, and bridging autonomous agents to external protocols and databases (ClickHouse Cloud and Grafana).
  • Deterministic Media Automation: Learned how to safely sandbox system binaries with AI agents, balancing creative flexibility in natural language with strict structural validation.

What's Next for MediaCraft AI

  • Distributed Cloud GPU Rendering: Decoupling compute-dense rendering from container CPUs by routing high-resolution exports and heavy codecs (HEVC/ProRes) to worker pools equipped with hardware acceleration (NVENC / Google Cloud Transcoder API).
  • Asynchronous Queueing Architecture: Implementing background worker queues (Redis/BullMQ) with live Server-Sent Events (SSE) progress tracking for long-running batch renders.
  • Intelligent Dynamic Saliency Reframing: Upgrading the 9:16 vertical reframe engine to use Gemini frame sampling for speaker and action face-tracking rather than static centering.
  • Automated Silence & Dead-Air Stripping: Integrating FFmpeg silencedetect with Gemini cutlist generation to produce tight jump-cut edits from unedited dialogue tracks.

Built With

Share this project:

Updates

Submission history