Filmidi

A smart video editor that lets you edit videos using simple text commands. Just tell the AI what to cut, add, or sync, and it does the heavy lifting for you

Comment

Filmidi: AI-Native Video Editor

Inspiration

Video editing is repetitive. Hunting silences, matching beats, removing filler words, tweaking keyframes. Creators waste hours on mechanical tasks instead of storytelling. Filmidi turns natural language into precise timeline edits in real time.

What It Does

  • Prompt-Driven Editing: Say "cut dead air >1.5s", "sync cuts to music", or "add glowing captions". AI executes instantly.
  • Frame-Accurate Control: Handles multi-track placement, speed, keyframes, and effects automatically.
  • Native MCP Server: Exposes http://127.0.0.1:19790/mcp for external AI tools like Claude Desktop and custom agents to inspect frames and execute edits.
  • Text-Based Editing: Auto-transcribes speech for word-level trimming and filler word removal.

Quick Setup

git clone https://github.com/cpatra11/filmidi-q
cd filmidi-q/eb
bun install
bun run dev:hmr

Create .env:

OPENAI_API_KEY=your_key
ANTHROPIC_API_KEY=your_key
QWEN_API_KEY=your_key

License: GPL-3.0-or-later | Repo: github.com/cpatra11/filmidi-q

How Codex & GPT-5.6 Accelerated Development

  • 50+ MCP Tools: Codex generated and validated all tool schemas (toolDefinitions.ts) and executors (toolExecutor.ts) for clipping, keyframes, color grading, beat detection.
  • IPC & State: Codex architected Electrobun IPC handlers and Zustand stores for real-time sync.
  • Algorithms: Codex implemented waveform correlation, beat detection, and RNNoise noise suppression using WebAudio and WASM.
  • Agent Orchestration: GPT-5.6 reasons over timeline state and issues precise tool call sequences.

Key Algorithms

Audio Sync via Cross-Correlation: $$R_{xy}(\tau) = \sum_{t} x(t) y(t - \tau)$$

Finds optimal time lag (\tau) matching transients between audio tracks.

Frame Mapping: $$f = \lfloor t \cdot \text{FPS} \rfloor$$

Architecture

┌─────────────────────────────────────────┐
│         React WebView UI                │
│  Timeline │ Preview │ Chat │ Settings   │
│     └──────────┬───────────┘           │
│           Zustand Store                 │
└───────────────┼─────────────────────────┘
                │ IPC (Electrobun)
┌───────────────▼─────────────────────────┐
│         Bun Backend                     │
│  ┌─────────────────────────────────┐   │
│  │  MCP Server (Port 19790)        │   │
│  │  50+ Tool Executors             │   │
│  └─────────────────────────────────┘   │
│  Audio Processing │ Media Pipeline     │
│  (RNNoise WASM)   │ (VideoFlow)       │
│  AI Agents (GPT-5.6, Qwen)            │
└─────────────────────────────────────────┘

Core Tools (50+)

Category Tools
Clip cut_clip, split_clip, delete_clip, duplicate_clip
Audio sync_audio, detect_beats, denoise_audio, adjust_volume
Effects add_keyframe, apply_color_grading, add_transition, set_speed
Layout split_screen, add_overlay, apply_preset
Text transcribe_audio, remove_filler_words, add_captions

Challenges & Solutions

Challenge Solution
Low-latency IPC with frame drops Optimized Electrobun message batching
Deterministic LLM tool calls 50+ schemas with precise coordinate constraints
Cross-process frame inspection Real-time canvas capture over MCP

Accomplishments

  • Smooth Electrobun desktop experience
  • Full MCP server inside video editor
  • Voice-to-timeline automation
  • Real-time beat detection (client-side)

What's Next

  • Multi-Agent Pipelines: Specialized agents for color, sound, subtitles
  • Cloud GPU Export: 4K/6K renders to cloud clusters
  • Voice Co-Pilot: Hands-free editing during scrubbing
  • Collaboration: Real-time multi-user editing

Tech Stack

  • Desktop: Electrobun, Bun
  • Frontend: React 18, TypeScript, Tailwind, VideoFlow
  • AI: GPT-5.6, Qwen 3.7 Max, MCP SDK
  • Audio: WebAudio API, RNNoise WASM
  • State: Zustand

Built With

Share this project:

Updates