Vide

An in-browser code-based video editor empowering AI agents to author timeline scripts, manage local media assets, and export MP4s via WebMCP.

Comment

The 28 WebMCP Tools by Domain:

  1. Asset Metadata & Semantic Notes (3 tools): get_asset_metadata, set_asset_notes, search_assets — Attach and search semantic descriptions, mood tags, BPM tempos, spoken transcripts, and visual descriptions so the agent understands raw media files (e.g., matching "whoosh transition" to sfx_0294.mp3).
  2. File System Operations (4 tools): list_project_files, read_project_file, write_project_file, delete_project_file — Direct read/write access to project directory files and scripts.
  3. Script & Tab Lifecycle (8 tools): get_active_script, get_video_script, set_active_script_code, update_video_script, create_script, list_open_scripts, switch_active_script, save_active_script — Read, generate, and hot-reload DSL scripts with real-time agent reasoning banners.
  4. Engine & Playback Telemetry (3 tools): validate_script, get_scene_settings, control_playback — Perform dry-run syntax checks, inspect canvas dimensions/FPS, and drive timeline playback.
  5. Hardware Video Export (1 tool): export_video — Trigger client-side GPU video encoding with configurable codecs (H.264/AVC, AV1, VP9) and automatically save outputs to the project's export folder.
  6. Documentation & Knowledge Base (3 tools): search_vide_docs, search_vide_documentation, get_dsl_reference — Embedded vector-scored knowledge base for the agent to look up DSL syntax, math rendering, and composition rules on the fly.
  7. Multimodal Visual Inspection & QA (3 tools): capture_frame_screenshot, analyze_timeline, inspect_layers_at_time — Render in-memory frame buffers to base64 PNGs for multimodal vision models to visually verify text alignment, contrast, and layout, correcting bugs autonomously.
  8. Timeline Cues & Scene Navigation (3 tools): list_timeline_cues, seek_to_cue, add_timeline_cue — Manage timeline markers, gutter breakpoints, and playhead anchors across scenes.

How We Built It

Having no formal background in compiler design, AST manipulation, or custom DSLs prior to this project, I paired with ChatGPT and Gemini as collaborative engineering partners throughout development.

We established that compiling a deterministic subset of JavaScript using Acorn was the ideal architecture—giving users familiar syntax with mathematical operators, while completely eliminating non-deterministic loops and security risks in the evaluation sandbox.

Technical Stack:

  • Framework & Routing: Full-stack local-first application built with TanStack Start, React 19, and Tailwind CSS v4.
  • Agent Protocol: WebMCP Bridge exposing 28 structured tools over window.modelContext and message channels.
  • Language Runtime: Acorn for AST parsing, evaluated in a deterministic sandbox with Valibot schema validation at every trust boundary.
  • Code Editor: Monaco Editor with custom Monarch tokenizers, syntax highlighters, and DSL snippet completions.
  • Engine & Export: Custom HTML5 Canvas 2D engine paired with Mediabunny for WebCodecs MP4/WebM hardware encoding and Web Audio API mixing.
  • Storage & Hosting: IndexedDB with the File System Access API; deployed serverlessly on Cloudflare Workers.

Challenges We Ran Into

  1. Deterministic Timeline Architecture: Translating declarative composition (parallel, sequence, at, after, gap, offset) into an immutable, time-addressable AST graph while preventing timeline keyframe collisions.
  2. Synchronizing Video, Audio, and Canvas Lifecycles: Ensuring Mediabunny's frame-exact video reader and the Web Audio context stayed strictly aligned during interactive timeline scrubbing and offline hardware encoding.
  3. Cross-Platform WebCodecs Audio Encoding: Overcoming platform-specific audio encoder limitations (such as Linux Chromium lacking AAC encoders) by implementing dynamic AudioEncoder.isConfigSupported fallback routing between AAC and Opus inside MP4 containers.
  4. Designing a Clean 28-Tool WebMCP Surface: Structuring a comprehensive tool suite that allows deep multimodal inspection without overwhelming the agent's context window.

Accomplishments That We're Proud Of

  • Zero-Server Video Encoding: Rendering 60 FPS 1080p MP4 videos with custom fonts and LaTeX math entirely within the client browser using WebCodecs and Mediabunny.
  • Complete 28-Tool WebMCP Suite: Enabling multimodal AI models to discover assets, author scripts, visually inspect rendered frames, and export motion graphics autonomously.
  • True Dogfooding: The entire demonstration and challenge submission video was 100% scripted, animated, composed, and rendered inside Vide itself.
  • Local-First Privacy: Full file editing, local asset references, and MP4 generation without a single byte of user media ever leaving their machine.

What We Learned

  • How to design a clean Domain-Specific Language (DSL) and evaluate AST nodes deterministically.
  • How to leverage the WebCodecs API and Mediabunny for client-side, hardware-accelerated video export pipelines.
  • The transformative power of the WebMCP standard in replacing brittle visual UI scraping with structured, bidirectional tool interfaces between web applications and AI agents.

What's Next for Vide

  • Orama In-Browser Vector Database: Integrating Orama to provide client-side vector search and semantic similarity matching across asset transcriptions, visual descriptions, and DSL documentation with sub-millisecond local latency.
  • Web Worker OffscreenCanvas Rendering: Offloading the entire frame rendering and WebCodecs encoding pipeline into background Web Workers for faster export throughput.
  • Procedural Data Visualization Nodes: Built-in animated bar charts, line plots, and telemetry counter primitives.
  • Smart Audio Ducking: Automatic volume ducking on music tracks when voiceover layers are active.
  • Direct Canvas Transform Gizmos: Interactive handles to manipulate shapes on screen while two-way syncing coordinates directly back into the DSL code.

Built With

  • acorn
  • canvas-api
  • cloudflare-workers
  • file-system-access-api
  • katex
  • latex
  • local-first
  • mediabunny
  • monaco-editor
  • react
  • tailwindcss
  • tanstack-start
  • typescript
  • valibot
  • vite
  • web-audio-api
  • webcodecs
  • webmcp
  • zustand
Share this project:

Updates

Submission history