Voivent MV Studio

Agent-native 3D music video production suite powered by WebMCP with 38 browser-native tools.

Comment

Inspiration

While modern text-to-video AI generators can produce visually striking clips, they present a fundamental workflow bottleneck for creators: once a video is rendered into flat pixels, making precise micro-adjustments—such as rotating a camera angle by 5 degrees, nudging lyric timing by half a second, or reusing customized 3D assets across scenes—becomes nearly impossible without regenerating everything from scratch.

We realized true creative collaboration requires structural, parametric control. By exposing a procedural 3D scene graph, multitrack timeline, and audio signals directly to AI agents via WebMCP, creators and agents can co-direct, iteratively polish details, and reuse assets non-destructively through natural conversation. WebMCP unlocks unique, interactive creative workflows that take browser-based music video production to the next level.

What it does

Voivent MV Studio is an agent-native 3D music video production suite powered by WebMCP.

  • 38 Browser-Native WebMCP Tools: Autonomous agents (such as ChatGPT) inspect studio capabilities, build 3D voxel worlds, split timeline chapters, translate & synchronize lyrics, and trigger video exports.
  • Visual Self-Inspection (get_mv_preview): The agent captures real-time canvas snapshots to visually evaluate lighting, composition, and text safe-zones, autonomously refining parameters without human intervention.
  • Procedural 3D World Engine: Real-time voxel terrain generation, orbital landmarks, dynamic lighting, atmospheric fog, and cinematic cameras rendered directly in the browser via Three.js, WebGPU, and WebGL.
  • Multitrack Timeline & Lyric Sync: Non-destructive multi-scene splitting (intro, verses, chorus, drops, narrative chapters, or time-of-day transitions), multilingual lyric translation with syllable-precise timing alignment, and audio-reactive beat pulses.
  • In-Browser 1080p MP4 Export: Full hardware-accelerated video rendering inside the client browser via WebCodecs and MediaBunny.

How we built it

  • WebMCP Layer: Built 38 structured tools registered on document.modelContext spanning 3D scene graphs, timeline actions, audio stems, and diagnostics.
  • Frontend & 3D: React 19, TypeScript, Vite, and Three.js with WebGPU/WebGL acceleration.
  • Client-Side AI & Audio: Web Audio API beat detection, and on-device AI source separation running locally via ONNX Runtime Web.
  • Cloud Infrastructure: Cloudflare Workers for edge hosting, R2 storage for streaming AI model assets, and Cloudflare Workers AI (Whisper) for optional vocal transcription.

Challenges we ran into

  • Designing a tightly coupled visual feedback loop where the AI agent can iteratively validate its own 3D scene compositions via Base64 canvas snapshots.
  • Orchestrating non-destructive multitrack timeline operations while maintaining seamless audio-video synchronization across multi-scene splits.
  • Achieving fast 1080p MP4 video rendering entirely inside standard web browsers using WebCodecs and MediaBunny without offloading video encoding to backend servers.

Accomplishments that we're proud of

  • Fully operational suite of 38 WebMCP tools operating reliably with ChatGPT and autonomous browser agents.
  • Autonomous multi-scene direction: splitting music tracks into distinct narrative scenes and musical chapters with dynamic camera paths and audio-reactive pulses.
  • Heavy computing in the browser: running real-time 3D graphics, local ONNX stem inference, and 1080p video muxing directly on the user's device without requiring expensive GPU rendering servers.

What we learned

WebMCP proves that the open web can host rich, stateful, and non-destructive creative ecosystems for AI agents. Instead of treating AI as an isolated prompt-in/pixel-out generator, WebMCP unlocks an unprecedented paradigm: humans and agents building, inspecting, and iteratively polishing unique 3D interactive experiences together.

What's next for Voivent MV Studio

  • Native DAW Integration: Bridging Voivent MV Studio with our standalone digital audio workstation (Voivent DAW), allowing creators to stream multitrack audio and lyric maps directly into the WebMCP environment.
  • Intuitive "Co-Pilot" Real-Time Jamming: Deepening the collaborative experience where creators and AI agents run side-by-side—intuitively directing 3D lighting, camera cuts, and visual shaders in real-time.
  • Multi-Agent WebMCP Sessions: Enabling multiple specialized agents (e.g., dedicated lighting, camera, and typography agents) to coordinate simultaneously through WebMCP tool calls.

Built With

Share this project:

Updates

Submission history