Vector — Agentic Video Studio

• 

  SpacetimeDB edit history records each instruction revision, before/after values, author, and timestamp.

• 

  Three ElevenLabs sound layers, independently mixed and exported without regenerating the source video.

Inspiration

A small creative change should not require starting a whole video over. We wanted an editor where objects, actions, sounds, and the agents responsible for them remain inspectable and editable. Vector brings that idea into a collaborative desktop workspace: change what you intend, preserve what you do not, and keep a record of who changed what.

What it does

Vector combines a visible multi-agent workflow, revision-controlled editing instructions, and independently editable AI-generated sound. Select a worker to see its assignment, execution type, timing, and output. Edit an instruction and retain the previous version, author, and timestamp. Adjust a sound layer and export an MP4 without asking a video model to generate the scene again.

Our demo contains 17 completed workers across a 14-role protocol, including three sound specialists. Gemini supplies specialist briefs; the interface makes the distinction between a completed brief and an executed scene edit explicit. Agent illustrations and task cards make a complex pipeline tangible rather than hiding it behind a single progress spinner.

How we built it

The desktop shell uses Electron, React, TypeScript, Vite, and Tailwind CSS, with a Python backend for orchestration and media processing. We adapted the existing AgentVideo protocol for planning, perception, assignment, motion, sound, and verification. Roles can expand into multiple specialists. Provider credentials are encrypted with Windows DPAPI outside the repository and are never shipped in the frontend.

SpacetimeDB: collaboration that has a memory

SpacetimeDB is the collaboration layer for instructions and history, not a store for video pixels. Private tables and membership-filtered subscription views restrict project visibility. Owners can grant editor or viewer roles. An instruction edit carries an expected revision; a transactional reducer checks it, updates the instruction, and appends an attributed before-and-after history entry. Repeated operation IDs are idempotent, and stale revisions are rejected instead of silently overwriting another editor's work.

The Agent Observatory exposes timing records for bottleneck investigation. Execution measurements and database receipt timestamps are separate, so synchronization latency is not presented as model inference time. The demonstrated collaboration service runs locally; the module and client bindings are included in the repository.

ElevenLabs: three specialists, three reusable sound layers

Three sound agents generated outdoor ambience, donkey chewing, and fruit/bowl interaction effects using ElevenLabs. Each output is stored with duration and checksum metadata. Independent gain controls let the editor rebalance those effects without regenerating the video or requesting new sound.

Our export path verifies the artifacts, applies gains, limits the combined signal, and copies the existing video stream into the final MP4. The demonstrated remix makes zero additional model API calls. Frame-by-frame verification confirmed that the remixed export preserves the source video frames.

Gaussian scene memory: the research direction

Our visual-editing research represents scene parts with persistent 3D Gaussian primitives carrying position, appearance, orientation, and scale. The objective is to assign a bounded group of primitives to a specialist agent and update that part while retaining the rest of the scene. This is a research pipeline, not a claim that general-purpose object replacement is solved: correspondence through motion and occlusion still needs validation. The sound-enhanced demo is not presented as a successful Gaussian orange-to-apple replacement, and we do not claim an unmeasured video-generation speedup.

Challenges we ran into

The hardest visual problem was maintaining object identity across whole fruit, slices, fragments, and occlusion. We also had to distinguish planning success from execution success, protect concurrent edits from silent conflicts, and separate model timing from database synchronization. Packaging a Python research backend inside a secure Electron shell required manifest verification and keeping credentials and generated media out of source control.

Accomplishments we are proud of

We connected agent planning to an inspectable interface, implemented attributed revision history and membership-aware collaboration, generated three real ElevenLabs sound layers, and exported a sound-enhanced video with its visual content preserved. We validated the collaboration behaviors against a running SpacetimeDB server, including revision conflicts, permissions, history, and revocation—not just an in-memory mock.

What we learned

Useful agentic creative software needs more than model output: it needs ownership, durable artifacts, measurable execution, and a clear boundary between an agent's plan and the change it actually performed. Reusable outputs also matter: the sound workflow already demonstrates that remixing an existing artifact can avoid another model request.

What's next for Vector

Train and validate reliable Gaussian correspondence, connect visual specialists to bounded part-level updates, and test identity and detail preservation across moving scenes. Extend the collaboration deployment beyond localhost, improve action-aligned sound timing, and benchmark end-to-end quality, compute, and memory against full regeneration.

Source and attribution

This project builds on the existing h3clinic/ai-video-studio shell and h3clinic/agentvideo protocol, alongside our Gaussian-video research. The integrated demo branch contains the integrated application, collaboration module, backend source, tests, and editing tools. Upstream code, artwork, and model assets retain their original terms and attribution. Model weights, credentials, and private media are not distributed. Review the repository README and DEMO_STATUS.md for setup requirements and current limitations. We're building an ambitious editing system—and making every contribution inspectable along the way.

Built With

• electron
• elevenlabs
• gemini
• python
• spacetimedb
• typescript