Agent Studio

AI agents edit. Humans stay in control.

Comment

Inspiration

We wanted to explore how AI agents could assist with creative work without taking control away from the creator. This inspired us to build Agent Studio: a WebMCP-enabled video editing prototype where an agent can perform structured editing operations while the creator remains involved in creative decisions.

What it does

Through WebMCP, an agent can interact with the video editor to:

  • 🔍 Inspect the current project and scene state
  • ✏️ Modify scene captions
  • ⏱️ Adjust scene durations
  • 🎬 Search Pexels for relevant portrait stock footage
  • 🖼️ Replace scene visuals
  • ▶️ Review the updated project in playback order

The key idea behind Agent Studio is human-agent collaboration. The agent handles repetitive editing operations through structured tools, while the creator can inspect the resulting changes and review stock-footage options in the visual editor.

The agent operates structured tools. The creator directs the creative outcome.

How We Built It

We built Agent Studio as a web-based video editor using React, TypeScript, TanStack Start, Tailwind CSS, and WebMCP.

The editor exposes structured WebMCP tools through the browser's model-context API. These tools allow compatible agents to read the current project state and perform editing operations without relying on screenshots, page scraping, or fragile button-click automation.

For stock-footage discovery, a server-side endpoint securely sends search requests to the Pexels Videos API and returns portrait-oriented results. The API key remains on the server and is not exposed to the browser.

The application is deployed on Vercel. We also implemented an in-page development shim that makes the tools inspectable and callable in browsers without native WebMCP support.

WebMCP Implementation

Agent Studio registers seven structured WebMCP tools:

  • get_project — returns the project, timeline, duration, and scenes
  • get_scene — returns the current state of a selected scene
  • update_caption — updates a scene's caption text
  • change_scene_duration — changes scene timing within validated limits
  • replace_scene_visual — replaces a scene's current visual
  • search_stock_visual — searches Pexels for portrait stock footage
  • preview_project — returns a project summary in playback order

Each tool includes a description, structured input schema, validation, and a result that the agent can interpret.

WebMCP allows an agent to interact with structured editor state directly. Changes made through the tools are immediately reflected in the visual editor, allowing the creator to see the same state that the agent is operating on.

Example Human–Agent Workflow

A creator can ask:

Inspect the project, rewrite scene two's caption, set its duration to six seconds, and find neon-city footage for it.

The agent can then:

  1. Call get_project or get_scene to inspect the timeline.
  2. Call update_caption to change the caption.
  3. Call change_scene_duration to update the timing.
  4. Call search_stock_visual to retrieve portrait footage from Pexels.
  5. Present the returned options in the editor for the creator to review.
  6. Call preview_project to verify the updated timeline.

This is more reliable than asking an agent to interpret the interface visually and guess which controls to use.

Challenges We Ran Into

One of our biggest challenges was connecting WebMCP tool calls to live React editor state. Tool actions needed to update the same project state displayed to the creator while returning useful structured results to the agent.

We also had to:

  • Validate scene identifiers and tool arguments
  • Keep WebMCP registrations synchronized with changing editor state
  • Search Pexels securely without exposing the API key
  • Select appropriate portrait-oriented MP4 files
  • Handle missing scenes, invalid durations, empty searches, and API failures
  • Support development testing in browsers without native WebMCP availability

Accomplishments That We're Proud Of

We created a working WebMCP-enabled editing environment rather than only a static interface.

Our prototype includes:

  • Seven functional WebMCP tools
  • Read and write access to shared editor state
  • Validated caption and duration updates
  • Real Pexels stock-video search
  • Server-side protection of the Pexels API key
  • A deployed web application
  • A development shim for inspecting and testing tool calls

We are especially proud that edits made through WebMCP appear in the same visual editor used by the creator.

What We Learned

We learned that agent-friendly applications need more than conversational interfaces. Agents work more reliably when applications expose clearly named tools, structured inputs, validated operations, and useful results.

WebMCP allowed us to represent video-editing operations as explicit capabilities instead of requiring an agent to guess its way through the interface.

We also learned that human control requires careful product design. The current prototype allows creators to review the editor state and stock-footage results, while more comprehensive approval gates, previews, and undo history remain important next steps.

Current Scope

Agent Studio is a hackathon prototype built around a bundled three-scene sample project.

Currently:

  • Caption, timing, and visual changes affect the active browser session.
  • Project changes are not persisted to a database.
  • The project does not yet export a newly rendered video file.
  • Not every direct editing mutation is technically approval-gated.
  • Stock-footage search requires a configured Pexels API key.

These limitations define the current scope and provide a clear path for future development.

What's Next for Agent Studio

We plan to:

  • Add explicit proposal, approval, rejection, and undo operations
  • Add WebMCP tools for transitions, audio, effects, and layouts
  • Provide side-by-side previews before applying visual changes
  • Add persistent projects and edit history
  • Support video rendering and export
  • Improve stock-footage recommendations for individual scenes
  • Support multiple specialized editing agents
  • Allow creators to save, reuse, and share agent workflows

Built With

Share this project:

Updates

Submission history