ShipReel

Inspiration

Every demo video starts the same way: a raw voiceover, a few screen recordings, and forty minutes of dragging clip edges around.

The work isn't really creative. It's finding the twenty-three places where I paused, trimming each one, then hunting for the moment I said “here is the terminal” so I can put the terminal footage there.

That is exactly the kind of tedious, constraint-satisfying work an agent should do.

But most AI video editors solve it by moving your footage to a server. For a screen recording of an unreleased product, that can be a non-starter.

WebMCP changes the shape of the problem. The agent doesn't need your files — it needs your editor.

What it does

ShipReel is a video studio that an agent can drive through WebMCP while I stay in control.

I can set a target runtime and give the agent a goal such as:

“Cut this down to 60 seconds and put the terminal recording where I talk about the terminal.”

The agent can inspect the timeline, detect dead air, search the transcript, place relevant footage, trim clips, add zooms, and optimize the edit until it fits the runtime budget.

In a test run, a single optimize_duration call took the bundled 108.7-second edit to the 60-second target.

Every action appears in an Agent Activity panel in plain English:

• “Removed 26.4s of dead air”
• “Dropped 4 lines of narration — 20.3s”
• “Found "Here is the terminal" in the narration at 17.4s”
• “Placed the terminal recording at 17.4s”

The underlying WebMCP tool name and arguments remain visible underneath as evidence.

My own manual edits appear in the same activity feed, so a drag I make by hand and a cut made by the agent become one shared history.

The footage doesn't have to be the bundled kind. Dropping a screen recording onto the timeline adds it to the project — nothing uploads, and it takes exactly the same path as the bundled clips, so every tool treats it identically. The clip id comes from the filename, so login-flow.mov lets me say, “put the login flow clip at 45 seconds.” The agent sees the imported footage on its next get_project_state, the same way it sees a manual edit.

Why this had to be WebMCP, not a backend MCP server

Most agent integrations can use a remote MCP server. This one is different.

The media is held in page memory. HTMLVideoElement decodes video, Canvas composites the preview, the Web Audio graph handles audio, and encoding uses MediaRecorder — all inside the tab.

There is no remote media service that needs to own the edit.

A backend MCP server would not naturally have access to the live timeline state, decoded audio, or the rendered frame the user is looking at.

The clearest example is preview_at.

preview_at does not change the edit. It moves the user's playhead so the agent can show the human exactly what it is referring to.

Over a backend-only protocol, that tool has little meaning. With WebMCP, the agent can operate on the same live interface the human is watching.

During testing inside ChatGPT's built-in browser, the agent independently called preview_at at 17.4 seconds to show the clip it had placed. I had not explicitly asked it to preview that timestamp.

That was the point where the premise stopped being an architectural argument and became a visible demonstration.

The agent is a guest in the editor, not the other way round.

How I built it

ShipReel is built with React, TypeScript, Zustand, Tailwind and Vite. The editing engine is written from scratch without a video-editing library.

Ripple delete across tracks

remove_ranges accepts an array of time spans and closes all the gaps in one operation, keeping narration and footage aligned.

Instead of making the agent issue a separate request for every silent section, it can remove a whole set of ranges in a single tool call.

Silence detection

find_silences analyzes the decoded audio with windowed RMS and derives a threshold from the recording itself.

The bundled voiceover contains 28.3 seconds of detected dead air.

Runtime optimization

optimize_duration is the high-level tool.

It first reclaims low-cost time such as silence, then escalates to progressively more expensive edits such as filler removal and whole-sentence removal until the edit fits the requested runtime.

It protects the opening sentence from becoming a candidate because saving a few seconds is not worth destroying the hook.

When multiple lines could close the remaining gap, the optimizer prefers the shortest sufficient line, avoiding unnecessary loss of narration.

Real-time in-tab export

The export path drives the same player/compositor used for preview and captures its canvas and audio graph with MediaRecorder.

The render clock runs in a Worker instead of relying on requestAnimationFrame, so rendering is not tied directly to the visible frame loop.

Fourteen WebMCP tools

Fourteen tools are available across two states.

At boot — 7 tools:

• get_project_state
• search_transcript
• find_silences
• optimize_duration
• remove_ranges
• place_clip
• preview_at

After footage is present — 7 additional tools:

• trim_clip
• move_clip
• set_clip_speed
• add_zoom
• remove_clip
• get_export_status
• export_video

The clip-editing tools register only when there is footage on the video track, so the agent is not offered actions that are meaningless in the current state.

Tool descriptions are designed around one job per tool, non-overlapping responsibilities, bounded outputs, and recovery-oriented error messages.

Human + agent collaboration

ShipReel is not designed around replacing the human editor.

The agent handles repetitive operations and constraint satisfaction. The human can still inspect the result, move clips, change timing, or intervene at any point.

For example:

“Preview the terminal section.”

The agent can call:

preview_at(17.4)

I see the exact part of the edit it is referring to, decide whether the placement is good, and continue directing the work.

That makes the browser itself the shared workspace between human and agent.

Challenges

Making “hit a target runtime” an agent problem instead of a guessing game

The first version reclaimed whatever time it could and then stopped, returning candidate sentences for me to choose between.

That worked, but it did not solve the actual problem.

The rewrite turned runtime optimization into an iterative constraint problem:

1. reclaim silence
2. reclaim low-cost filler
3. remove whole sentences only when necessary
4. stop immediately when the target is satisfied

A subtle selection detail mattered. When multiple lines could close the remaining gap, the optimizer chooses the shortest sufficient line instead of deleting more content than necessary.

That turned a 60-second target from a rough approximation into a controlled optimization process.

A silence-detection bug I only found by writing evals

find_silences originally truncated its returned list at a response cap while still reporting the full count.

That meant an agent could remove only the ranges it received and incorrectly believe the job was complete.

I changed the tool to make truncation explicit and to instruct the agent to call again for the remaining ranges.

The fix came from an agent-level evaluation, not from a traditional unit test.

Layout that only broke where it mattered

The preview frame used an aspect-ratio box with a max-height, which behaved correctly at desktop widths but failed inside the narrower layout used by ChatGPT's browser pane.

I found it in a responsive audit at 900×800 — and then realised ChatGPT's browser pane renders below that same breakpoint, so it would have hit exactly there.

The content-sized grid row allowed the preview to grow beyond its intended slot and paint over the timeline.

What I learned

Write evals for the tool surface, not just the code

I built a 17-task evaluation suite that drives real Chrome with WebMCP enabled and executes the same sequences a competent agent would perform.

The suite caught real bugs that ordinary unit tests did not expose.

It also runs against the production deployment, so I can validate the same behavior that a judge will see.

The seventeenth task covers bring-your-own-media end to end: a screen recording goes through the real file input, the agent is told about it, the footage is placed by name, and the resulting canvas is verified to remain untainted — an important check because imported media can otherwise silently break browser export.

Tool design is product design

The most useful WebMCP decision was adding the high-level optimize_duration tool.

Without it, an agent has to manually orchestrate silence detection, transcript search, filler removal and sentence selection across many calls.

With it, the agent can express the user's goal directly while still using the lower-level tools for the rest of the editor.

Show the reasoning, not the log

Raw tool calls prove that the plumbing works, but they are difficult to interpret.

The Agent Activity panel leads with plain-English outcomes and keeps the underlying tool invocation underneath as evidence.

That makes the agent's decisions understandable without hiding the implementation.

Validation & Testing

ChatGPT Desktop App

I tested ShipReel in ChatGPT's built-in browser using a WebMCP-enabled model.

In the successful end-to-end run, the agent:

• read the project timeline and current state
• called optimize_duration to reach an exact 60.0-second runtime
• used search_transcript to find “here is the terminal” in the narration
• placed the matching terminal footage at 17.4 seconds
• independently called preview_at to show the result without being asked

WebMCP discovery also behaved as intended: 7 tools were registered at page load, increasing to 14 once footage was present.

Google Chrome

I also built an automated evaluation suite in scripts/evals.mjs using Chrome with WebMCP enabled.

The suite drives ShipReel's WebMCP tools over CDP and asserts resulting project state instead of relying only on visual output.

It contains 17 agent tasks covering tool discovery, transcript search, silence detection, timeline editing, runtime optimization, preview behavior, export state, and bring-your-own-media.

The suite runs against localhost or the production deployment.

All 17 evaluation tasks pass against the live ShipReel deployment.

Potential impact

The first users are developers, founders, technical educators, and product teams who regularly turn screen recordings into product demos, tutorials, launch videos, and documentation.

The pain is specific: the footage already exists, but the editing requires dozens of small, repetitive, context-dependent decisions.

ShipReel turns those decisions into structured operations an agent can perform against the live editor.

The privacy benefit is equally specific: the editing pipeline is designed so the media can remain in the browser instead of requiring raw recordings to be uploaded to a remote rendering service.

The goal is not to replace professional editors.

The goal is to remove the mechanical work from technical video editing while keeping the human in control of the result.

What's next

The audio analysis itself is real: find_silences runs windowed RMS analysis over the decoded audio, with its threshold derived from the take's own loudness. And you can already bring your own screen recordings — drop them on the timeline and the agent places them by name.

What is not yet live is automatic speech recognition, which is why the voiceover still has to be the bundled one: the transcript is what the agent edits against.

1. In-browser transcription — generate the timestamped transcript locally with transformers.js, so bring-your-own-voiceover works without a pre-authored transcript.

2. Faster rendering — replace the MediaRecorder export path with a WebCodecs encode pipeline.

3. Richer editing — multi-track editing with transitions, once the core agent-native workflow is solid.

The core idea

The browser already contains the media, the timeline, the transcript, the rendered frame, and the user's decisions.

WebMCP gives the agent a way to operate on that environment directly.

ShipReel turns that into a practical workflow:

the agent edits, I direct, and the media stays in the browser.

Built With

• canvas
• chrome-devtools-protocol
• html5-video
• mediarecorder
• model-context-protocol
• react
• tailwindcss
• typescript
• vercel
• vite
• web-audio-api
• web-workers
• webmcp
• zustand