Handoff , an Agent Native Media Editor

Inspiration

I work in developer support for a media/API company, and I spend a lot of my day watching people struggle to get an AI agent to do something as simple as "resize this and remove the background." The agent has to fake mouse clicks through a UI that was built for us humans, it is slow and breaks the moment a button moves. When I saw this WebMCP Challenge, I thought about what if a webpage could just tell an agent what it can do, instead of making it guess.

What it does

Handoff is a photo editor where every action such as crop, resize, background removal, text overlay, format conversion, export is registered as a WebMCP tool via document.modelContext.registerTool().

The same canvas, same underlying functions, can be driven two ways, 1. By a person clicking buttons, or 2. By an AI agent working in English. Activity feed on the right side of the page tags every action as "You" Or "Agent".

How I built it

Very minimal coding, focused more on the outcome. Next.js on the frontend. Canvas API for crop, resize etc. @imgly/background-removal for background removal. Every transformation happens on-device, in the browser, including the background-removal model itself.

I originally planned to build this on Cloudinary's media API, since that's my day job. Eventually, decided to use the imgly background-removal.

Challenges I ran into

WebMCP wasn't detecting at all during testing. It turned out that the spec itself moved navigator.modelContext to document.modelContext on May 27, 2026, and Chrome only deprecated the old location in Chrome 150. I added a fallback mechanism to handle both.

Accomplishments that I am proud of

All 8 tools are registered, schema-valid, and verified working end-to-end. The whole tool runs with zero backend and zero API keys, which wasn't the original plan but turned out to be the right constraint.

What I learned

I learned that in-browser AI (background removal via a client-side segmentation model) is genuinely viable now, this runs comfortably in a regular browser tab with no server round-trip.

What's next for Handoff

Next big step would be thumbnail extraction and trimming via ffmpeg.wasm for videos. Exploring batch operations and social media export presets as natural tools to register.

Built With

• canvas-api
• css3
• html5
• imgly-background-removal
• javascript
• model-context-protocol
• netlify
• nextjs
• react
• typescript
• webmcp