Mini Paint WebMCP Extension

A real image editor where you and ChatGPT edit the same document. The agent sees the edits you just made by hand; you undo its edits with Ctrl+Z. One shared undo stack, 27 WebMCP tools, no backend.

Comment

Inspiration

Most AI integrations hand the agent a copy of your document and give you a result back. You are not editing together, you are taking turns. And undo belongs to whoever touched it last.

I wanted the opposite. One document, two editors. A real image editor where the person and the agent work on the same canvas, each can see what the other just did, and the same Ctrl+Z undoes either of them. That only works if the page itself says what it can do, which is what WebMCP is for.

What it does

Mini Paint WebMCP is a browser image editor with 27 WebMCP tools registered, so ChatGPT can drive it directly. It is build ontop the famous https://viliusle.github.io/miniPaint/ project.

Four of them read: the document, the layers, the selection, and an export. Seven work on layers, covering text, moving, resizing, duplicating, deleting, selecting and opacity. Three change the document itself through canvas resize, crop and rotation. Three are adjustments for brightness, contrast and saturation. One sits a level higher: create_social_variant reshapes the document into an Instagram, Facebook, LinkedIn, X or YouTube format and carries the existing layout across. Then undo and redo, which work on the same history a human uses.

The sequence that shows the point is short. You drop in a photo and drag it where you want it. ChatGPT inspects the document and tells you what you just did. Ask for a square Instagram version and the canvas reshapes with your layout scaled into it. Ask for a headline, then nudge it by hand into a better spot. Ask for a portrait version and the agent starts from your position, not the one it picked earlier. Press Ctrl+Z and its variant disappears in a single step.

How we built it

Adobe App Builder hosts the whole thing as a static SPA on Adobe's CDN. Nothing in the agent path touches a server. Every tool runs in the browser. Deploys are headless from credentials in .env, so they never depend on somebody being logged in.

miniPaint 4.14.3, MIT licensed, is the editor, vendored as its prebuilt bundle.

The layering matters more than it looks:

ChatGPT  ->  document.modelContext  ->  WebMcpAdapter  ->  ToolRegistry
                                                                |
                                                         EditorCommands
                                                                |
                                                            miniPaint

Every transport goes through the same ToolRegistry.invoke(). WebMCP, the debug panel inside the app, and the test suite all take that one path. No tool logic is duplicated anywhere, so a tool that passes a test is the same code the agent runs.

Challenges we ran into

miniPaint cannot be embedded as a library. Its maintainer says so directly in issue 169. The bundle initialises itself on window load, styles the page from body down, and needs its own HTML skeleton, so a same origin iframe is the only supported way in. The catch is that ChatGPT does not discover tools registered inside an iframe. So the tools are registered by the top level page and the handlers reach into the frame. Getting both of those true at once was the central design problem.

Then undo. miniPaint never exposes window.app, so its own action classes are unreachable from outside. What rescued it is that State.do_action() does no instanceof check. It only wants an object with action_id, action_description, do(), undo() and free(). So I define command objects in the host page and push them onto miniPaint's real history. No fork, no second history, and a human Ctrl+Z reverses an agent edit because it genuinely is the same stack.

The WebMCP API also moved under me while I worked. Chrome 149 and 150 expose navigator.modelContext, 151 and later expose document.modelContext. registerTool returns a promise from 151 but throws synchronously before that. executeTool takes a JSON string in Chromium and an object in the draft IDL. RegisteredTool.inputSchema is a string through Chrome 153 and an object from 154. The adapter handles all of it, and it is the only file in the project that knows WebMCP exists.

Three smaller things cost real time. Only ten of miniPaint's effects implement the filter render hooks, and passing any other name to add_filter throws inside the render loop and kills it permanently, so adjustments go through an allowlist. Dynamic text layers get measured during a requestAnimationFrame tick, which a background tab throttles, so an agent could read width: null and have nothing to position against; measurement is forced synchronously now. And on Adobe's CDN the cache header key has to be lowercase cache-control, otherwise it lands in S3 object metadata and never reaches the browser, which cost 30 seconds of load time until I found it.

Accomplishments that we're proud of

191 automated tests, and not one of them needs an LLM. 87 unit tests, 104 Playwright tests, and the Playwright suite runs unchanged against localhost and against the deployed Adobe URL. 104 of 104 on both. The assertions check miniPaint's own internal state rather than what a tool claims it did, so a tool that lies still fails.

That harness is why I trust the tool layer at all. It caught a spurious REVISION_MISMATCH that would have rejected the agent's own writes, and the throttled measurement bug above, before either reached anyone.

documentRevision came out of the same thinking. A write can carry an optional expectedRevision, and if the human edited in between, the write is refused instead of quietly overwriting their work.

What we learned

Designing tools for an agent is API design, not UI automation. Explicit units, explicit coordinate spaces, additionalProperties set to false so a typo produces a clear error instead of a silently ignored argument, and errors that carry structured detail. LAYER_NOT_FOUND ships the ids that do exist rather than a sentence about it.

The shared state idea only survives if you resist building your own history. Give the agent its own undo stack and it stops being one document and becomes two programs arguing over a file.

What's next for Mini Paint WebMCP

Selection aware tools, so "brighten just the sky" works. A load_image tool so the agent can bring in assets itself. And the orchestration this architecture was built for: applying one layout to six social formats as a single composed operation with a single undo.

Built With

Share this project:

Updates

Submission history