MCPixels

Inspiration

Every AI drawing tool I'd used treated the model the same way: describe a thing, receive a finished image, regenerate. None of them let me keep working on what came back, and none of them let the model keep working on what I made. AI gives you the speed and the general idea, but the details are the user's job, and the user should be able to change what the AI made without starting over. The interesting position isn't "agent makes art". It's two hands on the same document, where either can pick up where the other stopped.

WebMCP is what makes that possible in a browser, or at least much easier and cheaper. The page can hand an agent the same verbs the person has, so there's no separate API surface, no plugin, and no copy of the document living somewhere else.

What it does

MCPixels is a 1024×1024 pixel-art canvas that a person and an agent edit at the same time, in the same tab.

For the person it's an ordinary pixel editor: pencil, eraser, shapes, flood fill, selections, mirroring, PNG import and export, undo. In a WebMCP browser the page also registers six site tools, covering everything (mostly, TBD) a person can do:

• draw_pixel_art: the only tool that carries pixel data
• read_canvas: reads any region back in exactly the format a draw goes out in
• selection: duplicate, move, erase, flip, rotate, copy/cut/paste
• edit: undo, redo, clear
• view: frames a region for the person
• export_pixel_art: saves a PNG to their download folder

The two of you share one canvas and one undo stack, so the agent's edits sit on the same timeline as yours and either of you can step back through the other's work. Agent actions appear in a notice feed.

How we built it

A static React and TypeScript app on Cloudflare Pages. No server, no database. The tools are registered by the page itself with document.modelContext.registerTool, and their handlers call straight into the editor's own reducer, so an agent action and a mouse action take the identical path through the code.

The design constraint that shaped everything is that a tool surface lives in the model's context on every turn, competing with the art for room. So:

• Pictures, not coordinates. Sprites travel as rows of single-character palette keys, like "..kkkk..", rather than [{x, y, color}, …]. It's dramatically cheaper, and the model can see the shape it's writing. . leaves a cell alone and - erases, so stamping never scrubs the art underneath.
• A closed loop. read_canvas returns the same {origin, palette, rows} shape draw_pixel_art accepts, so read → edit the rows → draw back needs no reformatting in between.
• Geometry goes through an ops string. rect, ellipse, poly, path, bucket, recolor, and ordered-dither fills, so large flat areas cost almost nothing.
• Rearranging sends no pixels at all. selection moves art the canvas already holds.

The site also advertises itself to agents through static discovery documents: an RFC 9727 API catalog, an MCP server card, an agent-skills index and a SKILL.md, llms.txt, plus Link headers and DNS records pointing at them.

Challenges we ran into

The hard problems were all about budget and about two actors sharing one document.

Getting the encoding right took the longest. The obvious wire format for pixels is a list of coordinates, and it is completely unaffordable. A single sprite eats a meaningful fraction of a context window. Rows of palette characters fixed the cost, but then the read path had to return that exact shape or the agent would spend its remaining budget reformatting.

Reads also needed a hard ceiling. A read has to cost about the same whether the canvas is empty or covered, so read_canvas caps the region it will return exactly and falls back to a coarser overview beyond that, with an exact flag saying which you got.

Then the shared-document problems. One undo stack for two actors means an agent call has to be exactly one history step, or none, if it changed nothing. Edits that land off-screen are invisible, so the view eases across and the area flashes; but that must never fight the person, so it's skipped mid-gesture or while a panel is open, and can be switched off entirely. And view deliberately only ever zooms out. An agent should be able to show you something, never shove your workspace around.

Accomplishments that we're proud of

That the agent is a participant rather than a pixel pipe. It reads what you drew, rearranges what it drew, undoes your mistakes and its own, and hands you a PNG at the end, all through the same six verbs, with no backend anywhere.

The encoding, in particular. Sending art as pictures instead of coordinates is roughly an order of magnitude cheaper, which is the whole reason a full-featured tool surface fits alongside real work. A ten-frame walk-cycle spritesheet gets built from one draw call plus a handful of selection ops. The other nine frames never travel over the wire at all.

What we learned

Token budget is a design constraint, not an optimisation you do later. Every choice here came from the fact that descriptions and schemas are re-sent every single turn: the row format, the ops string, the read ceiling, keeping the surface to six tools instead of twenty.

The second lesson was that shared-state UX is most of the work. Making an agent able to draw is easy. Making its edits legible, undoable, and non-disruptive to someone already holding the mouse is where the actual product is.

What's next for MCPixels

• Accounts and saved sessions, so a canvas survives the tab closing and you and the agent can pick the same piece back up days later.
• Layers, so a person and an agent can work without painting over each other.
• Animation as a first-class object rather than frames laid out on a grid, with onion skinning and an export that knows what a frame is.
• Image import for agents, the one editor capability the tool surface doesn't yet cover.
• Tighter reads. Run-length rows already exist on the write path; the read path could use them to return much larger regions inside the same budget.
• Multiplayer, which the architecture already half-implies. The canvas doesn't care whether the second set of hands is a model or another person.

Built With

• biome
• cloudflare-pages
• html5
• model-context-protocol
• react
• typescript
• vite
• webmcp