PixlPal

The Problem with diffusion models

AI image editing is getting popular because it allows people without any experience in using image editing software to create beautiful edits of their photos.

However, the problem with AI image editing approaches right now is that they regenerate the photo. This approach is prone to mistakes such as the image generation model adding or removing certain details, or introducing diffusion noise. And since it "edits" by regenerating the whole photo, it leaves little control for the user to tweak certain image editing parameters themselves. It is not a true edit.

PixlPal's Approach

PixlPal believes that AI should instead use deterministic tools (like in photoshop) to truly edit the images directly. So I created a custom Rust engine that exposes tools/APIs for an AI agent orchestrator to edit photos. The coolest part: I added a segmentation model that agents can call so it can select specific objects to apply specific transformations to as well.

Think of it like watching an AI agent use Photoshop or Lightroom while you watch and tweak, instead of having it one-shot regenerate the whole image.

People and Agents working together

PixlPal's approach also creates a better user experience because the user has more control over the editing process. With image gen models, this was impossible: they need to regenerate the image as a whole. It is not a "true" edit. But with PixlPal, users can tweak edits directly if they want.

AI Agents and People working together.

How we implemented WebMCP, and what it brings

I already had a custom-built Rust engine to do image/pixel transformations set up, as well as tool definitions exposed to my built-in AI agent. For this challenge, it was just a matter of bridging those tools to WebMCP.

WebMCP turned out to be a great complementary approach to the built-in agent. Users can now effectively "Bring Your Own Agent" (BYOA) to PixlPal, and their agents might have better specific context of their tastes and what kind of photos they like. Or if they're just more comfortable using their own agent, WebMCP allows users to use their own agent.

Challenges we ran into

Usually people want to apply changes only to specific objects instead of the whole image (think of the lasso tool). Without that capability, PixlPal was just a filter-applier rather than a true agentic image editor. At first, I thought this would mean needing to get an AI agent to use a lasso tool (very hard). Eventually, I realized I can just add an Image Segmentation Model as a tool-call for the agent. So the agent does a natural-language prompt to the segmentation model i.e. "select the red snack bag", and the segmentation model returns the exact pixels of the red snack bag, and then the agent can apply pixel transformations specifically to those pixels.

Accomplishments that we're proud of

• Figuring out how to make AI agents select/segment specific objects easily.
• Creating a working product that lets anyone edits their photos. No background in professional editing needed.

What we learned

Human UI (clicking buttons, moving around cursors, drag and drop) are quite clunky for AI agents to interact with. In contrast, I think of WebMCP as like a UI for agents. WebMCP allows agents to easily navigate and use web apps and websites.

What's next for PixlPal

Image Segmentation Model is currently run client-side via WebGPU. This makes edits where PixlPal needs to segment a specific object slow unless the user has a GPU. In the future, I would like to host and run the segmentation model in the cloud.

Built With

• cloudflare
• florence
• openai
• react
• rust
• typescript
• web-assembly
• webgpu
• webmcp