Roomwright

Moving in shouldn't take a tape measure and a weekend, nor $1000. Roomwright lets you and an AI agent design a room together in 3D, with 38 WebMCP tools that refuse bad moves and learn from each one.

Comment

Why WebMCP, and what's new: A room planner is a 3D canvas. There are no buttons for an agent to click, so typed tools are the only way it can act on the room at all. And because every call comes back with a real answer, the agent can fix its own mistakes instead of guessing again. That changes what collaboration looks like. You are not handing the room over and waiting for it to come back. The agent places furniture while you drag your own, both of you are refused by the same engine in the same inches, and the log says who did what.

Inspiration

Everyone has done this: You measure the wall, buy the dresser, carry it up the stairs, and find out the drawers hit the bed. Rules exist that decide whether a room works stay invisible until you have already spent the money. A dresser needs 30 inches of clear floor. A door sweeps a 32-inch arc. A desk needs 36 inches behind it for a chair. You have to do meticulous work to find the numbers manually and ai agents can't autonomously interact with simulation apps without prior configuration and set up.

The tools meant to help do not close that gap. CAD planners enforce the geometry but have no idea what you want. AI design tools render rooms that cannot be built, because a model drawing a picture was never solving a constraint problem. One is rigorous and useless, while the other is beautiful and false.

But that change with WebMCP... with Roomwright.

The agent proposes. The engine disposes.

What it does

Roomwright is a browser-based 3D room planner. You drag furniture with a mouse. An agent places furniture by calling the page's WebMCP tools. Both go through the same geometry engine, so both get the same measured refusals and the same suggestions.

Write a one-sentence brief — "nursery plus home office, I'm 6'2", under $1,200" — and OpenAI turns it into clearances and a budget the engine enforces. Real dimensions and prices from real product pages, multi-storey buildings with doors, windows, radiators, closets and staircases, saved builds, undo/redo, a PDF plan, a shopping list, live multiplayer. Sensitive agent requests, like moving a wall, wait for your approval.

The moment that matters: the agent places a desk whose chair clearance runs through the radiator. The engine returns:

{
  "ok": false,
  "violations": [{
    "type": "clearance", "subject": "desk", "against": "radiator",
    "overlap_in": 11, "axis": "z",
    "message": "The 36 inch chair clearance behind the desk overlaps the radiator by 11 inches."
  }],
  "suggestion": { "x_in": 142, "z_in": 30, "rotation_deg": 90 }
}

The agent moves the desk 12 inches. Nobody re-prompted. That refusal is then written into place_item's own description, so the next agent in the room already knows.

Why this use case fits WebMCP

We'd rather be precise than overclaim. Reading dimensions out of a page is solvable without WebMCP. Writing is not. move_item({id, x, z}) is one call against a stable contract, and the structured return is what makes self-correction possible — a synthesized click on a canvas gets no return value. A tool surface is also a permission surface: mutation tools don't exist until a room is loaded, so the app decides what an agent may do and when.

We measured it

Same agent (Claude driving Chrome), same task on the live deployment, run twice: add a crib, desk and dresser to the hard room, crib on a wall away from the window, zero conflicts. Once through the page's WebMCP tools, once through the UI alone. Method, raw logs and caveats in BENCHMARK.md.

WebMCP tools UI only (screenshots, clicks, drags)
Agent actions 10 tool calls 34 browser actions
Wall clock 4.9 s 299 s, then stopped — task incomplete
Placed with zero conflicts 3 of 3 2 of 3, one conflict left
Rejections, and recovered 2, both on the next call 5, none recovered

The gap isn't click speed. A rejected place_item returns the violation, the overlap in inches, and a position that would validate, so the retry is one call. A rejected drag returns a red line of prose and a new guess at a screen coordinate on a perspective floor. Four drags in the UI run were refused and the crib never moved.

What people and agents can do together that was hard before

  • Shared live state, not turn-taking. The agent works in the room while you watch. Grab a piece mid-run and move it — nothing resets, and its next placement accounts for the change.
  • Negotiation in both directions. You state intent in English, the agent turns it into numbers, the engine says what's impossible, and the agent reports back which constraint had to give.
  • Multiple people and agents in one room. Every change is attributed ("You moved the dresser", "Claude placed the chair"). Big changes arrive as a propose_layout ghost anyone can accept or dismiss. Never a silent overwrite.
  • A tool surface that learns. Every refusal becomes a lesson stored with the room, returned in get_room, in later refusals, and inside the mutation tools' own re-registered descriptions. No training — memory plus feedback, shared by everyone in the room.

One sentence replaces a tape measure and a weekend. The agent handles the search space, the engine guarantees the answer is physically real, and you end with a room that fits and a list you could actually buy.

How we implemented WebMCP

document.modelContext.registerTool is called directly in apps/web/src/tools/register.ts, behind a shim handling Chrome 150 semantics (registerTool returns a Promise; removal via AbortSignal):

document.modelContext.registerTool({
  name: "search_catalog",
  description: "Search the furniture catalog. Returns items with real dimensions in inches, price, and required clearances. Filter by max_width_in / max_depth_in to only get items that could physically fit a given space.",
  inputSchema: { /* ... */ },
  execute: async (input) => { /* ... */ }
});

A mount-scoped React hook registers and unregisters tools as state changes: 12 tools on the landing page, 38 in a room, undo only when there's something to undo, camera tools only while the 3D view is live. The brief form uses the declarative API (toolname, tooldescription, toolautosubmit, respondWith), so the form is the set_brief tool.

Group Tools
Read get_room, get_placements, search_catalog, measure, check_clearances, get_budget, describe_view, recall_lessons, list_rooms, list_composed_tools
Write place_item, place_items, move_item, rotate_item, remove_item, swap_item, undo
Space create_room, extend_room, reshape_room, add_fixture, remove_fixture, add_floor, set_active_floor, connect_floors_with_stairs, open_room, delete_room
Self-extension create_catalog_item, compose_tool, remove_composed_tool
Collaboration propose_layout, comment, snapshot, export_plan, set_camera, record_lesson, forget_lesson
Declarative set_brief

compose_tool is the one to look at: an agent captures a routine as a new tool that registers with WebMCP live and persists.

The engine. packages/engine is pure TypeScript — no React, no Three, no fetch. Geometry, walls, floors, validation, ops, and a command pattern with inverses for undo. It's the only place rules live; pointer handlers and tool handlers are both thin callers. State is integer inches; meters exist only at the render boundary.

pointer drag ─┐
              ├─→ packages/engine (ops.ts) ─→ Result{ok | rejected}
WebMCP tool  ─┘        │
                       ├─ ok       → Zustand → R3F re-render → WebSocket fan-out → persist
                       └─ rejected → structured violations back to the caller; state unchanged

Stack. Next.js 15, React Three Fiber, Zustand, Zod, Tailwind on Vercel. FastAPI + Motor + MongoDB Atlas on Render, with Atlas Vector Search over catalog and lesson embeddings and a WebSocket per room. OpenAI Responses API with a strict JSON schema for brief-to-constraints, the Agents SDK for an in-app copilot running the same tools (so the app works in browsers without WebMCP), text-embedding-3-small for search and lesson recall. Key stays server-side.

Challenges we ran into

  • The namespace moved mid-trial. The rules show document.modelContext; some docs show navigator.modelContext. We verified against Chrome 150: registerTool returns a Promise, there's no unregisterTool, removal is via AbortSignal. Everything goes through one shim.
  • Chrome aborts an in-flight executeTool when its tool is unregistered. Re-registering a tool to add a new lesson had to be deferred until nothing is executing — getting the menu to learn without killing the call that taught it took a few tries.
  • Physics wanted to be the truth. Rapier makes drops settle and drags push against walls, but a simulation that varies run to run would eventually differ during judging from rehearsal. Physics is presentation only; snapped integer inches are the truth.
  • Agents invent things. The copilot made up catalog categories until search_catalog validated them, and tried to build a separate room per floor. Multi-storey is now one building, with floors on placements and fixtures.

Accomplishments that we're proud of

The rejection-and-recovery loop works end to end with no re-prompt, and the refusal shows up in the tool's own description afterwards. A person can grab furniture mid-run while an agent is placing, both hit the same rules, and the log says who did what. 38 dynamically registered tools, imperative and declarative, plus agent-composed tools that register live. The product is fully usable with zero AI. 80 tests green (49 engine, 31 API), typecheck, lint and production build clean, deployed.

What we learned

Structured refusals are the whole product. "11 inches into the radiator, try x=142" turns an agent from a guesser into a collaborator; prose errors don't, and the benchmark is what that difference costs. A tool surface that changes with app state is both a capability boundary and the clearest signal an agent has about what's allowed right now. And don't overstate WebMCP: an agent can read a DOM. What it can't do is manipulate a canvas whose coordinates depend on the camera, or get a typed answer back from a click.

What's next for Roomwright

Server-side re-validation of geometry (today the server persists what the client computes, and the README says so). Real CC0 furniture models in place of dimension-accurate kitbashes. A CRDT instead of last-write-wins for larger groups. Retailer integrations so the shopping list becomes a cart. Proposal diffs and comment threads, so agent-to-agent negotiation over a layout stays reviewable by the people in the room.

Built With

  • atlas-vector-search
  • fastapi
  • mongodb-atlas
  • motor
  • next.js
  • openai
  • openai-agents-sdk
  • openai-responses-api
  • pydantic
  • python
  • react
  • react-three-fiber
  • render
  • reportlab
  • tailwind
  • text-embedding-3-small
  • three.js
  • typescript
  • vercel
  • vitest
  • webmcp
  • websockets
  • zod
  • zustand
Share this project:

Updates