Spatialize

The floor plan that talks back.

Comment

Inspiration

Blind and low-vision visitors rehearse a route before entering an unfamiliar building, because getting it wrong means a wasted trip or being stranded. The data they need usually exists — as a flat PDF nobody can ask questions of. And it rots, because updating it means a human opening an app and filling in a form.

I had already built the part that makes a plan answerable: upload a floor plan, a vision model proposes rooms, doors and a route graph, and a deterministic topology validator decides whether it ships. What that version couldn't do was let the person standing at the door change it, or let their own assistant ask it anything.

That's the gap WebMCP closes, and it's a strange one: the answer a disabled visitor needs is not on the page. There is no screen anywhere that says "can I get from the north entrance to the quiet room without stairs, through doorways at least 760 mm wide?" That answer has to be computed from the building's geometry, in the venue's own session, at the moment somebody asks. A DOM-scraping agent can't find it, because it was never rendered.

What it does

Spatialize publishes a venue's floor plan to any agent in your browser as fourteen WebMCP tools on document.modelContext. With them, a person and their agent can do six things that were difficult or impossible before:

  • Ask a building a question it couldn't answer. find_step_free_route runs Dijkstra over the validated route graph. Google Maps knows whether a building has an accessible entrance; it doesn't know the topology inside. The route is computed, never estimated.
  • Hear why, not just no. When no step-free route exists, the tool re-runs against the unrestricted graph and names the exact door that blocks it, with its clear width — instead of returning nothing.
  • Check a route against one specific body. check_route_clearance takes the narrowest doorway that person can pass, in millimetres, and answers CLEAR, BLOCKED or UNKNOWN — saying unknown rather than clear whenever a doorway on the route is low-confidence or disputed, and saying how old the data is. CHI 2025 (N=190) found scooter users judge 46% of barriers impassable against 28% for cane users, so a single accessible-yes/no bit is wrong for somebody by construction. The tools give the measurement and leave the threshold to the person, because the threshold is theirs.
  • Fix the building's data by talking, and be refused when wrong. Four propose_* tools file what a visitor found on the ground. Every one runs through the same topology validator that guards the rest of the app. An impossible change comes back with the exact rule and field path, so the agent self-corrects — where a DOM-scraping agent would have gotten a silent success and left a corrupted building behind.
  • Disagree with the building, on the record. A venue can decline a report. It cannot delete one. Declined reports stay as disputed claims and list_disputed_claims tells the next agent both sides — because in the 2024 Euan's Guide survey (n=6,665), 77% of disabled respondents found venue-published access information misleading. The venue is the least reliable source on its own accessibility, so it doesn't get the last word.
  • Rehearse a closure before it happens. simulate_closure answers which destinations lose step-free access if a lift goes out of use, and changes nothing. It's the question an access manager faces before every works notice, and no button on any venue site does it.

Nothing an agent says goes live. Every write is a proposal: gated in the browser for fast feedback, re-applied and re-validated on the server from a five-field mutation, then decided by a human. You can also just talk to it — gpt-5.6-luna runs a function-calling loop over the same tools, and a spoken "mark the gallery door as not accessible" files a proposal through the same review ledger, with the spoken sentence kept as its provenance.

Why this has to live in the browser rather than behind an API: when my agent reports a step, the proposal card lands in the review queue on the screen I'm looking at, the 3D view moves to the door it means, and when the venue declines it, the dispute is visible to me and to the next visitor's agent in the same window. The venue's validator, the venue's record and the person's own eyes are in one tab.

The economics are the reason to bother. Detailed venue accessibility data is sold today by human surveyors — AccessAble covers about 70,000 venues at 30–60 minutes of surveyor time each; standalone audits run £3,000–£6,000. That model is labour-bound, which is why it covers 70,000 venues and not 15 million. WebMCP turns the update into a sentence spoken at the door, and because the validator refuses incoherent edits before a human ever sees one, quality goes up rather than down.

A demo venue loads itself for every visitor, in its own isolated server-side run, so you can try all of this with no upload, no login and no setup.

How I built it

Tools are registered directly on document.modelContext — no wrapper library — tied to an AbortController, and they're a function of page state, not a static list: the four propose_* tools are only published when there's a venue record to propose against, because a proposal with nowhere to be kept would be a lie. Every change fires toolchange so an idle agent learns the surface moved. Results come back in canonical MCP content form. Reads carry readOnlyHint; anything carrying visitor-authored text carries untrustedContentHint.

The scene contract is written twice on purpose — Zod in the browser, Pydantic on the server — so no unvalidated geometry crosses the wire in either direction. The browser's gate is fast feedback; the server is the boundary. A proposal is five fields, never a scene: the backend applies it to its own copy of the venue, re-validates, and computes the accessibility impact from that. So a "rename this room" cannot also widen a door.

I built against Chrome's published guidance, and it reshaped the surface. Write tools were originally named report_*, which implied they changed things — they queue proposals, so all four became propose_*. And propose_landmark originally demanded metre coordinates, which made models invent numbers; it now takes a room's name and computes the centroid. A test asserts that no tool in the surface accepts an x or z parameter.

Front end is React 19 + Three.js; backend is FastAPI; scene versions, review ledgers and provenance live in Backblaze B2 with SHA-256 manifests. The frontend is on Vercel, the API on Render.

Challenges I ran into

Trusting the browser was a vulnerability, and a review proved it. The first version of the review ledger accepted a candidate scene built by the client. A "rename this room" proposal could therefore smuggle in a widened door and clear the review issues, and the description the venue reviewer approved would say nothing about it. The server now applies the mutation itself and re-validates; a test asserts the installed scene differs from the base only where the mutation says.

A one-second race that only a real browser could find. All my unit tests passed, so I wrote a script that drives the tools through Chrome's own WebMCP surface. It scored 8/13. The store was hydrating from the server by replacing local state about a second after the tools went live, so an agent that acted in that second had its proposal wiped mid-flight. Hydration merges now.

Chrome changed the host API underneath me, mid-project. Running that same script against the deployed app scored 2/13. Chrome had auto-updated 151 → 152, which removed navigator.modelContextTesting and folded the host surface onto document.modelContext: getTools() in place of listTools(), and executeTool() taking the RegisteredTool object rather than a tool name. The page itself needed no change — the tools registered and ran correctly the whole time — but my harness did, and it now runs on 149 through 152. Building on an origin trial means the ground moves.

Auditing myself against Chrome's guides found three real defects. I checked the surface against all 181 checkable rules across the five WebMCP guides, including the newer tool-security and agent-security pages:

  • propose_doorway filled an unmeasured doorway with a 0.9 m default and stamped the evidence as a human observation — laundering my own placeholder into the venue's record as a visitor's measurement. Width and step-free status are now optional; an unmeasured width is evidenced as derived below the confidence floor, so a clearance check answers UNKNOWN rather than CLEAR, and the reviewer's card says "width not measured" beside what they approve.
  • minimum_clear_width_mm: "760mm" parsed to null, which meant "no width given", which meant no doorway was ever compared against it — so the tool could answer CLEAR to a wheelchair user whose chair had never been checked. Unreadable values are now refused, naming the expected format.
  • The 1.5K per-result budget held on my small demo venue and would have broken on a real building. Lists are budgeted line by line now, ordered so a cut drops the widest doorways rather than the narrowest.

Smaller landmines. A proposal and its own audit entry arriving together lost one of the two, every time, until every write to a run's ledger went behind a per-run lock. And gpt-5.6 refuses function tools on chat completions unless reasoning is switched off — a 400 my own catch-all was hiding behind a vague "provider unavailable" until I made it log the real error first.

Accomplishments that I'm proud of

  • A gate an agent can't talk its way past. The critic is a deterministic validator, not another model's opinion, so it never hallucinates approval — and its error messages double as perfect self-correction prompts.
  • Declining without deleting. Giving the venue a veto over first-hand visitor accounts would hand the least accurate party authority over the most accurate one. Making a decline a recorded disagreement rather than an erasure is the design decision I'm happiest with.
  • Refusing to guess. The tools say UNKNOWN when the data is unconfirmed, report widths instead of verdicts, and now refuse to record a measurement nobody took. In a field where the FTC fined an accessibility-overlay vendor $1M for overstating AI capability, being boring about what the data supports is the whole job.
  • 20 out of 20 on tool selection. Given only the published contract and nothing else, gemini-3.6-flash picked the right tool for 20 of 20 things a person might say, with the right arguments in 12 of 12 where one was implied. The cases were written before the first run and never adjusted.
  • 13 out of 13 through a real browser, against the live deployment, with a real click on Reject and a real page reload.
  • 162 automated tests across both halves of the mirrored scene contract.

What I learned

A deterministic validator makes a far better agent critic than an LLM judge: it can't be persuaded, and its field paths are exactly what a model needs to fix itself.

Driving a real browser isn't optional. Two defects — the hydration race and Chrome's changed host API — were invisible to a fully green unit suite and obvious within one run of a script that used the browser's own tool surface.

And the honest limit is worth writing down: the gate checks that a change is coherent with the plan. It can't check whether a report is true of the building. No software can — the best published agentic accessibility auditing scores about F1 0.60. So "validated" here means coherent, never verified on site, and the tools say so in their own output rather than letting an agent assume otherwise.

What's next for Spatialize

Speak IMDF, the OGC indoor format venues already export, so this becomes a validation and agent layer over data venues already have rather than a new format to adopt. Venue accounts instead of a shared reviewer token. Multi-floor plans with lift-aware routing. The review workflow is already built for teams; connecting it to real venue operators is the step that matters.

Prior work disclosure

Spatialize existed before this challenge: it was built for the Backblaze Generative Media Hackathon, where it earned a special mention — that prior project is the 3D canvas, the plan extraction loop, the voice pipeline and the B2 provenance store, and its last pre-challenge commit is 04e9ad8, 3 August 2026. The rules allow an existing app when significant new WebMCP functionality is added during the Submission Period, so this submission is everything after that commit: the entire WebMCP tool surface, the server-side review ledger, the OpenAI voice stack, and 135 of the tests. WEBMCP.md documents the boundary commit by commit, and git log 04e9ad8..HEAD lists every commit in it.

Built With

Share this project:

Updates