Spatialize

The floor plan that talks back.

Comment

Inspiration

Blind and low-vision visitors often rehearse a route before entering an unfamiliar building. The data they need usually exists — as a flat PDF floor plan nobody can ask questions of. Meanwhile, venue accessibility information goes stale because updating it means editing CAD files. Both problems have the same shape: the floor plan can't listen and can't speak. We built the version that does.

What it does

Spatialize turns a venue plan into a spatial twin you can talk to.

  • Upload a plan; a vision model proposes rooms, doors, landmarks, and a route graph in metric coordinates. A deterministic topology validator — doors must sit on room boundaries, routes must pass through their doors, distances must match geometry — decides whether it ships. Failed attempts get their exact errors back and try again.
  • Ask questions by voice or text: "How far is the studio from the entrance?" The agent computes over validated geometry only, and answers in a generated voice with the route drawn in 3D.
  • Edit by speaking: "Mark the gallery door as not accessible." The change goes through the same validator, and if it strands a destination without a step-free route, the venue tells you so out loud before flagging the new scene version for human review.
  • Everything — plans, transcripts, scene versions, narration — is stored in Backblaze B2 with SHA-256 genblaze manifests. Any vertex in the scene traces back to either a model run or a quoted human utterance.

A demo venue loads for every visitor, so judges can try the voice features with zero setup.

How we built it

FastAPI + LangGraph backend, React + Three.js front end, one Docker container on Render. The scene contract is written twice on purpose — Pydantic on the server, Zod in the browser — so no unvalidated geometry crosses the wire in either direction. Extraction uses genblaze's AgentLoop with our validator as the Evaluator; speech-to-text and all three narration tiers run as genblaze pipeline steps, two of them through provider adapters we wrote ourselves (Sarvam Bulbul, self-hosted Kokoro) because they weren't in the SDK's matrix. The demo video was produced by the project's own tooling: Playwright drove the live app, and Kokoro — our open-source narration tier — read the script.

Challenges we ran into

Free-tier engineering was the real boss fight. Gemini's 5-requests-per-minute ceiling starved a multi-step agent mid-conversation, so we added client-side pacing and cross-provider failover. Long voice requests blocked the server's event loop, so Render's health checks killed the instance mid-answer — moving the endpoint into the threadpool fixed a whole class of 502s. Running Kokoro-82M in-process blew the 512MB memory limit, which is why the deployed fallback is Sarvam's API and Kokoro is reserved for larger self-hosted deployments. Two smaller landmines: B2's S3 API rejects master keys outright, and our private bucket returned 401s on "public" sink URLs until we re-signed them server-side.

Accomplishments that we're proud of

  • An extraction loop that cannot ship a geometrically broken scene, because the critic is a validator, not another model's opinion
  • A voice edit that severs a step-free route gets announced out loud — the gate catching a real accessibility regression is the demo's best moment, and it's not scripted
  • Three-tier narration failover, all inside genblaze pipelines, all manifested: commercial primary, API fallback, open-source floor
  • 27 automated tests across both halves of the mirrored scene contract
  • Per-visitor demo isolation, so no judge inherits another judge's edits

What we learned

A deterministic validator makes a better agent critic than an LLM judge: it never hallucinates approval, and its error messages double as perfect refinement prompts. We also learned that "works in a curl test" and "works in a browser" are separated by autoplay policies, presigned URLs, and event-loop starvation — the last mile of a voice product lives in that gap. And that the best demo insurance is making the empty state real: the server-side demo venue turned every blank-slate visitor into a working session.

What's next for Spatialize

Streaming transcription so words appear as you speak, multi-floor venues with elevator-aware routing, and publishing approved scenes as portable packages a screen-reader can consume offline. The review workflow is already built for teams; connecting it to real venue operators is the step that matters.

Built With

Share this project:

Updates