Inspiration
The car pulls away. You are on a kerb outside a clinic you have never visited. Which way is the door? Eleven steps or forty? A kerb to step up, a lane where cars drive? Inside, is the desk left or right? A sighted visitor answers all of that with one glance. If you are blind, you bring someone, or you walk in and find out. Over 7 million Americans live with vision loss glasses cannot correct, and more than 700,000 of them are blind (CDC, 2024).
The method already exists. In September 2012, a team at Harvard Medical School and Massachusetts Eye and Ear showed that blind users of an audio only virtual environment built an accurate spatial map of a real building from sound alone. Follow up work found the transfer of navigation skill to the physical building was markedly high (Connors and colleagues, 2014).
So why can nobody do this today? Because each virtual building was rendered by hand from architectural plans: plans, a researcher, weeks of authoring, one building at a time. The research proved the method on a handful of buildings and there was never a way to make the fifteenth. Beacon based indoor navigation has the same flaw: it works, and someone has to install hardware in every building. What is new here. We kept the validated interaction and removed the authoring. Commercial buildings are already scanned in volume to lease office space, and a Matterport capture contains the wall graph, panoramas and floor plan; aerial imagery covers the approach; a property record ties it to an address. Every input that method needed a human to produce already exists, paid for by someone else. So the contribution is not the concept. It is that the concept now applies to any building that has already been scanned, and that number grows every week for reasons unrelated to accessibility.
Who it is for
People who are blind or have low vision, visiting an unfamiliar building:
- Independent cane and guide dog travellers, who want to arrive knowing the layout. Every instruction is in steps and clock directions, the vocabulary orientation and mobility training already uses.
- People newly blind or losing sight, for whom rehearsing in private is less exposing than being escorted.
- People with low vision, who can move but cannot read a sign at distance. Also the companion watching live and coaching without taking over, and the building owner, who gets a layout recommendation backed by a number. What it does not claim. It is not a navigation aid. It does not replace a cane, a guide dog, a sighted guide or a front desk. It runs at home, before the visit, where a mistake is free.
What it does
Keyboard only, one voice, captions that always match the speech word for word. No mouse anywhere. Outside. Type an address and the camera drops out of the sky onto real aerial photography of that property, landing at the drop off. "You are at the drop off, on the southwest side of the building. The entrance is at one o'clock, about eleven steps. Listen for cars before you cross." Press B and a beacon pulses continuously from the real door, panning as you turn and speeding up as you close in. Where the data does not know something, it says the safe thing rather than guessing: feel for a change in the ground with your cane. Inside. You are in the building's actual scan, in first person. Q names what is around you from its true direction. E reports what is ahead. Tab picks a target and the beacon follows it. Walk into a wall and you hear which side you hit. Cross a threshold and the room is announced and the reverb changes. Then it rehearses the route one stretch at a time, with recovery lines if you stall and a spoken warning at the 0.8 metre doorway every route here must pass. Then it tests you. A chime on arrival, the score in plain words, then point to where you think the desk is and hear your error in clock hours, answer three questions about the route, and run it again to compare. The point is not hearing the route once; it is remembering it on the day. Two other views. A companion page where a family member or an orientation and mobility instructor watches live and reads exactly what is being heard. A venue page for the owner: twenty properties ranked by navigability, plus a layout recommendation.
Why this one can reach people
A proven method sat unused for fourteen years because of economics, not science: somebody had to author each building, so coverage stopped at a handful. Here nobody authors anything, and the capture was already paid for to lease office space. The twentieth building costs what the first did. That means coverage grows on its own: every scan added for a leasing reason silently adds a building someone can rehearse. It also gives owners a number instead of an adjective, since the layout advisor says moving one reception desk here takes simulated arrival from 58 percent to 91 percent. And for clinics, courts and campuses it is cheaper than the escort they currently provide one visitor at a time. None of that is the point of the project. It is why the point survives contact with the real world.
How we built it
One pipeline. Three data sources in, a rehearsal out. Every fact carries a chip naming its origin: MP, VX, CS, CALC, AI, HAND. Map. Matterport's wall graph becomes a 10 cm occupancy grid. Fifteen never scanned offices get doorways carved in and interiors blocked, so a route never promises a room nobody captured. Routes. A cane friendly A star that prefers trailing a wall, keeps clearance from furniture, and retries down a ladder of relaxations rather than failing. Routes are split into stretches, scored, then walked by 300 simulated travellers who drift, bump, trail surfaces and get lost. Every resulting number is labelled simulated. Landmarks. A vision pass reads the cached panorama faces and ray casts each detection into the grid; duplicates seen from several panoramas merge and weak ones drop. 47 survive on the clinic, each with a short spoken name derived by code and disambiguated by geometry, so you hear "door by the sink counter", not "door" four times. Language. Every model sentence is checked against the geometry before it may be spoken. The model can phrase a fact; it cannot invent a door. Agents. A LangChain Deep Agents orchestrator runs five subagents: site investigator, route planner, annotator, layout advisor, finaliser. It streams its trace to the UI and falls back to labelled scripted and recorded replays, so a demo never depends on a live model call. Sound. Web Audio HRTF throughout: distance falloff, low pass behind you, per room reverb, speech ducking everything. A continuous beacon marks your target and a separate tick reports the nearest obstacle ahead, so you can hear yourself moving. Voice. One voice, Kokoro am_michael, rendered offline and keyed by a hash of its own text computed identically on both sides, so caption and speech cannot drift. No browser speech anywhere: a missing clip gives a soft tone, never a different voice. Front end. Vite, TypeScript, three.js. One page at a time, no scrolling, keyboard only, WCAG pass on every page.
Challenges we ran into
A graceful fallback hid a dead feature. The first person photo view never connected, and the fallback was so calm that every check read as success. Three causes stacked, and the last took longest: we kept the viewer hidden while it loaded, and a browser will not render a hidden cross origin frame, so it could never report being ready. A swallowed import. The layout advisor reported zero percent arrival for every desk position. One silently caught ImportError had left every route unsegmented. A real doorway, not a bug. Travellers kept failing on one route. A rendered crop of the grid showed a genuine 0.8 metre gap between the reception counter and a glass wall. The fix was not to widen it in software, but to make the simulated traveller trail surfaces the way a cane user does, and to speak the doorway with its step count. The scan did not label its own rooms. Every room came back as "Other" or "Void", so the names a visitor needs most did not exist in the source data. Sponsor data could not leave the laptop. That ruled out cloud GPU work and shaped the pipeline to run offline from a local cache, pulling the mesh out of its bundle with range requests rather than downloading a 65 MB point cloud we did not need.
Accomplishments that we're proud of
- It is automatic, and we proved it. A second building found five bugs one could not, and the first building's output stayed byte identical through every fix.
- It is honest. Provenance on every fact, "possibly" below 0.6 confidence, "simulated" on every modelled number, and safe wording where the data does not know.
- It never needs the network. Offline fallbacks on every path, verified by a 16 of 16 check.
- Accessible from the first commit, not retrofitted: keyboard only, one voice, captions matching speech exactly.
- Built solo, overnight, 257 engine and 545 web tests green.
What we learned
- Make fallbacks loud in development and quiet in production, or they hide what they protect.
- Geometry should outrank the model. The grounding check is why the spoken output can be trusted.
- A second test case is worth more than a hundred more tests on the first.
- An honest 56 percent on a hard route is more useful than a route that always looks perfect.
What's next for SecondVisit
- Test with blind and low vision users through an orientation and mobility instructor. Nothing here has been validated with the people it is for, and that matters more than anything else on this list.
- On demand generation: any address, built in minutes rather than ahead of time.
- Gaussian splat rendering per building, for photo quality visuals with genuinely free movement.
- A day of visit mode on a phone, honest about GPS being accurate to roughly ten metres.
- The layout advisor as its own product for building owners.
Built With
- claude
- costar
- deepagents
- fastapi
- kokoro-tts
- langchain
- matterport
- matterport-showcase-sdk
- numpy
- onnx
- pillow
- pydantic
- pytest
- python
- shapely
- three.js
- typescript
- vexcel
- vite
- vitest
- web-audio-api
