Wander

Wander turns any phone into a wearable sixth-sense for the blind: AR localization, 3D mapping, voice, and haptics help you navigate indoors hands-free, built first for blind users.

Comment

Inspiration

GPS stops at the door.

Outdoor navigation for blind and low-vision travelers works reasonably well. Indoors there's almost nothing. A white cane is great at finding the ground in front of you, but it can't tell you which hallway leads to room 3B, or that the elevator is 14 meters ahead on your left.

We didn't want to replace the cane. We wanted to add the layer it was never meant to cover, and let someone you trust help out from anywhere.

What it does

Wander turns a real building into a walkable 3D map and guides a person through it using a chest-worn phone, live obstacle cues, and a voice agent.

  • Scan once. Walk the building with Scaniverse. The export becomes a Gaussian splat for the web and a Niantic VPS map for centimeter-level localization.
  • Map it in the browser. A sighted helper opens the Wander web app, orbits the splat, measures distances, drops waypoints, and pins notes like "Bed 1" or "Water fountain." Wander can also propose the entire waypoint graph automatically from the scanned mesh.
  • Wear it. An iPhone on the chest localizes against the VPS map, streams its pose to the backend, and watches the floor ahead with LiDAR. Obstacle detection runs entirely on-device.
  • Talk to it. "Where am I?" "What's around me?" "Guide me to room 101." A full-duplex voice agent answers from the map data and speaks turn-by-turn cues as you walk: distances counted down, landmarks named, and quiet when there's nothing to say.
  • Get help remotely. A friend anywhere can see the wearer's live position on the splat and click a destination. The wearer hears the route.
  • Feel it. Extra phones on the sides and back act as haptic buzzers, so directional cues don't have to go through the ears.

Put a chair in the path and the guidance tells you what's there and offers a way around. If the camera doesn't agree with the route, Wander says it's unsure and stops rather than guessing.

A note on the hardware

The shipping version of this is a haptic band: four small buzzers, one strap, one phone. We didn't have 36 hours to spare for hardware, so we prototyped the band with the thing we had four of, which was iPhones. The phones on the left, right, and back run no logic at all. They receive a channel and a buzz pattern over the local network. Treat them as a stand-in for a fifteen-dollar component, not as a requirement.

How we built it

Capture and localization. Scaniverse produces the .spz Gaussian splat and a mesh; Niantic Spatial hosts the VPS site. On the phone, the Niantic Spatial SDK consumes our ARKit session and turns anchor updates into a 6DoF pose in the site frame. We mirror every VPS image query (frame, status, resulting pose) back to the backend so we can debug localization from the browser.

iPhone app (Swift, SwiftUI, ARKit). One app, four roles. The front phone runs ARKit with LiDAR depth, obstacle detection (gap-profiling in the depth image, inspired by the Shepherd smart cane), speech, and the voice call. The left, right, and back phones are haptic-only. Obstacle warnings never touch the network, because a late warning is useless: depth frame to haptic buzz measures [FILL: measured ms, e.g. "~80 ms"] end to end, entirely on-device.

Backend (Python, FastAPI, Modal). Worlds live on a Modal Volume as a manifest plus versioned assets. Routing is A* over a waypoint graph, with virtual start and destination nodes snapped onto the nearest walkable edge, so you can navigate to a pinned note and not just a waypoint. From the aligned Scaniverse mesh we rasterize an occupancy grid, erode it by the walker's radius, and grow a clearance-centered visibility graph, which gives a reviewer a proposed graph to start from instead of a blank map. The splat itself is voxelized into a static obstacle map the phone can ray-cast against in any direction. WebSockets carry poses in and cues out. GitHub Actions deploys to Modal on every green push.

Companion web app (Next.js, TypeScript, Three.js + Spark). Import a splat, align it to the VPS frame, measure, draw the graph, pin notes, generate a QR code that configures a phone for a world, and watch the wearer move in real time.

OpenAI. LiDAR knows something is 1.2 meters ahead, but it doesn't know it's a chair you can pass on the left. A Responses API agent with ten function tools turns route state and map facts into something you can act on. Routes are computed only by the deterministic backend; the model never makes up a position. Voice runs on GPT Live with client delegation: the phone streams PCM over our WebSocket relay, the backend bridges to the Live session, and every navigation cue is fed to the voice as commentary so guidance and conversation come from the same voice. Spoken destination names ("bed one") are resolved with a standard-library fuzzy matcher. If a name is ambiguous, the model gets the candidates back and asks.

Elasticsearch. Building documents, map entities, and live session events are indexed with BM25 and dense kNN, fused by RRF and optionally reranked. That way a question like "where is the nearest accessible washroom?" is answered from the building's own guide.

Codex. We mirrored the Niantic SDK documentation into the repo and wrote an AGENTS.md with our conventions so Codex could work against an unfamiliar SDK without inventing API surface. It built most of the companion web app against our style guide while we were busy on the phone and backend. That's how four people covered iOS, localization, backend, and web in one weekend.

Timings

Path Latency
LiDAR depth frame to haptic buzz (on-device, no network) [FILL: ms]
VPS cold localization on entering a world [FILL: seconds]
Spoken question to first word of spoken answer [FILL: seconds]
Backend route computation (A* over the waypoint graph) [FILL: ms]

What happens when it fails

A navigation aid for blind travelers has to be judged on its worst case, not its demo case. Four things can go wrong, and each has a defined behavior rather than a best guess.

The camera disagrees with the route. Wander says it's unsure and stops. It does not pick the more likely of two interpretations and keep walking you.

VPS localization is lost mid-route. [FILL: what actually happens. Something like: "Guidance is suspended immediately and the user is told they've lost position, with the last confident location spoken aloud. Turn-by-turn does not resume on a single re-localization frame; it waits for N consecutive consistent poses." Write what your code does, not what would sound best.]

The phone throttles. ARKit, LiDAR, and audio streaming together will heat an iPhone until iOS reduces performance. Obstacle detection is the last thing to be degraded, because it is the only subsystem where a delay is a physical hazard. [FILL: what you actually shed first — voice? pose streaming rate? frame rate?]

The network drops. Obstacle detection and haptics are entirely on-device and continue working. Voice conversation, remote assist, and knowledge lookup stop. The wearer is told that they've lost the assistant and still has the layer that keeps them from walking into things.

The order of that list is the design. The cane always works. The buzzers work without a backend. The voice works without the network only to say that it can't help. Every layer above the physical one is allowed to fail loudly.

Privacy

The wearer is carrying a camera through a building, and a remote helper can watch their position move through it. That deserves a straight answer.

  • Remote assist is session-scoped and wearer-initiated. [FILL: confirm or correct. How does a helper get access? QR code, invite link, always-on? Say exactly.]
  • Camera frames go to a vision model during mapping, not during navigation. Scene understanding runs on captured frames at map-build time and produces candidate annotations for human review. Live navigation streams pose, not video. [FILL: confirm this is accurate for your build.]
  • VPS image queries are mirrored to the backend for debugging. This was a development affordance and would be opt-in or removed in anything real.
  • Session event logs are stored per world. They're what makes obstacle hotspots possible, and they're also a movement history of a specific person inside a specific building. Any real deployment has to let a user turn that off and still navigate.

We didn't solve this in a weekend. We're naming it because a system that watches where blind people walk should not get to be vague about who else is watching.

Accessibility of Wander itself

The companion web app is intentionally sighted-only. It's a mapping tool for a helper, and the visual splat is the point.

The wearer-facing iPhone app is the one that has to be usable without sight. [FILL: state the truth. "Fully VoiceOver-navigable" / "VoiceOver-navigable for the core flow: start a world, start navigation, stop navigation, ask a question" / "Partially, and here's what's missing."] Most of the app's surface is deliberately not visual at all: once a world is loaded from a QR code, the primary interface is speech in and speech and haptics out.

Challenges we ran into

  • Scan processing takes hours. The scan had to start before anything else could be built, and every mistake in the walkthrough cost an afternoon.
  • Three coordinate frames. The splat, the VPS map, and our waypoint graph all disagree until you align them. Getting p_world = R · (s · p_splat) + t right, and keeping the graph in the frame the phone actually localizes in, took longer than the routing did.
  • Voice delegation has no "end of utterance" event. GPT Live tells you a delegation happened and gives an offset; you have to assemble the request from transcript fragments yourself. We built a grace-window transcript collector and pass recent dialogue along, so "yes," "the first one," and "not that one" still work.
  • ARKit + LiDAR + streaming makes an iPhone throttle. Rehearsals had to be planned around cooldown, and it forced us to decide explicitly which subsystem degrades first.
  • Two things want the speaker. Obstacle warnings and navigation cues both matter. Making them sound like one voice, and moving obstacle warnings to haptics while a call is active, took a while to get right.

Accomplishments that we're proud of

  • A destination clicked in a browser or spoken into a chest-worn phone becomes spoken directions in a real building, with live tracking and obstacle warnings fast enough to act on.
  • Obstacle detection stays on-device, so the safety-critical path never waits on the network.
  • Wander doesn't give instructions it can't verify. When it's unsure, it says so and stops.
  • A waypoint graph can be proposed from the scan geometry and validated against walls and floor height before a human accepts it.
  • Every failure mode has a defined behavior, and the ones that matter most degrade last.
  • 160+ backend tests, offline simulators for navigation and the assistant, and CI deploys to Modal.

What we learned

Don't cover the ears. Blind travelers navigate by ear, and a device that fills your ears is taking away the sense it's supposed to support. Once we understood that, everything got quieter and moved toward haptics.

Geometry and language are good at different jobs. LiDAR is fast and precise but has no idea what it's looking at. A language model understands meaning but is too slow for a safety loop. Splitting the work between them made both halves better.

Design the failures before the features. The useful question wasn't "what can it do?" but "what does it do when it doesn't know?" Answering that first is what let us ship something that stops instead of guessing.

Give your coding agent the docs. Thirty minutes spent mirroring SDK documentation saved hours of debugging invented method signatures.

What's next for Wander

  • Live scene description. Today, vision runs at map-build time: frames are analyzed once and become reviewed annotations. The next step is answering "what's in front of me right now?" from the live camera during navigation, which is a different and harder problem because there's no human reviewer in the loop.
  • Multi-floor worlds with elevator and stair transitions.
  • A real haptic band instead of three borrowed iPhones.
  • Real users. This is our best guess at what helps. The next version should be shaped by blind travelers, not by four sighted people at a hackathon.
  • More buildings. Scaling means scanning more places, not rebuilding the pipeline.

Sponsor Track Submissions

These two sections are written to stand alone. A judge who has never seen the project should be able to read either one cold and understand what was built and why.

OpenAI Challenge

Wander is a wearable indoor navigation assistant for blind and low-vision travelers. OpenAI is the reasoning layer that turns a camera frame and a spoken question into guidance a person can safely act on.

The central design decision: OpenAI handles perception and language; deterministic code handles anything a blind user has to trust their safety to. The model never invents a position, a route, a distance, or an accessibility claim. It decides what to look up and how to say it. Everything it says about the world comes back from our navigation graph.

That constraint is what made the rest possible, and it shows up in three separate places in the pipeline.

1. A grounded navigation agent (Responses API)

A strict function-calling agent sits between the user and the building. It has ten tools and no ability to answer from its own knowledge:

Tool What it's allowed to return
get_current_location Live VPS pose from the phone, nothing else
resolve_destination Only real nodes in the waypoint graph
set_destination Starts deterministic A* routing
stop_navigation Halts guidance
search_places Navigable destinations only
search_context Non-navigable entities (hazards, room contents, pinned notes)
search_building_knowledge Chunks of the building's own documentation
get_recent_events This session's event log
get_obstacle_hotspots Historical obstacle aggregations, explicitly labeled as history
[FILL: 10th tool] [FILL: what it returns]

The separation between search_places and search_context is a safety decision, not an ergonomic one. A blind user asking "what's around me?" and a blind user asking "take me there" need answers drawn from different evidence. A hazard is something to be told about. It is never something to be routed to. Splitting the tools makes that impossible to get wrong at the model layer.

When a spoken destination is ambiguous ("bed one" could match three beds), the tool returns candidates rather than a guess, and the agent asks. The model is allowed to be uncertain out loud. It is not allowed to resolve uncertainty by picking.

2. Real-time voice guidance (GPT Live)

The phone streams PCM audio over our WebSocket relay into a Live session, so a user can just talk: "where am I?", "guide me to the elevator," "never mind."

Two things make this more than a voice wrapper:

  • Delegation to the grounded agent. Anything requiring a real lookup is silently handed to the Responses agent above. The user never experiences a handoff, but no fact reaches them without passing through the deterministic layer.
  • Backend-controlled wording for safety-critical cues. "Arrived," "off-route," and "lost" are spoken with exact fixed phrasing supplied by the backend. The model narrates them; it does not decide when they're true or how to soften them. Navigation cues are injected into the Live session as commentary, so turn-by-turn guidance and free conversation come out of the same voice rather than fighting over the speaker.

The hardest part was that Live's client delegation has no end-of-utterance event. You're told a delegation occurred and given an offset, and you have to reassemble the actual request from transcript fragments yourself. We built a grace-window transcript collector that accumulates fragments and passes recent dialogue along as context, which is what makes follow-ups like "yes," "the first one," and "not that one" resolve correctly instead of being treated as fresh queries.

3. Vision-based scene understanding (Structured Outputs)

Captured frames run through a vision model constrained by a strict Pydantic schema. This runs at map-build time, not during live navigation, which matters: there is a human reviewer in the loop for every single output.

Each proposed destination, landmark, or hazard comes back with:

  • an uncertainty note in the model's own words, and
  • a permanence label: fixed, movable, or temporary.

Permanence is the field that matters. A doorway is fixed and worth routing around forever. A cardboard box is temporary and should decay out of the map. A chair is movable and belongs in warnings, never in routes. A vision model that returns "chair, 85% confidence" gives you no way to make that call; a vision model forced to commit to permanence does.

Nothing from vision reaches a blind user's map unreviewed. Every finding is a candidate until a sighted reviewer approves it in the web app. That is deliberately slower than the alternative, and it's the right tradeoff when the failure mode is walking someone into a wall.

Why this split was the right call

We could have let the model do routing. It would have demoed fine and been unsafe. The version we built can say "I'm not sure, I've stopped" and mean it, because the thing that knows where you are is not the thing generating sentences. Grounding wasn't a constraint we worked around. It's the feature.

If you're judging remotely

The clearest evidence is in the demo video, where a spoken destination becomes deterministic turn-by-turn guidance, and in the ambiguity handling, where the agent asks instead of picking. Our offline assistant simulator in the repo replays agent conversations without hardware, if you want to see the tool-calling behavior directly.

Elasticsearch Challenge

Wander guides blind and low-vision travelers through buildings. Elasticsearch is how it answers questions about a building that aren't on the map: "where's the nearest accessible washroom?", "what's in this room?", "does this hallway usually have something in the way?"

Those three questions need three different kinds of retrieval over three different kinds of evidence, and getting them confused would be dangerous. That's the problem ES solves here.

Index design

Index Contents
building_knowledge Chunked building documentation (floor plans, accessibility guides, building PDFs)
map_entities Every waypoint, destination, reviewed annotation, and context note, with x/y/z, category, permanence, navigation_role, is_destination
live_events Obstacle and navigation event log from live sessions

Hybrid retrieval (elastic/search.py)

RRF fusion of BM25 (multi_match over title / text / name / aliases) with kNN over a dense_vector embedding field, optionally reranked through text_similarity_reranker when ELASTIC_RERANK_ENDPOINT is configured.

Both halves earn their place. Room numbers and building jargon ("3B," "Lazaridis," "accessible washroom") are exactly where lexical search wins and embeddings blur. Natural spoken phrasing from a user who doesn't know the building's vocabulary is where dense retrieval wins. A blind user talking to a phone produces both in the same sentence.

Embeddings are generated through Elastic's own inference.inference endpoint (Jina AI by default, OpenAI if configured), so Elasticsearch owns the embedding call rather than the application calling an embedding provider directly. One less place for index-time and query-time embeddings to drift apart.

Spatial rescoring without geo_point

When the wearer is localized, a custom script_score rescore re-ranks results by true Euclidean distance in the building's local x/y/z frame.

This is deliberately not geo_point. There is no real-world lat/long in a Gaussian splat scanned indoors; the entire world lives in a local coordinate frame defined by the VPS map. Forcing it into geospatial types would mean inventing an earth anchor and inheriting a distance metric that ignores floors. Euclidean distance in the native frame is both more correct and cheaper, and it means "nearest washroom" resolves to the one on this floor.

ES|QL for event analytics

Two tools run on ES|QL rather than the search DSL:

  • get_recent_events reconstructs a session's event history.
  • get_obstacle_hotspots runs a real STATS ... BY nearest_waypoint_id, floor aggregation over recent obstacle reports, surfacing places that repeatedly cause trouble.

Hotspots are framed in the UI and in the agent's tool description as historical pattern, never a live safety claim. "People have reported obstacles here before" is useful context for a router. "There is something there now" would be a lie that a blind person might act on. ES|QL gives us the aggregation; the framing discipline is ours.

Retrieval that can't leak across the safety boundary

The tooling splits navigable from non-navigable evidence at the query layer:

  • search_places and resolve_destination return only real graph destinations. A route can only ever terminate somewhere the deterministic router has verified as walkable.
  • search_context filters to is_destination: false and returns category and role aggregations. This is the "what's in this room" tool: hazards, pinned objects, room contents. It is descriptive, and it is never a live sensor feed.

An LLM with one general-purpose search tool over all three indices would eventually route someone to a hazard, because nothing in the index would stop it. Enforcing the split in the ES query rather than in a prompt is what makes that structurally impossible.

Honest limitations

Hotspot data is only as good as the sessions behind it, and a fresh building has none. Building knowledge retrieval is only as good as the documentation the building publishes, which in practice varies enormously. Session logs are also a movement history of a specific person in a specific building, so a real deployment needs to let a user disable them and still navigate. None of those are solved by better retrieval.

Built With

Share this project:

Updates

Submission history