Isaac

Inspiration

In a nursing home, losing your sight can mean losing your independence. Something as simple as finding the dining room, going to the bathroom, or getting back to your own room can become difficult without someone guiding you.

About half of nursing home residents fall every year, and poor vision roughly doubles that risk. For residents with vision loss, even a short walk down the hallway can mean waiting for a staff member to help, in facilities that are already short on staff.

We wanted to change that. We wanted residents to be able to get around on their own, knowing that help was always one call away.

What it does

Isaac has three parts: Isaac Path, a voice navigation assistant that guides travellers through a building; Isaac Belt, a wearable that taps left or right to tell them where to turn; and Isaac Care, an app that lets caregivers follow along and help when needed.

Talk, don't type. Travellers can simply say where they want to go, and Isaac Path guides them with spoken directions in metres. They can use the app or just call Isaac's phone number. Photon picks up the call and connects them to an ElevenLabs voice agent, which starts navigation on the phone they're wearing. No need to look at a screen or press anything.
Feel where to turn. Isaac Belt has two servos, one on each side of the waist. Two taps on the left mean turn left, a light tap means a slight turn, three taps on both sides mean you've arrived, and a long press warns that you've gone off route. This way, travellers can follow directions even when they can't hear the voice instructions clearly.
Let caregivers follow along. With Isaac Care, staff can pair with a resident using a six-digit code, send them a destination, and watch their progress on a live floor plan. Staff and family members can also text Isaac over iMessage to ask how a trip is going. Isaac automatically sends updates when someone starts walking, changes floors, arrives, or needs help.
Help is one sentence away. If something goes wrong, the traveller just says "I need help." Isaac immediately alerts their caregiver and shares their location on the map. We made this a fixed rule rather than an AI decision, because asking for help shouldn't depend on whether a model understands you correctly.
It notices when something's wrong. If a traveller suddenly stops, walks backward, or keeps shuffling in the same spot, Isaac checks in and asks, "Is something in the way?" If multiple people get stuck in the same location, Isaac remembers that spot and avoids it when planning future trips.

We built and tested Isaac inside Princeton's Julis Romo Rabinowitz Building (JRR), using its actual rooms, corridors, stairs, and elevators.

How we built it

The map. We started with Matterport, turning a building's 3D scan into a graph that Isaac can use for navigation. Each scan position becomes a node, and we connect nodes wherever there's a clear path without a wall in between. Stairs and elevators connect different floors, while room labels become destinations travellers can ask for naturally, like "the seminar room."

For JRR, we ended up with 66 scan positions across 2 floors and 18 named destinations. We also used the point cloud to estimate doorway widths and spot potential hazards, particularly objects at waist or head height that a traditional cane might miss.

The survey. A map alone isn't enough. We also wanted Isaac to understand what a building is actually like before anyone starts walking through it.

We built a building-survey system using LangChain Deep Agents. A lead agent organizes the work with write_todos and delegates it to five specialists. One identifies and names destinations, one checks potential hazards against the actual Matterport panoramas, one finds landmarks travellers can hear or feel, one verifies entrances using Vexcel facade photos and CoStar records, and one connects separate scans of the same building.

The survey only needs to run once per building. We cache the results so every traveller after that can use the same information without running the whole process again.

The three CoStar layers. One of the most interesting parts of building Isaac was figuring out how Matterport, Vexcel, and CoStar could work together.

Matterport gives us the inside of the building. Vexcel helps us understand the outside, from entrances to crosswalks and curb ramps. In one of our Los Angeles test buildings, Vexcel's facade imagery revealed five steps leading to the entrance, with no visible ramp.

The CoStar record adds context about the building itself, including when it was built, whether it predates the ADA, its elevators, parking, and tenants that might become destinations.

Together, these three sources tell Isaac much more than just where a hallway leads.

The route. We use A* search on the building graph to find routes. But the shortest route isn't always the safest, so we also account for hazards, stairs, and unmarked crossings. We adjust these costs depending on the traveller's needs, meaning a blind pedestrian and a wheelchair user may get different routes through the same building.

The wearable. Isaac Belt runs on a Raspberry Pi connected to two servos through a PCA9685 driver board. We designed the mechanism so the servo arm releases between taps instead of constantly pressing against the body, keeping the belt quiet, cool, and comfortable.

Tracking someone indoors turned out to be much harder. GPS doesn't work reliably inside buildings, and we wanted Isaac to work without installing sensors everywhere. So we use the phone worn on the traveller's back to collect step counts and compass headings. The phone sends that information through our server to the Raspberry Pi, which estimates where the traveller is along the route. To reduce drift, we keep the position attached to the planned path and only reconsider it when the compass clearly suggests they've gone another way.

Keeping AI out of the loop. This was one of our biggest design decisions. We use AI extensively to understand buildings and handle unexpected situations, but we don't let it make turn-by-turn navigation decisions.

A model hallucinating a chatbot response is one thing. A model hallucinating a turn into a stairwell could seriously hurt someone.

So routing, tracking, and haptic feedback are all deterministic. AI does its work before the trip, during the building survey, and alongside it when something unexpected happens, like a locked door or broken elevator. Even then, Isaac asks the traveller before making changes.

One hub. Everything runs through a FastAPI server: Isaac Path, Isaac Belt, Isaac Care, iMessage, the phone line, and our live operations dashboard. Keeping everything connected through one backend means the traveller, their caregiver, and the wearable are all working from the same trip information.

Challenges we ran into

Every Matterport edge looks like a wall. Matterport's scans don't explicitly tell us where doors are, so we had to identify doorways from the point cloud ourselves. We also found that the API schema differs between models, so we made our queries automatically drop unsupported fields and retry rather than failing altogether.
Small models kept giving up early. During longer building surveys, smaller models would sometimes inspect one panorama, write a convincing summary, and stop, even though most of the building hadn't been checked. We stopped trusting the agent's final response and instead rebuilt the survey from the specialists' individual output files, checking what had actually been completed.
No GPS indoors. Counting steps sounds straightforward until the phone is strapped to someone's back and their walking speed changes. Small errors add up quickly. We ended up keeping the estimated position on the planned route and using significant compass disagreements to detect when someone might have gone off course. It's not perfect yet, but it gave us a working prototype without additional building infrastructure.
Getting everything connected. Campus Wi-Fi prevented some of our devices from talking directly, so we moved the phones, Raspberry Pi, and laptop onto a shared hotspot and used a tunnel to make the server reachable from the phone line. We also spent hours debugging iMessages that seemed to vanish into nowhere, only to discover that Photon's shared number accepts texts only from explicitly added phone numbers.

Accomplishments that we're proud of

We built an end-to-end system where a blind traveller can call a phone number, say where they want to go, and be guided there through voice and taps while their caregiver follows along on a live map.
We made safety part of the architecture. No language model ever chooses a turn, and emergency alerts don't depend on AI.
We got Matterport, Vexcel, and CoStar working together, with all three contributing information that changes how Isaac guides a traveller.
We mapped and tested Isaac in a real building at Princeton rather than relying entirely on a simulated environment.

What we learned

AI doesn't have to make every decision to make a system intelligent. We found that AI is most useful when it has time to understand a building, identify hazards, and prepare useful information before someone starts walking. Once the trip begins, predictable algorithms are much better suited for decisions where mistakes could cause physical harm.
Agents work better with clear tasks. Giving a smaller model a broad goal like "survey this building" often led to incomplete results. Breaking the work into specific tasks and verifying the outputs made the system much more reliable.
More data doesn't always mean better data. Matterport, Vexcel, and CoStar each have their strengths, but they don't always agree. We learned to cross-check sources, understand where each one works best, and be upfront when we couldn't verify something.

More than anything, building Isaac made us think about what independence actually means. It's not about replacing caregivers. It's about giving people the confidence to do more on their own, while making sure someone can still be there when they need help.

What's next for Isaac

Test in a real nursing home. JRR was a great place to build and test the prototype, but a university building isn't a nursing home. We want to scan an actual care facility, survey it, and test Isaac with residents and staff to understand what works and what needs to improve.
One screen for every resident. Right now, Isaac Care lets caregivers follow individual trips. We want to expand that into one live dashboard showing everyone who's walking, who's arrived, and who might need assistance.

Ultimately, we want Isaac to make something as simple as walking to the dining room feel simple again.