Inspiration

GPS gets you to the building — not to the room.

For someone who is blind or has low vision, the hardest part of a trip often starts after the front door: finding Radiology on the third floor, the right lecture hall, or the accessible restroom. Signs are hard to read, announcements are hard to hear, and layouts are unfamiliar. More than 7 million Americans live with vision loss and about 1 million are blind (2021 CDC-funded estimate), and millions more older adults struggle with signage, hearing, or mobility.

Existing apps can describe what a camera sees. We wanted something that can actually take you somewhere — an assistant you can talk to, that knows the building and guides you to the exact room.

What it does

IndoorAbility is an AI navigation agent you talk to.

  • Ask for a destination by voice or text — "Take me to room A71" or "Find the nearest accessible restroom."
  • The agent plans the trip: it chooses the destination, picks a route, and changes it when you ask (e.g. "avoid stairs").
  • Outdoor to indoor handoff: GPS walking directions to a mapped entrance, then indoor guidance once your position is verified there.
  • Indoor localization with iPhone LiDAR maps and ARKit tracking.
  • Accessible feedback your way: spoken turn-by-turn directions, haptic vibrations, large text and visual arrows.
  • Destination confirmation: the camera reads room signs with computer vision to confirm you've arrived.
  • Safety first: when tracking is uncertain, Indoorbility pauses instructions instead of guessing and helps you recover.

How we built it

  • iOS app in Swift / SwiftUI, designed for VoiceOver, with AVFoundation speech and iOS haptics.
  • Indoor mapping & localization: routes are recorded once with iPhone LiDAR and saved with entrance and destination labels; ARKit relocalizes the user against the saved map at a verified entrance.
  • Computer vision: Apple Vision OCR recognizes room and entrance signs.
  • Outdoor navigation: Apple MapKit / Core Location and Google Places API.
  • AI agent: built with the OpenAI Agents SDK. It interprets natural-language requests and calls real navigation tools (set destination, change route, apply accessibility constraints) instead of just chatting.
  • Backend: Python + FastAPI on Railway; saved maps stored locally on the device so recorded routes work offline; credentials in the iOS Keychain.

Challenges we ran into

  • Indoor positioning is hard. GPS doesn't work inside, and ARKit tracking drifts in long, featureless hallways, on glossy floors, and under changing light. We learned to keep routes short, start from a verified entrance, and treat low confidence as a reason to stop rather than guess.
  • Designing for people who can't see the screen. Every instruction had to make sense through audio and haptics alone: short, timely, and never overlapping.
  • Making the agent act, not just talk. Connecting language like "somewhere I can sit down" to concrete destinations and route changes took careful tool design.
  • Honesty under time pressure. Most features are still experimental

Accomplishments that we're proud of

  • Ability to pivot when stuck
  • A voice request turned into a planned route and step-by-step guidance inside a real building.
  • Haptic, spoken and visual feedback working together, so users can pick what works for them.
  • A safety model that pauses instead of guessing.
  • LiDAR scanning of an indoor environment.

What we learned

  • Accessibility is about choice: the same instruction needs to work as speech, vibration, or large visuals.
  • In assistive tech, knowing when you're uncertain matters as much as being right.
  • Agents become useful when they have real tools — the value is in what they can do, not what they can say.
  • Testing in a real building beats any simulator.
  • Plans change at hackathons; picking one strong idea and committing to it beats spreading effort thin, but it is important to know when to pivot.

What's next for IndoorAbility

  • Skip manual mapping: import buildings that are already 3D-scanned. CoStar alone has about 15,000 Matterport-scanned buildings in Houston and Los Angeles, with floor plans and room labels.
  • More reliable tracking in long corridors and multi-floor routes (elevators, stairs).
  • Testing with blind and low-vision users and orientation & mobility specialists.
  • Partnerships with hospitals, universities and transit hubs, where finding the right room matters most.

We're not replacing the cane. We're giving it a map of the inside.

Built With

Share this project:

Updates

Submission history