Inspiration

Insight comes from two of our own stories.

Gerard got lost in an airport and missed his flight, since the signs in the terminal were arbitrary and vague. Where exactly do they go and those minutes of doubling back caught up.

Jeeyan is about to spend a semester in South Korea. This spring, every train station, store and campus building Jeeyan walks into will have signs in a language Jeeyan doesn't speak.

Google Maps gets you to the front door of most buildings, but not navigating expansive buildings. You're squinting at signs, hunting for a map on the wall, or asking a stranger, and none of that works when you can't read the signs or speak the language. It isn't just us, where in a survey of more than 1,000 international passengers at eight U.S. gateway airports, finding their way ranked as a top priority (ACRP Research Report 161). For someone who is blind or has low vision, it can make a building impossible to navigate alone.

We're not the first people to try indoor navigation but we've been there and want to fix that. The problem sounded really challenging but at the same time really interesting. So much to learn about localization, embedding models, and maximizing Matterport by Co-star.

What it does

Insight is indoor navigation on your phone, calibrated by streamed photos from the user's device. Using these photos, panorama data, camera positions, and 3D meshes of Princeton's Rabinowitz Hall offered in MatterPort, it estimated the user's position. From there, we do basic path finding to get to a landmark like a bathroom, specifically in JRR. We used a basic augmented reality to display the path a user might take along with a mini-map for a general overview. For accessibility, these instructions are also provided by Gemini.

How we built it

The general architecture is this:

flowchart LR
  subgraph Phone["iPhone app (Expo / React Native)"]
    CAM["camera + motion sensors"]
    MIC["hold-to-talk mic"]
    MAP["venue map, search,<br/>route, banner"]
  end
  subgraph Server["Laptop server (Python, FastAPI)"]
    DET["object detector<br/>(YOLO)"]
    LOC["localizer: match photo to<br/>scan panoramas, position, heading"]
    VEN["venue rooms + route<br/>across levels"]
    AGT["voice agent<br/>(Gemini Live) + tools"]
  end
  DATA[("Matterport scan<br/>+ venue floor plans")]
  CAM -- "photos + sensors (websocket)" --> DET & LOC
  MIC -- "voice clip" --> AGT
  LOC --> VEN --> MAP
  AGT -- "spoken answer" --> Phone
  DATA --- LOC & VEN
  Server -- "live view" --> VIEW["browser viewer<br/>(debugging)"]
  • Phone app: Expo (React Native) on an iPhone. It streams camera frames to our server with the compass heading, the phone's tilt, and motion data.
  • The map: CoStar's Matterport scan of Rabinowitz Hall: 360° photos at all 66 scan points and a 3D model that we used localization techniques to estimate user position.
  • Finding you: DINOv2 takes the streamed photos from the phone which SuperPoint with LightGlue checks the match feature by feature. A place counts as found after 3 verified matches within 5 seconds. We use a PnP depth techinque using the 3D model gives your position in meters, allowing us to cross-reference. , There are also Kalman filters that remove noise from the different positions calculated.
  • The route: We used Djikstra's algorithm for each pixel with negative weights for edges and positive weights for things in bounds.
  • Voice: Gemini Live is the voice that can speak the instructions in a friendly manner.
  • Obstacles: We used Yolov11 model to track different objects and mask them during inference, producing less noisy output for better predictions.
  • Server and viewer: Python and FastAPI over WebSockets, and a laptop viewer with 2D and 3D maps of the floor, the route and the matched points for every frame. We used a Tailscale funnel to let the phone reach the laptop from any Wi-Fi.

Challenges we ran into

Almost everything broke at least once. In roughly the order it happened:

  • Data access: our first Vexcel aerial-imagery key failed (we had pasted it twice), and the next request was refused because our building sat outside the hackathon's map area until the organizers widened it.
  • 3D data we couldn't use at first: the building scan we wanted didn't exist as a model we could access, there were no splats, and the 3D meshes were locked. They were not provided to use until yesterday and thus we had less time to iterate.
  • Slow inference on CPU: We ran it on MPS instead.
  • Venue Wi-Fi: on the guest network the phone couldn't count on reaching the laptop directly, so we routed everything through a tunnel, and learned the hard way that closing the tunnel window kills the address the phone is using.
  • A scanner's photos versus a phone's camera: matching worked on simulated views and fell apart on real iPhone frames: different lens, field of view and lighting, crowds blocking the view, and many hallways that look the same (the scan labels almost every room "Hallway"). We added a shortlist-then-verify matcher, checks against the phone's tilt, the 3-matches-in-5-seconds rule, and memory cuts so it all runs on a laptop.
  • Directions that were wrong: position from the nearest scan point and facing from a compass that can be far off indoors added up to wrong directions. We added Kalman filters for position and heading, blocked jumps to far-away scan points, and kept tuning on site.
  • The voice assistant: recordings raced their uploads, the phone gave up after 20 seconds while Gemini was still answering, and for a while the assistant couldn't tell where you were because two parts of our system disagreed about the location format.

Accomplishments that we're proud of

  • A full loop in one weekend: phone camera, place recognition on a real building's 3D scan, a route to the entrance, and spoken directions.
  • Gemini Live as the voice for answers and directions, streamed to the phone as it talks.
  • It runs on the laptops we had, including an Intel Mac with no PyTorch (there, SIFT and HOG stand in for DINOv2 and LightGlue).
  • More than 450 automated tests across the server, the app and the viewer.

What we learned

  • A 3D scan made for property tours can be a map for navigation, but a phone camera sees a building very differently from a scanner.
  • Good voice directions are short and well timed, and saying nothing beats saying something wrong.
  • Test on the real device early. Simulations hid the problems that real phone frames showed us straight away.

What's next for Insight

  • Tune it on site: walk Rabinowitz Hall until every turn is right.
  • UI and UX: pick a destination, see your route, and haptic cues alongside the voice.
  • Any destination: gates, exhibits, restrooms and rooms, not just the entrance, starting with airports, museums, apartment and house tours, and other large buildings.
  • Everything on the phone: run recognition on the device so no laptop is needed.
  • Your language: directions in the language you speak. Gemini Live already speaks many languages.
  • Real users: test with travelers, students abroad, visitors, and blind and low-vision people.
  • Business model: venues such as airports, museums, property managers and universities pay a setup fee to use the building's 3D scan (or make one) and a monthly subscription per building; visitors use it free.

Built With

Share this project:

Updates

Submission history