Fuga | Devpost

Mapping of phone path onto UW floor plan
LiDAR hallway reconstruction
FUGA Ui
FUGA Path generation

Inspiration

As new students, we all got lost inside UW buildings. Finding the building was easy; finding room 430 once inside wasn't. GPS stops working the moment you walk through the door, and every hallway in Mary Gates Hall looks the same. We'd also been wanting to build something with the LiDAR sensor and were thinking, if room signs are how sighted people find their way, how does someone who's blind or low-vision find a classroom?

What it does

It uses LIDAR scans of the hallways and public floor plans, to do hallway reconstruction from a phone, and then can pinpoint where you are in a specific building. Then the app would show a visual path in the hallway on where to go to your specific classroom destination

How we built it

Scanning: We walked Mary Gates Hall with an iPhone Pro's LiDAR, a laser that measures depth. The scans recorded video, depth, and the phone's path as we went. Mapping: Our Python pipeline turns UW's floor plan into a map of walkable hallways. Then it lines up our scan with the plan, rotating and shifting it until our path fits inside the hallways. The scan's depth pins room signs to their real positions. The result is a hallway map of rooms, doors, and corners for the 4th floor. The app (Swift): You start in front of a door Fuga knows. From there, Apple's ARKit uses the camera and motion sensors to follow every step, with no GPS or beacons. Fuga finds the shortest hallway path to your room and drives the arrow, the floor line (SceneKit), the voice, and vibration (Core Haptics) from where you're facing. The voice clips ship inside the app, so it works offline. Finding you from a photo (experimental): A neural network called DISK picks out distinctive points in a camera photo, like corners, door frames, and sign lettering. A second network, LightGlue, matches them to the same points in our LiDAR scan, and geometry works out where you're standing. We only accept a result when two photos agree.

Challenges we ran into

Pathway misalignment, and the app not being able to identify where you are in the building 100% of the time.

Accomplishments that we're proud of

Turning our own LiDAR walks into a working indoor map lined up with UW's real floor plan. Full guidance on one floor: route planning, a live arrow, a line drawn on the actual hallway floor, spoken turns, and vibration you can follow without looking. Matching a phone photo against our 3D scan and getting real position estimates. Early results, but it works. Being honest about what works. We tested recognition on photos held out from the reference map instead of reporting our best-looking result.

What we learned

LiDAR data is huge. Our scan archive is about 4.5 GB, and the raw scans are too big for GitHub. Because LiDAR measures in real meters, lining a scan up with a floor plan only needs rotation and position, not guessing the scale. Check your data before trusting your results. One wrong frame index invalidated our first experiment. Indoor navigation is mostly about knowing exactly where you're standing. Drawing the route is the easy part.

What's next for Fuga

Full campus coverage, and beyond. Potential use cases for places outside of college campuses, like train stations, airports, malls. Fuga is useful for people who are unfamiliar with their new surroundings, making exploring them as easy as possible.