marko

app startup screen
room image display
camera overlay
destination found
look down feature
AR view
digital twin psb

Inspiration

GPS stops at the front door. Inside malls, campuses and hospitals, people still rely on directory boards, and venues have no idea where visitors get lost. Indoor maps have always needed beacons or costly hand surveys. We wanted one built automatically from floor plans and a single phone scan.

What it does

marko builds AR maps of indoor spaces from floor plans and one phone scan.

Visitors say where they want to go: any of 61 places, a person, or "the nearest restroom".
- marko finds them from the camera in ~0.4 s, or from where they say they are.
- It guides them with an AR path, a Find My-style arrow, and spoken turn-by-turn directions, across floors.
Venues upload, build and publish their map. marko for Venues shows:
- sessions, arrival rate and time to destination;
- 3D traffic heatmaps;
- top searches, plus searches that found nothing;
- "I'm lost" hotspots and visitor-reported problems.

How we built it

Floor plans become walkable grids automatically.
iPhone LiDAR scans (Stray Scanner) become a localization map of 7,753 posed photos and 589k 3D points, plus a 1.89M-point 3D twin.
Compute: processing thousands of scan frames was too slow on our laptops, so we rented RTX 4090 GPUs on RunPod for the project.
- Feature extraction ran at about 12 images/s.
- The same GPUs serve live camera fixes 80× faster than a laptop, using SuperPoint, LightGlue and NetVLAD.
Routing: Dijkstra over a 0.5 m grid with wall avoidance, landmark-based instructions, and re-planning if you're more than 5 m off route.
Orientation: the compass and the building's bearing (179°, from OpenStreetMap) orient routes when the camera can't.
App: SwiftUI, ARKit and RealityKit, with 104 tests. Voice: an ElevenLabs agent with 9 in-app tools. Portal: React and three.js.

Challenges we ran into

Compute and bandwidth: our laptops couldn't process the scan data, so we moved to rented cloud GPUs. Campus Wi-Fi uploads at about 2 MB/s became the bottleneck. We uploaded the raw videos and recomputed features on the GPU, and shrank the 7.7 GB map to 1.0 GB.

Night lighting: the map was recorded in daylight; at 4 AM, 54% of frames matched.
Confidently wrong fixes: cut from 15% to ~4% with geometric checks and compass cross-checks.

Accomplishments that we're proud of

Building to AR map, end to end, on 2 real buildings.
~1 cm median accuracy across 1,000 benchmark queries.
A working 4 AM walk: located from the camera on both floors and routed correctly.
A voice-first flow usable without seeing the screen.

What we learned

Indoor positioning is a data problem: lighting and coverage matter more than the model.
No position beats a wrong one. Know when to ask the user.
Fusing camera, motion, compass and voice beats any single signal.
Renting cloud GPUs turned overnight processing into minutes.

What's next for marko

Drone mapping: camera and LiDAR drones that scan large indoor spaces autonomously (malls, airports, warehouses), with no walk-through needed.
Wi-Fi and Bluetooth positioning: the phone locates itself instantly from a venue's existing Wi-Fi access points (Wi-Fi RTT) and Bluetooth beacons, with the camera refining it to the metre.
Fully automatic labelling from floor plans, and live, anonymous analytics in the venue portal.
Night-ready maps, step-free routes and commercial pilots with malls.