Marco Polo

dashboard
Blind person testing the harness
GIF
architecture diagram gif

Inspiration

Losing your stuff is annoying for anyone. If you're blind, it can stall your whole morning: you can't glance around the room, so you sweep every surface by hand or wait for someone to help. Apps like Seeing AI can describe what the camera sees right now, but they only search when you ask and forget everything when you close them. We wanted something that quietly keeps track of your things and, when you ask, walks you to them.

What it does

Marco Polo is a wearable for blind and low-vision people: an iPhone mounted on a chest vest, AirPods, and a server that remembers.

It watches without being asked. The phone's camera and LiDAR take in what's in front of you mounted on a harness. The server recognizes your registered items and remembers where each one was last seen.
You ask, it guides. "Marco, where are my keys?" "Polo!" A beep then plays from the keys' actual position through head-tracked AirPods. A distinct beep tells you when you're facing the right way. The beeps speed up as you get closer and change tone when your hand is near. When you touch the keys, Marco says "Got it."
Family teaches it. A caretaker texts a photo and a name ("wallet") over iMessage, and the item is registered. If you touch something Marco doesn't know and ask "what is this?", it texts the caretaker a picture and reads their answer back to you.
A live dashboard shows the room map built from LiDAR, with walls and furniture drawn as clean shapes, where every item is, and what the wearer is hearing.

How we built it

iPhone app (Swift, ARKit, Vision): streams camera frames once a second, the phone's position 10 times a second, and hand joints with LiDAR depth 15 times a second. It plays the guidance as 3D spatial audio, combining ARKit's body position with the AirPods' head tracking.
Server (Python, FastAPI, PyTorch) on a Vultr GPU: OWLv2, an open-vocabulary detector, finds each item from a single photo. DINOv2 double-checks the match, and LiDAR depth places the item in 3D. A world model tracks whether items are seen, moved, missing or picked up, and detects touches from the hand stream. Each frame takes about 200 ms.
Caretaker bridge: Photon over iMessage. Voice: ElevenLabs, with Apple speech as a fallback.
Dashboard: React and TypeScript, updated live over WebSockets.
Security: the server listens only on its own machine. Tailscale makes it reachable on our private network, and only approved accounts get in. Nothing is open to the public internet.
Evidence first: before building the app, we ran 29 proofs of concept and emulators covering LiDAR, ARKit drift, head tracking and guidance with simulated users. We benchmarked detectors on ORBIT, a dataset of blind people filming their own belongings.

Challenges we ran into

"Is it actually there?" The detector always returns a best guess, often at near-full confidence even when the item isn't in view. We built a three-part check to separate your keys from everything else. Look-alike objects are still our biggest source of false alarms.
The chest is an awkward place for a camera. Held upright, it can't see a table within a metre, so we tilted the mount down. When you reach for something, the back of your hand hides your fingertip, so we also count a hand held steadily on the item as a touch.
Small geometry bugs, big misses. A frame rotated the wrong way, or placed using the phone's current position instead of the one it was taken from, puts an item metres off. Each of these bugs now has a unit test.
Sound can't tell you "how far" through loudness. Up close, volume barely changes and people can't hear height, so we use tempo to carry distance.
Real conditions: much of our testing happened late at night under dim indoor light, which is hard on both the camera and ARKit's tracking.

Accomplishments that we're proud of

The whole loop works on real hardware: ask, follow the sound, touch, "Got it."
Memory, not just search: Marco notices where things are without being asked.
Our design choices come from measurements, and we kept the ones that held up on data we hadn't tuned on.
A family member can teach Marco a new object with one photo and a name.

What we learned

One good photo matters most. With the best registration photo, the item was found 97% of the time in its owner's clutter. With an average photo, it was about 62%.
Pace beats volume when guiding someone by ear.

What's next for Marco Polo

Fewer false alarms: comparing the whole object instead of just its middle cut them by about 40% in testing. A larger detector on a faster GPU should cut them further.
Proactive alerts: "You're leaving without your keys."
Real users: testing with blind and low-vision people in their own homes.