Cyclops

Inspiration

Finding an object is a constant problem for blind and low-vision people: an empty chair, the trash can, the door in an unfamiliar room. A cane finds what's at your feet, not what you're looking for. Most phone apps can name what's in front of you but can't walk you to it, and the distance features that do exist need a Pro iPhone with LiDAR. We wanted something that works on any phone, using only the camera.

What it does

Cyclops is a voice-first guide for finding things indoors.

Ask for anything. Press and hold anywhere and say "find the chair", "find the red backpack" or "find the exit sign". It knows 79 everyday objects, matches descriptions, and searches for anything else by name.
It locks on. Turn slowly; when it's sure, you hear a chime and feel a buzz. Then the voice guides you: "Chair, turn a little right and walk about 3 meters." and "Straight ahead." The gyro remembers where the target is, so if you turn away, you hear "Chair at 2 o'clock", with pulses that get stronger as you face it again.
It watches the way. Once you're lined up, it checks the straight line to the target: "Path to it looks open." or "Something in the way, about 2 meters. Go around it on the left." Anything close starts a detour ("Something close on your right. Step to your left."), and low things like boxes and bins are called out before they slip out of the bottom of the picture. The wall the target stands against doesn't count.
You arrive. Within reach it dings and says where to reach: "You're there. Reach forward and a little left, at waist height."
Walker runs it all by voice. A voice assistant handles setup, finding, and settings (units, warning distance, speech speed, haptics, voices) without looking at the screen.
Trusted voices. With their consent, record someone for 15 seconds and the app guides you in their voice.

Cyclops is clear about its limits: stationary obstacles, indoors, and never a replacement for a cane.

How we built it

App: Expo and React Native in TypeScript: camera frames, the gyro, haptics and audio. A tracker follows the target across frames in gyro coordinates, so the box doesn't jump between similar objects, and locks on after two confident sightings.
Backend: FastAPI on a Hugging Face Space with a T4 GPU, every model in half precision: about 0.1 to 0.3 seconds per frame on the server, with two frames in flight.
Models: Depth Anything V2 Metric for distance at every pixel, scaled against tape-measured photos from the venue; DETR for the 79 built-in objects; OWLv2 for open-vocabulary search; SigLIP for matching descriptions like "red bench".
Obstacle logic: a walking lane split into foot, waist and head height; the floor removed by comparing each pixel with the rest of its row; a 0.8 m wide corridor along the line to the target; the target and anything behind it ignored.
Voice: ElevenLabs Flash for speech, cached per voice and phrase; Scribe v2 speech-to-text with speaker separation, so it follows whoever said "find"; and instant voice cloning. The server only speaks the app's own phrases, so a cloned voice can't be made to say anything else.

Challenges we ran into

Depth from one camera is wrong up close. A box half a meter away read 1.3 m. We tuned the depth scale against tape-measured photos and use object size and the tracker for the last meter.
The floor in front of you isn't in the picture. Held up toward a target, the phone can't see roughly the last 1.5 m of floor, so a low box leaves the bottom of the frame before it ever reads as close. Low things are now warned about early, and the detour holds after they leave the frame.
False alarms next to the target. The wall behind a chair set off "something close on your left" right before arriving. Now only things clearly nearer than the target count.
Latency. The first version took over a second per frame on a CPU. A GPU in half precision, two frames in flight and timeouts for stragglers brought the server to about 0.2 seconds.
Talking too much. Directions ran into each other. The voice now stays quiet while you're on course and speaks when there's something to do; safety warnings always cut in.

Accomplishments that we're proud of

It's the first hackathon for all three of us.
It finds the thing you asked for and walks you to it, end to end, on an ordinary phone.
The whole app works without looking at the screen.
Safety comes first: warnings interrupt everything, and it only suggests stepping to a side the camera actually saw was open.
We personally prefer the accessibility settings provided by Cyclops compared to the iOS accessibility features

What we learned

How monocular depth behaves in real rooms, and how much it matters to tune against measured ground truth instead of trusting the model.
How to design for non-visual use: every state spoken or felt, short phrases, never talking over a warning.
How to serve several vision models on a GPU and measure latency from the camera to the speaker.

What's next for Cyclops

One language-model call per request to map "my MacBook" or "the thing I drink from" onto objects the app can find.
Remembering where an obstacle is from the phone's motion after it leaves the picture.
LiDAR depth on phones that have it, and a native build for faster photos.
Outdoors, once curbs and drop-offs can be detected reliably.