Reach

Inspiration

There are already apps that can tell a blind person what's on a table. But knowing your keys are "somewhere in front of you" doesn't get them into your hand. You still end up patting around and knocking things over. We wanted to help with that last part.

What it does

Reach is an iPhone app you wear on your chest with earbuds in. Say "find my keys" and it talks you there: "Farther and to your right… a little left… down." Soft pings get faster as your hand gets closer, and when your finger touches the keys you hear a chime and "Got them."

You can also ask "what's on the table?" to hear everything in front of you, then run your hand across the table and Reach names whatever you touch. Hold something up and ask "is this okay for me to eat?" and it reads the label, warning you first if it mentions one of your allergies. It can also count money, check that a pill bottle is yours, and start from Siri.

How we built it

It's a Swift app for iPhone. Gemini finds the object in the camera image, ARKit and the phone's LiDAR sensor pin it in 3D so it stays put while you move, and Apple's Vision framework tracks your fingertip. ElevenLabs gives Reach its voice. We used Claude Code to help us write code faster.

Challenges we ran into

Our first version lost track of objects every time you leaned forward, so we rebuilt it around 3D pinning. Fingers are hard for a depth sensor to see, which made "are you touching it?" surprisingly tricky. We also rewrote the spoken directions several times. Exact distances were too much, saying nothing was too little, and "out" and "in" confused everyone. "Farther" and "closer" finally clicked.

Accomplishments that we're proud of

Guiding a hand to one specific object on a messy table using only sound. And touch to explore: sweep your hand over the table and hear each thing named as you touch it.