Inspiration

We built Iris around a simple everyday problem: how does a blind person find and reach a water bottle when they do not know where it is?

An object might be right on the table, but finding its exact position can mean searching by touch or asking someone for help. We wanted to make that interaction more independent: say what you need, hear where to move your hand, and reach the object.

What it does

Iris helps blind and visually impaired people find nearby objects and guides their hand toward them.

Imagine a water bottle sitting on a desk. You tell Iris, “Find my water bottle.” Using the phone’s camera, Iris detects the bottle and tracks your hand. Spoken directions, directional audio, and haptic feedback help you move toward it in stages: reach forward, move across, and take hold.

Once you have the item, Iris can also read its label and answer questions about it. The central experience is simple: find the object, reach it, and understand what you picked up.

How we built it

We built Iris as a native iOS app using Swift and SwiftUI. We combined:

  • Core ML and YOLO segmentation to detect supported objects, such as bottles.
  • Apple Vision to track the user’s hand and read label text.
  • ARKit and LiDAR depth on compatible iPhones to estimate the distance between the hand and the object.
  • ElevenLabs for spoken instructions and conversational assistance.
  • Figma to design and refine the interface.

These systems turn camera observations into short, actionable cues that help the user reach toward a target.

Challenges we ran into

Detecting a bottle was only part of the problem. We also needed to locate the user’s hand relative to it and translate that relationship into understandable instructions.

Aligning camera coordinates, hand landmarks, and depth measurements was challenging, especially when the hand covered the bottle or tracking became unstable. We also had to balance responsive guidance with enough smoothing to prevent constantly changing directions.

Another challenge was coordinating listening, speech, and audio cues so the feedback remained clear.

Accomplishments that we're proud of

We built a prototype that connects finding an object with physically reaching for it.

We are proud of bringing object detection, hand tracking, depth sensing, and spoken guidance into one experience focused on a concrete task: helping someone locate and pick up an everyday item like a water bottle.

What we learned

We learned that recognizing an object and helping someone reach it are very different engineering problems.

Useful guidance requires reliable timing, consistent spatial measurements, and clear instructions. We also learned to make uncertainty explicit: when tracking or depth is unavailable, the app should acknowledge that instead of giving misleading directions.

What's next for Iris

Our next step is testing with blind and visually impaired users to understand how well the guidance supports real everyday tasks.

We want to improve bottle detection in cluttered spaces, make hand guidance more stable, expand the range of supported objects, and evaluate performance across physical iPhones. Our goal is to make finding a water bottle—or another misplaced essential—a more independent everyday action.

Built With

Share this project:

Updates

Submission history