StepSight

StepSight Hackathon submission by Samrah Riaz Inspiration My uncle is blind, and I watched how much of his daily life depended on other people. Even finding a cup or a phone meant asking someone. I wanted to build something that gives that independence back, one small "where is it?" at a time. What it does StepSight helps blind people find everyday objects on their own. The user says the name of an object into the microphone, and the app tells them where it is: how many steps away and in which direction. For example, "Your cup is 4 steps away, slightly to the left." They can then walk to it without asking anyone. How we built it StepSight is a four-stage pipeline:

Detect: Google OWLv2, an open-vocabulary detector, finds whatever object the user names in the camera frame. It isn't limited to a fixed list of objects.
Measure: Depth Anything V2 (Metric Indoor, Small) estimates depth. The median depth inside the detection box gives the distance.
Convert: Distance ÷ stride gives steps. The box's horizontal position gives the direction: far left, slightly left, ahead, slightly right, or far right.
Speak or show: Meta MMS-TTS produces the voice, panned toward the object's side so the user can hear which way to turn. A large-text view is also available. Challenges we ran into The hardest part was handling detection errors and raising the model's confidence. With limited time during the hackathon, I chose to get a working end-to-end pipeline first. I'll focus on improving accuracy after the event. Accomplishments that we're proud of This was my first time testing a model on real-world images, and seeing it work was exciting. It gave great results on 6 out of 9 real images. Seeing the app turn a camera frame into "3 steps, slightly right" was the moment the idea felt real. What we learned • How to properly test a model, and why datasets are split into training and testing sets • That real images behave very differently from clean examples, and testing on them shows what actually needs fixing What's next for StepSight • Calibration: stride calibration and distance calibration from measured data, for more accurate step counts • Haptics: vibration patterns so users can feel direction as well as hear it • Safety: a clear-path hint (experimental) and warnings about dangerous situations • Navigation: guiding users along a route, not just to a single object • Real users: testing with blind and deaf-blind users, so the app is shaped by the people it's built for • Speed: a faster model for real-time use Editing notes • The inspiration now opens with a concrete moment, which is the strongest part of the story. • An example sentence in "What it does" lets judges picture the app in use. • "How we built it" is a numbered list so a skimming judge can follow the flow quickly. • The challenge is framed as a deliberate choice (working pipeline first). • Navigation and danger warnings moved into "What's next" so the page ends on the vision. • Suggestion: if possible, add the 3 images that failed and one sentence on why (lighting, small objects, clutter). Honest failure analysis impresses judges and strengthens the "what's next" plan.