StepSafe

Travel safe. Together.

Comment

StepSafe

Tagline: Travel safe. Together.

Live map: https://stepsafe.miami

Code: https://github.com/xVedara/shellhacks-2026

Live Demo: https://youtu.be/2hzeV3CCp_g

Inspiration

Dev's grandmother is visually impaired. For her, a normal walk down the block comes with risks most of us never think about: a sign at head height, a curb that drops away, or a car rolling into the crosswalk.

Around the same time, Ara came across an app that connects blind users with sighted volunteers over a video call. That got us asking a question. What if the phone could be the lookout itself, with no call, no wait and no internet connection needed?

So we set out to build a lookout that runs on an iPhone people already own, plus a shared map where every walk makes the neighborhood safer for the next person.

What it does

StepSafe helps blind and low-vision people walk public sidewalks more safely. It combines the iPhone's LiDAR depth sensor with on-device AI for object classification.

While you walk, the phone scans the path ahead and speaks through your AirPods:

  • Path guard warns about obstacles on the ground, obstacles at head height, and drop-offs like curbs.
  • Crossing assist warns when a car, bike or cart is closing in.
  • Two AirPods presses give you "what's ahead" and mute. There is nothing to look at and nothing to type.

Every obstacle the phone spots is also pinned to a community map at https://stepsafe.miami, so other walkers hear about it before they reach it. When a walker passes a pin and the hazard is gone, the pin loses weight and eventually clears itself.

How we built it

  • Depth: ARKit scene depth from the LiDAR sensor feeds path guard. It works from raw depth, which gives it real distances rather than guesses from a camera image.
  • On-device AI: YOLO11s runs in Core ML on the phone for classification. It is paired with a depth-based closing detector for crossing assist.
  • Alerts: Spatial audio plus Core Haptics. English and Spanish phrases were pre-generated with ElevenLabs, so alerts play instantly and work offline.
  • Backend: Node and TypeScript on Fastify, with MongoDB Atlas. Reports within 10 meters on the same height band merge into one pin. Otherwise they become a new pin.
  • Naming: In the background, Gemini (gemini-flash-lite-latest) looks at each new pin's photo and picks a label from our list of 67 hazard types.

Challenges we ran into

Speed versus accuracy on a phone. A warning that arrives late is useless, but a warning that is wrong teaches people to ignore it. Running local AI on an iPhone meant constantly trading model size and frame rate against accuracy.

Making sense of raw LiDAR point clouds. A depth sensor gives you thousands of distance points, not "there is a curb." We had to turn raw points into conclusions: what is ground, what sticks up, what hangs at head height, and what drops away.

People are different heights. Head height for one walker is chest height for another, and the phone sits at a different height on every body. Our thresholds had to adapt to each walker instead of assuming one height.

Turning your head looked like a car coming. A detection box grows when the camera turns, just as it does when a car gets closer. Growth-only alerts now wait until the head is holding still.

Accomplishments that we're proud of

We honestly weren't sure this idea was possible. Going from a far-fetched idea to a working MVP in 36 hours genuinely surprised us. It includes live depth sensing, on-device AI, spoken alerts and a live community map.

We are also both first-time hackers.

What we learned

Sleep is optional.

Time matters in two ways. The first is our own time: 36 hours forces you to cut scope hard and build what matters first. The second is the walker's time. Our first plan was to send everything to Gemini in the cloud. A network round trip is too slow for someone about to step off a curb, so we moved classification and segmentation onto the device. Gemini stayed on for something that is not urgent: naming hazards on the map after the walker has already been warned.

What's next for StepSafe

We want to take StepSafe from a hackathon MVP to a real, polished app. We plan to keep committing after ShellHacks. That means adding more obstacle types, tuning path guard and crossing assist on more outdoor walks, and making it more reliable for walkers of every height.

Longer term, we want to build our own hardware: a sleek pair of glasses with the depth sensor, on-device AI and audio built in. You just put them on and walk, and the lookout rides at eye level all day without anything extra to carry or mount.

Built with

Swift, SwiftUI, ARKit, Core ML, YOLO11s, Node.js, TypeScript, Fastify, MongoDB Atlas, Google Gemini API, ElevenLabs, Next.js, Leaflet, OpenStreetMap.

Sponsor tracks

Best Overall (auto). Live depth sensing, on-device AI, spoken alerts and a shared hazard map, built in 36 hours.

Best First-Time Hacker. Dev and Ara are both first-time hackers.

Microsoft — What's Missing? There is no chat interface: two fixed AirPods commands, not a conversation. AI runs inside perception, never as a dialogue layer.

Waymo — Mobility Challenge. Uses OpenStreetMap crossings, curbs and tactile paving, plus community hazard reports, with no turn-by-turn route.

MLH / ElevenLabs — Best Use of ElevenLabs. English and Spanish alert phrases play instantly and offline.

MLH / Google Cloud — Best Use of Gemini API. Gemini names each new pin in the background, picking from our 67-type taxonomy. The report is saved first, so a slow response never delays an alert.

MLH / MongoDB — Best Use of MongoDB Atlas. Hazards and votes live in Atlas, using a 2dsphere index, TTL expiry by category, and change streams.

MLH / GoDaddy Registry — Best Domain Name. The community map is live at https://stepsafe.miami.

Built With

Share this project:

Updates

Submission history