WayFinder

Inspiration

We kept running into the same gap: tools for blind and visually impaired users either narrate the world (Seeing AI, Be My Eyes) or route you somewhere (Google Maps), but nothing does both while also solving the screen problem. If you can't see the screen, setting a destination is already a barrier before you've taken a step. We also noticed that most of these tools treat the user as if they're navigating completely alone, when in reality a lot of blind and visually impaired people already have a caregiver, family member, or aide who helps out day to day. We wanted to build something that works with that relationship instead of ignoring it.

What it does

WayFinder turns a phone into a real-time hazard-detection and navigation system. The back camera watches the path ahead and flags curbs, potholes, and drop-offs before you reach them, with voice guidance telling you exactly what to do — stop, step down, turn. For routing, a caregiver can set or change the destination remotely, so the blind user isn't stuck typing into a screen they can't see. It's a Progressive Web App, so there's no app store step — install it straight from the browser.

How we built it

The core hazard-detection pipeline combines MiDaS for real-time monocular depth estimation — critical for judging distance to curbs, potholes, and drop-offs from a single phone camera — with MediaPipe and TensorFlow for on-device object and obstacle detection, so the system doesn't depend on a round-trip to a server for something safety-critical. Gemini handles higher-level scene reasoning where raw detection isn't enough to describe what's actually ahead. Voice guidance runs through the Web Speech API. The caregiver's remote destination-setting works over WebRTC and WebSockets, keeping the connection between the caregiver's session and the user's phone live and low-latency. Maps handles routing once a destination is set. On the backend, Firebase manages auth and real-time state, Postgres and Redis handle persistent data and session/caching needs, and Cloudinary stores and serves images. The whole frontend is Next.js, TypeScript, and Tailwind, packaged as a PWA and deployed on Vercel — installable straight from the browser, no app store.

Challenges we ran into

The biggest one was getting MiDaS to run fast enough for real-time use. Depth estimation models like MiDaS are usually built for offline or server-side inference, not for running in-browser on a phone against a live camera feed — so getting it to keep pace with the camera without lag, while still being accurate enough to trust for something safety-critical like spotting a curb or a drop-off, took real tuning. A missed detection here isn't a minor bug, it's a real risk to the user, so we couldn't just trade accuracy for speed to hit real-time performance. On top of that, keeping the WebRTC/WebSocket connection between the caregiver's session and the user's phone reliable — so a remote destination change actually reaches the user without a noticeable delay — added its own layer of complexity on the networking side.

Accomplishments that we're proud of

Getting hazard detection and voice guidance working together in something that feels usable in real time, not just as a tech demo. We're also proud of the caregiver-remote-destination piece specifically — it's a small feature on paper, but it's the part that actually removes the biggest daily friction point we heard about.

What we learned

Building for a screen-free, safety-critical use case forces different tradeoffs than a typical consumer app — false positives are annoying, but false negatives are dangerous, so we had to think about failure modes differently than we would for a normal feature.

What's next for WayFinder

The current version runs entirely on a phone's built-in camera and sensors. Next is dedicated wearable hardware — smart glasses paired with bone-conduction earphones — so the whole system is fully hands-free and doesn't require holding or wearing a phone at all.

Built With

• cloudinary
• firebase
• gemini
• maps
• mediapipe
• midas
• nextjs
• postgrel
• pwa
• redis
• tailwindcss
• tensorflow
• typescript
• vercel
• webrtc
• websockets
• webspeech