-
-
Prototype of iCane
-
Landing Page for iCane
-
Monitoring Page for iCane
Inspiration
A white cane is the most trusted mobility tool for blind and low-vision people, but it has a fundamental blind spot: it only tells you what's underfoot, not what's ahead. It can't warn about a person walking into your path, a flight of stairs, a crosswalk signal changing, or a sign posted at eye level. We wanted to give an everyday cane the one thing it's always been missing — eyes.
What it does
iCane turns any cane into a smart navigation aid using nothing but a mounted phone. As the user walks, the camera continuously scans the path ahead and speaks up only when something matters:
- Hazard detection with actionable guidance — a hazard on the left tells you to move right (and vice versa), something approaching head-on tells you to stop, and stairs tell you to climb or step down carefully.
- Sign reading — recognizes and reads aloud Entrance, Exit, Restroom, Emergency Exit, and Danger signs, including which direction they're in.
- Pedestrian traffic signal detection — reads crosswalk signals and tells the user when to stop, when to walk, and the countdown if it's clearly legible.
- On-demand full description — a shake, double-tap, or button press triggers a fuller spoken description of the whole scene.
Every interaction is designed to work without needing to see the screen — a single tap anywhere starts monitoring.
How we built it
The backend is a FastAPI service deployed on Render (Dockerized), which sends each camera frame to Google Gemini for vision analysis and ElevenLabs for natural text-to-speech. The frontend is a single-page vanilla JavaScript app — no install required, runs directly in a phone browser — that continuously captures downscaled frames, polls the backend every ~2.5 seconds, and plays back audio through a debounce/hysteresis system so the user isn't bombarded with repeated or flickering alerts.
A large part of the work was prompt engineering: getting Gemini to return short, decisive, immediately actionable output (under 12 words, no hedging, no filler like "I can see") instead of descriptive but unhelpful scene narration, and building a priority-ordered decision tree so a single vision call could handle hazards, traffic signals, and signage without exceeding output length.
Challenges we ran into
We initially built a full voice-activated navigation feature — geocoding, POI search, turn-by-turn walking directions triggered by spoken destination requests. Under real-world testing conditions it became unreliable enough (compounded by hitting Gemini's free-tier rate limits and then fully exhausting our ElevenLabs TTS quota mid-hackathon) that we made the call to roll it back entirely rather than ship something we couldn't trust. That was a hard but necessary decision with our submission deadline close.
Debugging was its own challenge — distinguishing a client-side network failure from a backend error required digging through server logs directly rather than guessing from symptoms, which is how we eventually traced "Load Failed" errors all the way back to a 429 rate limit and later a 401 quota-exhaustion error from ElevenLabs.
We also iterated through several rounds of prompt engineering to get Gemini's hazard descriptions from vague scene descriptions down to short, confident, directly actionable instructions — including handling cases like crosswalk countdown timers, where guessing an unclear number is worse than saying nothing.
Accomplishments that we're proud of
We shipped a fully working, deployed, real-time assistive device — not a mockup — running on hardware that costs about $15 beyond a phone the user already owns. We made the disciplined choice to cut a feature that wasn't reliable enough rather than ship it broken, and used that time instead to make the core hazard-detection experience genuinely fast and useful.
What we learned
We learned how real free-tier API limits shape system design in practice — our hazard-detection polling interval is directly tuned around Gemini's rate limits, and hitting ElevenLabs' quota mid-build taught us to plan for that ceiling rather than discover it live. We also learned that prompt engineering is as much about what you forbid a model from saying as what you ask it to say, and that a smaller, reliable feature set beats a larger, flaky one — especially under a hackathon deadline.
What's next for iCane
Revisiting voice-activated destination navigation with a more robust architecture, testing with real blind and low-vision users for feedback on the instruction phrasing and timing, exploring a dedicated low-cost hardware mount with haptic feedback as a phone-free option, and multi-language support for the spoken guidance.
Log in or sign up for Devpost to join the conversation.