GuideSight

harness
architecture

💡 Inspiration

A guide dog is great at getting you around things, but it can't tell you what is ahead. It won't read a sign, say that the light is red, or warn you that the curb drops off in two steps. We wanted to give guide-dog users that missing layer of information without adding one more thing to hold.

🦮 What it does

Sighty is a small sensor that clips onto the guide-dog harness, plus a web app on the walker's phone.

📡 Obstacle sensing: an ultrasonic sensor on the harness buzzes the handle faster as things get closer, even with no internet.
📷 Scene description: the phone's camera sends a frame to Gemini every few seconds, and right away when the harness senses something.
⚠️ Only the hazards that matter: Sighty speaks ground hazards only. Trip hazards (curbs, ramps, potholes) get a warning. Stairs and drop-offs are always urgent. A red light or moving traffic opens a full stop screen. Doors, people and furniture are left to the dog.
🧭 Guide mode: spoken turn-by-turn walking directions.
📍 Where am I?: the street address you're at and which way you're facing.
🗣️ Ask and Read: ask a question out loud, or have Sighty read signs, menus and bus numbers.
🔓 No login: open the page and go. Setup is fully voice-guided.

🛠️ How we built it

🔌 Harness: ESP32 with an HC-SR04 ultrasonic sensor and a buzzer, talking to the server over WebSockets.
🖥️ Server: Flask + flask-sock. It runs every camera frame through Gemini Vision, then a hazard filter sorts what it sees into danger, drop and trip tiers and decides what to say and how to buzz.
🔊 Voice: ElevenLabs for speech-to-text and the spoken alerts, with the phone's built-in voice as a fallback.
🗺️ Maps: Google Maps Places, Routes and Geocoding APIs for directions and "Where am I?".
📱 App: React + Vite + Tailwind web app, with Firestore for storage.
🧪 Every external service falls back to a mock when its key is missing, so we could keep building even when one piece wasn't ready.

🧗 Challenges we ran into

🤐 Too much talking. Our first version announced everything it saw, which is exhausting when you're trying to listen to traffic. We cut it down to ground hazards and built urgency tiers so only real danger interrupts you.
📶 Phones and hardware. Phone browsers only allow the camera, mic and GPS on HTTPS pages, and iPhone Safari won't speak until the user taps once. The ESP32 also can't join eduroam, so we ran everything off a phone hotspot.
🔄 Changing course mid-hack. We swapped our speech-to-text provider to ElevenLabs and dropped account login entirely, because every extra step is a barrier for someone who can't see the screen.

🏆 Accomplishments that we're proud of

✅ A working harness that buzzes, talks to the server and triggers a camera check in real time.
🚦 The hazard tiers. Deciding what not to say turned out to be the hardest and most important part.
🎙️ A full voice-first setup flow that someone can get through without looking at the screen.

📚 What we learned

Designing for blind users is mostly about restraint: less speech, fewer taps, no accounts. We also learned a lot about wiring a microcontroller, a phone and a server together over WebSockets in under two days.

🚀 What's next for Sighty

🐕‍🦺 Calibrate the sensor distances on a real harness with a real guide dog team.
👂 Hands-free voice with a wake word instead of tap-to-ask.
🔒 Keep the walk running with the screen locked.

🤝 Community impact

Sighty gives handlers information to make their own decisions. The system supports the existing partnership between the handler and the guide dog rather than replacing it. The next phase is co-design with guide-dog teams and mobility professionals. We will measure if the system improves navigation and reduces effort while preserving the handler's awareness of ambient traffic sounds. Success is useful information, delivered when wanted, that makes everyday journeys more manageable.