Smart Cane - Guardian Ecosystem

Expanded setup of the cane with the Pi and wiring detached
Top-down view of the prototype attachment contraption

Inspiration

Over 2.2 billion people worldwide live with vision impairment or blindness, and millions more face cognitive challenges like dementia that affect their ability to navigate the world safely. Traditional white canes detect physical obstacles but offer no awareness of who is nearby, whether it be a family member, caregiver, or a stranger. We wanted to build something that gives these individuals not just physical safety, but social and emotional context, especially for those who cannot rely on their eyes. So, we developed Smart Cane, an extra set of eyes and an outstretched hand.

What it does

Smart Cane is an AI-powered, modular assistive device and companion mobile app for visually impaired and dementia patients. It is the premier of an ecosystem of "guardian devices," a collection of equally compassionate and accessibility-first companion accessories.

Face recognition: An onboard camera identifies registered people (family, caregivers, friends) and announces them aloud through the user's phone using natural, warm language generated by Gemini AI.
Voice announcements: Gemini 2.0 Flash crafts personalized phrases ("Your daughter Sarah is nearby and coming to greet you") delivered via ElevenLabs TTS through the mobile app.
Live companion app: A React Native app connects over BLE and WiFi to show real-time obstacle distance, battery level, last detected face, GPS tracking, and safe zone geofencing.

How we built it

The system has two parts:

Smart Cane

Raspberry Pi 5 (the cane brain)
Python multithreaded orchestrator with 4 concurrent threads: face recognition, BLE GATT server, and a WebSocket TTS server
OpenCV and DeepFace for real-time face detection at 2 Hz
bless library for BLE peripheral mode, exposing 7 GATT characteristics
google-generativeai SDK for Gemini Flash phrase generation with a 1-hour phrase cache
websockets server on port 5001 pushes TTS text to the mobile app instead of doing audio locally

React Native Mobile App

BLE central via react-native-ble-plx for live sensor data
WebSocket client receives TTS text from the Pi → calls ElevenLabs API → plays audio locally (clean separation of concerns: Pi generates text, phone generates speech)
react-native-maps with Google Maps for GPS tracking and geofence management
react-native-config for secure API key management via .env

##Challenges we ran into

BLE on Linux: Getting bless to act as a stable GATT peripheral on the Pi required careful asyncio event loop management across threads. Thread-safe WebSocket broadcasting from synchronous threads was solved with asyncio.run_coroutine_threadsafe.
Android build environment: Gradle dependency conflicts between react-native-maps, react-native-screens, and the Android Gradle Plugin 8.x required careful version pinning. Emulator performance was too slow on our machines, so we built standalone APKs and deployed directly to a Samsung S22 Ultra.
TTS latency: Running ElevenLabs on the Pi introduced ~3 s delays and used significant memory. Moving TTS to the phone eliminated Pi-side latency and enabled Bluetooth audio routing to hearing devices automatically.
Face recognition in real time: Running dlib on the Pi at usable frame rates required downsampling frames and running detection at 2 Hz rather than full video rate.

##Accomplishments that we're proud of

A fully working end-to-end pipeline: camera detects a face → Gemini generates a warm announcement → ElevenLabs speaks it through the phone, all in under 2 seconds
- Clean separation of the Pi (AI brain) and phone (UI + audio output) over both BLE and WebSocket simultaneously
- Gemini-generated announcements that are genuinely warm and context-aware, not robotic
- A polished caregiver companion app with live BLE monitoring, GPS tracking, face registry, and SOS — built and deployed to a real device in one hackathon weekend

##What we learned

Designing for accessibility forces you to think about latency, reliability, and clarity in ways regular apps don't. Every half-second delay matters when someone is walking toward an obstacle
Raspberry Pi 5's improved performance makes real-time computer vision genuinely viable at the edge
Splitting AI workloads between edge (text generation on Pi) and cloud (TTS synthesis on phone) is a practical pattern for latency-sensitive assistive tech

##What's next for Smart Cane

Raise-to-wake functionality, so that the guardian is watching over you when you need it.
A greater ecosystem of comfort-first guardian contraptions, including glasses equipped with object avoidance and facial recognition for users with no trouble walking
A vibration motor added to the attachment, signaling
- On-device face recognition using a quantized model to work without WiFi
- Fall detection using an IMU (accelerometer/gyroscope) with automatic SOS trigger
- Expanded mobile dashboard interface and congruent web interface
- Voice commands: the user can ask "Who was that?" or "How far is the obstacle?" via the phone mic
- Clinical trials with assisted living facilities in partnership with local health organizations
- Obstacle detection: Dual ultrasonic sensors measure distance in real time and drive a PWM buzzer with dynamic beep patterns: slow pulses at 3 m, rapid pulses at 1 m, and a continuous tone at under 20 cm.
- SOS system: A 2-second button hold on the cane triggers an emergency alert to registered contacts with the user's GPS location.