AI Assistant for visualy impaired

InspirationCommercial assistive vision devices cost between $1,500 and $4,500 USD, pricing out over 90% of visually impaired individuals worldwide. We set out to eliminate dedicated hardware barriers entirely by turning the ubiquitous smartphone into a real-time, voice-guided spatial co-pilot using standard web technologies.What it doesThe web application provides eyes-free, hands-free urban navigation on any modern smartphone:Real-Time Obstacle Detection: Uses the camera to detect walking hazards and delivers instant spatial voice alerts (e.g., "Stop! Vehicle approaching ahead").Stationary-Aware Audio: Uses ground-plane tracking ($y_2$) to alert the user to moving threats while remaining silent when standing still to avoid audio fatigue.Turn-by-Turn Pedestrian Navigation: Provides OSRM-powered walking instructions with 12-meter turn triggers and 35-meter off-route recalculation.Echo-Safe Voice Control: Enables full-screen, single-tap voice commands ("Home", "Repeat", "Stop") without audio feedback loops.How we built itFrontend: Vanilla JavaScript, WebRTC camera capture ($320 \times 240$), Geolocation API, and Web Speech API on a full-screen tactile canvas with haptic feedback.Backend: FastAPI and Uvicorn running in-memory frame ingestion with zero disk I/O latency.Computer Vision: Ultralytics YOLOv8n running on CPU alongside OpenCV 5 SIMD decoding, calculating ground-contact proximity ($y_2$) and looming velocity:$$\Delta y_2 = y_{2, t} - y_{2, t-1}$$Routing: OSRM pedestrian API integrated with planar equirectangular distance math for sub-millisecond waypoint checks.Challenges we ran intoMicrophone-Speaker Echo: The phone's speech output was triggering false voice commands. We engineered an Exclusive Audio State Machine that cancels active TTS immediately upon screen tap before opening the microphone.Network Latency: Initial video streams caused ~350 ms latency. Downscaling video capture to $320 \times 240$ with $0.25$ JPEG compression dropped round-trip latency to ~135–160 ms.Audio Overload: Announcing every detected object masked critical ambient street sounds. We added a Single-Threat Priority Queue and Stationary Memory Lock that only warns of the closest threat and mutes static obstacles.Accomplishments that we're proud ofBuilt a zero-hardware, zero-cost accessibility solution accessible via standard mobile browsers.Achieved sub-160ms end-to-end inference over mobile networks on CPU hardware.Solved the mobile web audio feedback loop using gesture-synchronized state transitions.Modeled pedestrian hazard distance accurately using bottom-edge ground kinematics ($y_2$).What we learnedSilence is Safety: Visually impaired pedestrians rely on ambient traffic sounds; assistive audio must be concise, urgent, and quiet when paths are static.Kinematics Beat Brute-Force ML: Simple geometric rules like ground-plane tracking and velocity vectors solved false alarms better than larger neural networks.Web APIs Can Replace Hardware: Browser-native capabilities (WebRTC, Speech, Geolocation) are mature enough to build high-performance assistive systems.What's next for AI Assistant for visualy impairedOn-Device Inference: Porting YOLOv8n to client-side WebGPU/Wasm via ONNX Runtime for sub-30ms offline execution.On-Demand Scene VLM: Adding multimodal models (e.g., Gemini Flash) for static queries like reading signs or finding building entrances.Custom Hazard Weights: Fine-tuning detection for street hazards like open potholes, uneven curbs, and overhanging branches.

Built With

• css3
• fastapi
• html5
• javascript
• numpy
• opencv
• python
• pytorch
• yolo