VisionAssist

Inspiration

Navigating dynamic physical environments is a profound challenge for visually impaired individuals. While traditional white canes are excellent for detecting ground-level barriers, they completely miss head-height hazards and cannot identify what an obstacle actually is. We wanted to build a hands-free, wearable solution that restores spatial awareness and independence without forcing users to rely on expensive, proprietary hardware or cloud subscriptions.

What it does

"Visionary" is a multimodal, edge-AI navigation assistant. It continuously scans the user's environment to detect, track, and classify approaching obstacles. Using a custom proximity algorithm based on bounding-box scaling and screen position, it translates visual data into real-time audio cues (e.g., "Warning. Chair ahead. Very near."). It also triggers a low-latency physical buzzer for immediate safety alerts, allowing the user to seamlessly perceive their surroundings and avoid collisions.

How we built it

The core logic runs on a Raspberry Pi 4 equipped with a lightweight, quantized YOLOv8 model via Ultralytics and OpenCV. Visual input is captured by a chest-mounted Sony IMX323 USB camera, which is paired with custom 6000K LEDs and 3W drivers to ensure the camera can still see in dim environments. To offload critical safety alerts from the Pi's operating system, an Arduino microcontroller acts as a dedicated deterministic controller to drive the passive buzzer.

On the software side, we utilized Python's threading library to run the pyttsx3 Text-to-Speech engine asynchronously. This ensures the camera stream maintains a steady frame rate and never blocks while the system is speaking to the user.

Challenges we ran into

Running real-time computer vision on a Raspberry Pi 4 CPU is heavily resource-constrained. We initially struggled with severe latency and single-digit frame rates. We overcame this by downsizing to the YOLOv8-nano model, reducing the inference resolution, and implementing a frame-skipping tracker logic. Additionally, managing the audio buffer—specifically preventing the TTS engine from stacking overlapping speech alerts or freezing the main camera loop—required engineering a custom, non-blocking asynchronous message queue.

Accomplishments that we're proud of

We are incredibly proud to have built a fully offline, edge-computed system. Because Visionary processes all video locally on the Raspberry Pi without sending data to an external API, it protects user privacy, operates with lower latency, and works reliably in areas with zero internet connectivity.

What we learned

Building this prototype taught us how to optimize machine learning models for edge ARM CPUs, how to orchestrate hardware communication between a Linux SBC and a microcontroller, and how to write robust, multi-threaded Python architectures to handle concurrent sensor data and audio streams.

What's next for Visionary

The next steps involve migrating the compute to a smaller form factor (like a Raspberry Pi Zero 2 W paired with a Coral TPU) for better wearability, swapping to a stereo camera array for true depth-map estimation, and integrating a lightweight local Vision-Language Model (VLM) so users can ask conversational questions like, "Is there an empty seat on this bus?"