FOCUS VisionAssisted Project

Inspiration

Imagine walking through a crowded area while an AI constantly tells you about every object it sees. For someone who is visually impaired, too much information can be just as overwhelming as too little. And a camera feed is someone’s everyday viewpoint, so it shouldn’t have to leave their device.

That’s why we created F.O.C.U.S. (Filtering Objects by Context, Urgency, and Space).

F.O.C.U.S. runs entirely on-device, so no images are sent to the cloud and it works without an internet connection. It combines two modes: real-time hazard alerts that stay quiet until something matters, and on-demand scene descriptions when the user asks. Instead of announcing everything, it answers "what does this person need to know right now?"

What it does

F.O.C.U.S. is an on-device AI vision assistant. No images are sent to the cloud, and it works without an internet connection. It combines two modes:

Hazard alerts (automatic, every frame): detects and tracks obstacles, measures their distance and direction, and speaks short alerts like “Person, 2.5 meters, ahead.” It stays quiet until it detects a "hazard."

How we built it

We built our system using two paths: fast and slow.

Fast Path (Object detection using YOLO and depth models): YOLO26 and YOLO-World with BoT-SORT tracking, Depth Anything V2 for metric depth, a geometry fuser for position, bearing and closing speed (with smoothing), and a rule-based prioritizer with decision-making based on object type, position, motion and distance.

Slow Path (VLM detected scenery): a local Gemma model via Ollama, plus a router that decides whether a request goes to the sensors, the model, or both.

Challenges we ran into

Detecting the right things. A single standard detector missed many real-world hazards. We combined two: YOLO26 for common classes and YOLO-World for extra obstacles like staircases and curbs. Open-vocabulary detectors score lower, so we also had to tune confidence thresholds so detections weren’t silently dropped.
Combining the detection pipeline with a VLM. Real-time detection and a slow, descriptive language model work on very different timescales. We split them into two independent paths and added a router that decides which one answers each request, so the VLM can add detail without ever delaying a hazard warning.
Designing the priority stack. The hardest question was what the user needs to hear, and when. We settled on explicit rules for each object category (distance, time-to-contact, whether it’s in the walking path).

Accomplishments that we're proud of

Hazards always take priority over descriptions: Our prioritizer judges each object by its type, position relative to the walking path, motion (closing speed and time-to-contact) and distance. A hazard alert cuts off whatever is being said, including a scene description in progress.
Hazard alerts are independent of the language model: If the VLM is slow, wrong or crashed, warnings still work.
Our multi-model perception stack: YOLO26 and YOLO-World detect and track objects, including hazards that standard detectors miss, and Depth Anything V2 adds real distance in meters to each one.

What we learned

Choosing and comparing models: We learned how different object detection and depth estimation models work, where each one is strong, and why combining several covers more than any single model.
Turning detections into real-world information: We learned how to combine depth with tracked object IDs to estimate distance, direction and speed, and how much smoothing that data needs before it’s reliable.
Combining fast and slow AI: We learned how to pair a real-time detection pipeline with a slower vision-language model so each does what it’s best at.
Deciding what to say: For accessibility, the hardest part was choosing what information matters most and when to share it, not just detecting objects.

What's next for FOCUS VisionAssisted Project

Ask questions about the environment. We want users to be able to ask things like “what does the sign say?” or “where is my phone?” and get an answer, using sensor measurements when possible and the vision-language model when the picture is needed.
Test with blind and low-vision users, and tune alert wording, timing and length from their feedback.
Wearable hardware, such as a chest or glasses camera with bone-conduction audio