Inspiration

Our opening clip shows one collision: a blind pedestrian struck by an e-scooter. It’s one incident, but it makes it very clear the danger that visually impaired individuals are subject to, just when walking down the street. Turn-by-turn directions can guide someone toward a destination, yet still put you in harm's way.

RNIB’s 2024 survey of 405 UK adults registered blind or partially sighted found that 82% had collided with a street obstacle in the previous three months; 24% reported a collision with an e-scooter and 20% with a regular bicycle.

What it does

StreetSight combines walking directions with camera-based obstacle alerts. A user can enter or speak a destination, receive walking directions, and get short spoken and haptic warnings about likely obstacles. The camera preview can show object bounding boxes and an estimated hallway path.

How we built it

The iPhone app is built with Expo and React Native. It sends camera frames over a WebSocket to a local Python aiohttp server, which runs on a nearby laptop. Walking routes use OpenStreetMap geocoding and routing services.

For object detection, we use Ultralytics YOLO11n with a 320-pixel inference size and a 0.45 confidence threshold. It returns object classes and normalized bounding boxes. A lightweight heuristic flags likely obstacles when a recognized object is in the center of the frame and its box is large enough, or reaches far enough toward the bottom. The box’s left and right margins inform a short avoidance suggestion. Alerts are rate-limited to avoid repeating the same warning too often.

We integrated Qwen2.5-VL 3B through Ollama as an optional vision-language step. YOLO can identify a likely obstacle first; if enabled, the server sends that frame to the local Ollama endpoint and requests a concise instruction in JSON. Because VLM inference is slower, the app waits for it rather than queueing additional frames. The default live demo uses YOLO-only warnings.

For the path estimate, OpenCV resizes and blurs each image, applies Canny edge detection, then uses Hough lines to estimate converging boundaries in the lower part of the frame. The Arduino sketch samples an HC-SR04 every 250 ms, activates a vibration motor below 100 cm, and clears it at 115 cm to reduce rapid toggling. It sends distance and hazard readings as JSON over USB serial.

Challenges we ran into

Connecting the phone and laptop took troubleshooting. In our setup, Georgia Tech eduroam allowed internet access but blocked direct peer-to-peer communication, so we switched to a personal hotspot. The phone needed the laptop’s hotspot IP, not localhost, and the server needed to listen on 0.0.0.0. We also had to distinguish Metro’s port 8081 from the inference server’s port 8765 and check for an existing listener when the port was already in use.

Expo Go could not provide the native speech-recognition functionality we needed, so we moved to a local iOS development build through Xcode and updated Node to meet Expo’s requirements. On the vision side, carpet patterns created distracting edges, and a path overlay could look too fixed. Restricting line detection to the lower image and smoothing boundary estimates helped, but did not turn the overlay into a true map of walkable space.

On the vision side, the pretrained YOLO classes didn’t include important navigation hazards like stairs, doors, or speed bumps. We trained and integrated custom classes for all three, then connected their detections to our bounding-box overlays and hazard-specific alert logic. Carpet patterns also created distracting edges, so we restricted path-line detection to the lower part of the image and smoothed boundary estimates across frames. Those lines remain visual estimates, not a depth map or a guarantee that the marked space is clear.

Accomplishments that we're proud of

We brought together an iPhone interface, walking routes, camera streaming, local object detection, spoken and haptic alerts, and a hardware sensor path in one prototype. We’re especially proud of separating the fast YOLO warning loop from optional VLM reasoning, so richer language generation does not have to block the basic alert path.

What we learned

We learned that recognizing an object is not the same as knowing it is dangerous or how close it is. YOLO gives us a class and a bounding box, but a monocular camera does not directly provide distance. Box size and position are only rough proximity cues, so our collision logic has to be conservative

YOLOv11 doesn't currently support stair, speed bump, or door classes. To implement these, the model needs representative labeled examples, and its class mapping, confidence handling, visual overlays, and alert behavior all need to agree. A model that works in one hallway may behave differently with another camera angle, lighting, floor pattern, or partially hidden obstacle. We need broader testing to understand those failure cases.

What's next for StreetSight

Next, we want to evaluate the prototype with blind and low-vision participants, under appropriate supervision, and measure false alerts, missed obstacles, and end-to-end latency. We also want to improve depth and walkable-space estimation, test on varied indoor and outdoor paths, and refine the wearable hardware and tactile feedback.

StreetSight is an experimental prototype, not a certified mobility aid. We have not established that it reduces collisions, and it is not a replacement for a cane, guide dog, or established mobility practices. Ultimately, we hope for StreetSight to be a suitable addition for a cane that can give visually impaired individuals a heads up for objects in their way.