-
-
Internal Components
-
Left Side
-
Belt View
-
Front Side
-
Complete View
-
Right Side with Internal Components Visible
Inspiration
What if you could feel an obstacle before you reached it?
Wayfeel began with the idea to address the needs of people without vision. For blind users, nearby hazards should be communicated without requiring a screen or overwhelming the environmental sounds they already depend on.
Our vision pairs Ray-Ban camera input with a directional vibration belt, providing an additional source of information alongside established mobility aid without replacing it.
What it does
Wayfeel is an experimental collision-awareness prototype with three main components: an iPhone camera bridge, a Python perception hub, and an ESP32 vibration belt.
The hub combines object detection, estimated depth, and motion information to assess potential hazards across five directions. Its haptic policy maps obstacle direction to the corresponding belt motor: a vibration on the left means an obstacle on the left.
The iPhone app provides session controls, connection status, and VoiceOver announcements. The system distinguishes paused sessions, deliberate stops, stale camera input, and connection failures rather than treating silence as evidence of a clear path.
We have tested the software pipeline with simulated glasses, replayed video, and simulated belt output. A separate MacBook webcam experiment has also completed live cloud inference for proof of concept. Live Ray-Ban capture and the complete physical camera-to-belt experience remain in testing.
How we built it
We built the iPhone app in Swift and SwiftUI, using Meta’s Wearables Device Access Toolkit for the intended glasses connection. Camera frames travel over WebSocket to a Python hub, which sends bounded haptic commands over Bluetooth Low Energy.
The hub uses YOLO11n and ByteTrack for object detection and tracking, Depth Anything V2 for depth estimates, and experimental OpenCV optical flow for additional motion evidence.
For NVIDIA-powered visual labeling, we integrated Nemotron 3 Nano Omni. It labels tracked regions as people or objects in the background, while local perception handles collision alerts.
Our earlier browser-based vision experiment uses NVIDIA Cosmos3-Edge on a Brev A10G GPU, combining object detections with measured optical flow to produce short collision assessments.
We wrote the belt firmware in C++ with PlatformIO, added a browser dashboard, and built recording, replay, and simulation tools to investigate failures.
Challenges we ran into
Recognizing an object does not establish whether it presents a collision risk. Camera movement, object motion, depth errors, and changing bounding boxes can produce misleading signals.
In synthetic experiments, our initial vision model sometimes described stationary or receding objects as approaching. This led us to separate semantic recognition from measured motion evidence.
Cloud latency also shaped the architecture: background labeling should not delay local alerts, and when hosting the LLM through brev, we had latency that capped the video at around 1 FPS. Hardware communication introduced another challenge: successful Bluetooth delivery does not prove the firmware applied a command. We added belt status reporting, command-expiry handling, and explicit fault states.
Accomplishments that we're proud of
- Built modular phone, perception, and belt components with documented communication protocols.
- Verified live MacBook-camera inference and real Nemotron requests using replayed inputs.
- Created simulations that connect the real hub to the belt’s actual control logic compiled for a computer.
- Added accessible phone controls, an operator dashboard, and distinct session and fault cues.
- Built automated tests, continuous integration, recording tools, and a hardware bring-up checklist.
What we learned
A convincing AI response is not sufficient evidence for a physical alert. Timestamps, measured motion, and explicit uncertainty matter just as much as the model's output.
Also, we learned that accessibility includes communicating the system's state. A wearer needs to know when alerts are paused, disconnected, or unavailable, not just when an obstacle is detected.
Simulation helps expose problems early, but passing simulated scenarios does not establish real-world reliability. Repeating experiments, fine-tuning the model, and accounting for changing environmental variables leads to consistent results in real world situations.
What's next for Wayfeel
Our immediate priorities are refining depth measurements, implementing reliable non-wired connectivity, and adding navigation that processes high definition maps to keep our users updated with their surroundings.
Work currently under review adds floor-relative depth processing, revised approach estimates, clearer uncertainty reporting, optional spoken obstacle warnings, and stationary practice controls. An experimental spoken campus-route feature is also being explored separately from obstacle vibrations.
We want to evaluate vibration patterns with blind users and test failures involving head movement, low light, and disconnections. Walking demonstrations will follow bench testing and controlled, supervised evaluation.
