Waymaker

Inspiration

A white cane is great at finding curbs, steps, and anything on the ground, but it can't warn you about a branch at head height, a bike closing in from the side, or tell you what a sign says. The AI tools that can, like smart glasses and phone apps, mostly depend on the cloud, which disappears in subways, tunnels, and rural areas. We wanted to build something that fills the gaps a cane leaves and never needs a signal to keep someone safe.

What it does

Waymaker is a backpack wearable for blind and low-vision people that runs fully offline on an NVIDIA Jetson Orin Nano.

Feel obstacles: two ultrasonic sensors and a chest camera drive vibration motors under the left and right backpack straps. Pulses speed up as obstacles get closer, and turn steady when it's time to stop.
Understand the street: an on-device YOLO11n model tracks people, bikes, poles, and other obstacles, and predicts whether each one is actually on a collision course. A second model reads pedestrian crossing signals and announces when they change.
Ask about the view: hold the button on the strap, ask "What does that sign say?", and Waymaker answers through your AirPods using a local vision-language model.
Safety first: crossing-signal alerts interrupt the assistant mid-sentence, and the motors never wait on the AI. If the camera fails, the ultrasonic sensors keep working.

How we built it

Hardware: Jetson Orin Nano 8GB, a 130° USB camera on the backpack strap, two Grove ultrasonic sensors, two PWM vibration motors, a Grove button, and AirPods Pro over Bluetooth, all wired directly to the Jetson.
Obstacle detection: YOLO11n, exported to TensorRT, runs on the Jetson's GPU. Because a chest camera sways with every step, we measure that rotation from the background and remove it before tracking.
Collision prediction: we estimate time to collision from how fast an object's bounding box grows, $$\text{TTC} \approx \frac{1}{\frac{d}{dt}\ln s(t)}$$ then project where the object will be when it reaches the user. Both motors fire only if it will cross the walking path.
Sensor fusion: a 100 Hz haptics thread takes whichever source is more urgent per side. Ultrasonic gives true distance up close and catches glass and plain walls; the camera sees farther and knows what's coming.
Voice assistant: pressing the button grabs the sharpest recent frame and starts encoding it in Qwen3-VL-2B (via llama.cpp) while the user is still talking. Whisper transcribes the question, the VLM streams its answer, and Piper speaks each sentence as soon as it's ready.
Demo view: a live MJPEG stream with bounding boxes, motor states, and Q&A timings, served to a laptop over a local hotspot.

Challenges we ran into

Memory: the CPU and GPU share 8 GB, so every model competes for space.
False alarms: early versions buzzed at people just walking past.
A dead board: our first Jetson wouldn't boot, and it took us a while to realize the board was broken.
New to hardware: it was our first time with circuits, so we learned by trial and error.
Wiring: at 11 p.m. Saturday we realized we needed Grove cables. We asked other teams for spares, then made our own with tape.
Loose wires: our chained jumper wires kept disconnecting and were hard to track.
Scarce data: we couldn't find enough labeled data, so we annotated images by hand.

Accomplishments that we're proud of

A complete pipeline, from sensors to vibration to spoken answers, that works with zero internet.
Warnings that can tell the difference between someone walking toward you and someone walking past.
A custom 3D-printed case for the Jetson Orin Nano.
Training our own classification model on data we labeled by hand.

What we learned

Safety alerts must be instant while answers can take a few seconds.
Two cheap sensors that cover each other's blind spots can beat one expensive one.
On edge devices, memory runs out before compute does.
Test the hardware first.
Plan and label your wiring before building.