Inspiration

Most tools for blind and low-vision pedestrians alert on proximity: something is near, so they buzz. In a city, everything is near. The result is a device that buzzes all the time, and people switch it off.

Smart canes already handle the ground and fixed obstacles well. What they don't handle is the thing that is hardest to hear coming: a bike, a scooter, or a quiet electric car moving toward you. We wanted a warning that fires only when something is actually going to reach you, and stays silent for everything that is just passing by.

Sailors have had a rule for this for centuries: constant bearing, decreasing range means collision. HeadsUp is a rule, worn on the face and the wrist.

What it does

HeadsUp is a wearable that warns only about things on a collision course.

The wearer has Meta Ray-Ban glasses and a wristband with two vibration motors. The glasses camera streams video to an iPhone. The phone finds people, bikes, and cars, decides which ones are heading at the wearer, and then:

  • buzzes the side of the band that matches the hazard, more strongly as it gets closer in time
  • speaks a short cue through the glasses, such as "car, left"

Someone walking past produces nothing. Someone walking straight at the wearer produces a buzz.

It also gives voice navigation: press a button, say a destination, and hear turn-by-turn walking directions. Hazard warnings always interrupt directions, never the other way round.

Two rules shaped everything. HeadsUp works alongside the white cane, never instead of it. And it only reports what it saw. It never says that something is safe or tells the wearer to go.

How we built it

Glasses camera → iPhone: detect → track → judge risk → decide alerts → band + speech

Everything in the hazard loop runs on the phone, with no network. A round-trip to a server is too slow for something coming at you.

Detect. A YOLO11n model converted to Core ML finds people, bicycles, motorcycles, cars, buses, and trucks. If a frame is still being processed when the next one arrives, the new one is dropped, so we never fall behind.

Track. An IoU tracker gives each object a stable ID and keeps it alive for about a second when it is hidden.

Judge risk. This is the core, and it is plain geometry, not AI.

1. Is it getting closer? An approaching object's box grows. Let \( s = \sqrt{\text{box area}} \). We fit a least-squares line to \( \ln s \) against time. Its slope \( k \) is the expansion rate, and the time to contact is

$$\tau = \frac{1}{k}$$

If \( k \le 0 \), the object is not approaching. This needs no depth sensor.

2. Is it heading at me? We fit lines to the box's left and right edges to get their velocities \( v_L \) and \( v_R \), then predict where the edges will be at contact:

$$L^* = L + v_L\,\tau \qquad R^* = R + v_R\,\tau$$

The object is on course if \( [L^, R^] \) overlaps the wearer's corridor \( [-c, +c] \), with \( c = 0.15 \) in frame units from −1 to +1. If the box grows but slides out of the corridor, it is passing, and we stay quiet.

3. Is it actually moving? Walking toward a standing person also makes their box grow. We estimate distance from the box height \( h \), a typical real height \( H \) for the class, and a focal length \( f \) we measured ourselves:

$$Z = \frac{f\,H}{h}$$

The closing speed is \( Z / \tau \). We subtract the wearer's own walking speed, and what is left is the object's speed:

$$v_{\text{object}} = \frac{Z}{\tau} - v_{\text{wearer}}$$

Below 0.5 m/s it counts as standing still.

4. Is it soon enough? \( \tau \) sets the level:

Time to contact Level Output
more than 5 s 0 nothing
3.5–5 s 1 gentle buzz
2–3.5 s 2 medium buzz and a spoken cue
under 2 s 3 strong buzz

Decide alerts. A level must hold for 3 frames to rise and 5 to fall, so the band doesn't flicker. With several objects, the lowest \( \tau \) wins.

The band. A Seeed XIAO ESP32-C3 on a breadboard, two vibration motors, and a battery, fixed to a wrist brace. The phone connects over Bluetooth Low Energy and sends each motor a strength and a short run time every 200 ms. If the phone goes quiet, the motors stop by themselves within 400 ms.

Navigation. Apple Maps for routes, on-device speech recognition for the destination, and a small FastAPI server on a laptop that checks aerial imagery for marked crosswalks at turns. It sits beside the hazard loop and never touches it.

The app. Swift and SwiftUI, about 6,900 lines, with 139 unit tests that run on synthetic scenes with no camera.

Challenges we ran into

  • Head turns look like motion. When the wearer turns their head, every object slides across the frame, which fakes or hides a collision course. We measure the shift of the whole image between frames and, when it is fast, hold our judgments until the view settles.
  • Walking into a standing person buzzed. That is a real approach, but not the hazard we care about. It led to the "is it moving?" check and the distance estimate.
  • Apple's walking label was too slow. It took around 10 seconds to notice that the wearer had started walking. We wrote our own step detector on the accelerometer.
  • We had to measure our own camera. Distance needs a focal length, so we ran a tape-measure test with a person at known distances.
  • Merging boxes. When two people overlap, the box suddenly jumps in size, which looks like something rushing at you. We added a check that keeps those jumps out of the estimate.
  • App and band spoke different languages. The app was written for one Bluetooth message format, and the band firmware used another. We changed the app to speak the band's format instead of reflashing hardware close to the deadline.
  • Power. The first band design needed a power bank in a pocket and a cable up the arm. We moved to a battery on the band.
  • A new SDK. Meta's Wearables Device Access Toolkit is recent and changes often, so the glasses stream was the first and riskiest step.

Accomplishments that we're proud of

  • It stays quiet. A person walking past does nothing; a person walking at you triggers the band. That difference is the whole project, and it works.
  • Fast. Detection takes about 12 ms a frame and frame-to-band about 17–19 ms on the phone (not counting the glasses' own Bluetooth delay).
  • Distance from a single camera was within about 7–12% of a tape measure between 3 and 8 m.
  • A working end-to-end system in a weekend: glasses, phone, custom wristband, speech and navigation.
  • The risk logic is explainable. It is a few equations we can write on a whiteboard, covered by 139 tests, not a black box.

What we learned

  • Deciding when not to alert is harder and more valuable than detecting things.
  • Classic geometry can beat a bigger model. Time to contact from a growing box is decades-old vision science, and it runs in microseconds.
  • Estimates stack. Time to contact, distance and walking speed each carry error, and the errors add up, so we lean toward warning.
  • Hardware and software need an agreed message format on day one.
  • For an assistive device, the wording matters as much as the code. "Car, left" is a report. "Safe to cross" is a promise we can't keep.

What's next for HeadsUp

  • Test with blind and low-vision users. Everything so far was tested by us.
  • Pulse patterns on the band, so levels differ by rhythm and not only by strength.
  • Say when the band disconnects, and fall back to phone vibration by itself.
  • A real band. A printed circuit board and a low-power Bluetooth chip in place of the breadboard, for days of battery life.
  • Crowds, low light and range. Overlapping people and darkness are our weakest cases, and detection fades past about 8 m.
  • Coverage behind the wearer, where the glasses cannot see.

Built With

Share this project:

Updates

Submission history