Instinct

Inspiration

Between 61 and 93 percent of EMTs are physically assaulted at some point in their careers, and about a quarter of those assaulted are injured. We've worked in clinical settings and have seen firsthand how dangerous they can be.

When an EMT is focused on a patient, instinct alone isn't enough to protect them. So we built them a better one. That's INSTINCT.

INSTINCT is a lightweight computer-vision wearable. It watches the space behind a first responder, predicts when someone is closing in, and sends a strong vibration on the side they're coming from. The responder never needs to take their eyes off the patient.

How we built it

Pipeline: rear camera -> YOLO person detection -> ByteTrack multi-object tracking -> physics-based world model -> task-aware attention policy -> directional haptics.

Due to the resources required for training a model, we instead built an interpretable estimator that we could reason about and debug.

An object approaching the camera grows in the image, and the rate of that growth contains information about time-to-contact directly. For each track we fit the looming rate from bounding-box height $ h $ over a half-second window:

$$ r = \frac{d}{dt}\ln h $$

Time-to-contact is its inverse, corrected for the lag of the fitting window ($ \bar{t} $ is the window's mean timestamp):

$$ \tau = \max\left(0,\; \frac{1}{r} - (t - \bar{t})\right) $$

Looming alone can't separate someone walking at you from someone passing close by. For that we use the classic collision rule from navigation: constant bearing, decreasing range. From rough depth $ Z $, bearing $ \theta $, and bearing rate $ \dot{\theta} $, we predict the lateral miss distance at arrival:

$$ m = \frac{Z\,\dot{\theta}\,\tau}{\cos^2\theta} $$

Approach evidence, urgency (from $ \tau $), and predicted conflict (from $ m $) combine into a risk score in $[0, 1]$. Each task carries an interruption cost, and a cue fires only when risk exceeds the cost for the current task. CPR's cost is the highest.

[When our haptic hardware failed late in the event, we moved to phones: one phone as the rear camera and two as left and right haptics.] The whole pipeline ends in a single command (LEFT, RIGHT, or NONE), so changing the hardware didn't touch the prediction code.

Challenges we ran into

Monocular depth. Once someone's feet leave the frame, width is the only size signal left, and arm swing on a person's last step can look like a rapid approach. We still see this in a few recorded cases and have documented them.
Hardware. Our haptics failed late in the weekend, and we rebuilt the output on phones.
Networking. Campus Wi-Fi made it hard to connect the Pi, so we turned the Pi into its own access point.

What we learned

Deciding when not to alert turned out to be harder than detecting people. We also learned how far explicit, physics-based modeling can go without training data, and how useful replayable logs are when you can't keep re-running live tests.