Allos

Inspiration

Allergen cross-contact is invisible. A cook can touch a bowl of peanuts, pick up the tongs, wash their hands properly, and still plate a peanut-allergy order with tongs that carry peanut. Nobody saw a mistake, because the risk was created minutes and several touches earlier. Kitchens rely on memory and good habits, and the one thing everyone remembers ("I washed my hands") is exactly what hides the problem.

We wanted a system that keeps track of where a risk came from across time and objects, not just what's happening in the current frame. Our pitch in one line: "Washing your hands didn't clean the tongs."

What it does

Allos is a real-time computer-vision prototype for a food-prep station. An overhead webcam watches a worker's hands, a pair of tongs, a (simulated) peanut source, a sink and a plate for Order #4, peanut allergy.

It detects contacts. When fingertips stay in an object's zone long enough, or when the tracked tong tips dip into the peanuts or touch the plate, Allos records a contact.
It tracks potential exposure across every contact. Exposure moves both ways between the two things that touched. The tong tips and handle are tracked separately, so gripping a handle that only clean hands ever touched doesn't spread anything.
It knows a hand wash cleans only the hands. Hands held in the sink for 6 seconds count as a wash, which clears the hands and nothing else. Washing hands and the tong jaws together in the sink clears both.
It explains every alert. When potential peanut exposure reaches the allergy order's meal, the dashboard shows "Action required — hold meal", chimes, and lays out the full chain, for example: Allergen source → Hands → Tong tips → [Hands washed] → Tong tips → Hands → Meal
It follows the tongs, not just the hands. If the worker never touches a peanut but dips the tongs and serves with them, Allos follows the tongs instead: Allergen source → Tong tips → Meal.
It has a physical alarm. An ESP32 with an RGB LED and buzzer flashes red and beeps on an alert. One press of its button acknowledges the alert, and a second press resets the meal for a replacement.

How we built it

Everything runs in one browser tab on the laptop. There's no server, and no video ever leaves the device.

Vision (TypeScript, MediaPipe, ArUco)

MediaPipe Hand Landmarker (WebAssembly, GPU delegate) tracks 21 landmarks on each of two hands. We use the five fingertips for contacts.
Each object has a calibrated polygon zone in normalized camera coordinates. A dwell-time state machine per actor-and-target pair turns noisy per-frame overlaps into clean contact events.
The tongs carry an ArUco marker. Each frame, the visible marker's four corners give a planar homography that projects our calibrated tip outline, handle outline and jaw-end points into the image, so the tong zones move with the tongs.
Height gate: from overhead, hovering looks like touching. A marker closer to the camera looks bigger, so we record the marker's size with the jaws resting on the plate and in the peanuts. Tongs carried more than about 4–5 cm above a target don't count as touching it.
Sink detection: 6 seconds of hands in the sink is a hand wash. Hands and tong jaws in the sink together, at bowl depth, is a utensil wash.

Exposure engine (pure TypeScript, event-sourced)

Vision, dashboard buttons, the hardware button and a simulator all emit typed events. The engine is a pure reducer over an event log, so the same events always produce the same state.
Every exposure record stores a provenance pointer to the record it came from. Records are marked cleared, never deleted. When an alert fires, we walk the pointers back to the source, check each link against the logged contact, and insert any wash that cleared an earlier step. That's how we can explain a risk that started several touches and a hand wash ago.

Dashboard (React 19)

Live video with an SVG zone overlay, status cards with exact wording rules, the alert chain, wash progress rings and tong-tracking badges.

Hardware (ESP32, Web Bluetooth)

An ESP32 DevKit runs Arduino/C++ firmware with a BLE GATT service: one characteristic receives commands (GT:ALERT, GT:ACKED, GT:IDLE), and another notifies button presses.

Challenges we ran into

Hover vs. touch from above. A single overhead camera can't tell tongs resting on the plate from tongs carried over it, and our first build often raised false alerts. Using the marker's apparent size as a height estimate fixed the carry without making real touches harder to catch.
The handle problem. On every normal scoop, the tong handle's hinge end looked like it was inside the peanut bowl, which exposed the handle and then the hands gripping it. We stopped the vision side from treating the handle as touching the bowl or plate, and kept the rule in the engine in case detection improves.
Small tongs, hidden markers. The arms stack in a grip, so a single marker was often face-down. We moved to one marker per arm face, calibrated each face separately, and never average the two, because the arms move relative to each other.
Jittery landmarks. Fingertip landmarks wobble by a few pixels, and a hand passing over a zone briefly looked like a touch. We tuned the dwell on the rig, from 400 ms to 800 ms and then 700 ms.

Accomplishments that we're proud of

The chain. Allos doesn't just say "risk". It shows why, back to the source.
Tracking a moving tool from one webcam, with printed markers, a homography and a height estimate, and no depth camera.
A full loop from camera to physical alarm: vision events → engine → dashboard → an ESP32 over Bluetooth.
Privacy by design: all inference runs on the device, and no video is uploaded.

What we learned

Practical computer vision on a webcam: homographies, marker dictionaries and error correction, parallax, and lens distortion.
Separating perception from reasoning made both better. Vision only reports "these two things overlapped long enough", and the engine never touches the camera, so we could build and test the logic before the rig existed.
How to write a spec as a team, amend it as the rig taught us things, and keep three people building against one contract.

What's next for update later

Multiple workers, utensils, allergens and orders at once, plus a recall view of every meal a contaminated tool touched.
Depth sensing or a second camera, to replace the marker-size height estimate and track hands in 3D.
Marker-free utensil tracking with a trained detector.
Integration with a kitchen order system, so allergy orders arrive automatically.