Tempo

A vision agent that reads what's happening inside the pots and conducts the order, so every dish on a ticket is ready together — built on OpenCV 5 and AWS Graviton.

Comment

Inspiration

Kitchens time everything that happens after water boils. Nothing tells them when it will boil — and that unknown, minutes that change with the burner, the pot and the water, is why the dishes of one table finish at different times and why pots are left alone until they boil over. We wanted a camera that answers one small question very well — "when will this pot boil, and is it about to overflow?" — and an agent that turns that answer into the right move for every cook at every station.

What it does

  • A camera above each burner turns the liquid surface into stage events (shimmer → simmer → rolling boil) and a foam-overflow forecast, using OpenCV 5 only.
  • A first-order heating model turns those events into a time-to-boil estimate; each burner's real power is learned as the service goes on.
  • An order agent splits every ticket into prep, burner and plate steps and re-plans all open tickets so each table's dishes finish together. When the camera changes an estimate, the plan changes — and the board says why.
  • The live demo runs a whole service with a camera clip on every burner (ramen, kimchi stew, egg soup, pasta); each clip drives the timeline of the pot it watches.
  • A kitchen twin (/sim) shows the simulated kitchen from above — cook, burners with LED rings, the four cameras, prep and pass — next to the screens it drives, all running on one clock.
  • Screens at every station: a prep board (what to cut, how much), a long burner-line strip (countdown, progress, LOAD / ADD / PLATE NOW), single-burner and pass screens, LED rings (simulated), and a whole-kitchen board.
  • When foam is about to overflow, Tempo lowers the heat. That is the only automatic action; everything else is a prompt a cook can ignore.
  • Cameras look at pots, never at people.

How we built it

  • Perception (OpenCV 5.0): pot opening via Canny + contours + fitEllipse (HoughCircles fallback); high-pass temporal energy and bubble births (connectedComponentsWithStats) for early stages; Farneback optical flow with camera-shake removal for the rolling boil; HSV + morphology for foam coverage and its trend; a monotone, persistence-filtered stage tracker relative to each pot's own cold baseline.
  • Physics: C·dT/dt = P − h(T − Tenv); stage times → time-to-boil with uncertainty; per-camera calibration from a few clips that run to a boil.
  • Agent: backward scheduling with burner constraints and hysteresis, event-triggered re-planning that never reads the future, keeping the cook free for predicted boils, foam → heat down; every decision traced.
  • AWS: CloudFormation → EC2 t4g (Graviton, arm64) running Docker; S3 (encrypted, expiring) for results and traces; SNS for safety alerts; CloudWatch Logs; least-privilege IAM; SSM instead of SSH; one-command deploy from CloudShell.
  • Evaluation: a kitchen simulator (4 policies × 120 services × 2 load levels, common random numbers) and a photo-realistic pot renderer with ground truth, because we wanted numbers we could check.

Challenges we ran into

  • Camera shake hid the boil on our first realistic clip — subtracting the median optical flow fixed it.
  • Compression shifts when a stage becomes visible; we made stages relative to each pot and calibrate per camera.
  • We found a leak in our own simulator (planners could read a pot's future boil time). Every result we report was produced after the fix, with a regression test.
  • Real stock clips taught us two things the simulator could not: a hand or ladle in the pot reads as boiling, and absolute motion thresholds depend on camera distance. We report both instead of tuning on those clips.
  • Honest trade-off: in a rush, finishing a table together costs about 14 % longer order-to-table time.

Accomplishments that we're proud of

  • Quiet service: −66 % time that cooked food waits; rush: −38 %; boil-overs from 7.5 to 0.2 per 100 dishes (simulator).
  • On photo-realistic simulated footage: rolling boil detected in 8/8 pots within 1.1 s; time-to-boil error 30 s ahead ±7.9 s vs ±34.5 s for recipe timing; foam overflow warned 13 s ahead in 4/4 with zero false alarms.
  • Sanity check on 8 real stock clips (no ground truth, judged by eye, nothing tuned): 5 correct, 1 ambiguous, 2 wrong (hand/ladle motion, phone close-up) — both now on the roadmap.
  • Screens that a cook can read from three metres away without reading a word.

What we learned

Most of the benefit comes from planning a table to finish together at all; the camera makes that plan robust and makes the kitchen safer. Detecting the boil already captures most of the scheduling value — the prediction earns its keep in countdowns people act on.

What's next for Tempo

Hand/utensil masking; validate on full cold-to-boil real pot footage next to the simulated results; a POS integration; a "speed first" mode for rushes; Cloud-Optimized OpenCV on Graviton for multi-camera kitchens; prompts projected onto the cutting board.

Disclosure

All camera results come from simulated footage generated by our renderer and labelled "Simulated footage" in every frame. Kitchen results come from a simulator with stated assumptions. The real-footage sanity check used free-licence Pixabay / Pexels clips, judged by eye. The demo video narration is an AI voice cloned from the author's own voice. The project was developed with the help of an AI coding assistant; it is not part of the submitted agent.

Links

Share this project:

Updates

Submission history