Inspiration
Last month, one of our team members
What it does
- You start the food fetching process with your voice (or through a web dashboard for administrators)
- The arm moves towards the food, grabs it, and moves towards your mouth, where you can lean in and eat it
- If it missed the goal at any point (missed the food, crushed it, etc), the same model receives multiple error metrics and retries the action
- Context is saved persistently, so any errors regarding specific food will never repeat
How we built it
We used the SO-101 robot arm, along with a harness-like loop:
- First, we calibrate by moving the arm around in a fixed grid and taking camera captures. This lets us build an inverse mapping from each camera input's pixels to the arm's internal coordinates.
- Next, we use the pixel -> coord mappings to move the arm to the target food item, whose bounding box is drawn by a vision model
- If it misses, we retry with multiple error metrics that the VLM learns from
We use MediaPipe to detect when someone's mouth is open, Vosk for listening to speech and faster-whisper for speech to text.
We also have a web dashboard for monitoring the cameras, annotating all the metrics and predictions.
Challenges we ran into
- The servos are rated for 6 to 7.4 V and our supply gives 5 V. They sometimes cut out at the start of a move.
- Calibration was off by several centimeters because the tape position on the gripper was a guess. Calibration now measures it.
- The vision model often returned boxes on a different donut hole in each camera view.
- Food more than about 30 cm from the base is out of reach at table height.
- The Gemini API returned high demand errors during the event, so we used Claude.
Accomplishments that we're proud of
- The arm picked up a donut hole on its own, using a plan from the vision model.
- After a miss, the next plan changes based on the measured error.
- [Add: success rate over how many tries]
- The
What we learned
Most of our problems were physical: power, calibration and table height. The vision model only returns a food box and a list of fixed moves, so every motion goes through code we wrote and could debug.
What's next for Byte2Bite
- Test a full delivery to a person's mouth.
- Use a 7.4 V power supply.
- Mount both cameras at about 45 degrees from two sides.
- Add simple gestures, like waving or going home, for the vision model to choose from.
