Kimchi Tacos

Inspiration

We wanted a robot arm you can talk to. You ask it for a colored ring out loud, and it finds the ring on the table, picks it up and drops it on the AMD sticker. We wanted the whole thing, voice, vision and arm control, to run on one AMD Ryzen AI MAX+ 395 PC, with no cloud.

What it does

You say a color (red, green, blue or purple) or press a button on a web page.
A camera above the table finds that ring anywhere inside a square marked with gray tape.
An SO-101 arm picks up the ring, drops it on the AMD sticker, goes back to rest and waits for the next command.
Before it carries the ring away, the arm checks the grab twice:
- By feel: the jaws must stop on the ring, not close on air.
- By camera: the ring must be gone from where it was.
- If either check fails, it finds the ring again and tries once more.
Bonus NBA Champion Picker page: Heat = red, Celtics = green, 76ers = blue, Lakers = purple. The robot "predicts" the champion by picking up a random ring.

How we built it

Hardware: SO-101 leader/follower arms (Feetech STS3215 servos), an overhead USB camera and a wrist camera, all on an AMD Ryzen AI MAX+ 395 running Ubuntu.
Arm control: Hugging Face LeRobot, using PyTorch on ROCm.
Voice: faster-whisper speech-to-text, running locally, wired to a small web server with color buttons and a live log.
Vision: OpenCV color detection that uses hue, so it still works in sunlight. It also locks the camera exposure, ignores the sticker, and treats the gray tape as the edge of the work area.
Learning from demonstration: we drove the leader arm by hand to show the robot dozens of pick-ups. For a new ring position, the robot blends the poses from the nearest successful demos with an earlier grid of taught positions.
Camera-shift correction: if someone bumps the camera, we click the 4 tape corners again. The system then converts what it sees into the camera view it was trained in, so none of the training has to be redone.

Challenges we ran into

Sunlight and color: strong sun washed out the colors and made red look orange. We switched to hue-based detection and dropped the colors that were hard to tell apart.
The camera kept getting bumped. After a bump, every trained position pointed at the wrong spot. We fixed this with tape-corner calibration and a conversion back to the training view.
Fooled grip check: we found an empty gripper physically can't close past a certain point, so it looked like it was holding something when it wasn't. Measuring that limit fixed the check.
Height and reach: going lower to reach the ring also pushed the arm forward, so we changed the motion to go more straight down.
Motor overload faults and flaky Wi-Fi/SSH during a live hackathon.

Accomplishments that we're proud of

Full voice-to-pick-and-place running locally on one AMD PC.
A robot that checks its own grab and retries when it misses.
Training by demonstration with our own teleop recordings, not hand-coded positions.
Vision that works in sunlight, plus a fun NBA demo.

What we learned

Calibration is everything in real-world robotics. A few pixels of camera drift is the difference between a grab and a miss.
Simple checks that use more than one signal (motor position plus the camera) catch failures a single check misses.
Learning from demonstration works best when you keep only the clean, successful demos.