Bagel-Bot

Inspiration

Ever since we were served bagels for breakfast at 9am on Hackathon Day 1, we have always dreamed of having robots make us bagel sandwiches.

BagelBot Presentation

What it does

Bagelbot is trained to execute two action sequences:

Pick up bagel, put it in the toaster, activate the toaster
Take nutella knife out of nutella, spread it on the bagel in the pan, put the knife back

It can do task #1 with about 65% reliability and task #2 with about 10% reliability!

How we built it

The hackathon generously provided us with a reBot arm B601, a leader arm, a Jetson orin nano for inference and driving the robot with LeRobot preconfigured, and a workspace camera, making imitation learning easy. We focused our energies on collecting demonstrations, designing the mechanical task setup, and training/testing a few different model typologies (9 ACT policies, 1 diffusion policy, 2 smolVLA policies).

In the end, our most successful policies were ACT policies. You can see our GitHub repo here

For the toaster task, we collected 95 demonstrations across two separate days, with slightly different workspace setups across the two days.

For the nutella spreading task, we collected 55 demonstrations over one day.

Challenges we ran into

The knife is hard to hold onto with a parallel, non-force-controlled gripper - and especially hard to hold onto at a consistent angle. We 3D printed a knife handle which improved this problem, but did not solve it Bagels and bread degrade over time (especially when you’re repeatedly picking them up and putting them down!)
Bagels are more durable than bread, so we went with bagels -- The dirty bagel problem – for nutella spreading it would be ideal if all training data starts from a clean bagel. Alas, we did not have infinite bagels, so most of our training data involved spreading nutella over and over again on the same bagel. -- We initially started the spreading task wanting to use jelly or cream cheese, but these had difficulty sticking to our knife. To simplify this task we switched to nutella as it was easier to stick to the knife.
We initially wanted the spreading task to contain three swipes to try to cover more of the bagel. This was a difficult task to teleoperate reliably, so to improve our chances at a good demonstration we reduced this to one motion. This task is still pretty difficult to do as it is very dependent on if you have a good grip on the knife.
Our cameras and setup got moved around over the course of the weekend
Because models are not easily interrogable, when a policy failed, we couldn’t easily determine whether it was due to the setup being out of distribution, a bug in deployment, or the task simply being hard
We trained a diffusion model and a SmolVLA model, but were unable to successfully deploy these. The diffusion model had issues with jerky and slow execution and the VLA had issues running out of memory.

Accomplishments that we're proud of

Sai is an incredible teleoperator
We were able to adapt our setup when we ran into roadblocks (e.g. switching from bread to bagels, and jam to nutella) as we tried a variety of approaches early on
We trained three different kinds of models - ACT models, diffusion models, and a SmolVLA model
Our ACT model sometimes exhibited “self-correcting” behavior - it would try to press the toaster lever multiple times if it failed the first time - without any explicit demonstration
The model was robust to the bagel being placed in a few different ways - not infinitely robust, but at least somewhat!
We built in a safety feature to slowly return the robot to home when a button is pressed during policy execution. This was motivated by there not being a clean way to shut down the robot (the default behavior of hitting the estop or CTRL+C would both drop the robot aggressively).

What we learned

Whether a robotics task is easy or hard is extremely dependent on mechanicals - gripper design, orientation, etc. Picking up an open-faced bagel is much easier than a closed-face bagel! Spreading nutella is much easier when the bagel is elevated over the table surface! Etc.
Another thing that matters is chunk size and inference schedule - even with the same model outputting 100 action steps, different policies were more and less effective. For instance, we tried a temporal ensemble averaging approach, but found it was super jittery. What worked better was only using the first 90 outputted steps and then re-inferencing.
Most robots drop most objects most of the time
Mark down or bolt prop and camera locations on the table so they don’t move
Blood is thicker than water, but nutella is thicker than jelly

What's next for Bagelbot

If we had more time, we would have loved to train a third policy that moved the bagel from the toaster onto the plate for nutella-ing. This would make the whole demo continuous.
A better gripper adapter for the knife might have resulted in better grasps for spreading