SafeServe Devpost Submission

Inspiration

We wanted to build something that was both technically challenging and practically useful. After discovering a pretrained imitation learning model for robot manipulation, we realized we could build on existing capabilities instead of starting from scratch. That made it possible to tackle a more ambitious problem within the limited time of a hackathon.

Our goal was to teach a robot to perform food preparation tasks that can transfer to similar kitchen workflows. We focused on actions such as pouring oats and then extending the learned behavior to pouring water, demonstrating that robots can efficiently acquire new skills by building on previously learned knowledge rather than relearning everything from the beginning.

We were inspired by existing imitation learning success demonstrations and graph-based workflow pipelines that show how complex robotic behaviors can be composed from simpler learned skills.

What it does

SafeServe teaches a robotic arm to perform food preparation tasks through imitation learning.

The robot first learns how to pick up and pour oats from human demonstrations. Instead of training a completely new model for the next task, we fine-tune the existing model to pour water as well. Because the model has already learned the environment, grasping behavior, and robot motion, it requires significantly fewer demonstrations to learn the new task.

This approach shows how robotic assistants can continually learn new kitchen skills while reusing knowledge from previous tasks.

How we built it

We began by configuring the robot and understanding its software stack before collecting demonstrations.

Our workflow consisted of:

Configuring the robot and calibrating its environment.
Recording approximately 50 demonstrations of pouring oats.
Training an imitation learning policy using the collected demonstrations.
Extending the policy by collecting only about 30 demonstrations for pouring water.
Fine-tuning the previously trained oats model instead of training from scratch, allowing the robot to leverage its existing understanding of the environment and manipulation tasks.

By reusing the pretrained model, we reduced the amount of data needed while demonstrating effective transfer learning between similar manipulation tasks.

Challenges we ran into

The majority of our challenges came from working with unfamiliar robotic hardware and infrastructure rather than the machine learning itself.

Some of the biggest obstacles included:

Understanding how the robot was assembled and why various software libraries were installed.
Hardware safety limits that restricted the robot's yaw angle.
Intermittent Wi-Fi connectivity throughout development.
File upload limitations that prevented us from using the normal workflow.
Long delays during demonstration collection—waiting nearly two minutes between recordings, often longer than the demonstrations themselves.
Ethernet restrictions that prevented SSH access to the robot.

Despite these issues, we adapted quickly. One teammate focused on reading the robot documentation so we could get the system running within the first hour of the event. We modified manufacturer configuration values where necessary to work around motion limits, relied on our own collected datasets, used personal accounts to transfer files when standard uploads failed, and simply worked through the unavoidable delays in the data collection pipeline.

Accomplishments that we're proud of

We're proud that we successfully trained a robot to learn a real manipulation task through imitation learning and then demonstrated transfer learning by extending the model to a second task with substantially fewer demonstrations.

Beyond the machine learning results, we're also proud of how quickly we became productive with unfamiliar robotic hardware. We went from having little knowledge of the platform to collecting demonstrations, training models, and running successful experiments within a single hackathon.

What we learned

This project reinforced how powerful imitation learning can be when combined with pretrained models. Instead of collecting massive datasets for every new task, existing manipulation knowledge can be reused to significantly reduce training time and data requirements.

We also learned that robotics projects are often constrained as much by hardware setup, networking, and deployment infrastructure as by the machine learning algorithms themselves. Careful system engineering was just as important as model development.

What's next for SafeServe

Our next step is to expand SafeServe beyond two tasks into a growing library of transferable kitchen skills.

Future work includes:

Teaching additional food preparation tasks such as stirring, scooping, and ingredient placement.
Reducing demonstration collection time through a more efficient recording pipeline.
Improving deployment and networking workflows for faster iteration.
Investigating continual learning techniques that allow the robot to acquire new skills without forgetting previously learned behaviors.
Building a complete autonomous food preparation workflow by combining multiple learned manipulation policies into a single system.