The Brackey Way

Inspiration

After our first week of classes as university freshmen, we found that this program often left us inadequate time for maintaining a routine three meals a day. We realized a lot of other students were dealing with the same problem: skipping meals, grabbing whatever was fastest, or eating at odd hours. That's what led us to build The Brackey Way, a way for students to enjoy a healthy meal during a busy day.

What it does

The Brackey Way is a fully autonomous sandwich-assembly robot built on a BracketBot. The robot runs makes an entire sandwich-on its own, picking up a toasted pieces of bread from the toaster, placing it onto a plate, adding a piece of lettuce and then finishing by adding a second slice of bread to close the sandwich.

How we built it

For complex, high-precision actions like grabbing bread from the toaster, we fine-tuned a pi0 Vision-Language-Action (VLA) model on 100+ demonstration episodes, collected by manually teleoperating the Bracket Bot's arm via a Meta Quest to grasp bread slices and place them on a plate. To make the model robust to real-world variation, we recorded these episodes across a range of toaster, plate, and bread orientations, allowing it to generalize to slightly different setups rather than a single fixed configuration.

For simpler, well-defined actions like grasping a leaf of lettuce, we opted for a CV pipeline instead of a learned model. Using the head camera, we apply OpenCV to generate a binary mask based on pre-calibrated HSV thresholds tuned to the lettuce's green hue, isolating it from the rest of the scene. We then compute the centroid of the masked region and use it as the target goal pose for the gripper. As the arm rises toward that target, we run closed-loop correction using the wrist-mounted camera, continuously fine-tuning the gripper's angle and position for a precise, reliable grasp.

To tie all of these individual skills together into one coherent sandwich-making process, we built a state machine that sequences each action in order. Each stage of the pipeline is treated as a discrete state, with the script triggering the appropriate action depending on the task at hand as determined by what is missing in the sandwich through the camera. Once a state completes successfully, the state machine transitions to the next step automatically, with no human intervention required.

Challenges we ran into

We hit two major infrastructure bottlenecks: our WiFi connection struggled to keep up with uploading 100+ recorded teleop trials, leading to interruption of multiple uploads and since our GPU for training/inference was hosted remotely, we dealt with significantly slow round-trip latency that made the robot's movements sluggish and iteration cycles frustratingly long.

Accomplishments that we're proud of

Given the time constraints, we're most proud of getting our fine-tuned pi0 VLA model consistently. Going from zero experience with VLA models to collecting our own teleoperated dataset, fine-tuning the model, and deploying it on real hardware within this timeframe is a huge accomplishment for our team.

What we learned

We learned that building with real robotics hardware surfaces problems no amount of planning can fully anticipate, infrastructure constraints like WiFi and GPU latency can matter just as much as the ML or mechanical design itself. We also learned the value of optimizing our time for more complex tasks. A fine-tuned VLA model was worth the investment for high-variability actions like grabbing bread from a toaster, while a simpler CV approach was faster and more reliable for well-defined tasks like picking up lettuce. We also learned techniques in teleoperated collection that result in more adaptive, more consistent models. Adding environmental variance between each episode, such as moving around the locations of the toaster and plate, increases the depth of the robot's learning, and results in a model that can perform in previously unseen situations.

What's next for The Brackey Way

Next, we want to expand our sandwich menu with more ingredients beyond bread and lettuce. We're also aiming to give the robot the ability to operate the toaster itself and make the entire sandwich-making pipeline end-to-end autonomous. Finally, we want to extend the robot's capabilities beyond the kitchen counter so it can navigate to and deliver the finished sandwich directly to the person who ordered it, turning The Brackey Way into a true start-to-finish meal service.