BracketButler

Team photo with the BracketButler.
Custom website to visualize all metrics of the robot remotely.
Environment setup for demo.
Training the position for the arms to pick up the tray from the chair.
More training for moving the arm to the correct positions during the demo.

Inspiration

Having had experience in high school working in restaurants, we know firsthand how exhausting and repetitive carrying loaded trays across a busy dining room can be. This is exactly the kind of routine task that should be automated; yet, commercial service robots remain expensive. Transitioning from the restaurant floor to studying Engineering Science and hands-on hardware development, we realized the solution to affordable automation often lies in solid mechanical design rather than overpriced computing. Using the low-cost BracketBot platform, which allows developers to write code straight in the application layer, we set out to build an autonomous robotic waiter. Our goal was to prove that a small team with accessible hardware could build a reliable solution to support these manual serving tasks. We wanted to show that affordable robotics is finally ready to fundamentally transform traditional restaurant operations.

What it does

At the core of our system is the tray perception and carrying algorithm. Using a custom docking station and tray designed for the BracketBot, we are able to scoop the tray using the robots’ two arms and carry a delicious breakfast to the user. One standout feature is the adaptability of the task. Currently, using ArUco codes for the docking station, tray bottom, user position, and more, we are able to achieve pseudo-depth and variable object position data. This means our algorithm will be able to work with different tray heights, tray positions, and user positions. In addition, the range of the BracketBot’s base and arms also allow it to work with different end users, such as customers in other rooms, or of different heights. Another feature is the options for accessibility. We added a lightweight voice model to the BracketBot, ensuring easy customer interaction using commands like “I want my food!” and “Thank you!” We designed this specifically because customers might not want to touch the robot directly, or it might be cluttered in a restaurant setting.

How we built it

The robot needs to find a tray and a person without us hand-measuring exact spots every time, so we stuck ArUco tags on the pickup chair and gave one to the person receiving delivery. The head camera (fisheye, calibrated once with a checkerboard) picks up a tag, solves its pose with OpenCV's PnP, and a small visual-servo loop drives the base to a set distance from it, turning to face it and driving forward while centered.

Because the tag makes the parking spot repeatable, we use mimic once the base arrives to move both arms to one hardcoded joint pose tuned by hand for that exact spot. The wrist cameras confirm both grippers are actually on the tray's markers before it lifts. The robot then picks up the tray, backs off, searches and drives to the dropoff tag, and waits for a spoken "thank you". These voice commands are captured on a laptop mic, transcribed locally with Whisper, and sent to the robot over HTTP.

Challenges we ran into

After a long night of debugging, the two biggest challenges that we faced were item-level manipulation and environmental calibration. Our original scope included the robot autonomously picking up individual items from a table and placing them onto the tray. However, translating the vision data (ArUco marker poses) into precise end-effector trajectories was complex under the strict time constraints. Adding on, calculating perfect, collision-free paths to maneuver objects onto the tray without hitting items that were already in the way required a level of dynamic path-planning of the arms that exceeded our timeline, leading us to move to a pre-loaded tray approach. Furthermore, we underestimated the difficulty of standardizing the physical testing environment across trials. Fine-tuning the exact placement of the ArUco markers, designing the custom tray to perfect spatial tolerances, and calibrating the robot's reach to the specific height of the target platform required lots of trial and error. Navigating these interacting variables reinforced our biggest takeaway from the build: when faced with a complex spatial or control issue, redesigning the physical setup or modifying the hardware is often a far more efficient and reliable solution than attempting to write convoluted code to compensate for physical inconsistencies.

Accomplishments that we're proud of

Drawing on our background in Engineering Science, we addressed the inherent challenge of reliable robotic manipulation by optimizing the physical interface itself. Rather than developing error-prone algorithms for gripping arbitrary objects, we designed and engineered a standardized tray adapted specifically to our end effectors. This hardware-first approach gave a highly tolerant, easily graspable target that significantly increased the reliability of the pick-and-place mechanics. By simplifying the physical interaction, we successfully streamlined the grasping process, which allowed us to dedicate our limited timeline entirely toward perfecting the broader autonomy layer. Transitioning from raw hardware components to a fully autonomous, closed-loop system within a single hackathon timeline was complex, and achieving this was our most significant accomplishment. We successfully implemented a robust, multi-stage finite state machine that operates entirely independently. Seeing BracketButler successfully navigate the environment, visually identify its targets, grasp the custom tray, and deliver the tray with zero teleoperation or human intervention was the most rewarding part of the hackathon and it demonstrated the viability and stability of our integrated architecture.

What we learned

We gained invaluable experience building on top of an established hardware and software architecture. Rather than starting from scratch, we had to rapidly understand the baseline BracketBot framework and seamlessly interface our finite state machine and autonomy logic with its existing control systems. This process taught us how to effectively read, adapt, and extend a pre-existing codebase under strict time constraints without breaking the underlying functionality. We learned the power of using markers (ArUco/AprilTags) for robotic perception. Instead of relying solely on inaccurate dead-reckoning or attempting to train machine learning models, using these markers gave our three-camera system highly reliable 3D perception right away.

What's next for BracketButler

Since our current architecture relies on a custom tray to ensure a reliable grasp, our immediate next step for BracketButler is implementing advanced object recognition and dynamic grasp planning. This next step would allow the robot to autonomously identify, select, and handle a varied array of open-set items, such as distinct cups, plates, or utensils, directly from a serving station to the table. Additionally, to successfully deploy BracketButler in an unpredictable restaurant environment, we plan to significantly upgrade our navigation stack. We aim to transition from our current method of visual marker navigation to a full dynamic mapping system, such as SLAM. By integrating active spatial awareness, BracketButler will be able to map its environment in real-time, detect moving obstacles like staff or patrons, and autonomously recalculate safe, collision-free paths across the dining floor, ensuring safe and robust operation alongside human workers.