Bracketbot

Inspiration

Everyone has a bad roommate story. Too messy, antisocial, weird. We decided we had enough of the passive aggressive sticky notes, and built Wingman, the roommate you wish that you had.

Built on Bracketbot's affordable and modular 3D-Printed hardware and BBOS, having a roommate has never been so frictionless.

What it does

Wingman is the roommate you wish you had, one who actually helps clean up.

It spots trash like empty cans, finds its way between tables, picks it up, and takes it to the trash can. Instead of leaving you with the mess, Wingman lends a hand. Literally.

But a good roommate isn’t just there for chores. You can dance with Wingman, play Shadow, trade rock-paper-scissors throws, or dap it up when the cleanup is done.

Wingman helps with the boring stuff and joins you for the fun stuff.

How we built it

Wingman builds on Bracketbot’s BBOS runtime, connecting perception, navigation, manipulation, and conversation.

Perception: YOLO-World detects objects such as cans. For pixel-level segmentation, we integrated alternative SAM 3 and YOLO-World + FastSAM backends. Synchronized stereo depth turns those image observations into 3D geometry.
Mapping, visualization, and navigation: We built on Bracketbot’s existing mapping and navigation stack. It turns stereo depth into 3D point clouds, uses visual SLAM to locate the robot, and produces a voxel map and traversability grid for Dijkstra path planning. Its interactive Three.js viewer displays the room geometry, robot position, waypoints, and planned routes. This infrastructure was provided by the Bracketbot team, not built from scratch during our hackathon. Our work focused on applying and extending that foundation for Wingman’s object-aware tasks, navigation experiments, and cleanup experience.
VLA manipulation: We fine-tuned π0.5 on recorded robot demonstrations to perform trash pickup, using camera observations and task instructions to guide the robot’s arm actions.
Conversation: A Discord interface uses OpenAI Codex to select supported actions through structured JSON. Application code validates requests and tracks execution, the language model does not directly generate motor commands.
Social interaction: YuNet face detection and MediaPipe palm detection support visual interactions, while scripted arm trajectories use quintic interpolation to give smooth motion replay via BBOS for smooth dance, dap, and game movements.

Challenges we ran into

Calibration is not optional, and it is not fun. Before Wingman could pick anything up, we had to calibrate the stereo cameras and get depth lined up with the robot's actual coordinate frames. A bounding box around a can means nothing if you don't know exactly where that can is relative to the gripper. We spent real hours here before we ever got to the "cool" parts.

Four of us, one robot, way too many agents. At some point we were all running coding agents on the same Jetson that was also trying to run SLAM and perception in real time. Turns out a robot's onboard compute is not built for four people's dev tools plus localization plus object detection all fighting for the same CPU/GPU. It crashed. More than once. We'd be mid-navigation and the whole thing would just fold because someone's agent decided now was a great time to load a model. Worktrees kept our code from colliding; they did nothing for the hardware.

Getting the pieces to actually agree with each other. Detection tells you "there's a can." Depth tells you "here's a point cloud." SLAM tells you "here's where the robot is." None of that automatically adds up to "the gripper should close at this exact point in space." Wiring those handoffs together, and making sure a stale detection or a moved base didn't quietly break a pickup, took way more care than the individual pieces did.

Accomplishments that we're proud of

We got Wingman to actually do the thing: find an object, drive to it, pick it up, and put it somewhere else. Physically. On real hardware. That's the bar we set for ourselves and did it!

We're proud that we built this on top of Bracketbot's mapping, SLAM, and navigation stack instead of reinventing it, and used that foundation to ship something that feels like a product idea, not a pile of robot demos duct-taped together. Wingman cleans up, but it also dances, plays Shadow, throws rock-paper-scissors, and daps you up. It has a personality, not just a task list.

We also started fine-tuning π0.5 on our own recorded pickup demonstrations, which is a real step toward a learned manipulation policy instead of scripted motion.

What we learned

The physical world does not care how good your code looks. A confident detection can still be a meaningless number if your calibration is off by a centimeter. A command that returns "success" doesn't mean the wheels actually turned, or the can actually got picked up. We learned to trust telemetry and visual verification over API responses.

We also learned that shared hardware is a shared resource, and treating a robot's Jetson like a personal dev laptop is how you get crashes mid-demo. Four agents plus SLAM plus perception is a great way to find out exactly how much headroom you don't have. :'(

Honestly, we learned that a robot that reliably does one small useful thing beats a robot that sort of does ten big things. Wingman picking up one can and putting it in the trash, every time, is worth more than five flashy features that only work when nobody's watching.

What's next for Wingman

Introduce more object types, messier rooms, failed-grasp recovery, and re-detecting things that moved since the last time we looked. We want success rates across many trials, not just a great run we happened to record.

We also want to get our dev tools off the robot entirely so nobody's coding agent ever competes with SLAM for GPU again. Lesson learned the hard way...

Ultimately, we want Wingman to feel less like a robot you're operating and more like a roommate you're asking a favor from. This means greater integration into the existing ecosystem people use, and cleaning up the user experience!