Butler[bot]

An affordable, smart home robot that can identify, pick up, and move around objects, all while letting the user control it from their phone using a joystick, text, or voice commands trained on an LLM.

Comment 2

Inspiration:

Most robotics these days are aimed at industrial-level facilities and warehouses; as a result, robots are still rarely affordable for the everyday person. Inspired by the question "what would we love to have a personal robot do for us?" and united by the answer "help us with our daily chores," we set out to create a personal assistant for everyday life. Using BracketBot's framework as our starting point, we set out to build a butler that's actually affordable, one that could take on the burden of personal life while giving the user time to focus on the things they love.

What it does:

Butler[bot] is a household robot that can spot objects, drive over to them, pick them up, and carry them where they need to go. But, we didn't want it to be a black box that only works when everything goes perfectly, so we built a companion phone app that puts you in the driver's seat whenever you want it.

You can:

  • Drive it around with a virtual joystick
  • Type it a command (Eg., Hey Butlerbot, move the yellow bottle into the basket!)
  • Talk to it
  • Switch between its cameras to see what it sees
  • Press the emergency shutoff if accidents occur

The result is a robot that can act on its own, but also one that you can take over instantly from your phone, laptop, or any web-connected device.

How we built it:

We split the project into four pieces...

  1. Computer vision — recognizing objects, planning a path, avoiding obstacles
  2. Scene reasoning — 3D environment reconstruction, LLM-based smart reasoning, human recognition
  3. Manipulation — object grasping, carrying, dropping, dexterous handling
  4. Remote control (Voice/Text/Joystick) — man-in-the-loop controls, voice-to-command reasoning, joystick-based direct movement.

Simulation and scene. Everything runs in MuJoCo. We started with main_mujoco handling arm IK, cube picking, and obstacle-aware navigation, then split vision work out into its own comp_vision_sim module so the two could evolve independently. From there we built out a proper environment: a furnished-apartment scene, real object meshes, a plant that actually looks like a plant, and eventually a randomized house generator so we could stress-test navigation and grasping across layouts we hadn't hand-built.

Computer vision. Object detection went through a few iterations: we trained our own YOLO net first, fixed channel-order bugs and shipped those weights, then switched to pretrained open-vocabulary YOLO-World so the robot could search for objects by name instead of being limited to a fixed label set. We also added an LLM-based scene-reasoning detector as a second, more flexible way to find a goal in 8-photo survey mode that gives the robot a full look around a room, and a detector LLM flag that lets the LLM pick out the target instead of a trained classifier. Later we sped the LLM path up and added re-searching for when the goal gets lost mid-task.

Navigation. The navigator went through several rounds of hardening: recovering from displacement instead of parking forever, re-seeding its internal clock when sim time jumped backwards or ran on every physics step, fixing it going blind after a sim reset, and widening its planning grid so it could reach farther goals, and reporting clearly when a plan simply wasn't possible instead of failing silently.

Manipulation. This lived in its own Hand_and_Wrists module: grasp planning and a pick skill for seven household items, finding items with the head camera, hardened floor picks, and later, "chores" behavior: putting things down, handing them over, and tidying up.

Remote control. We added a mobile BracketBot remote so a person could take over from their phone at any point, plus a natural-language search command (command, e.g., "find me a key") so the robot could be told what to look for in plain language rather than through a fixed API.

Integration. Once each piece worked on its own, the real work started: merging feature/manipulation, feature/llm-scene-reasoning, and the computer-vision branch into main, fixing the navigation and model bugs that only showed up once everything was running together, and building out seam tests (Seam C, Seam D) specifically to catch integration breaks between vision, reasoning, manipulation, and remote control, including making sure a failed search got handed off cleanly to the phone remote instead of the robot just stopping.

Testing and CI. In the final phases, we added a full CI pipeline: linting with ruff at a 120-character line length, tests running on Python 3.12 and 3.13, and an end-to-end smoke test with evals — plus a comprehensive test suite, tests that render and measure rather than just assert on guesses, and a pinned CLIP dependency so the YOLO tests would run consistently in CI. One known robot-mask failure got marked as a strict xfail rather than silently ignored, so it stays visible until it's actually fixed.

We kept these systems as decoupled as we could, so perception, reasoning, manipulation, and control could each keep working even if we were still tinkering with the others.

Challenges we ran into:

The real challenge was getting all components to agree on what the robot was doing at any given moment. Navigation, object recognition, arm movement, reasoning, and user commands all need a shared, up-to-date picture of the robot's state, and that took more effort than we expected. Additionally, simulating a robot fully in MuJoCo, including computer vision and object detection, was not easy, on top of integrating and merging all different aspects of this project. Merge conflicts caused us a LOT of headaches.

Accomplishments that we're proud of:

We're proud that Butler[bot] isn't just a simulation, and it isn't just a remote-controlled toy; it's both an autonomous robot and a fully manual one, in the same system. It can see, recognize objects, reason about what to do, move, manipulate things, take spoken or typed commands, and hand off control to a person the moment they want it.

The phone remote is probably what we're most proud of: joystick driving, arm and gripper control, live cameras, text and voice commands, an emergency stop, and safety behaviors that kick in automatically.

What we learned:

Building a robot taught us that the hard part is getting a handful of very different systems to trust and talk to each other. Vision, reasoning, networking, control, and manipulation each come with their own headaches, and integration ends up being the real engineering challenge. We also came away with a strong appreciation for fallback controls. No matter how good your autonomy gets, the person using the robot should always have a fast, obvious way to take the wheel.

What's next for Butler[bot]:

Our next goal is teaching Butler[bot] to understand looser, more natural requests and carry out longer tasks with less hand-holding. We also want to push forward on object detection, navigation, grasping accuracy, live camera feedback, voice interaction, and autonomous task planning.

The dream is a robot you can just ask, "Bring me the water bottle from the kitchen", and it figures out the rest: finds the bottle, navigates there, picks it up, brings it back, all while you can still grab the controls from your phone the second you want to.

Built With

Share this project:

Updates

Submission history