Crafter

• 

  Successful ground pickup by the bot.

• 

  Building in Minecraft and scanning

• 

  Generated Schematic from Minecraft

• 

  Animated Schematic with live progress updates

• 

  Local perception reference showing boxes in view and remembered positions

Inspiration

Have you ever wished you could bring your Minecraft builds into the real world? We wanted to make that happen, with a robot doing the heavy lifting.

Meet Crafter: a robot that turns virtual block designs into physical cardboard structures. You build it in Minecraft. Crafter builds it in real life.

What it does

Crafter takes a block structure as a blueprint, locates physical objects, and plans how to assemble them. It chooses which pieces belong where, builds from the bottom up, and executes the necessary pick-and-place movements.

The player designs the result rather than controlling the robot’s arms. Crafter handles the steps between the virtual blueprint and its physical counterpart.

How we built it

We built Crafter around BracketBot, using Python modules for structure input, perception, planning, and execution.

We built a Minecraft mod that allows you to scan your builds with a click of a button, encoding them as machine readable blueprints. These blueprints are then interpreted by our central reasoning engine powered by OpenAI. The robot scans it's environment using camera and depth data locate nearby objects. The LLM based reasoning creates a plan to construct the blueprint from the available objects. It then builds the structure step by step via a tool calling loop which triggers the robot to carry out actions.

Challenges we ran into

Minecraft makes stacking look easy. Physical boxes introduce questions the game never asks: Where exactly should the gripper grab? Can the arm reach that position? Is the next block supported?

Bridging grid coordinates, camera detections, and arm movements required careful calibration.

Accomplishments that we're proud of

We turned a playful idea into a concrete robotics pipeline spanning perception, planning, and physical execution.

What we learned

The hardest part of embodied AI is connecting a reasonable plan to a messy physical world. Choosing where a box belongs is only useful if the robot can locate, grasp, and place it reliably.

We also learned how much clear interfaces matter under hackathon pressure. Shared data formats and explicit hardware ownership made parallel development more manageable.

What's next for Crafter

We plan on expanding Crafter beyond cardboard boxes to support a wider range of block types, colors, and textures. To make assembly faster and more reliable, we plan to implement closed-loop visual feedback so Crafter can monitor its own work, detect slipped or fallen blocks, and correct mistakes in real time. We hope to scale up so we can handle larger structures, turning Crafter into a true partner that builds right alongside you as you design in-game.

Built With

• bracketbot
• fabric
• python