Out of the Box

Ducky
Piano
Car

Inspiration

LEGO sets are getting way too expensive, but almost everyone has had a big bucket of random LEGO pieces at some point. For a lot of kids, that pile is full of possibilities, but actually turning it into something cool can be hard. Sometimes you run out of the pieces you need, sometimes the build falls apart, and sometimes you just get stuck because you have no idea how to move forward.

That inspired Out of the Box: a tool that helps people turn the LEGO pieces they already own into creative, buildable models. Instead of needing a brand-new set, users can start building outside the box, straight out of the box.

What it does

Out of the Box scans and stores your LEGO inventory using computer vision. Once your pieces are recognized, the app suggests build ideas based on the shapes, colors, and variety of bricks you actually own.

The user can choose one of the suggested ideas or enter a custom prompt. From there, our AI generates a real working LEGO model using only pieces from the user’s inventory. The model is designed with realistic constraints: no floating pieces, everything connected, and every brick accounted for. Finally, the app generates step-by-step LEGO-style instructions so the user can physically build the model at home.

How we built it

For the computer vision pipeline, we built a multi-view LEGO scanning system using an iPhone's RGB camera, LiDAR depth, and camera tracking. As the user scans their pile, we detect the table plane and create a shared physical coordinate system across every camera view. From hundreds of captured frames, we automatically select a small set of complementary views that cover the workspace from useful angles.

We then use a custom-trained YOLO instance segmentation model to separate individual pieces, including same-colored pieces that are touching. Detections across different views are projected into the shared table coordinates and fused so the same physical brick is only counted once.

For identification, Brickognize proposes a part identity, which we cross-check using additional evidence such as physical footprint, visible stud count, and agreement across multiple views. Color is estimated independently across views using quality-weighted voting. The result is a structured inventory containing the part, color, and quantity of each detected piece, which becomes the constraint for the generation pipeline.

We built an agentic AI pipeline using Backboard and the Gemini API. The first agent analyzes the user’s inventory and turns it into possible LEGO ideas, considering factors like brick shape, color, quantity, and variety.

After that, we use a three-agent pipeline: the Planner, the Builder, and the Repairer.

The Planner takes either a suggested idea or custom prompt and turns it into a structured design plan, including key features and which types of bricks should be used.

The Builder converts that plan into an actual LEGO model, returning brick IDs and coordinate placements.

Then we render the model, generate an isometric image, collect any build errors, and send everything to the Repairer. The Repairer fixes structural issues and improves the model until it matches the original prompt as closely as possible.

Challenges we ran into

One of our biggest challenges was making the computer vision pipeline work on dense piles of LEGO, where pieces touch, overlap, occlude one another, cast shadows, and often look nearly identical. A single camera view was not enough to resolve these ambiguities, so we built a multi-view pipeline combining learned instance segmentation, physical workspace geometry, and cross-view fusion. This also required us to match detections of the same brick across angles, prevent duplicate counts, and identify pieces reliably despite changes in appearance between views.

Another challenge we faced was prompt engineering. We had to make sure the model could design and build recognizable LEGO objects without hallucinating pieces, creating floating bricks, or ignoring the user’s inventory.

At first, we tried getting the AI to do everything in one step, but the results were inconsistent. We solved this by breaking the process into multiple specialized agents, each responsible for one part of the workflow.

Another major challenge was token usage. Our prompts required a lot of context, including inventory data, design constraints, and model feedback. We used Backboard.io’s memory features to reduce repeated context, save tokens, and improve speed.

Accomplishments that we're proud of

We are proud of the quality of the generated LEGO models and how quickly the system can create build suggestions from a user’s inventory.

On the CV side, we are proud of building a system that goes beyond simply recognizing pieces from one image. Our pipeline reasons across multiple camera views, separates densely arranged pieces, reconciles repeated observations of the same physical brick, and combines multiple sources of evidence before committing to an identity.

We are also proud of building a true agentic workflow. Our Repairer agent does not just make a single correction; it loops through the model, checks errors, learns from previous mistakes, and keeps improving the build until it becomes structurally valid and matches the prompt as closely as possible.

This made our pipeline feel less like a one-shot generation system and more like an iterative building process.

What we learned

We learned a lot about prompt engineering, especially how to give an AI enough context to build something recognizable without overwhelming it or limiting its creativity. We also learned how memory systems like Backboard can improve speed, reduce token usage, and make multi-agent workflows more efficient.

What's next for Out of the Box

Next, we want users to be able to share their generated models so other people can build them too. We also want to improve the overall user experience, make scanning inventories smoother, and generate even better LEGO instructions.