Brick by Brick

LEGO bricks are the building blocks of our childhood. Brick by Brick lets you relive this past & reconstruct your future. Step into our generation playground to build moments defining your identity.

Comment 3

Inspiration

While building the LEGO mascot during the opening ceremonies, we were reminded of a simpler time, before university and internship applications. These moments were pivotal in forming the identities that we would eventually adopt, so we wanted to build something to rekindle some of the nostalgia.

What it does

Brick by Brick is a dynamic LEGO generation playground that transforms your environment into buildable sets, along with providing an instruction manual composed of the LEGO pieces needed (fetched from Rebrickable API) and the instructions to build it. Powered by Twelve Labs’ video understanding and Gemini 3 Pro’s spatial reasoning, the platform maps your physical world into a digital voxel grid. Start by capturing a video of your space or specific objects; our pipeline handles the complex geometry conversion and part-mapping in real-time. By leveraging Backboard AI’s stateful memory, the system switches between models based on the complexity of the conversion, using lighter models for standard structures and scaling up to more powerful reasoning engines for intricate sub-assemblies (more details in how we built it).

How we built it

  • Twelve Labs (Pegasus & Marengo): Leverages the Pegasus model to analyze multiple video frames and "photos" from different angles to extract deep semantic data on room layout and textures. Marengo provides the underlying video-to-text search and rapid feature extraction (descriptions).
  • Vooitrex Pixel System: A custom voxelization engine that quantizes 3D environments into a $0.1m$ "logical pixel" grid, creating a unified coordinate system for LEGO conversion.
  • Backboard AI Orchestrator: Acts as the "Master Builder" brain, managing stateful build sessions and routing tasks between high-reasoning architectural phases and high-speed bulk placement.
  • Greedy Optimization Algorithm: A backend logic engine that prioritizes the largest possible bricks from the Rebrickable database to minimize part counts while ensuring structural integrity via interlocking joints.
  • Gemini (Pro & Flash): Generates architectural logic for complex shapes (like slopes and curves) and transforms raw manifests into polished, LEGO-style instruction manuals with tips and visual guides.
  • Solana Blockchain: Utilizes the Memory BB Coin system to store project metadata, including piece counts and shows up directly on-chain via Solana Memo for permanent, decentralized build history. Also includes links to buy the LEGO!

Challenges we ran into

The primary hurdle was the "Perfect Fit" problem. Our algorithm had to map irregular, scanned 3D volumes to a rigid, discrete LEGO grid. We struggled with the Backboard Model's initial tendency to "over-brick," where it would place too many small 1x1 plates, leading to massive performance lag and a cluttered aesthetic.

Accomplishments that we're proud of

  • Multi-Angle Video Analysis: We successfully utilized Twelve labs to process complex video data, extracting precise spatial measurements and descriptive metadata from multiple perspectives
  • The Backboard Piece-Selection Algorithm: We developed a proprietary algorithm for Backboard.io that maps raw spatial data against the Rebrickable database. The system intelligently analyzes volumes to identify the most efficient LEGO bricks, prioritizing large-format pieces to minimize complexity while maintaining high-fidelity detail.
  • Web3 Integration with Solana: We successfully integrated the Memory BB Coin on the Solana blockchain, enabling a decentralized layer for our ecosystem and ensuring fast, low-cost transactions for our users.

What we learned

We learned about the Vooitrex Pixel System, which teaches us how to map 3D "pixels" to standard LEGO units. We also learned how to optimize combinatorial selection; the algorithm doesn't just find a part that fits, it keeps in mind the optimization and physics to support the build.

What's next for Brick by Brick

The time for generation is still very slow; we would like to speed it up, possibly with LPUs and by reducing the complexity of the algorithms. We can also connect to the database better, the Rebrickable database is hug,e and we are only connecting to around 10% of it to shorten wait times and computing power.

Built With

Share this project:

Updates