LLM (Large Lego Model)

LegoLanguageModel logo
generated model of the Empire State Building
generated model of the Taj Mahal
generated model of the Morehead Bell Tower

Inspiration

As engineers and builders, we all have a secret–or not so secret–love for the creative institution that is LEGO. From Formula 1 cars, the Death Star, or a bouquet of flowers, LEGO allows us as users to create a fun and tangible project that engages our cognitive capabilities. Yet, have you ever wanted to build something that LEGO has not created yet? Our team sure has, especially our teammate Arsh who possesses an avid love of rockets and building his own rocket models. We bonded over this idea of unlimited LEGO fun to create a project that we are excited to share.

What it does

LargeLegoModel, or LLM for short, is an iMessage linked agent used as a method of receiving written and image prompts of objects, through iMessages, that users desire to build with existing LEGO blocks that they either already own or could own. The agent creates a 3D rendering of the desired object via text and visual data it sources from the internet and sends it back to the user via iMessages for confirmation. A user manual is then created upon approval that guides the user through a step-by-step build of their original LEGO, detailing the required blocks needed.

How we built it

We researched BrickLink Studio, Rebrickable, LDraw, and LegoGPT, then built our own architecture, Large LEGO Model. We started with a React and TypeScript website, using Three.js for 3D viewing and Node.js/Express for the backend, before connecting the workflow to iMessage.

From a text prompt or uploaded image, our backend retrieves photographs from different angles and written references. Gemini uses these to produce a JSON plan describing the model’s dimensions, colors, curved sections, openings, repeated features, and specialty parts.

Our compiler converts that plan into a grid measured in studs and plate heights, then selects and places real LEGO pieces. Rebrickable supplies part identities, produced colors, and inventories; LDraw supplies geometry; and LDCad provides supported connection information. Our code checks collisions, stud connections, and assembly order, repairs local arrangements, and sends larger corrections back to Gemini.

One saved brick list drives the preview, instructions, and exports. We also match required pieces against users’ confirmed collections, identify which sets supply them, and show shortages.

We integrated Photon Spectrum’s TypeScript SDK so users could send text and photos through iMessage and receive progress updates, previews, and files. Our agent tracks active builds and revisions, accepts feedback, and supports stopping generation. We tested USDZ attachments that users could open and rotate on their phones. Approval selects the exact previewed revision for delivery.

For assembly, we built a LEGO Builder–inspired mini-app designed to open from a Photon app card inside iMessage. It combines whole-model rotation and zoom with step-by-step piece callouts, orientations, animated placements, replay, repeated-pattern instructions, and saved progress. Users can check owned pieces and shortages, download instructions and models, or choose optional AR viewing.

The website, mini-app, previews, and manuals follow the hackathon’s Frutiger Aero/Y2K theme through sky blues, greenery, glossy controls, and dimensional lighting, while retaining accurate LEGO colors.

Challenges we ran into

We initially had our architecture source an understanding of the prompt strictly via relevant images that it found online. However, we eventually realized that randomly selecting only a handful of images representing the object makes it incredibly difficult to perceive a 3D understanding of the object at hand. A clever solution we found was to also source textual information and existing 3D models that provided details to fulfill this knowledge gap in addition to the images that it already found. This allowed us to maximize the agent's access to context such as geometry, dimensions, angles, etc in order to optimize its outputs.
Photon's API limits us to 100MB files. Our 3D files, which comprised of renderings of individual LEGO bricks stacked upon each other, were extremely large and over this limit. To solve this, we maintained from now on that the rendered model only strictly contains are strictly only on the peripheral of the model. As a supplement to that, we also made it such that they are not rendered as individual blocks but rather as larger, continuous bricks.

Accomplishments that we're proud of

A big focus for our tea, early on, was an in depth brainstorming session that spanned four hours. In this brainstorming session we threw out many ideas, but LLM is the one we are most proud of for its sheer creativity. We decided to tackle a really technically complex problem–essentially creating a vision model that builds 3D renderings from pictures and descriptions alone–in only 24 hours. By using AI for a majority of the code building, we freed ourselves up to be critical of our product, in a vigorous way, that we take pride in.

What we learned

HackNC was two of our teammates’ first hackathon. From their perspectives, they learnt a lot regarding the broader AI landscape, including what constitutes a frontier AI company, and general code and AI-derived vocab such as a wrapper. For the rest of the team, AI-aided problem solving and code building were big areas of growth. Persistence through mental blockers and lack of motivation were additional learnings for the entire group.

What's next for LLM

We believe that LLM has a strong foundation with plenty of room to grow further. Much of this room for growth comes from the issues we faced during this hackathon. One such example is the generation of complex shapes, such as flower petals. We found that generating these while maintaining structural integrity was both difficult and time-consuming, due to the impossibility of creating these shapes with purely rectangular pieces. The future of the LargeLegoModel is one in which it works naturally with the entire global catalog of LEGO pieces instead of the limited selection we had to work with due to the time constraints of this challenge.