TerraVision

Inspiration

Construction mistakes are expensive, and they almost always happen for the same reason: someone looks at a blueprint, looks at a messy job site, and just starts building. If a wall ends up framed a foot to the left, or the wrong materials go in, nobody notices until weeks later when it costs thousands of dollars to tear out and redo. We wanted to build something that catches those mistakes before they happen. A second set of eyes that looks at the plan, looks at the room, and tells the crew exactly where things stand.

What it does

TerraVision compares what should be built with what actually got built. You upload a blueprint or an IFC model along with your materials spec, take a photo of the job site, and get back a plain English report in seconds. Overall completion percentage, a live audit of what is done and what is missing, an issues log flagging errors down to the measurement, and a prioritized list of exactly what the crew needs to do next.

How we built it

The app runs a four stage pipeline. OpenCV reads the blueprint and converts every wall, opening, and dimension into 2D coordinate data. YOLOv8 scans the site photo and detects what is physically present in the frame. Our diff engine compares the two datasets and flags every gap and mismatch. Claude then takes those mismatches and turns them into a human readable action plan. Everything is wrapped in a Streamlit web app that works on any device on site.

Challenges we ran into

The sponsor provided first person camera footage from Ironsite workers on site, and our first instinct was to build around that. The problem was that the footage was shot from arbitrary angles and rarely captured the full context of what was being built, making it very hard to extract reliable spatial data. We made the call early to pivot to still photos paired with blueprints, which gave us a much cleaner input to work with. The other big challenge was the spatial problem itself: a blueprint is a perfect 2D bird's eye drawing and a site photo is a distorted 3D snapshot, and getting those two things to talk to each other accurately took a lot of iteration on our diff logic. We also quickly realized that finding real construction blueprints online to test with is surprisingly hard, they are not publicly available the way other datasets are, which made testing and validation harder than expected.

Accomplishments that we're proud of

We went from idea to working pipeline in the first day, which gave us time to actually refine the output rather than just scramble to finish. We are proud that the app accepts two completely different input formats, standard blueprint images and IFC models, the industry standard file format used by real construction firms, and handles both correctly. We also got a working AR prototype where structural and systems data from the IFC model is anchored directly onto the physical space. And we are proud that the app does not just flag problems, it explains them in plain language a worker can act on immediately.

What we learned

We learned that the hardest part of this problem is not detecting objects or reading blueprints, it is the translation layer between the two. A model can find a stud in a photo and OpenCV can find a line on a blueprint, but connecting those two things to understand actual construction progress requires a dedicated reasoning step. We also had to think hard about a problem that sounds simple but is not: how does the app know which part of the building it is even looking at? Are we in the kitchen or the living room? Which section of the blueprint applies? We designed a QR marker system to solve this, where physical markers placed around the site tell the app exactly which room and blueprint section the photo corresponds to. And we learned that real construction blueprints are surprisingly hard to find online, which is itself a sign of how closed off and analog this industry still is.

What's next for TerraVision

The immediate priority is implementing the QR marker anchoring system fully so room identification is automatic rather than manual. Beyond that we want to close the feedback loop with real time progress tracking so site managers are not just getting a snapshot but a running picture of how a project is trending against its timeline. Expanding the AR experience into a full on site tool where workers can see the digital blueprint anchored onto the physical space in front of them is where this is ultimately going.