Backyard Intelligence

First prompt to genration
Thinking animation when aksing for revision
Output of revision
Safety Report
Bill of Materials + Cost Estimate
full shed AR
shed inside AR
cutting tool AR

Backyard Intelligence

Backyard Intelligence turns a natural-language backyard project idea into a structured, inspectable build concept. A user can ask for something like a pergola, shed, or raised garden bed, and the system converts that request into a typed construction spec, deterministic 3D component model, verification report, bill of materials, and safety checklist. An AR companion app allows users to see the structure in real-life size, filter by specific materials and see how they fit together, and precisely fabricate parts using computer vision cutting hints.

Inspiration

We were interested in a version of AI for physical-world projects that does more than produce a nice-looking answer. Backyard construction is full of details that matter: dimensions, stock lengths, support paths, openings, fasteners, roofing layers, and material quantities. A language model is great at understanding messy human intent, but it should not be trusted to hallucinate geometry or a shopping list.

So we built a pipeline where AI interprets the request, but deterministic code owns the buildable representation. The combination of AI generation and the AR app provides the speed of vibe coding, the safety of a hired contractor, and the spatial intuition of building a lego set.

What it does

The core flow is:

natural language → AI-interpreted design spec → deterministic component generation → verification → BOM + GLB + AR handoff

Backyard Intelligence currently supports:

Pergolas
Sheds
Raised garden beds

For each generated design, the app produces:

A typed construction specification
A true-scale 3D model with named parts
A verification report for dimensions, connections, stock fit, openings, support paths, and overlaps
A bill of materials traced back to generated component IDs
A safety/building-guideline checklist
A downloadable GLB model
An AR handoff path for viewing and fabrication guidance on iOS

How we built it

The web app is built with React, TypeScript, Vite, Three.js, React Three Fiber, and Drei. Gemini is the primary AI interpreter, with Claude available as a fallback. The AI route lives in Vite middleware so API keys stay server-side.

The important architectural choice was separating interpretation from generation. The AI produces a structured DesignSpec, then deterministic TypeScript code validates the spec, generates every component, derives the BOM, verifies the assembly, and exports the GLB. The BOM is not guessed by the model; it is calculated from the generated parts.

The companion iOS app was built using SwiftUI, ARKit, RealityKit, Metal, OpenCV/ArUco marker tracking, and GLTFKit2. It can fetch the latest exported GLB and render placement/cutting guidance in AR.

Challenges

The hardest part was making the output inspectable instead of just visually plausible. We had to model real construction details like rafters, studs, siding, roof layers, gable/lean-to geometry, deck blocks, joists, openings, trim, fasteners, and stock-length segmentation.

Another challenge was verification. A model can look fine while still having disconnected parts, impossible stock lengths, invalid openings, floating geometry, or overlapping solids. We built checks for required part groups, finite dimensions, connection references, physical joint fit, support paths, roof bearing, BOM correspondence, and more.

The AR side also had its own challenges: true-scale unit conversion, loading GLB files on device, marker-based placement, LiDAR occlusion, and making the guide readable over a real camera feed.

What we learned

We learned that AI becomes much more useful for high-stakes physical domains when it is paired with deterministic systems. The language model should help translate intent, ask clarifying questions, and repair invalid specs, but the final geometry, quantities, verification, and exports should come from code.

We also learned how much value there is in traceability. When every visible part has a node ID, every BOM item traces back to components, and every verification check explains what passed or failed, the result becomes something a user can inspect instead of blindly trust.

What is next

Next, we would expand the supported structure library, improve permitting/local-code awareness, add more complete cost and store availability integrations, and tighten the AR workflow so users can move from generated concept to guided layout and cutting more smoothly.

Backyard Intelligence is not a permit package or structural certification tool. It is a prototype for a safer AI-assisted design workflow where human intent, deterministic generation, verification, and AR visualization work together.