Hello

Inspiration Commercial buildings are complex, living systems where structural integrity, mechanical equipment, and operational leases are deeply interdependent. Yet today, building management operates in silos: facility managers look at 2D floor plans, technicians sift through scattered sensor dashboards, and engineers use isolated calculation spreadsheets. When critical equipment degrades—like a vibrating chiller causing micro-fractures in a load-bearing wall—traditional workflows fail to catch the cross-system ripple effect until catastrophic downtime occurs. We were inspired to build ForeSite (architecturally powered by Cascade) to bridge the gap between physical spaces, real-time telemetry, and structural engineering. We asked: What if you could physically step inside an interactive digital twin of a building, accompanied by an autonomous team of specialized AI agents that reason about structural physics, track load paths, and simulate cascading failures before they happen? What it does ForeSite is a walk-in, mixed-reality digital twin that actively reasons about buildings. Wearing a Meta Quest headset, facility managers and field technicians step inside a full-scale or tabletop model of their facility to collaborate with an autonomous team of eight NVIDIA Nemotron agents. Key capabilities include: Full-Scale & Tabletop Mixed Reality: Walk through facilities at 1:1 scale with dynamic thermal mapping and visual inspection cues (cracks, leaks, and pulsing alarms), or shrink the building to an interactive tabletop model in passthrough. X-Ray & Structural Health Modes: Toggle X-ray vision to inspect interior conduits and pipes, or switch to Structural View to see color-coded load-bearing walls based on real-time stress and capacity utilization. Multi-Agent Blackboard Reasoning: A team of eight specialized agents (Structural Engineer, Signal Analyst, Lease Analyst, Simulation Engineer, Maintenance Historian, Planner, Critic, and Narrator) autonomously investigate anomalies, debate hypotheses, verify load paths, and propose actionable fixes. Structural Engineering Engine: Automatically executes tributary load analysis, evaluates crack degradation, checks building codes, and calculates beam-sizing formulas (e.g., deflection limits under L/360) to provide concrete verdicts (go, go with beam, not feasible). Cascade Outage Simulation: Runs 24-hour forward-looking Monte Carlo simulations to model how single mechanical failures propagate across air handlers, zones, and tenant leases, calculating financial risk clouds and demonstrating ROI before and after mitigation (e.g., reducing expected losses from $244k to $4.8k). Role-Based Collaboration & On-Device Voice: Technicians and managers share spatial presence with synchronized viewpoints. On-device voice transcription (whisper.cpp) lets users command the twin hands-free, receiving role-tailored spoken briefings. How we built it ForeSite's architecture pairs an immersive frontend with an unified, database-native backend: Spatial Client (Frontend): Developed in Unity 6 using the OpenXR standard, Meta OpenXR composition layers, XR Interaction Toolkit, and XR Hands. It features custom unlit shaders, runtime-generated geometry for dynamic stress mapping, and world-space UI. Database as the Operating System (Backend): Built entirely inside SpacetimeDB 2.14 (TypeScript module hosted on Maincloud). Rather than maintaining separate agent orchestrators, cache layers, and calculation servers, the structural engine, Monte Carlo simulations, blackboard tables, and agent loops execute directly inside database reducers and procedures next to the data. Knowledge Graph Generation: Ingests 2D architectural floor plans and structural framing sheets via POST /blueprint, compiling walls, bays, point loads, and footings into an ACID-compliant knowledge graph connected by relational edges (supports, carries, bounds, separates_fire). NVIDIA Nemotron Agent Collective: Powered by NVIDIA Nemotron 3.5 Super (120B) in reasoning mode for high-level planning, structural critique, and hypothesis evaluation, alongside Nemotron 3 Nano for low-latency specialist tasks. Tool calls execute natively via SpacetimeDB procedures (ctx.http.fetch) writing directly to blackboard tables. Edge Speech Processing: Integrated whisper.unity (whisper.cpp running the base.en model) directly on the Meta Quest headset for zero-latency, private, offline speech-to-text, paired with Android TextToSpeech for spoken debriefs. Challenges we ran into Eliminating Agent Hallucinations in Engineering Contexts: Structural decisions cannot tolerate hallucinated component IDs, false capacities, or fabricated material costs. We enforced strict runtime grounding: every claim posted to the database blackboard must cite verified database row IDs, and all sizing/cost metrics are computed deterministically by the structural engine rather than estimated by LLMs. Synchronizing Spatial Mixed Reality in Real Time: Maintaining low-latency state synchronization across multiple headsets without visual jitter was critical. By leveraging SpacetimeDB's real-time live subscriptions, we pushed state changes directly from ACID reducers to connected headsets with a median latency of ~38 ms (p95: 48 ms). Running Heavy Simulations at Interactive Speeds: Simulating 24-hour cascading thermodynamic and electrical failures across 10-minute increments across 50 Monte Carlo runs is computationally intensive. We wrote deterministic reducers utilizing seeded pseudo-random number generation directly in the database engine, executing ~1,700 full 24-hour simulations per second. Agent Deadlocks & Latency: Multi-agent chains often suffer from sequential bottlenecks. We designed an asynchronous blackboard architecture where specialists act in parallel from pre-fetched briefings, backed by deterministic fallback procedures to ensure investigations conclude even if a model call times out. Accomplishments that we're proud of 100% Grounded Tool Calling: Achieved 15/15 on our comprehensive structural evaluation benchmark with zero hallucinated IDs and zero rejected tool calls across 133 executed Nemotron calls (median latency: 1.2 s). Blueprint-Aware Structural Superiority: Our framing-aware engine correctly identified 8 out of 8 load-bearing walls, whereas traditional floor-plan-only baselines identified only 6 and missed the two most structurally critical walls. Zero-Server Backend: Consolidated the entire backend stack—knowledge graph, agent loop, spatial subscriptions, and physics engines—into a single SpacetimeDB module. Seamless Mixed-Reality Utility: Delivering a complete workflow where an anomaly detected by an automated background watchdog triggers an agent investigation, sizes a structural replacement beam in 3.7 ms, and guides a technician to the physical problem via voice in headset. What we learned Framing Data is Non-Negotiable: Floor plans alone are insufficient for operational and structural reasoning. Knowing joist orientations, tributary bay areas, and rooftop mechanical placements makes the difference between an accurate diagnosis and a catastrophic oversight. Colocating AI with State Changes the Game: Moving agent execution inside the database via procedures drastically simplifies distributed architecture, eliminating network hops, cache invalidation bugs, and synchronization drift between agents and headsets. Multi-Modal Specialization Works: Assigning deep step-by-step reasoning (Nemotron 3.5 Super 120B) strictly to the Planner and Critic while utilizing lightweight models (Nemotron 3 Nano) for domain specialists optimizes token efficiency without sacrificing analytical depth. What's next for ForeSite BIM / IFC Pipeline Integration: Expanding beyond 2D plans and framing sheets to ingest full Industry Foundation Classes (IFC) and Autodesk Revit models directly into the SpacetimeDB knowledge graph. Expanded Predictive Mechanics: Integrating dynamic finite element analysis (FEA) for complex seismic, wind-load, and foundation settlement predictions directly inside database modules. Automated Work-Order Dispatch: Bridging the manager approval gate directly to enterprise facility management platforms (such as Procore, Maximo, or ServiceNow) to automatically generate stamped contractor work orders, order structural steel members, and send tenant compliance alerts. Field Drone Telemetry Integration: Enabling autonomous inspection drones to capture high-resolution photogrammetry and thermal scans, feeding real-time crack-width growth tracking back into ForeSite's live twin.