Inspiration
Industrial plants today still run on spreadsheets, paper work orders and dashboards that sit far from the machines they describe. When a robot's bearing starts to wear, the shift lead has to decide quickly: keep running, stop for maintenance, or move work to another line. Getting it wrong means late orders, scrap or a crash stop. We wanted a digital twin that thinks (agents that diagnose and plan), that you can operate (commission, simulate, maintain) and that you can step into (3D and WebXR on Meta Quest). We also wanted every predictive number it shows to come from a model you can check.
What it does: Live twin: lines, machines, sensors, inventory, orders, cameras and mobile robots, all running on a deterministic plant model. It is shown as a topology, isometric, floor-plan or 3D view, plus VR and passthrough AR on Meta Quest. Agentic operations on Meta's Muse models: a coordinator and three specialist agents work over 32 deterministic tools, and the same tools are available over MCP. Every figure in an answer is checked against tool results. Plans are only proposed; a person approves them. What-if and experiments: maintenance, speed and reroute plans are replicated under uncertainty, with confidence intervals and the odds of each plan winning. Commissioning: live signals with limits, OPC UA and Modbus connectors with drag-and-drop tag linking, manual axis jog, a sequencer, per-unit product flow, snapshots, and guided maintenance procedures signed step by step. Spatial AI: ask Muse Spark about what the 3D view shows, or let the copilot fly the 3D view to a machine. Open standards: Asset Administration Shell (AAS v3) export, glTF export with twin metadata, and MCP.
Predictive Maintenance: Real-time vibration and thermal analysis forecasting Remaining Useful Life (RUL) and machine trips, proposing repairs before downtime occurs. Supply Chain Intelligence: Dynamic material runway forecasting synced with line speeds to predict stockouts and draft automated purchase orders. Visual Quality Inspection: High-speed camera inspection with Meta SAM 3.1 and OpenCV to segment solder bridges, defects, and assembly flaws in real time.
What TWIN is Capable Of:
- Live 3D Kinematics: Real-time conveyor flow, dynamic part queuing, photo-eye blockage detection, and machine motion synced to speed setpoints.
- WebXR (VR & Passthrough AR): Walk the factory floor on Meta Quest with 6DoF hand tracking and floating spatial telemetry panels.
- Agentic AI & 32 MCP Tools: Coordinator and 3 specialist agents using 32 deterministic tools over MCP, with zero-hallucination fact checking.
- Spatial AI & Camera Navigation: Interrogate 3D scenes in natural language via Muse Spark, with auto-flying cameras focused on machine anomalies.
- Field Commissioning: Live OPC UA and Modbus TCP connectors, drag-and-drop signal linking, manual axis jog, and a visual step sequencer.
- Signed Guided Maintenance: 8-step standardized work orders with 3D component isolation and live sensor-gated test runs.
- Intralogistics & AGV Fleet: Discrete per-unit tracking, collision-free grid routing, dock pacing, and automated deadlock resolution.
- Monte Carlo What-Ifs: Seeded simulation experiments using Common Random Numbers (CRN) with Student-$t$ 95% confidence intervals.
- Open Industry 4.0 Standards: Native Asset Administration Shell (AAS v3 JSON) export and 3D
.glbexport with embeddedextras.twinmetadata.
Agentic AI Core:
1. Spatial Agents (Embodied & 3D Grounded)
- Holographic In-Scene Guidance: Agents don't live in a disconnected text box. In the 3D twin and WebXR (Quest), the agent acts as an embodied co-pilot—spatially flying the operator's viewport to the exact machine, highlighting degraded components with 3D status halos, and projecting floating telemetry boards above physical equipment.
- Spatial Scene Understanding (
/api/twin-view/ask): The agent "sees" what the operator sees. Powered by Meta Muse Spark, operators can ask spatial questions ("Why is Line 2 backing up?" or "What's causing this vibration?"), and the agent inspects the 3D scene camera frame paired with real-time telemetry to diagnose the issue visually.
2. Voice-First Factory Interaction
- Hands-Free Shop-Floor Voice: Inside a VR headset or while wearing safety gloves on a factory floor, typing is impractical. TWIN implements voice-first control (powered by Muse Voice Transcribe / Web Speech API).
- Conversational Action Commands: Operators speak naturally:
- "Twin, show me the thermal hotspot on SMT Line 1."
- "Jog robot arm axis 2 by minus 15 degrees."
- "Run a Monte Carlo experiment on shifting production to Line 3."
3. Procedural Content Generation
- Blueprint-to-Twin Procedural Layouts: Upload a raw 2D floorplan image, CAD file, or sketch, and the Builder agent procedurally extracts room geometry, machine placements, and conveyor routes—automatically assembling a live, physics-driven 3D factory in seconds.
- Procedural Maintenance Workflows: When an anomaly is detected, the Maintenance Agent procedurally constructs an 8-step, signed SOP tailored to the machine model, failure mode, and safety lockouts, complete with interactive 3D component isolation.
4. Context-Aware Assistance (Multi-Agent Swarm with Zero-Hallucination Tools)
- Specialist Swarm Architecture: A master coordinator delegates autonomously to three domain specialists:
- Maintenance Specialist: Diagnoses vibration harmonics, thermal drift, and RUL.
- Quality Specialist: Segments micro-defects using SAM 3.1 computer vision.
- Supply Chain Specialist: Monitors material depletion and forecasts stockouts.
- 32 Deterministic MCP Tools: Every claim, figure, and recommendation is calculated via deterministic tools before being presented. Plans are only proposed; human operators retain final cryptographic approval.
How we built it
The web app is Next.js 15 and React 19. The 3D views use React Three Fiber on three.js, and the VR and AR modes use @react-three/xr. Everything persists in a single SQLite database. A Python FastAPI backend hosts the intelligence features, DuckDB analytics and the field connectors (asyncua, pymodbus). AI runs on the Meta Model API: Muse Spark for agents and vision, Muse Wonder for briefings, SAM 3.1 for segmentation, and Muse Voice and Image.
The core rule: the engine computes and the AI explains. Simulation, KPIs, traffic and statistics are pure TypeScript functions with tests. The models choose tools and write the text, never the numbers.
A few of the models:
Line pace: a line runs at the speed of its slowest running machine:
Fleet traffic: zone control on a grid. A robot may enter zone (z) only if no robot in (z) shares its heading or crosses it. Wait-for cycles (deadlocks) are detected and broken deterministically.
Challenges Keeping AI honest: we built a grounding check that pulls every figure out of an answer and traces it to a tool result. Anything it can't trace is flagged. Real-time 3D in the browser: product flow, machine motion, labels for dozens of objects, live virtual-camera feeds and WebXR all have to keep a usable frame rate. The answers were one label projector for all labels, capped pixel ratios, and automatic quality fallback. Industrial protocols without hardware: we wrote a simulated PLC that serves the same tags over OPC UA and Modbus. This let us test linking, scaling and polling end to end. Realistic rules that don't block real plants: strict material-port rules flagged valid template lines, where some machine types stand in for others. We kept the checks that matter physically and inferred each line's raw material. What we learned Deterministic engines plus tool-using agents beat letting a model "reason" about numbers. Commissioning features such as snapshots, sequences and signal linking matter as much as dashboards for trust. WebXR makes spatial access to a plant one link away, with nothing to install.
Built With
- asset-administration-shell
- duckdb
- fastapi
- gltf
- mcp
- meta-model-api
- meta-quest
- modbus
- muse-spark
- next.js
- opc-ua
- playwright
- python
- react
- react-three-fiber
- sam
- sqlite
- tailwindcss
- three.js
- typescript
- vitest
- webxr
- zod
- zustand