Inspiration

Warehouse operators shouldn't have to walk back to a desktop terminal every time a shelf bin runs empty. Traditional ERP workflows force floor workers to leave the aisle, log into legacy web tables, manually compare supplier quotes, and chase invoices—creating costly stockout delays. Sovereign Supply Chain XR Walk brings accountable, agentic AI directly into the operator's physical warehouse aisle using Meta Quest Passthrough and hands-first interaction.

What It Does

Sovereign XR Walk anchors a spatial digital-twin inventory rack directly onto the user's real-world floor within comfortable seated reach—zero paired controllers or room-scale setup required:

  1. Spatial Stockout Detection: Pulsing red bins (such as Widget-A at 0 units) highlight critical stockouts in 3D space.
  2. Autonomous Multi-Agent A2A Negotiation (Meta Muse): A bare-hand pinch triggers our procurement agent (SAM), powered by Meta Muse (muse-spark-1.3), to run a real-time Agent-to-Agent (A2A) negotiation across competing supplier agents (Apex Logistics, Global Parts Co, FastShip Inc). SAM evaluates unit cost, lead time, and reliability live in 3D space—streaming every turn into a floating A2A Negotiation Log beside a drafted Purchase Order.
  3. Quest OS-Style Draggable Spatial Panels: Both floating 3D panels feature dedicated top grab bars so operators can pinch-and-drag either card anywhere in their passthrough workspace.
  4. Human-in-the-Loop Governance & Invoice Settlement: Agents never spend enterprise funds without human sign-off. Operators pinch Approve PO (PO-2026-847), review the extracted supplier invoice (INV-2026-847) against the locked Purchase Order terms (500 units @ $8.20), and pinch Verify & Pay to release payment and commit the inventory restock (500 units in green) to SQLite (mock_sap.db).
  5. Voice-First Intent Classification: On desktop WebXR browsers supporting the Web Speech API, operators can also drive the entire workflow hands-free by voice ("restock", "approve PO", "pay invoice"), backed by a FastAPI + SQLite orchestration backend.

How We Built It

  • Spatial WebXR Frontend (Three.js + WebXR): Single-file immersive AR client (frontend/vr/index.html) supporting immersive-ar, local-floor auto-anchoring (1.35m ahead at y=0), 25-joint hand tracking, raycast + near-field finger-poke detection, dynamic canvas-textured 3D spatial cards, and Quest OS-style draggable grab handles.
  • Agentic AI Backend (FastAPI + Meta Muse + SQLite): Low-latency Python FastAPI endpoints (/api/vr/negotiate, /api/vr/interpret-voice, /api/vr/invoice-scan/demo, /api/vr/invoice-pay/demo, /api/vr/reset-demo) powered by Meta Muse (muse-spark-1.3) for real-time multi-supplier RFQ evaluation, natural-language voice intent classification, and SQLite (mock_sap.db) inventory persistence—with deterministic fallback simulation so the hosted cloud link remains 100% playable even when shared cloud server IPs are blocked by upstream API firewalls.

Challenges We Ran Into & What We Learned

  • Quest Browser Web Speech Limitations: Meta Quest Browser exposes a stubbed SpeechRecognition object that fails inside immersive-ar. We added user-agent detection (OculusBrowser + Android) to hide the voice button on standalone headsets, center the 3D Reset Demo control, and dynamically switch all HUD prompts to pure hands-first pinch instructions.
  • Preventing Duplicate Agent Loops & Late Responses: Rapid pinch jitter and mid-flight resets initially caused duplicate A2A runs. We implemented a 900ms selectstart debounce, state guards on Widget-A, and an AbortController + monotonic generation counter on Reset Demo to cancel in-flight Muse calls cleanly.

What's Next

  • Integrating live barcode/QR bin scanning via Quest Passthrough camera access and persistent spatial anchors (anchors WebXR module) across multi-aisle warehouse walks.

Built With

Share this project:

Updates

Submission history