AETHER-LOOP: Autonomous Physical AI & Agentic SRE Platform

Decoupled 500Hz on-robot momentum observer, NVIDIA OpenShell & Nemotron 3 SRE agent with live Tavily grounding, and async Nebius H100 VLA orchestration.

Comment

Inspiration: Solving Physical AI's "3-Computer Problem"

Modern industrial robotics is bottlenecked by the 3-Computer Problem:

  1. Computer 1 (Edge Control Plane): Deterministic 500Hz real-time motor control loops running on embedded hardware where even a 10ms network stall causes physical instability.
  2. Computer 2 (Simulation & Synthetic Data Plane): Photorealistic physics and optical domain randomization (NVIDIA Isaac Sim & Cosmos 3).
  3. Computer 3 (Hyperscale Training & Agentic Reasoning Plane): Multi-tier LLM reasoning (NVIDIA Nemotron 3) and Vision-Language-Action (LeRobot SmolVLA) policy fine-tuning on cloud H100 GPUs.

Today, these three computers operate as disconnected islands. When an industrial manipulator encounters an unmapped physical edge case—such as an aviation hydraulic fluid spill (Eastman Skydrol LD-4), intense specular LED glare on BOPP blister trays, or non-rigid LDPE polybag deformation—recovery requires weeks of manual incident triage, datasheet lookup, and offline retraining.

Furthermore, a naive architecture that places LLM inference or web search inside a 500Hz control loop violates real-time control theory. AETHER-LOOP was built to solve this cleanly: we strictly decouple deterministic 2ms edge mechanics from asynchronous cloud SRE triage, live web tribology grounding, and sandboxed Monte Carlo safety verification.

What It Does & Decoupled 3-Subsystem Architecture

Subsystem 1: 500Hz On-Robot Deterministic Mechanics & Generalized Momentum Observer (Edge)

Running with zero LLM or network dependency, Subsystem 1 evaluates 6-DOF manipulator dynamics under a strict $\Delta t = 2\text{ ms}$ budget using the Recursive Newton-Euler (RNE) formulation:

$$M(q)\ddot{q} + C(q, \dot{q})\dot{q} + G(q) = \tau + \tau_{\text{ext}}$$

To detect contact anomalies without expensive 6-axis force/torque sensors, we implement a Generalized Momentum Disturbance Observer over generalized momentum (p = M(q)\dot{q}):

$$\hat{r}(t) = K_O \left( p(t) - \int_0^t \left( \tau + C^T(q,\dot{q})\dot{q} - G(q) + \hat{r}(s) \right) ds \right)$$

Simultaneously, the kernel monitors kinematic dexterity via the Yoshikawa Manipulability Index:

$$w(q) = \sqrt{\det\left(J(q)J^T(q)\right)}$$

and evaluates the Inclined-Plane Coulomb Friction Cone:

$$F_{t,\text{req}} = m g \sin(\theta_{\text{incline}}) + m a_{\text{lat}} \le \mu_k \left(m g \cos(\theta_{\text{incline}}) + F_{\text{preload}}\right)$$

When the baseline policy (SmolVLA v1.0) hits a Skydrol spill at $t = 1.24\text{ s}$, the residual spikes to $\hat{r}(t) > 6.0\text{ Nm}$ and Joint $J_2$ experiences a $73.88\text{ Nm}$ overload. Subsystem 1 executes a deterministic local safe-stop in $<2\text{ ms}$ and asynchronously flushes a ROS2 telemetry bag to the cloud.

Subsystem 2: Sandboxed SRE Agent, Multi-Tier NVIDIA Nemotron 3 Router & Live Tavily Grounding (Control Plane)

Once the robot is safely halted, the offline SRE Agent takes over inside an NVIDIA OpenShell runtime sandbox:

  • Live Tavily API Grounding (/search & /extract): Before synthesizing simulation parameters, the agent queries the live Tavily Search API to retrieve authoritative Material Safety Data Sheets (MSDS) and tribology specifications—extracting real kinematic viscosity $\nu = 11.42\text{ cSt}$ at $38^\circ\text{C}$, refractive index $\text{IOR} = 1.478$, and boundary kinetic friction $\mu_k = 0.08$.
  • 4-Tier NVIDIA Nemotron 3 Cascade on Nebius Token Factory:
    • Tier 0 (nvidia/Nemotron-3-Nano-30B-A3B): High-speed telemetry triage and CUSUM residual classification.
    • Tier 1 (nvidia/nemotron-3.5-super-vl): Multimodal optical IOR and specular depth-dropout inspection.
    • Tier 2 (nvidia/NVIDIA-Nemotron-3-Super-120B-A12B): 1M-context cross-correlation across historical ROS2 logs and trajectory kinematics.
    • Tier 3 (nvidia/Nemotron-3-Ultra-550B-A55B): Synthesizes strict JSON domain-randomization specifications and an executable Python controller patch.
  • NVIDIA OpenShell Syscall & AST Guard: Enforces capability-based isolation—allowing read-only telemetry inspection and outbound HTTPS calls to Nebius and Tavily while explicitly blocking unauthorized hardware syscalls (EXEC_SHELL_DIRECT_HARDWARE_WRITE on /dev/can0 returns EPERM).

Subsystem 3: Asynchronous Nebius H100 Batch Orchestration & (N=100) Monte Carlo Attestation (Cloud Batch)

  • Nebius Serverless H100 Jobs (gpu-h100-sxm): Dispatches two sequential cloud jobs: (1) generating 50 physically grounded synthetic video rollouts via NVIDIA Cosmos 3 (cosmos-transfer1-7b) and (2) fine-tuning LeRobot SmolVLA for 12 epochs (lr=1.5e-4).
  • 100-Rollout Box-Muller Monte Carlo Gate: Before any Over-The-Air (OTA) deployment is approved, the candidate policy (v1.1) must pass (N=100) stochastic trials perturbed via Box-Muller Gaussian noise:

$$Z_0 = \sqrt{-2 \ln U_1} \cos(2\pi U_2)$$

We compute the exact Wilson Score 95% Confidence Interval over survival count (k/n):

$$CI_{95\%} = \frac{\hat{p} + \frac{z^2}{2n} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}}, \quad z = 1.96$$

Only when peak joint torque stays strictly within $\tau_{\max} \le 18.0\text{ Nm}$ and survival exceeds $\ge 92\%$ does AETHER-LOOP issue a cryptographically signed Dual WebCrypto SHA-256 Attestation Receipt.

How We Built It

  • Backend Control Plane: Built in Python 3.12 and FastAPI, implementing the 6-DOF Recursive Newton-Euler solver (backend/openshell_gate.py), the multi-gateway NVIDIA Nemotron 3 router (backend/router_nemotron.py), the live Tavily client, and the Python ast.parse() execution sandbox.
  • Interactive 3D Digital Twin & Edge Kernel: Built with React 18, TypeScript, Three.js, and Vite. To guarantee 24/7 zero-latency accessibility for judges even on static hosting (github.io), we also compiled the 6-DOF RNE dynamics solver, Box-Muller Monte Carlo engine, and CORS Tavily bridge into an edge-optimized TypeScript kernel.
  • Continuous Verification: Automated via GitHub Actions CI (.github/workflows/ci.yml), running 11 pytest unit/integration tests and a standalone CLI benchmark (scripts/benchmark_mechanics.py).
Contact Regime (\mu_k) RNE Compute (ms) Peak (\tau) (Nm) Manipulability (w(q)) (k/n) Survival Wilson 95% CI
skydrol_spill 0.08 0.343 ms 14.44 Nm 0.4729 97/100 (97.0%) [91.55%, 98.97%]
biopharma_glare 0.25 0.061 ms 16.02 Nm 0.4729 100/100 (100.0%) [96.30%, 100.0%]
deformable_garment 0.20 0.056 ms 14.56 Nm 0.4729 100/100 (100.0%) [96.30%, 100.0%]

Challenges We Ran Into

  1. Preventing LLM Latency from Violating Real-Time Control: Early architectural iterations risked coupling cloud inference with kinematic evaluation. We solved this by architecting a strict hardware/software boundary where Subsystem 1 runs pure Lagrangian mechanics at 500Hz (<0.35ms compute time) and communicates with Subsystem 2 exclusively via asynchronous incident bags and verified OTA weight swaps.
  2. Extracting Reliable Physics Constants from Unstructured Web Text: Raw web search snippets often mix temperature regimes (e.g., viscosity at (-54^\circ\text{C}) vs. (38^\circ\text{C})) or emit reasoning <think> tags before JSON payloads. We solved this by combining Tavily's search_depth="advanced" chunking with a deterministic JSONDecoder.raw_decode parser and physical range clamping (_clamp_finite) in Pydantic.
  3. 24/7 Judge Testability: Tunneling local ports can drop when a machine sleeps. We engineered a dual-mode architecture that pairs our FastAPI backend with a self-contained GitHub Pages build so every slider, 6-DOF Jacobian matrix, and 100-rollout Monte Carlo swarm works 24/7.

What We Learned

  • Routing sub-tasks across specialized NVIDIA Nemotron 3 model sizes (30B Nano for fast triage vs. 550B Ultra for structured physics JSON synthesis) dramatically reduces end-to-end SRE remediation latency while preserving mathematical accuracy.
  • Grounding generative world models (NVIDIA Cosmos 3) in real manufacturer MSDS data retrieved via Tavily bridges the sim-to-real gap far more effectively than ungrounded prompt guessing.

Built With

  • control-theory
  • fastapi
  • github
  • github-actions
  • lerobot
  • monte-carlo
  • nebius-ai-cloud
  • nebius-token-factory
  • nvidia-cosmos
  • nvidia-isaac-sim
  • nvidia-nemotron
  • nvidia-openshell
  • python
  • react
  • ros2
  • tailwindcss
  • tavily
  • three.js
  • typescript
  • vite
Share this project:

Updates

Submission history