Inspiration

On an oil rig, a pump fault is rarely a one-person problem. Someone has to notice it, someone has to confirm it, a technician has to be dispatched with the right tools, the repair has to follow the right procedure, and someone has to check that the fault is really gone. Today those steps are handled by separate people, radios and paper checklists, and every handoff costs time in a hazardous area.

Our team builds digital twins for industrial plants, and we kept asking: what if that whole loop, detect, confirm, deliver, repair, verify, were run by a fleet of robots that each do one job well, coordinated by an AI that can reason about the whole site but is never allowed to move a robot directly?

That became Multi-Agent Physical AI Fleet Twin:

  1. A quadruped walks to the pump and reads its pressure gauge with a vision model.
  2. If the pressure is out of limits, an incident opens and a second inspection can confirm it.
  3. An AMR carries the tool kit to the pump, after an operator approves.
  4. A humanoid walks with the technician and guides the repair by voice from the site's written procedures.
  5. A quadruped re-reads the gauge, and the incident closes only when the pressure is back to normal.

We built the whole fleet in NVIDIA Isaac Sim first, so the logic can be designed and tested safely, then connected it to real robots.

What it does

Physical_AI_Fleet is an Isaac Sim extension for event-driven, multi-robot inspection and response. You build a fleet from its UI panels (Go2 and Spot quadrupeds, H1 and G1 humanoids, Nova Carter / MiR AMRs), place equipment on a site map, and run it in Simulation or Digital Twin mode, inside your own environment USD if you have one.

In simulation

  • Every monitored asset carries a pressure gauge from its USD. INJECT FAULT turns the gauge needle into the red zone. Nothing tells the robots the reading: they have to go and see it.
  • A quadruped walks to the asset, frames the gauge with its camera and reads it with a vision-language model. If the image is unclear, the model can ask the robot to turn left, turn right or step back, then it tries again.
  • Nemotron, the fleet brain, decides which robot handles each task: who inspects, who re-inspects, which AMR delivers tools.
  • Every decision passes a deterministic safety gate. High-risk tasks such as tool delivery wait for operator approval in the UI.
  • The AMR plans a path inside the walkable area, avoiding no-go zones and equipment, and delivers the tools. After MARK REPAIRED, a quadruped re-reads the gauge, and the incident closes only if the pressure is back in range.
  • Everything is written to an audit log: each model call with its latency, each snapshot the brain saw, each decision, each safety-gate result, and the camera frames the vision model looked at.

Extended to real robots

  • Real Go2 digital twin: a real Unitree Go2 streams its joints, IMU and odometry over UDP at 100 Hz, and its twin in Isaac Sim walks leg-for-leg with it, with no ROS needed on the PC. The real Go2 navigates to the pump on its own LiDAR SLAM map and waypoint graph, and the twin shows it moving through the site in Isaac Sim. From the extension we can also jog the real robot in small, slow steps.
  • Real G1 humanoid assistant: a real Unitree G1 follows the technician to the fault, guides the repair by voice from written SOPs ("next", "repeat", "what tool do I need"), identifies objects through its camera ("what is this?"), and gestures, including a high five when the repair is done.
  • AMR: Nova Carter in simulation, with a MiR200 as the real-world counterpart.

How we built it

The decision loop

The coordinator is pure Python with no Isaac Sim imports, so the whole fleet can run headless in tests:

alarm or event → incident → due tasks with their eligible robots → fleet snapshot → Nemotron (one decision) → re-check against the live fleet → safety gate → operator approval if needed → skill executor → result event → incident update

Incidents move through clear states: open, confirmed, tools picked up, tools dispatched, awaiting repair, verifying, closed. A gauge that is unreadable twice escalates to the operator instead of guessing.

Nemotron as the fleet brain, never the driver

Nemotron gets a compact JSON snapshot: due tasks, the candidate robots for each task (already filtered for capability, availability and safety), distances, priorities and recent decisions. It returns exactly one decision: which task goes to which robot.

  • The output is forced into a JSON schema, then validated: it can only use IDs that were in the snapshot.
  • If validation fails, Nemotron gets one corrective retry with the error message.
  • If it still fails, or no model is configured, a deterministic rule-based assigner with the same contract takes over, and the audit log records which one decided.
  • The skill catalogue has no velocity, joint, torque or trajectory fields, so the model has no way to output motion. Movement is always generated locally.

The safety gate

The gate is deterministic: the same inputs always give the same answer. It checks event freshness, robot availability and telemetry age, capability, whether the target is inside the walkable area and outside no-go zones, confidence for high-risk skills, and real-robot policy (by default, every action on a real robot needs operator approval). A mirrored real robot is never given tasks by the fleet at all; it is driven only by its own remote or the JOG buttons.

Reading a gauge with a vision model

  1. The robot stops in front of the dial and captures a frame.
  2. A first VLM call finds the gauge and returns a bounding box.
  3. The frame is cropped to 1.8 × the dial and enlarged to 768 px, so the needle is large enough to read.
  4. A second call, with thinking on, classifies the pressure as high, normal or low and gives a confidence.
  5. Answers below 0.75 confidence count as unreadable. The robot can make up to 3 adjustments within a 150 s limit before it gives up. Nothing is guessed.

Both the full frame and the crop the model saw are saved as evidence. The same pipeline works on a real quadruped's camera, because it needs no positions from the simulation.

Walking

skill goal → occupancy-grid A* (assets, environment USD obstacles, other robots, site map) → line-of-sight smoothing → pure-pursuit follower → smooth [vx, vy, wz] within the policy's limits → RL walking policy at 200 Hz → joint targets

The site map is a walkable polygon with no-go zones, in metres. Goals outside it are refused, never driven through.

Models and providers

The extension talks to any OpenAI-compatible endpoint through a small standard-library client written for Nebius Token Factory. An AI Models panel switches between NVIDIA and Nebius provider profiles, tests each connection, and keeps API keys outside the extension folder. [FILL: e.g. "In our demo, the fleet brain ran and the gauge reader ran on Nebius Token Factory."]

  • Fleet brain: Nemotron (our NVIDIA profile uses nvidia/nemotron-3-super-120b-a12b)
  • Gauge reader: a vision-language model (nvidia/nemotron-3.5-super-vl-120b-a12b in our NVIDIA profile; Cosmos Reason is also supported)
  • G1 assistant, self-hosted on one 48 GB GPU: Parakeet TDT 0.6B v2 (speech-to-text), Nemotron Nano 9B v2 (SOP answers), Cosmos Reason1 7B (camera questions)

Real-to-sim twin (Go2)

The Go2 runs a UDP state sender (port 9870, 100 Hz) and a TCP velocity server (port 9999) with a 0.5 s watchdog: if commands stop arriving at about 10 Hz, the robot stops. Jog speed is capped at 0.25 m/s. The extension reads the UDP stream directly, reorders the joints from Unitree to Isaac order, smooths them, and switches off any ROS graph in the twin USD.

To put the twin in the right place, we stand the real robot on a known home pose and press ALIGN TWIN. If the robot reports pose $(x_m, y_m, \psi_m)$ in its own odometry frame and the home pose on the site map is $(x_w, y_w, \psi_w)$, the extension stores one 2D transform:

$$ \theta = \psi_w - \psi_m, \qquad \begin{bmatrix} t_x \ t_y \end{bmatrix} = \begin{bmatrix} x_w \ y_w \end{bmatrix} - R(\theta)\begin{bmatrix} x_m \ y_m \end{bmatrix} $$

Every later odometry reading is then placed in the site as:

$$ \begin{bmatrix} x \ y \end{bmatrix}{\text{world}} = R(\theta)\begin{bmatrix} x \ y \end{bmatrix}{\text{odom}} + \begin{bmatrix} t_x \ t_y \end{bmatrix}, \qquad \psi_{\text{world}} = \psi_{\text{odom}} + \theta $$

The transform is saved in floor_layout.json, so it survives an Isaac Sim restart.

Humanoid voice agent (G1)

The G1 (Jetson Orin NX) handles audio, camera, walking, arms and hands, and calls a GPU workstation over HTTP. SOPs are plain Markdown files re-read on every request, so adding a procedure needs no restart. The assistant refuses rather than invents: unclear speech gets "please say it again", and anything outside the SOPs gets "contact the supervisor". Robot actions (follow, turn, gestures) are matched in code, separate from the LLM conversation.

Testing

The coordinator, brain, safety gate, planner, gauge pipeline and twin link are covered by 403 pytest tests that run without Isaac Sim. A headless runner plays the full loop with Go2, Spot, H1 and an AMR, and a mock Go2 bridge lets us test the digital twin without the real robot.

Challenges we ran into

  • Slow vision reasoning. With thinking on, our first gauge reads took 23-95 s per image. Cropping to the dial made answers faster and more accurate, and we tuned the timeouts to measured times (8-15 s per call).
  • A gauge too small to read. In a full camera frame the needle was a few pixels wide. We added a detect-then-crop step, and logged the camera-projected box alongside the model's box so we could compare them.
  • Bounding-box formats. Different models return boxes as 0-1000, 0-1 or pixels. We wrote a small tool that draws all three on a real frame so you can see which one fits.
  • An LLM that must not drive. We wanted Nemotron's reasoning without letting it output motion. A locked skill catalogue, schema-constrained output, ID validation, a corrective retry and a rule-based fallback solved that.
  • Idle robots falling over. Running walking policies for idle robots wasted compute, but switching them off made some sag. Robots now "park" only at home, are watched while parked, and wake up with a clean re-stand if they sink or tilt.
  • Timeouts under slow physics. Wall-clock limits cut off long walks when the simulation ran slowly. All task limits now use simulation time.
  • Robots stranded after a failure. A robot whose task failed used to stand where it stopped and block others. Now it walks home, and idle robots are cleared from a target spot before another robot is sent there.
  • The twin needed ROS on the PC. We removed that dependency: the extension now reads the Go2's UDP stream directly. We also locked odometry to one source, because mixing two made the twin jump.
  • Three models on one GPU for the G1. vLLM kept running out of memory until we found a working start order and memory split, and capped Nemotron's max_num_seqs to fit its Mamba cache.
  • A camera that looks at the floor. The G1's head camera points steeply down, so follow mode only sees the worker's legs. We estimate distance from foot position in the image.
  • Sim-to-real manipulation. We fine-tuned a GR00T N1.7 policy in Isaac Lab Arena for pick-and-place, but it did not transfer to real object positions. It stays on the roadmap.

What we learned

  • Let the model decide, not drive. Nemotron is very good at weighing priorities, distances and recent failures, but motion has to come from deterministic code. Splitting "which robot does what" from "how it moves" made the system both smart and predictable.
  • Validate everything a model returns. Structured output reduced errors but didn't remove them. Validation plus one corrective retry plus a fallback made the fleet keep working when the model was slow, wrong or offline.
  • Vision models need help to see. Cropping and enlarging the target helped gauge reading more than switching models did.
  • Simulation first pays off. Testing routes, no-go zones, failures and recovery in Isaac Sim, plus a headless test suite, let us iterate without risking hardware.
  • Coordinate frames are where twins break. Most "the twin is wrong" bugs were alignment or odometry-source problems, not rendering problems.
  • Safety has to be built in: deterministic gates, operator approval for high-risk and real-robot actions, watchdogs, speed caps and a person on the e-stop.

What's next

  • Close the real-world loop end to end: a real Go2 gauge reading opens the incident, and the fleet brain dispatches the real MiR200 and the G1.
  • Bring the G1 assistant into the fleet brain as a humanoid agent, so the technician's progress updates the incident directly.
  • Obstacle avoidance and interruptible commands for the G1.
  • Real SOPs and gauge limits from plant engineers.
  • Retry sim-to-real manipulation so the humanoid can hand over tools.

Built With

  • boston-dynamics-spot
  • cosmos-reason
  • fastapi
  • hugging-face
  • jetson-orin
  • lidar-slam
  • mir200
  • nebius
  • nebius-token-factory
  • nemotron
  • nova-carter
  • nvidia-isaac-sim
  • nvidia-nemo
  • nvidia-omniverse
  • omnigraph
  • opencv
  • openusd
  • parakeet
  • python
  • reinforcement-learning
  • ros2
  • unitree-g1
  • unitree-go2
  • unitree-sdk2
  • vllm
Share this project:

Updates

Submission history