A.T.I.D.E. - Agent Runtime for Rover Fault Recovery

LLM-powered diagnostics agent for autonomous rovers. Detects LiDAR faults, retrieves ROS 2 recovery knowledge via Tavily, and reasons through Nemotron to decide: retry, re-plan, or halt.

Comment

Inspiration

Rovers that inspect tunnels, collapsed structures and caves lose sensors as a matter of course. When a LiDAR drops out, a conventional system does the one thing it knows: halt. That's correct if the sensor is dead and wasteful or stranding if the signal would have returned a second later, or if the fault is a misconfiguration rather than failed hardware. I wanted to see what changes when the recovery decision is made by a reasoning agent, with a hardcoded rule next to it as a control.

What it does

A TurtleBot3 navigates a simulated tunnel in Gazebo Harmonic using Nav2. A fault-injection node degrades the LiDAR feed, either as a full dropout or as an intermittent, sparse signal. A diagnostics agent retrieves current ROS 2 recovery guidance with Tavily, then asks NVIDIA Nemotron 3 Nano on Nebius Token Factory to choose one action: RETRY_NAVIGATION, REVERSE_AND_REPLAN or HALT. A recovery executor acts on the decision through Nav2 (cancelling or re-sending the goal, or backing up through the behavior server). The reasoning trace is published as a topic, so every decision can be read afterwards.

A hardcoded baseline, any fault means HALT, runs next to the agent so the difference is visible rather than claimed.

How I used Nemotron and Nebius Token Factory

Every recovery decision is a call to Nemotron 3 Nano served from Nebius Token Factory. It receives the fault severity, the definition of each severity, and the retrieved guidance, and returns a decision, a rationale and a chain-of-thought trace. The trace arrives in the reasoning field of the response and is published to /reasoning_trace.

I use a reasoning model rather than a classifier because the right action depends on context, not only on the label. The model's output is also never trusted blindly: a deterministic validation layer checks it against the three-action vocabulary and falls back to HALT if it returns anything else. The model proposes; the validator decides what reaches the robot.

In the 30-run evaluation every one of the 60 calls returned a usable decision, and the median time from fault to decision was 5.7 s (intermittent) and 4.4 s (complete), including the Tavily search. I did not try the larger Nemotron models, so I cannot say how they would compare. Nano was enough for this bounded choice among three actions: it returned a usable decision every time, in a few seconds, so I had no reason to try a larger model.

How I used Tavily

Tavily is called at runtime before every reasoning step, with a query built from the fault severity and the operating context. The top three results go into the prompt as grounding, and the reasoning trace refers to them. I chose runtime retrieval because ROS 2 and Nav2 recovery practice changes, and a static prompt would freeze it at the model's training snapshot.

I also checked whether retrieval changes the decision. With the search switched off (same agent, model and prompt), RETRY_NAVIGATION on intermittent faults went from 9 of 50 runs to 14 of 30 (Fisher p = 0.010 against the pooled runs, 0.15 against the same-day control alone), so retrieval shifts the mix of decisions (suggestive, not conclusive). It does not show the decisions are better; there is no ground truth. The retrieved pages were often only loosely related to LiDAR dropout, and retrieval accounts for most of the latency (median 5.7 s with it, 1.5 s without).

How I built it

ROS 2 Jazzy, Gazebo Harmonic, Nav2 with a TurtleBot3 Waffle, a custom Gazebo tunnel world. Four packages: atide_bringup (launch files, config, bridge, world), atide_fault_trigger (fault injection, switched live with a ROS 2 parameter), atide_diagnostics_agent (Tavily + Nemotron + validation + the operating context node) and atide_recovery_executor. The Nav2 costmaps subscribe to the fault node's filtered scan, so a degraded LiDAR genuinely degrades navigation rather than only changing a log line.

Challenges

  • Making the fault real for navigation: the costmaps had to read /scan_filtered, not /scan.
  • Duplicate /clock publishers from orphaned ros_gz_bridge processes flooded every node with TF "jump back in time" errors until I found and killed the extras.
  • load_dotenv() did not find the .env file after colcon build, because the executed file lives under build/; the agent now takes an explicit env_file parameter.
  • One test run produced a decision string with the prompt's line label echoed into it. The validator caught it and fell back to HALT; I tightened the prompt and it has not recurred in later runs.
  • A wrong Docker apt repository URL, a .dockerignore that was silently ignored because the dot was missing from its name, and a 2.3 GB build context (VS Code IntelliSense databases) cost time before the demo ran in a container.
  • Getting real numbers also cost time: the first evaluation was 10 runs per severity and looked like a clean 8 / 2 split, but a larger 30-run sample on the same agent gave a different distribution and forced the whole README to be rewritten. A judge would only see the final numbers; the lesson was that early small samples on a stochastic system should not be trusted.

Accomplishments

  • The full loop works end to end in simulation: fault, retrieval, reasoning, decision, executor, rover.
  • All three actions verified through the executor. In a 30-run evaluation in a CPU-only container, intermittent faults led to a recovery action in 30 of 30 runs (26 REVERSE_AND_REPLAN, 4 RETRY_NAVIGATION) and complete faults led to HALT in 30 of 30.
  • The decision responds to operating context, not just the severity label. In four pre-registered scenarios (context values injected, 20 runs each), two predictions held clearly (more RETRY_NAVIGATION with free space ahead and a slow robot, p = 0.0038; more HALT with a close obstacle and a moving robot, p = 0.0033), one was suggestive, and with the LiDAR dead and the goal 1 m away the agent still halted in 19 of 20 runs.
  • A retrieval ablation shows the Tavily search changes the decision mix (suggestive), and quantifies its cost in latency.
  • The reasoning is readable and references the retrieved context, so decisions are auditable.
  • The same decision vocabulary drives a physical hexapod through a bridge to its wireless controller link. On HALT it holds still and hands control back to the operator. The fault is injected by hand and the hexapod has no LiDAR, so this shows the decision interface carries over, not full fault handling on hardware.

What I learned

Two things surprised me. First, that retrieval seems to change the decision, not just the explanation: with the Tavily search off, the same model chose RETRY_NAVIGATION in 14 of 30 intermittent runs, against 9 of 50 with it (p = 0.010, suggestive rather than conclusive). I expected grounding to make the model more cautious; it made it commit to the heavier recovery action instead. Second, that a 30B reasoning model served from Token Factory returned a usable decision with a median under six seconds, even when called from a CPU-only container, fast enough that latency stopped being the interesting problem.

What's next

An evaluation run across many fault scenarios on Nebius Serverless Jobs, a full-simulation container so the whole closed loop needs only Docker and a single launch file. All of these are future work, not part of this submission.

Built with

Nebius Token Factory, NVIDIA Nemotron 3 Nano, Tavily, ROS 2 Jazzy, Gazebo Harmonic, Nav2, TurtleBot3, Python, C++ (fault trigger, recovery executor), Docker.

Feedback

Nebius Token Factory / Nemotron 3 Nano

  • The reasoning trace is returned in the reasoning field, exposed through the SDK's extra model fields, rather than reasoning_content. The docs did not make this obvious. I found it by trying reasoning_content first (it returned nothing) and only later discovering the correct field. A one-line note on the model card would save the search.
  • Median fault-to-decision time was 5.7 s (intermittent) and 4.4 s (complete), including the Tavily search. For a fault-recovery decision that is acceptable, the alternative is an instant halt and no human is waiting on a control loop, but it would not be acceptable for a controller reacting to obstacles at 20 Hz.
  • All 60 calls in the 30-run evaluation returned a decision, with no timeouts. No rate limits, errors or model-ID confusion were observed.
  • Occasional format drift: in one run the model echoed the prompt's field label into its answer. I did not try structured output or guided decoding; the response has to be one of three known tokens, so a regex validator in application code is cheaper and catches the same failure modes. If the output schema were richer I would reach for structured output first.

NVIDIA tools and models

  • For the same intermittent fault, the model chose REVERSE_AND_REPLAN 26 times and RETRY_NAVIGATION 4 times in 30 runs, so the decision is not deterministic per run; the reasoning trace is what makes each choice inspectable. The trace is much longer than the answer and hedges before committing. Reading it is what makes the model's behaviour legible; without it the agent would look like a black box that emits one of three strings.
  • Isaac Sim and Isaac Lab were not used in this submission; Gazebo Harmonic is the simulation layer.

Nebius AI Cloud / Serverless

  • I built the agent and the evaluation harness as a container image meant to run as a Serverless Job, but I only ran it locally, so I have no hands-on feedback on Serverless Jobs.

Tavily

  • search() is the supported call; get_search_context() is deprecated, which was easy to miss when following older examples. I confirmed this during the build.
  • Many of the pages returned for a query about LiDAR dropout in Nav2 were only loosely related (other robots' documentation and blog posts).
  • The search accounts for most of the latency: the median fault-to-decision time was 5.7 s with it and 1.5 s without.

Built With

  • ai-agents
  • c++
  • gazebo
  • llm
  • nav2
  • nebius
  • nemotron
  • nvidia
  • physicalai
  • python
  • retrieval-augmented-generation
  • robotics
  • ros-2
  • simulation
  • tavily
  • ubuntu
Share this project:

Updates

Submission history