Project Story: AeroPatch - NIM-LLM Supported Robot Self-Healing & Local Fast Reaction Control

Inspiration

Despite the rapid proliferation of Autonomous Mobile Robots (AMRs) in settings ranging from warehouses to hospitals, we realized that current control architectures remain surprisingly brittle. A single unexpected environmental change — such as a newly formed puddle on a warehouse floor — causes the robot's wheels to slip ($\text{slip}$).

Under these conditions, standard PID controllers typically assume the robot is "falling behind" its trajectory and react by increasing power to the motors. On a wet surface, this causes the wheels to spin freely even faster, leading to path deviation, motor overheating, and ultimately the jury's biggest fear: a complete vehicle tumble.

We developed AeroPatch drawing inspiration from human biology, specifically the interaction between local "reflexes" and deferred "brain analysis." When you touch a hot stove, your spinal cord pulls your hand back via a local reflex long before your brain processes the sensation of pain (Local Reflex). Subsequently, your brain analyzes the situation and formulates a plan to treat the burn (High-Level Analysis). We applied this exact principle to AMRs operating in wet and unpredictable environments.

What It Does

AeroPatch is a unique hybrid control architecture for autonomous vehicles that combines a Local Safety Reflex with Deferred Cloud Intelligence.

Local Response (Local Fast Fallback): Onboard sensor fusion (currently based purely on odometry divergence) detects anomaly conditions within milliseconds at the local "Edge" layer. When a $\text{slip}$ state is detected, the "Local Reflex" triggers with 0 ms latency without waiting for the cloud: an ESCALATION alarm is raised, the controller's proportional gain $K_p$ is instantly reduced from $2.2$ down to $1.0$ to prevent an immediate rollover, and a maximum speed limit ($0.12\,\text{m/s}$) is strictly enforced.

Deferred NIM LLM Analysis: Simultaneously, a telemetry packet is asynchronously dispatched to the NVIDIA NIM LLM Agent (nvidia/nemotron-3-super-120b-a12b) in the cloud. The cloud-based "brain" analyzes the telemetry stream, validates the slip diagnosis against historical pattern models, resynchronizes the robot's spatial state (resynchronize), and generates an optimized dynamic patch ($K_p = 1.5$, $K_d = 0.22$, $\text{slip_comp} = +0.400$).

Hybrid Application & Self-Healing: Seconds after securing its physical safety locally, the robot receives the cloud-generated patch. It validates the patch content, cross-checks it against external context data (such as live weather conditions), and applies the optimized Cloud Safe Mode parameters. The robot dynamically "heals" its control loop and resumes its trajectory with fine-tuned precision.

Multi-Cloud Fallback: If our architecture detects a service outage or timeout in external context providers (e.g., the Tavily Weather API), it automatically fails over to secondary services like DuckDuckGo Search (DDGS) to maintain overall system availability.

How We Built It

AeroPatch is built on a distributed ROS 2 Humble microservices architecture. Onboard Edge services and the cloud NIM LLM Agent operate fully asynchronously.

Edge (AMR):

• controller_node: Manages the robot's active PID control loop.
• monitor_node: Calculates the deviation between expected odometry motion and actual sensor feedback (currently Ground Truth) to compute a dynamic anomaly score ($S$). If $S > 0.70$, it triggers the Local Fast Fallback escalation logic.
• bridge_node: Manages asynchronous communication with the Cloud Agent.

Cloud (NIM LLM Agent):

• nebius_handler: A robust Python agent orchestrating the nvidia/nemotron-3-super-120b-a12b NIM model via a Nebius endpoint. It ingests telemetry, fetches live external context (weather) via Tavily and DuckDuckGo Search, and generates structured parameter patches linked with msg.event_id.

AeroPatch is grounded in classical control theory. The standard PID controller output is defined as:

$$u(t) = K_p e(t) + K_i \int e(t)\,dt + K_d \frac{de(t)}{dt}$$

where $K_p$, $K_i$, and $K_d$ denote the proportional, integral, and derivative gains, respectively. AeroPatch goes beyond merely adjusting these gains; it switches the active control regime to instantly transition the robot's dynamic profile to a safe state on wet surfaces (softening motor response by dropping $K_p$ to $1.0$).

Challenges We Faced

This project served as an exercise in engineering honesty and the pursuit of enterprise-grade robustness. The primary hurdles we encountered exposed our technical debt and service reliability boundaries:

Slip Accumulation and EKF Absence: Our single greatest technical hurdle was that despite dropping the PID gain $K_p$ to $1.0$, the visual slip_gap metric continued to accumulate on wet surfaces. As observed on our RViz diagnostics panel, our anomaly scoring engine currently relies strictly on raw odometry divergence. For an enterprise-grade autonomous driving architecture, the absence of an Extended Kalman Filter (EKF) or proper Sensor Fusion layer — fusing raw odometry with IMU, LiDAR, or camera feedback — acts as a critical bottleneck. Transitioning from pure odometry to robust EKF residual calculation represents our primary technical debt.

API Key Failures and DDGS Packaging Errors: Although we designed a multi-cloud fallback mechanism for external context services, the weather[fallback]: timeout error displayed on our diagnostic HUD revealed a dual failure: an invalid Tavily API key combined with a missing duckduckgo-search Python package on the server. Consequently, our secondary fallback mechanism crashed, forcing the system to operate purely on Local Reflex to maintain physical safety without external context awareness.

Accomplishments That We're Proud Of

Despite the lack of an EKF and external API failures, we are exceptionally proud that our architecture succeeded $100\%$ in preventing vehicle rollover (tumble) and motor burnout.

Even during worst-case scenarios shown on our RViz telemetry screens — where both Tavily and DDGS failed, incurring a $13\,\text{s}$ latency penalty — the HUD panel consistently maintained GLOBAL STATUS: Ok. This proves the absolute resilience of our Local Response mechanism. Thanks to our hybrid design, even when the cloud fails to respond, the robot guarantees its immediate physical safety within milliseconds via its local reflex (switching to $K_p = 1.00$).

What We Learned

AeroPatch taught us that self-healing control in autonomous systems is not simply about wrapping LLMs around robots — it is about orchestrating structural resilience.

LLMs are Context Orchestrators, Not Instant Drivers: LLMs like NIM Nemotron-3-Super excel at high-level telemetry analysis and contextual diagnostics. However, they cannot be placed directly in the real-time control loop for instant PID gain adjustments. A $13.7\,\text{s}$ delay for a cloud-generated patch is completely acceptable for deferred fine-tuning, but immediate physical safety must strictly reside on the Edge.

Fallback Robustness is Mission-Critical: Multi-cloud fallback mechanisms cannot themselves be brittle (e.g., failing due to a missing pip dependency). Every fallback tier, down to the final deterministic safety mode, must be rigorously verified and tested.

What's Next for AeroPatch

The next phase of AeroPatch focuses on elevating the system to an Enterprise-Grade Autonomous Driving Safety Architecture:

EKF / Sensor Fusion Layer Integration: Fusing raw wheel odometry with IMU and LiDAR data using robot_localization to calculate robust EKF residuals. The anomaly scoring engine will be driven entirely by sensor fusion rather than raw odometry alone.

Robust Recovery and Ramping Logic: Recovering from wet to dry surfaces must not rely solely on a static $10\,\text{s}$ stabilization timer. Following autonomous driving standards, controller gains (e.g., $K_p$) will be gradually ramped up ($1.0 \rightarrow 1.5 \rightarrow 2.2$) while cross-verifying surface friction state with context services to eliminate PID gain chattering.

Active Safety (AEB Layer): Integrating dynamic obstacle avoidance and emergency braking. AeroPatch will dynamically adjust stopping distances and $v_{\max}$ limits based on the surface friction coefficient ($\mu$) estimated by the NIM agent. If laser scanners detect an obstacle on wet terrain, the Autonomous Emergency Braking (AEB) layer will initiate earlier, smoother deceleration curves to safely bring the robot to a complete stop without slipping.

Built With

• ai-agents
• amr
• autonomous-robots
• duckduckgo
• gazebo
• llm
• nebius
• nvidia-nim
• pid-control
• python
• ros2
• rviz
• tavily