-
-
Nav2 costmap in RViz: LiDAR obstacles, inflation zones and free space used to plan safe paths.
-
Perception debug view: red box tracked at 1.43 m, while a range-mismatch check rejects a bad green box detection.
-
Gazebo test world: TurtleBot3 Waffle Pi, red/green/blue boxes and cylinder obstacles, fully in simulation.
-
Another angle of the arena as the robot works the red-box-to-green-box manipulation task.
-
RViz during navigation: Nav2 active, SLAM map growing live, 1.86 m left to goal in an unknown environment.
-
Prompt “find red box and green box”: the agent calls find_object, explores frontiers with no prior map, then task_complete.
-
Gazebo arena during the “put the red box near green box” run, with the robot beside the red box.
-
Failure handling: obstacle 0.25 m ahead halts the robot, the agent re-plans, then navigates the remaining 0.84 m.
-
Prompt “put the red box near green box”: agent stages, pushes toward the goal, verifies the gap (target 0.40 m), retries.
-
Full stack live: Gazebo, RViz/Nav2 and the perception node tracking boxes while filtering LiDAR false positives.
-
NIRVA’s closed loop: Observe → Understand → Plan → Act → Observe → Detect Failure → Re-plan, repeating until the goal is met.
-
Three layers: NVIDIA model on Nebius (reasoning), ROS 2 skill layer (action), TurtleBot3 sensors and compute (robot).
Inspiration
What it does## Inspiration
Most "LLM + robot" demos are a chatbot that emits a fixed list of waypoints. The robot never checks whether anything worked. We wanted the opposite: a robot you give a goal ("find the red box and put it next to the green box"), which then works out the steps, checks the result against what it actually measures, and recovers when something goes wrong.
NIRVA stands for Natural-language Intelligence for Reactive & Vision-guided Autonomy. The name nods to the Sanskrit prefix nir- (निर्), meaning "without" or "beyond". Here it means beyond hand-scripted missions. NIRVA is a project name, not a classical Sanskrit word.
What it does
You type any natural-language command. NIRVA runs this loop until the task is done or it honestly gives up:
Observe → Understand → Plan → Act → Measure → Detect failure → Re-plan
- It starts in an unknown world with no pre-built map.
slam_toolboxmaps online while Nav2 navigates. - An NVIDIA open model on Nebius Token Factory acts as the high-level reasoning layer. It picks the next skill by tool-calling. It never touches motors.
- Perception builds a live world model (where the red, green and blue boxes are, in the map frame).
- If a skill fails, the result goes back to the model, which has to choose a different action or call
give_up.
Example: "Place the red box next to the green box." NIRVA explores until it has located both boxes, stages behind the red box, pushes it toward the green one, backs off, then measures the gap from a fresh side view. If the gap is wrong, it re-stages and tries again.
How we built it
Stack: ROS 2 Jazzy, Gazebo Harmonic, TurtleBot3 Waffle Pi, slam_toolbox, Nav2, Python.
Brain. agent_node calls NVIDIA Nemotron-3-Nano-30B-A3B on Nebius Token Factory through its OpenAI-compatible API. Each step is a tool call over this skill set:
| Skill | What it does |
|---|---|
navigate |
Nav2 navigate_to_pose to named places or coordinates |
drive |
Closed-loop turn, straight or circle, driven by measured TF pose (not timers) |
explore / find_object |
Frontier-style exploration until a target is in the world model |
push_object_to |
Stage, push, back off, measure, retry once |
verify_relation |
Navigate to a side vantage point and measure the gap between two boxes |
detect_object / describe_surroundings |
Camera frame to a vision model |
replan / give_up / task_complete |
Explicit failure handling and completion |
Perception. The camera feeds HSV segmentation. The pixel column gives a bearing, the lidar gives the range at that bearing, and TF gives a map-frame $(x, y)$. A detection is only accepted if the lidar range agrees with the range implied by the box's apparent size, which rejects hits on the wall behind the box. Positions are the median of recent observations, so a box that gets pushed is tracked as it moves.
Don't trust the LLM's "done". Two code-level checks guard completion:
task_completeis rejected if the last physical action failed.- A second Nemotron call acts as a completion auditor. It gets the request, the measured final state and the action history, then returns a verdict. "Next to" is defined as a measured centre-to-centre distance of
$$0.30\ \text{m} \le \lVert p_A - p_B \rVert \le 0.50\ \text{m}$$
Safety. While pushing, the robot ignores the pushed box in the front cone and watches the flanks, with a hard safety stop. It also remembers stale occupied cells in the SLAM map where a moved box used to be.
Observability. Every model call is logged (model, endpoint, latency, prompt/completion tokens, chosen tool) to a JSONL file. The latency and token numbers in our demo come from that log.
Challenges
- Reasoning mode. Nemotron's reasoning is on by default and a system-prompt phrase did not turn it off. We had to disable it through the chat template (
enable_thinking: false) to get fast, reliable tool calls. - LLM overconfidence. The planner would call
task_completeearly orreplanin a loop. Prompt rules, code-level rejection and the auditor fixed this. - Moving objects break maps. A pushed box leaves phantom obstacles in the SLAM costmap, and the target box is hidden behind the one being pushed. We added a close look at the target before pushing and stale-cell handling afterward.
- Perception on an integrated-GPU laptop. We used lightweight classical vision plus lidar fusion instead of heavy detectors, so the whole stack runs in real time on modest hardware.
- Honest scoping. The TurtleBot has no arm, so "manipulation" is pushing.
pick_placeis a simulated stub, and we say so.
What we learned
- Put the LLM where it is strong (choosing skills, interpreting failures) and keep control loops, safety and verification in code.
- Return measured results to the model, not success flags.
- An agent that knows when to stop (
give_up) is more trustworthy than one that always claims success.
What's next
Replace HSV with a VLM/VLA-based detector, add voice input, test across other robots and worlds, and port the same skill interface to real hardware.