Inspiration

What it does## Inspiration

Most "LLM + robot" demos are a chatbot that emits a fixed list of waypoints. The robot never checks whether anything worked. We wanted the opposite: a robot you give a goal ("find the red box and put it next to the green box"), which then works out the steps, checks the result against what it actually measures, and recovers when something goes wrong.

NIRVA stands for Natural-language Intelligence for Reactive & Vision-guided Autonomy. The name nods to the Sanskrit prefix nir- (निर्), meaning "without" or "beyond". Here it means beyond hand-scripted missions. NIRVA is a project name, not a classical Sanskrit word.

What it does

You type any natural-language command. NIRVA runs this loop until the task is done or it honestly gives up:

Observe → Understand → Plan → Act → Measure → Detect failure → Re-plan

  • It starts in an unknown world with no pre-built map. slam_toolbox maps online while Nav2 navigates.
  • An NVIDIA open model on Nebius Token Factory acts as the high-level reasoning layer. It picks the next skill by tool-calling. It never touches motors.
  • Perception builds a live world model (where the red, green and blue boxes are, in the map frame).
  • If a skill fails, the result goes back to the model, which has to choose a different action or call give_up.

Example: "Place the red box next to the green box." NIRVA explores until it has located both boxes, stages behind the red box, pushes it toward the green one, backs off, then measures the gap from a fresh side view. If the gap is wrong, it re-stages and tries again.

How we built it

Stack: ROS 2 Jazzy, Gazebo Harmonic, TurtleBot3 Waffle Pi, slam_toolbox, Nav2, Python.

Brain. agent_node calls NVIDIA Nemotron-3-Nano-30B-A3B on Nebius Token Factory through its OpenAI-compatible API. Each step is a tool call over this skill set:

Skill What it does
navigate Nav2 navigate_to_pose to named places or coordinates
drive Closed-loop turn, straight or circle, driven by measured TF pose (not timers)
explore / find_object Frontier-style exploration until a target is in the world model
push_object_to Stage, push, back off, measure, retry once
verify_relation Navigate to a side vantage point and measure the gap between two boxes
detect_object / describe_surroundings Camera frame to a vision model
replan / give_up / task_complete Explicit failure handling and completion

Perception. The camera feeds HSV segmentation. The pixel column gives a bearing, the lidar gives the range at that bearing, and TF gives a map-frame $(x, y)$. A detection is only accepted if the lidar range agrees with the range implied by the box's apparent size, which rejects hits on the wall behind the box. Positions are the median of recent observations, so a box that gets pushed is tracked as it moves.

Don't trust the LLM's "done". Two code-level checks guard completion:

  1. task_complete is rejected if the last physical action failed.
  2. A second Nemotron call acts as a completion auditor. It gets the request, the measured final state and the action history, then returns a verdict. "Next to" is defined as a measured centre-to-centre distance of

$$0.30\ \text{m} \le \lVert p_A - p_B \rVert \le 0.50\ \text{m}$$

Safety. While pushing, the robot ignores the pushed box in the front cone and watches the flanks, with a hard safety stop. It also remembers stale occupied cells in the SLAM map where a moved box used to be.

Observability. Every model call is logged (model, endpoint, latency, prompt/completion tokens, chosen tool) to a JSONL file. The latency and token numbers in our demo come from that log.

Challenges

  • Reasoning mode. Nemotron's reasoning is on by default and a system-prompt phrase did not turn it off. We had to disable it through the chat template (enable_thinking: false) to get fast, reliable tool calls.
  • LLM overconfidence. The planner would call task_complete early or replan in a loop. Prompt rules, code-level rejection and the auditor fixed this.
  • Moving objects break maps. A pushed box leaves phantom obstacles in the SLAM costmap, and the target box is hidden behind the one being pushed. We added a close look at the target before pushing and stale-cell handling afterward.
  • Perception on an integrated-GPU laptop. We used lightweight classical vision plus lidar fusion instead of heavy detectors, so the whole stack runs in real time on modest hardware.
  • Honest scoping. The TurtleBot has no arm, so "manipulation" is pushing. pick_place is a simulated stub, and we say so.

What we learned

  • Put the LLM where it is strong (choosing skills, interpreting failures) and keep control loops, safety and verification in code.
  • Return measured results to the model, not success flags.
  • An agent that knows when to stop (give_up) is more trustworthy than one that always claims success.

What's next

Replace HSV with a VLM/VLA-based detector, add voice input, test across other robots and worlds, and port the same skill interface to real hardware.

Built With

  • agentic-ai
  • gazebo
  • llm-tool-calling
  • nav2
  • nebius
  • nebius-token-factory
  • nvidia
  • nvidia-nemotron
  • openai-api
  • opencv
  • physical-ai
  • python
  • robotics
  • ros2
  • ros2-jazzy
  • slam-toolbox
  • turtlebot3
Share this project:

Updates

Submission history