Dashboard
Nemotron agentic reasoning

Inspiration Language models are good at open-ended instructions and bad at knowing whether a physical action actually worked. We wanted a robot arm you can talk to, where the AI proposes and plain code checks. "Put the green cube on the blue cube, then the blue cube on the red cube" should work even though blue is buried under green.

What it does You type a sentence in a web GUI. A Franka Panda arm in NVIDIA Isaac Sim (real PhysX physics: cubes fall, slip and stack) picks and places coloured cubes into a tray, on each other, or beside each other. Each move is checked by safety rules, executed, and then measured against the cube's real final position (3 cm sideways, 3.5 cm vertically). If a move fails, the agent plans again from the live scene. A prompt naming a cube that is not on the table is refused and nothing moves.

How we built it

Nemotron (30B-class, nemotron-3.5-lightning-30b-a3b) reads the sentence, splits it into goals once, then proposes one move per turn against the live scene. Thinking is off and the answer is streamed, so a move takes about 1 s.
Code, not the model, decides when a goal is reached. When the model tracked its own progress it looped (14 moves, no result). With code tracking it took 7 calls and about 9 s.
A deterministic stack planner handles "stack the cubes" exactly from positions, with no AI. It is tested on 12,000 random layouts with 0 failures.
Isaac Sim 6.0.1 with NVIDIA's PickPlaceController and RMPFlow does the 10-phase grasp.
Camera-only perception** (OpenCV colour + depth, about 3 mm mean error, no neural network) can replace the simulator's exact positions.
NVIDIA Cosmos looks at a camera frame before and after each step and gives a second opinion in a background thread. It never gates the arm.
The Nemotron client speaks the OpenAI protocol, so Nebius Token Factory is a configuration change (NEMOTRON_BASE_URL, key, model id).
Backend and GUI: Python, FastAPI, a polling HTML page, MJPEG camera streams.