-
-
Attempt 1 drops the can, the repair stands it upright, with the real-world facts beside it.
-
Three repair candidates tested in the simulator; the sturdiest runs.
-
Real sizes from the web, the model's own guess under each, 4/4 sources agreeing.
-
16 perturbed re-runs as tiles, and a learned policy's report card.
-
Refused before building: a grapefruit 8.2 cm wide against an 8 cm gripper.
Inspiration
Testing a robot skill in simulation starts with a scene, and someone has to build it by hand: measure the objects, write the scene file, place everything where the arm can reach, write the success checks, then debug why the cube fell off. That's days of work per scene, before a single policy is tested. I wanted to type the task and get a working, checked scene — one that tells you whether a plan really works or just got lucky.
What it does
You type a task, e.g. "Put the soda can to the left of the Rubik's cube and keep it upright."
- Plan: NVIDIA Nemotron 3 Super turns it into objects and goals that code can check.
- Ground: Tavily finds each object's real size and mass, cross-checked across web pages (read in full with
/extractwhen search snippets aren't enough), friction and bounciness from engineering tables, and the Franka gripper's limits from its datasheet. Tasks the gripper physically can't do are refused before anything is built, with sources. - Build: Super places the objects and writes the plan as skills (pick, place, push). Our own controller turns them into motion — the model never writes code or robot XML.
- Simulate and check: MuJoCo runs a Franka Panda (MuJoCo Menagerie). Code checks every goal; a model never decides pass or fail.
- Repair: if a goal fails, Nemotron 3.5 Super VL explains what went wrong from the frames, Super writes three repair candidatesor tests each in seconds, and thesturdiest runs next.
- Stress-test: every success is re-run 16 times with objects moved, heavier, more slippery,and the robot's view slightly off.
- Use it: export the scene with a policy test harness (
evaluate.pyscores your own policyrandomized versions of the scene), o training data.
Everything streams live in the web app — which model ran each step, its time, tokens and cost — with a 3D replay of every attempt in/after view of the failure and thefix. You can drag objects or edit sizes and masses and re-run the same plan to try to break it.
How I built it
- NVIDIA Nemotron on Nebius Token Factory: Nemotron 3 Super (planning, building, repair), Nemotron 3 Nano (extracting numbers from web pages), Nemotron 3.5 Super VL (looking at the sceneand at failed runs). One OpenAI-compatible client, strict JSON-schema output validated with Pydantic, reasoning switched on or off per step, and every call's cost logged and shown in the UI.
- Tavily for every real-world number: advanced search,
/extractfor full pages, quotes thatmust appear word for word in what Tavily returned, a context check (a friction value must be about this material on a dry wooden table), cross-source agreement, and size-vs-weight sanity checks. - MuJoCo with the Franka Panda from MuJoCo Menagerie, my own IK controller and skill library,goal predicates modelled on NVIDIA RoboLab's vocabulary, and a three.js viewer that replays the physics in the browser.
- LangSmith traces every call (public trace links in the demo). FastAPI streams each run live; the demo runs on a free Hugging Face Space.
- Evaluation: 15 held-out prompts frozen before any run, plus 11 development prompts, 3 runs each — 104 runs per round, with 95 % confidence intervals.
Challenges
- Honest grasping. Physical grasping in simulation is fragile, so a weld holds an object once it is truly between the fingers. Reviewers found an early version "grasped" a cube the fingers only touched from above; I fixed it, turned the probe into a test, and now every stress test is alsoreported with the assist off.
- Bad web data. One farm guide's 8.9 cm apple could refuse a whole task; a soda can came back measured lying down. I added median voting across sources, verified agreement votes, orientationchecks and a size-vs-weight test.
- Measuring what actually helps. My own eval showed the vision Critic made no measurable difference to repair success, and escalating to the 550B Ultra fixed only 4 of the 10 runs sent to it. I replaced Ultra with three cheap Super repairs checked by the simulator — as good, for less —and say plainly that the vision model is there to explain, not to win.
- Deployment. My Nebius AI Cloud account returned PermissionDenied for registries andendpoints, so the demo runs on Hugging Face while every model call still goes through Token Factory.
Accomplishments
- 42 of 45 runs pass on 15 held-out prompts, at a median model cost of about half a cent per task.
- Test-time compute plus a verifier: three parallel Super repairs, judged by physics, matched a model four times larger.
- Reasoning pays for itself: with Super's reasoning off, success fell from 11/11 to 6/11, and tasks cost twice as much because of extra repairs.
- A policy test you can take home: the exported harness needs only MuJoCo and NumPy. A small policy trained on our generated demos matches the scripted plan on a simple stack but drops to 4/16 placing a tennis ball — exactly the weakness the test is meant to find.
- Every claim checked: five rounds of independent reviews; every number they couldn't verifywas corrected.
What I learned
- A cheap model plus a fast, exact verifier beats a bigger model guessing alone.
- Never let a model decide pass or fail; let it explain.
- Web grounding is only as good as its checks: quotes must exist, sources must agree, the context must match.
- Measure every component. Two of mine didn't earn their place, and the data said so.
What's next
- Physical grasping without the weld; more robots from MuJoCo Menagerie (UR5e first) and sidegrasps.
- Mesh objects with sizes from the web instead of primitive shapes.
- Export to Isaac Lab / USD, and a regression suite that scores a policy on every saved task in CI.
- A Nebius Serverless Job that generates thousands of verified demonstrations overnight.
