Triage: Autonomous VLA Bug Repair

Triage detects if robot simulation failures stem from code, data, or the simulator via deterministic checks. It then uses Nemotron models on Nebius Token Factory to fix code bugs for pennies.

Comment

Inspiration

When a robot policy fails in simulation, the first question is the hardest: what broke? It could be a one-line bug in the client code (a flipped image, a wrong normalisation file), a gap in the training data (the policy never saw this lighting), or the simulator itself. Each needs a completely different fix, and guessing wrong is expensive: a fine-tuning run spent on a one-line bug, or an afternoon of patching code when the data was the problem. I wanted a system that answers "what broke" before it spends money on a fix.

What it does

Triage has two parts:

  1. A deterministic router. It runs contract checks on the policy's real inputs and outputs and feeds the results into a fixed decision table. The checks look at image orientation, channel order and value range; config against the training spec; and action delay. No model decides the route. A test enforces that the decision code imports no model layer. If evidence is missing, the router says undetermined and names what is missing, instead of guessing.
  2. hagent, a two-model repair agent for code bugs. It runs on Nebius Token Factory: Nemotron-3_5-Lightning drives almost every step, and Nemotron-3-Ultra-550b-a55b is consulted rarely, for planning, when the agent is stuck, and for the final review. It has hard caps on steps, cost and big-model calls.

The policy under test is NVIDIA GR00T N1.6 in Isaac Lab Arena (GR1 robot, put an item in the fridge and close the door).

How I built it

  • Router and verification ladder (Python, about 835 tests against mocks): Stage 1 checks, counterfactual replays, the decision table, and Rung 0/1/2 verification of patches.
  • hagent: a bash-only coding agent with no dependencies beyond the standard library. It talks to Token Factory's OpenAI-compatible endpoint and has a loop detector and history compaction.
  • The simulation side: an Isaac Lab Arena backend, instrumentation on a GR00T client wrapper, and a ledger of 14 injected bugs, each with a patch, a symptom and the check that should catch it.
  • A real GPU session on an RTX PRO 6000 Blackwell, which was a Spot VM preempted several times. Over it, I ran clean episodes, injected bugs, the router checks on real frames, automatic repair and a lighting sweep.

Success rates use Wilson intervals and Fisher tests. Repair success uses the unbiased pass@k estimator:

$$\text{pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

Challenges I ran into

  • Blackwell (sm_120): GR00T's shipped torch had no kernels for the GPU, so I had to install torch 2.7.1+cu128. Isaac Sim crashed at camera init until I installed the NVIDIA Vulkan driver package.
  • Spot preemptions wiped running jobs and /tmp several times. I rebuilt the bugged repos deterministically from the ledger and bundled results to my laptop between runs.
  • Isaac Lab Mimic would not start (a render-graph error on both CPU and GPU). That blocked the reference controller my router needs to rule out simulator faults.
  • Honest scoring: my first scoring of one bug was wrong, and I had to rescore it with the router's own config check.

Accomplishments that I'm proud of

  • Clean policy: 32/40. With the 180° image-rotation bug: 0/30. After hagent's fix: 28/40, within noise of clean; the fixed repo is byte-identical to the clean one.
  • The router flagged 6 of 7 injected bugs on real data, including 4 that barely change success, which a success-rate monitor would miss. It had 0 of 3 false positives on held-out clean runs.
  • Automatic repair with no hand-written hint: 4/5 attempts on the rotation bug (about $0.09 per attempt), 2/5 on a normalisation-stats bug, and 1/5 on a bug given only as a symptom.
  • I found a real vision failure with no code change: dimming the dome light took success from 17/20 to 0/20.
  • The router refused to guess. Without the reference replay it returned undetermined, exactly as designed.

What I learned

  • Detection is not routing. The observation check also fires on the dark-lighting scene, so contract evidence alone cannot tell a vision gap from a code bug. Only a counterfactual replay can, which is why the design has them.
  • Localization matters more than model size. hagent fixed located bugs far more often than the unlocated one. On my SWE-rebench-V2 benchmark, adding the big model did not beat the small one (13/39 vs 14/39).
  • Small samples make it easy to fool yourself, so I report per-attempt rates and intervals, not best cases.

What's next for Triage: Autonomous VLA Bug Repair

After my end-semester exams:

  1. Get a reference controller working, either by fixing Mimic or using the stock Arena client as a stand-in. Then run the one-factor-restore replay, so the router can actually separate the dark-lighting case from a code bug.
  2. Run the full incident loop end to end on the simulator, with Rung 1 as a selector over repair attempts.
  3. Calibrate thresholds on no-fault runs, and run the remaining ledger bugs with more episodes.
  4. Close the vision half: a short fine-tune on the dark-lighting failure and a held-out evaluation, compared with a fixed fine-tuning recipe on cost.

Built With

Share this project:

Updates

Submission history