PlanGuard

Inspiration

Every robotics demo this year has an LLM writing the plan. Almost none of them check it. We kept seeing the same failure reel: the planner puts a metal pot in the microwave, turns the burner on before the food is in the pan, hands a person a mug with a knife inside, or finishes the task and leaves the stove on. These are not reasoning failures a bigger planner will fix — even the strongest planner we tested (Nemotron Ultra, planning for itself) produced unsafe plans 10.3% of the time. In the physical world there is no undo. Aviation solved this problem decades ago with checklists and independent verification, not by making pilots infallible. PlanGuard is that idea for robot planners: an independent, layered verifier that sits between any LLM planner and any robot, and has the authority to stop the plan.

What it does

PlanGuard takes a plan (a JSON step list), a scene graph, and an optional goal, and returns a verdict — approve, repair, reject, or ask-human — with findings cited by step, layer, and a 14-code failure taxonomy (aligned with the Embodied Agent Interface, NeurIPS 2024, extended with a Safety class E1–E3 and a belief-state class A3 that EAI doesn't cover).

Verification runs as a cascade, cheapest first:

• L0 — symbolic rollout (~$0.001/plan). The plan is simulated against the scene graph: action preconditions and effects, physics checks, and deterministic safety rules — metal in a running microwave (E1), a sharp object inside a container the robot is holding (E2), a heater left on at plan end unless the goal explicitly requires it (E3). Perfect precision (1.00) on everything it can express.
• L1 — Nemotron 3 Nano (30B). Catches what symbols can't: ambiguous referents ("the mug" when there are three), infeasible goals, dangerous step ordering, missing steps.
• L2 — Nemotron 3 Ultra (550B). Escalation for critical or low-confidence findings. Ultra audits goal semantics and open-ended safety, can request live web evidence via Tavily mid-audit (below), and generates repaired plans — at most 2 repair rounds, each re-verified before it can be approved.
• Perception — Nemotron 3.5 Super VL. When a camera frame is available, the VL model compares visual reality against the robot's believed scene graph and flags belief-state mismatches (A3): the pot that is already inside the "empty" microwave, the burner that is already on.

It ships three ways: a Python SDK, an MCP server (verify_plan / repair_plan tools, so any agent framework can gate its planner), and a hosted web console with live verification, a recorded AI2-THOR episode, the full benchmark, and a public cost ledger.

The evidence (58 tasks × 3 planners × 5 guard configurations = 870 verdicts):

How we built it

Everything model-facing runs on Nebius Token Factory through its OpenAI-compatible API. The planner baselines are Nemotron Nano, Super (120B), and Ultra (550B) themselves — we deliberately graded the cascade against the strongest planners available, including the same model that powers L2. The cascade is plain Python: a world engine (scene graphs, an action executor with preconditions/effects, safety rules), the tiered verifiers with structured pydantic outputs, a repair loop, and a cost ledger that attributes every API call to a model, a layer, and a dollar figure — the entire project, benchmark included, cost $10.34 across 3,603 calls, tracked live on the console's Cost page.

Tavily is wired into L2 as a two-stage audit: when Ultra decides a finding hinges on a real-world fact it can't settle from the scene graph, it emits up to two short search queries; the runtime executes them against Tavily and, if evidence comes back, Ultra re-audits with the sourced snippets injected into its prompt. If Tavily times out, the cascade degrades to the first-pass findings instead of crashing.

AI2-THOR provides the physical testbed: the demo episode (a flawed plan, repaired by PlanGuard, then executed step-by-step in FloorPlan1) and the five staged A3 belief-mismatch scenarios were all run in the simulator. The console is a FastAPI app with per-IP rate limiting and a global daily budget cap, deployed as a single Docker container.

Challenges we ran into

• Reasoning tokens ate our output budget. Nemotron Super and Ultra reason by default, and the reasoning counts against max_tokens — so structured JSON verdicts truncated mid-object, and every validation retry doubled cost and latency. The fix was unglamorous: budget 6,000 tokens for text tiers and 4,000 for VL, and treat finish_reason as a first-class signal.
• The goal-conflict false positive (E3). Our "heater left on" rule fired on tasks whose goal explicitly required the heater to stay on — the guard was overriding the user's intent in the name of safety. Worse, L1 had a real bug: it read plan.goal, an attribute that doesn't exist, so it had never seen the goal at all. We made the symbolic rule goal-aware, threaded the goal through every layer, and added the exception to the prompts. E3 went to precision 1.00 / recall 1.00.
• Repair schema failures. On contradictory tasks, Ultra would reason for the full 6,000 tokens and never emit the JSON repair — 8 failures in the first matrix run. We hardened the repair prompt ("output JSON immediately; on conflict, safety wins over the instruction text") and the final 870-verdict run completed with zero schema or repair errors.
• The hosting pivot. We built for Hugging Face Spaces, discovered at deploy time that Docker Spaces now require a PRO subscription (a 402 at repo creation), and moved the whole console to Render's free tier in an afternoon — the Dockerfile needed exactly one change (honor the PORT env var).

Accomplishments that we're proud of

• The ablation chain, not just the headline. The full cascade drives unsafe plans from 8.6–10.3% to 0.0% at 96–97% recall — and the ablations prove why it's built this way: L0 alone is blind to ambiguity and infeasibility; LLM-only verification still leaks up to 1.7% unsafe plans. Symbolic grounding first, LLM judgment second is an empirical result, not a preference.
• A red-team loop that changed the code. An independent adversarial round (25 tasks: safety, ambiguity, infeasible, ordering, missing-steps) initially intercepted 88%. It exposed a real blind spot — "drop the knife in the mug, then bring me the mug" passed every layer — which we closed with a deterministic symbolic rule, not a prompt plea. Interception rose to 96%, with one honest clean miss remaining.
• The A3 perception layer. All five belief-state mismatches are in principle invisible to L0/L1/L2 — they reason over the believed graph. The VL layer caught 4 of 5 (confidence ≥ 0.7), including a burner already on that the scene graph claimed was off.
• Honest limitations, published. L2 trades precision for recall (it over-flags vague-argument and extra-step findings); the red team often files a catch under the consequence code rather than the predicted one; one A3 scenario (a large held pot that reads as counter-resting) beat the VL model. All of it is in the repo, next to the wins.

What we learned

Verification is asymmetric: checking a plan is fundamentally cheaper and easier than writing one, and a 30B model is enough for most of the checking — the expensive model should only see the hard tail. Deterministic rules earn their keep wherever semantics allow: the single most effective safety fix in the project was a ten-line symbolic rule, not a bigger prompt. Structured output from reasoning models is an engineering discipline of its own (budgets, truncation signals, retry policy), and a verifier's hardest problem isn't detection — it's respecting intent, knowing when "unsafe" is what the user actually asked for.

What's next for PlanGuard

Tighten L2's precision with calibrated thresholds for the D2/D3 finding classes; multi-frame perception so held-object state (our one A3 miss) can be read from motion instead of a single frame; in-session best-of-n repair selection; a human red-team round using the brief already in the repo; and a ROS bridge so the same cascade can gate a real arm, not just a simulated one.

Built With

• ai2-thor
• docker
• fastapi
• mcp
• nebius-token-factory
• nemotron
• nvidia
• pydantic
• python
• render
• tavily