NemoGate

Does your agent team forget the rules you give it? NemoGate finds where a rule dies between agents, proves it with forked re-runs, and carries it across.

Comment

Inspiration

We were looking for gaps in multi-agent systems for an open source project, and the handoff between agents looked like the weak link. A planner agent receives a rule from the user, restates the task in its own words for a subagent, and the subagent only ever sees that restatement. If the rule gets dropped or softened on the way, nothing errors. The work just comes out wrong.

So we decided to test whether handoffs really are a problem. With Nemotron 3 Super as the agent team, they are. LangChain Deep Agents is being used for more and more real work, and we could not find a simple way to measure this failure or to stop it. NemoGate is our attempt at both.

What it does

A user says "don't touch migrations/". NemoGate records every handoff and tool call, checks each rule with a deterministic oracle (programs over the trace and the file diff, never a language model), and compares four arms: natural, mutated (the handoff is sabotaged), block-only (forbidden actions are refused) and gated (refused, and the rule is put back into the handoff).

It then reports, for every rule, whether the agents did the job without breaking the rule, with exact tests and intervals, and says what the numbers cannot tell you.

How we built it

  • Nebius Token Factory and NVIDIA Nemotron: three Nemotron 3 models on one endpoint, each with one job: Lightning as the helper, Super as the agent team under test, and Ultra as the planner for repair.
  • Nebius Sandboxes: every run happens in a fresh sandbox. A run is recorded, then forked at the handoff, so every arm sees the same history and only the handoff differs.
  • A gate (one Deep Agents middleware) that observes, sabotages, blocks or re-pins the rule, a typed ledger of rules, a results page, and a repair agent that proposes a fix.
  • More than 2,400 offline tests, and an audit anyone can run with no API key: uv run python -m replay.audit examples.

Challenges we ran into

  • Honest statistics. Small samples, so exact tests and intervals, and a "not significant" wherever that is the truth (for example, finishing the task: 10 of 14 against 7 of 14, \( p = 0.44 \)).
  • Zeros that are built in. Blocking gives a 0 by construction, so a 0 is not always a result. On the second app the gate also checks each reply, so that 0 is partly by construction too, and we say so.
  • Platform limits. 21 of the 160 runs on the second app errored (20 "System is overloaded" responses and one timeout). They are listed per arm and counted in no rate.
  • Keeping the test honest. Two held-out tasks were written before the pipeline and sealed until the final sweep, so we could not tune to them.

Accomplishments that we're proud of

  • Handoff loss, measured. With no sabotage the agents broke the rule by themselves in 4 of 15 runs on the first task. When the handoff drops the rule, they broke it in 9 of 14.
  • What carrying the rule adds on top of blocking. Fewer attempts at the forbidden action: 11 of 14 runs for block-only against 4 of 14 for gated (Fisher exact test, \( p = 0.0213 \)).
  • A second app. On a research team with web search (rule: don't cite Wikipedia), the rule was broken in 10 of 32 runs with block-only and 0 of 33 gated (\( p = 0.0078 \) on 29 matched pairs).
  • Checkable by anyone. The 300 runs behind the main numbers are in a public release, and the page's numbers can be rebuilt from them with no key.
  • Reporting what did not work. The repair agent barely moved the numbers (45% to 44% on one rule), and the page says so.

What we learned

  • Handoff loss is real and measurable, and a rule that arrives is not always obeyed.
  • A zero is not always a result. Much of the work went into saying which zeros are measured and which are built in.
  • A result that does not hold up is worth reporting.

What's next for NemoGate

This is a proof of concept: one reference app, one research app, and one model family, so the results are small samples. We intend to develop NemoGate into a proper open source project: more rule types, more handoff failures to test, other model families, and an easy way to run it on your own Deep Agents team. Our goal for the next version is zero forbidden actions from the agent when the rule is gated. Today, gated agents still tried the forbidden action in 4 of 14 runs on the first task (block-only: 11 of 14). The gate refuses those calls before they happen, but we want the agent to stop trying, so that carrying the rule across the handoff is enough on its own.

Try it: the results page and the code.

Built With

  • contree
  • deepagents
  • langchain
  • nebius
  • nemotron
  • pydantic
  • python
  • tavily
Share this project:

Updates

Submission history