Inspiration

Fine-tuned LoRA adapters from model repositories such as Hugging Face are a fast way to ship and integrate open-weight models, but they can hide sleeper backdoors that only activate upon encountering a secret trigger phrase. Research like Anthropic's Sleeper Agents has shown that this phenomenon is quite possible, and an NVIDIA mentor at their table yesterday told me this was something that was actively being researched by the company. I find open-models crucial to my workflow for when I need my data to be isolated on machines that I have full control over. To that end, I was curious to see if a team of Nemotron agents could audit such an adapter before deployment and back its decision with evidence. Essentially, the idea was for teams at NVIDIA to use trusted and vetted open-weight NVIDIA models to guarantee the behavior of third-party models before integrating them into the workflow.

What it does

TripWir3 audits a target third-party base model with an adapter attached to it, at a chosen risk tier (low, standard, or critical), using provenance checks, LoRA weight forensics, and paired trigger tests. The audit is done by the Investigator, Skeptic, and Report agents running NVIDIA Nemotron™ that pass their respective verdicts grounded in the output logs of the model. Every audit ends in QUARANTINE, FLAG, PASS, or INCOMPLETE, and any and all failures are marked as INCOMPLETE. In a reviewed end-to-end audit of a small test adapter on Qwen2.5-0.5B with one known trigger, it returned QUARANTINE in about 3 minutes: the backdoor's marker appeared on all 12/12 trigger prompts and on 0/20 clean prompts.

How we built it

A Nemotron 3.5 Super orchestrator on the NVIDIA API plans and does the tool calling, while the Investigator, Skeptic, and Report agents run locally on Nemotron 3.5 Lightning 30B-A3B through LM Studio. A FastAPI backend streams every step to an observation dashboard, and every tool call is logged so all the claims are grounded in the log outputs.

Challenges we ran into

My first three full audits ended INCOMPLETE because of a planning cap, token output limits, and schema mismatch. Agents also sometimes got the numbers right but told the wrong story, so I fixed that by coupling the report's key facts directly to the evidence log. Additionally, I had trouble with the slow spinning up and termination of compute instances on Brev, which limited the scale of my project.

Accomplishments that we're proud of

A multi-agent Nemotron pipeline that reached a deployment decision with inspectable evidence and verifiable reports. I am also excited by the fact that I'll be able to use this framework for my own research workflows, and I'm sure many practitioners and hobbyists will find it useful too!

What we learned

An agent can get its facts straight and still spin up a false story, so it has to be grounded at the code-level. Multi-agent pipelines break at the seams, and it is important to closely monitor at these points to identify where the workflow is failing.

What's next for TripWir3

I also trained 12 test adapters on Qwen2.5-14B-Instruct on an NVIDIA Brev H100 instance which seems to work (since it flagged 42/96 trigger prompts and didn't flag 8/176 matched no-trigger prompts) but haven't been audited yet due to the aforementioned time-constraints.

I also trained 12 test adapters on Qwen2.5-14B-Instruct on an NVIDIA Brev H100 instance, and their backdoors work: in a direct check, the marker appeared on 42/96 trigger prompts and only 8/176 matched no-trigger prompts. I wasn't able to fully audit them in TripWir3 due to time constraints. It would be interesting to run an audit on these and see how TripWir3 handles scale. In the long run, it could manifest as a CI or registry gate that checks adapters before being deployed into production, alongside runtime guardrails like NeMo Guardrails.

Built With

Share this project:

Updates

Submission history