Agent Trap

• 

  Agent Trap

• 

  Architecture

Inspiration

Developers now hand new repositories to AI coding agents with a sentence like "set this up and run the tests". That is convenient, and it means whatever is in the repository gets run, often by something that does not stop to read it first.

Two kinds of attack already aim at this. The first is code that steals as soon as it is installed. In October 2026 a backdoored release of LiteLLM sat on PyPI for about two hours and collected SSH keys, cloud credentials and AI provider keys from everyone who installed it. It was the second LiteLLM release this year to ship credential-stealing code. elementary-data, with over a million downloads a month, was hijacked in April. In 2025, malicious nx releases leaked over a thousand GitHub tokens, and were the first known malware to use the victims' own AI command-line tools to look for more secrets.

The second kind is text written for the agent. A malicious GitHub issue has been used to make an agent leak private repositories, and hidden instructions in the rules files for Cursor and GitHub Copilot have made the AI insert malicious code.

Reading the code does not reliably catch either. The stealing code is written to look like telemetry. The injected instruction is plain English, so there is no code to flag. The agent will not catch it either, because it is the one being fooled and it never sees what leaves the machine. So we stopped judging how code looks and decided to watch what it does when it runs, with an agent doing the work.

Sources: StepSecurity on LiteLLM, Snyk on elementary-data, Wiz on nx, Invariant Labs on the GitHub issue attack, The Hacker News on the rules file attack.

What it does

Agent Trap takes a GitHub link, a folder, or a pull request (through a GitHub Action) and runs the repository in a locked Docker sandbox. The sandbox has no internet and is filled with fake secrets: AWS, SSH, GitHub, npm and AI provider keys, plus a fake machine name and user name, all random for each run.

First Agent Trap runs the usual developer steps itself, with no model involved: it installs the project, imports every module and runs its tests. Code that steals when it is installed, imported or tested is caught here. Then an NVIDIA Nemotron coding agent is asked to set it up, run its tests and summarize it, the way a developer would use an agent on new code. All network traffic from the sandbox goes to a proxy that reads HTTP and HTTPS and to a fake DNS server that records every lookup. Nothing is forwarded.

If one of the fake secrets shows up in that traffic, the repository is marked MALICIOUS. We look for it as plain text, as base64, hex or a hash, inside a compressed archive, and inside host names. If nothing leaks but something looks wrong (the agent reports that text in the repo tried to instruct it, or an unknown server was contacted), the verdict is SUSPICIOUS. Otherwise it is SAFE. The result shows the exact request that was caught and the lines that caused it.

To find those lines, Agent Trap replays the run with lines removed (delta debugging) until only the lines needed for the leak are left. When the agent itself was tricked by text, it asks Nemotron the same question again with parts of the repo hidden, up to 12 times, and counts how often the model still goes for the key. That narrows it down to the sentence that fooled it. Then it removes those lines and scans the copy again as a check.

It comes as a command line tool, a local web page that streams the scan, and a GitHub Action that fails a pull request and marks the stealing lines.

What it does not do: it only handles Python. The sandbox is a container, not a virtual machine, so real malware should be scanned on a throwaway machine. Malware that waits for a date, another operating system or a real internet connection will not show itself, so SAFE means nothing was seen leaving, not that the code is harmless.

How we built it

Nemotron 3 Nano on Nebius Token Factory is the agent in the trap. It runs in a loop where each step is one JSON action (read a file, run a command, write a file, make a request), and every action is carried out inside the sandbox. After each task we ask it whether anything it read tried to instruct it, and we only accept a quote if we can find it in the repository. Its answers are shown as warnings and never decide the verdict. We use the OpenAI-compatible API, so one small client covers everything, and --model super or --model ultra swaps in a larger Nemotron. Every result below was measured with Nano.

Finding the sentence that tricked the agent needs many calls to the same model, which is why fast, inexpensive calls on Token Factory matter here. We made more than 20,000 model calls while building and evaluating this.

The trap is Docker with all capabilities dropped, no-new-privileges, and CPU, memory and process limits, on a per-run network with no route out. HTTPS is read with mitmproxy, using a certificate authority that only the sandbox trusts. A small DNS server answers every name and logs it.

Tavily is called during every scan, after the sandbox run. We search for the exact package name and version, to see whether it has already been reported as compromised, and for each server the sandbox contacted we search for what the web says about it. For LiteLLM 1.107.0 this finds the StepSecurity and Datadog write-ups. For elementary-data, Tavily returned the heise and Snyk reports and the library's own incident report for the server our fake keys were sent to, and the domain turned out to be registered a month before the hijack. Only host names, or a package name and version, are sent to Tavily, never code or secrets. The results are shown next to the verdict and never change it.

It is written in Python, with a FastAPI web page, a GitHub Action with SARIF output, and 466 tests. All of it was built after the hackathon started; the first commit is from 6 October 2026.

Challenges we ran into

Nemotron would not take the bait. Our first test hid simple instructions in a README, and Nano and Super ignored them in all 18 runs. That is good for the model and bad for a trap. The published AgentDojo attack texts do trick it, but not every time. So each task runs more than once, and we never trust the agent to say whether it was tricked.

Finding the guilty sentence needed counting, not one try, because the model does not answer the same way twice. Hiding every code comment to isolate the injected text changed the model's behavior completely: a leak that happened in 19 of 20 samples dropped to 0 of 20. So we only hide the remarks that name the secret or where it goes.

Real malware broke assumptions our own test repos never touched. Some stealers send the machine's name and user name instead of a key file. Some hide data in DNS names. One compressed everything it sent into an archive, and another was an already-installed package with no setup.py, whose .pth file runs whenever Python starts. We added support for each.

Proving the exact lines is slow, because it means hundreds of re-runs. On real malware it took from a few minutes to a few hours. A normal scan stops after a few minutes and shows the lines it could not prove as "likely". The --prove option runs the search to the end.

Accomplishments that we're proud of

We caught elementary-data 0.23.3, the hijacked library. In the trap it collected all 7 planted secrets, packed them into a .tar.gz and sent it out. Agent Trap caught the archive, unpacked it, found every secret and proved the one line responsible, in about two minutes.

We caught 5 real malware packages stealing, with the lines proven. Four came from Datadog's dataset of malicious PyPI packages, where we ran 11 likely credential stealers. Of the other 7, two download their real payload from the internet, which the sandbox cannot reach, four contain stealing code that never ran during install, import or tests, and one did not install. Those 7 were scanned before we added the machine-name and DNS checks and have not been rescanned. The 11 were hand-picked, so this is not a detection rate.

On our own 11 attack repos, Agent Trap flagged all 11. Bandit flagged 6 and Semgrep flagged 4, using their standard rules, and neither found any of the 5 README attacks, because those contain no code. Bandit also flagged 3 of our 8 harmless repos. These repos are ours, so this is a demo set and not a benchmark.

On 20 well-known open-source projects (click, attrs, tqdm, bottle and 16 more), none was flagged, and none came back with any warning. Only 8 of them could run their tests, because the sandbox has no internet for dependencies, so this means no false alarms on the code that ran.

We are glad to have numbers for what the AI agent adds. The install-time stealers fire without it. For the attacks that exist because of agents, it flagged 5 of our 6 instruction attacks as aimed at it, and it was actually tricked into sending the key in 2 of 5 README attacks (in 2 of 5 scans each). On a repo it fell for, Nemotron reached for the key in 19 of 20 sampled answers and the search named the exact sentence, README line 17.

Finally, the GitHub Action works on a real pull request in our repository: it fails the check and puts annotations on the stealing lines.

What we learned

A verdict should rest on what was seen leaving, not on what a model says about the code. The agent's opinion is a useful hint and nothing more. Small open models are a good fit when a method needs many repeated calls. And real attacks break assumptions that your own test cases never will, so we should have tested against real malware earlier.

What's next for Agent Trap

Support for npm and PHP. This year the same install-time stealing hit hundreds of npm packages (the Shai-Hulud worms) and PHP packages (Laravel-Lang and intercom-php). The trap itself works for any language; what changes is how a project is installed and tested.

Bigger evaluations, with hundreds of malicious and normal packages and the run-to-run variance published, since an agent does not behave the same way every time. Longer scans with the sandbox clock moved forward, for malware that waits. And a hosted version on Nebius AI Cloud, so a team can paste a link and get a verdict without setting anything up.

Built With

• ai-agents
• cybersecurity
• dns
• docker
• fastapi
• github
• github-actions
• html5
• javascript
• mitmproxy
• nebius
• nebius-token-factory
• nemotron
• nvidia-nemotron
• openai-api
• prompt-injection
• pytest
• python
• sarif
• supply-chain-security
• tavily