Inspiration
Every team shipping an LLM feature eventually asks: can a user talk it into leaking what it's guarding? Today the answer usually comes from a fixed list of attack prompts. But every model fails differently. Run the same list against a new model and you re-find the same three failures, and the new weaknesses are never tested.
We wanted a red team that learns what this particular model is weak to, proves every failure instead of trusting a guess, and then fixes and verifies, so the loop ends with a safer system rather than a report.
What it does
The Adaptive Red-Team Agent attacks a model behind your real system prompt and gives you evidence you can act on:
- Learns what to attack. A Thompson-sampling bandit chooses the attack family each step and shifts budget toward what this target actually falls for. With some probability it invents a new attack family, which becomes a policy arm only after it causes a new, reproduced failure.
- Writes each attack with NVIDIA Nemotron, from the target's replies and the attacks that already worked: 1 to 3 turns of strict JSON.
- Proves every break. A mechanical detector recovers the secret in any form (spaced, reversed, homoglyphs, base64, hex, ROT13, fragments). A Nemotron judge may only flag subtler breaches if it quotes the response verbatim. Every hit is replayed before it counts, and anything uncertain fails closed. The tester never grades itself.
- Fixes and verifies. Nemotron 3 Ultra reads how each leak happened and rewrites the system prompt. Every confirmed failure is then replayed 3× against the fix. If something still leaks, "harden again" iterates on just those modes. If no prompt fixes it, the failure stays an open finding.
- Operates like a security product. Findings map to the OWASP LLM Top 10 and MITRE ATLAS, export as SARIF or a JSONL dataset for guardrail training, transfer between models, and re-test on a schedule. There are roles, an audit log, and an MCP server so other agents can red-team themselves.
How we built it
Everything runs on Nebius Token Factory with one key and the stock OpenAI client. Each Nemotron is sized to its job:
| Role | Model | Why |
|---|---|---|
| Writes every attack | Nemotron 3 Super / 3.5 Lightning | High volume, 8 parallel workers |
| Judges every response | Nemotron 3.5 Lightning | Bounded task, must quote evidence |
| Invents attack families, hardens prompts | Nemotron 3 Ultra | Rare, high-stakes reasoning |
| Models under test | Gemma 3 27B, Qwen3 30B, Qwen3.5-397B, GLM-5.3, gpt-oss-120b | Swappable per run |
Each attack family \( f \) keeps a Beta posterior. Each step the bandit samples \( \theta_f \sim \mathrm{Beta}(\alpha_f, \beta_f) \), attacks with the highest draw, and updates fractionally with the shaped reward \( r \in [0,1] \): \( \alpha_f \leftarrow \gamma\alpha_f + r,\ \beta_f \leftarrow \gamma\beta_f + (1-r) \), where \( \gamma = 0.99 \) so mined-out families lose priority. The reward pays for useful information, not raw hit rate:
$$ R = \underbrace{1.0 \cdot \text{success} \cdot \text{severity}}_{\text{high }1.0,\ \text{med }0.7,\ \text{low }0.4} + 0.5 \cdot \text{novelty} + 0.3 \cdot \text{reproduced} - 0.01 \cdot \tfrac{\text{tokens}}{1000} - 0.4 \cdot \text{duplication} $$
Novelty is measured on failure modes (mechanism × leak form), so a paraphrase of a known exploit isn't new.
Stack: Python + FastAPI backend with SQLite, a Next.js console that streams every attack over SSE, Docker Compose, about 300 offline tests in CI, and an MCP server over stdio. The launch video was built in code: real console footage captured with Playwright, motion-anything scenes, Onyx narration, and a soundtrack synthesized in numpy.
What we found
- Same tester, same 40-call budget, five models (90 live runs, about 3,250 target calls): Gemma 3 27B leaked in every run, Qwen3 30B one or two times per run, and Qwen3.5-397B, GLM-5.3 and gpt-oss-120b essentially never.
- Most leaks were self-disclosure: 139 of Gemma's 142 confirmed leaks were the model printing the secret inside its own refusal.
- The right model for the fix matters. On held-out attacks the hardener never saw, Nemotron 3 Ultra's fix cut Gemma's leak rate from 80% to 0% (Qwen3 30B: 53% to 0%). Ultra was the only tier to return a usable fix 3/3 times; Super managed 0/3 and Lightning 1/3.
- Offline, over 50 seeds, the adaptive policy found more confirmed failures than random selection within the same budget: +1.74, 95% CI [+0.74, +2.78].
Challenges we ran into
- Fixes that kept failing. Our first hardener (the tester model) mostly returned prompts that weren't even usable, and the usable ones still leaked. The breakthrough came from an audit of our own results: almost every leak happened inside a refusal. Showing the hardener how each leak happened, and moving the fix to Nemotron 3 Ultra, took held-out leaks to zero.
- Not trusting an LLM's opinion. A judge that says "breached" with nothing to back it up is noise in the reward. We made the detector carry most verdicts and required the judge to quote the response verbatim with confidence ≥ 0.7, failing closed otherwise. Only 3 of 2,063 live interactions ended uncertain.
- Latency. Nemotron's reasoning trace is on by default: a judge verdict took 16 to 46 s. With thinking off for the high-volume roles it takes about 2 s, which made parallel live runs practical.
- Honesty about what didn't separate. At budget 40 the three policies don't separate on live targets; the adaptive advantage is proven offline at 50 seeds. We report both.
- Some failures can't be patched. In our recorded demo, the tester invented a "collaborative fiction" attack that survived every prompt fix. The product's job is to say so, and it keeps that as an open finding.
Accomplishments we're proud of
A complete find → prove → fix → verify loop, running live; a hardening result measured on held-out attacks rather than vibes; and a tester that invents attacks no list contained.
What we learned
Model choice is a design decision per role, not per app. Evidence beats opinion: our metrics only became trustworthy once every success was mechanically proven or quoted. And the most valuable output of a red team is sometimes "this can't be fixed with a prompt."
What's next
Detectors beyond disclosure (unsafe content, tool misuse), agent targets with real tools, threat-intel-seeded invention via Tavily by default, and large scheduled suites on Nebius Serverless Jobs.
Log in or sign up for Devpost to join the conversation.