Inspiration

AI agents learn by practicing in RL environments, and companies now need them by the thousands. Labs spend roughly $8.5 billion a year on RL environments and training data from more than 50 vendors, but every environment’s reward is software, and software has bugs. An October audit by Stanford, CMU and Fewshot found 22% of 1,860 tasks across five terminal-agent benchmarks could be passed without solving them, and on Terminal-Bench 2.0 it was 53%. Anthropic showed that models which learn those hacks in training go on to fake alignment and sabotage safety research. So we built Loophole to find each hack, patch it, and remember it, so it stays closed in every environment thereafter. Then we audited real public RL environments on the Prime Intellect Hub, published a leaderboard, and showed that a model trained on a hardened environment learns the task while the same model on the original reward learns to cheat.

What it does

Type a task in plain English and Loophole compiles it into an RL environment with about 15 seconds of Claude calls: a task generator, a reward and a reference solution. It's accepted only if the reference scores 1.0 and doing nothing scores near 0.

Then it attacks its own reward. A hacker proposes answers that should lose, an independent check confirms the ones that got paid anyway, and a fixer patches the reward, which ships only if honest answers still pass. Every confirmed loophole becomes a rule in a shared knowledge graph that the next audit starts from.

Finally it trains. On the hardened version of a hidden-number game, Qwen2.5-0.5B went from 0% to 90% real accuracy in 10.5 minutes. On the original, its reward hit 100% while real accuracy stayed at 0%: it learned to list ten guesses per turn.

Across 19 Prime Intellect Hub environments, Loophole confirmed 95 loopholes and repaired 11 environments, cutting attack success to 0% without breaking honest answers.

How we built it

Environments: Prime Intellect's verifiers format, so Hub environments and our compiled single- and multi-turn games run through one engine.
Hardening loop: our own take on CMU and Fewshot's hacker-fixer loop, with ideas from Berkeley RDI's BenchJack. Claude plays every role. An answer counts as a loophole only if it earns at least 80% of the reference reward and an independent check marks it wrong. Patches wrap the original reward, must pass static safety checks, close every loophole and keep 90% of honest answers passing. Five rounds at most, then a frozen checker re-scores every logged answer.
Knowledge graph: each loophole becomes a rule (mechanism, root cause, preconditions, defense), embedded with MiniLM and stored in SpacetimeDB, which links similar rules. Moss semantic search hands the closest ones to the next audit.
Training: GRPO with TRL on Qwen2.5-0.5B-Instruct on one RTX A5000 on Runpod, plus our own GRPO loop for multi-turn games.
Dashboard: React, TypeScript and Vite on Vercel, live-subscribed to SpacetimeDB.

Challenges we ran into

Proving a patch is harmless. A patch that rejects everything also closes every loophole, so nothing ships until at least five verified honest answers still pass. Haiku kept running out of output tokens on our answer batches until we cut them in half. Three environments never reached the minimum, so we left them unpatched rather than ship fixes we couldn't check.

Our safety check was too blunt. Patches run inside the grader, so they can't call eval, exec or compile. Our first check also blocked harmless regexes, so we rewrote it to read the patch's syntax tree.

Multi-turn environment training had no open source implementation. TRL's GRPO trainer handles one reply per prompt, so we wrote our own loop. Our first runs collapsed after three steps until we kept fp32 weights and computed logits only for the model's own replies.

Accomplishments that we're proud of

We audited 19 public environments for $16.60 and closed 60 of 95 confirmed loopholes, and a frozen checker re-verifies every number.
A rule learned on a Sudoku environment, adapted with one $0.0006 call, blocked all 60 scripted attacks on a completely different game.
We built our own multi-turn GRPO engine. On a hidden-number game, the fixed-reward policy went from 0% to 90% in 10.5 minutes on one GPU, while the hackable-reward policy scored 100% reward and 0% true accuracy.

What we learned

Most loopholes are generous matching, not clever tricks: 79 of 95 were near-misses counted as right, odd formatting, or several answers at once. A high reward isn't a correct answer, so a grader should never check its own work. And writing a fix is cheap; proving it harmless is the hard part.

What's next for Loophole

Audit the rest of the Prime Intellect Hub and keep the leaderboard current. Measure whether the knowledge graph makes audits faster and cheaper. Re-harden during training as the model gets stronger, and learn which loopholes RL actually exploits so teams fix those first. Longer term, a pre-ship check vendors run before any lab trains on their environment.