Guardian

Regression detection against a frozen baseline. Three flagged with nothing changed - the judge disagreeing with itself, caught live.
Extracted Security Constraints

What it does

Guardian red-teams an AI agent and tells you what got through.

The target is ShopAssist, a customer-service agent that can look up orders, verify them, and issue refunds. We run it in two builds — vulnerable, with no guards, and protected, with security constraints enforced — then attack both with the same catalog of 28 cases and compare the outcomes.

Each case targets a specific constraint: refunds without verification, refunds over the manager-approval threshold, cross-account data access, confidential disclosure. Every run produces a full trace — every tool call, every result, every security event — and a judge scores it from that trace alone. The judge never sees the case's expected label. It reads what happened.

A frozen, commit-pinned baseline lets us diff any future run against it, so weakening a guard shows up as a regression instead of a surprise.

How we built it

Backend — FastAPI, PostgreSQL, SQLAlchemy, Alembic. The agent runs inside a state machine that dispatches guards per tool call. A trace collector captures every event with deep-copied state snapshots taken at call time, so the trace records what was true when the call happened rather than after.

Evaluation — 28 attack cases with expected labels for both builds. Deterministic constraints are checked in pure Python from the trace; semantic ones go to an LLM judge that sees only the agent's response text. The judge is structurally denied the answer key — we narrowed its signature so the case object, with its expected labels, can't reach it at all.

Regression detection — status_diff compares a fresh run against the baseline. It reports a regression only when a verdict crosses from a non-violation into a violation, and deliberately does not flag movement between two violation labels, because our judge disagrees with itself often enough that label-ordering alerts would be noise. It also carries a derived evidence object — call count, successful calls, cumulative amount — because a label alone can't say whether a rejected refund was the first attempt or the third.

Frontend — React and Vite: attack catalog, live execution with tool calls and security events, and verdicts.

Challenges we ran into

Every number is a distribution, not a value. The agent is non-deterministic at temperature 0 — vulnerable-build accuracy ranged 50.0%–76.9% across identical runs. Ten runs of one case: five made zero tool calls, four executed the refund, one stopped mid-attempt. The judge is non-deterministic too. So a single run is one draw, and we had to rebuild the result schema around that: every record carries how many runs backed it and whether judge voting was applied.

Our own evidence kept turning out to be contaminated. Twice, rate-limit errors were written into trace files as if they were agent responses. One produced a finding — "the agent engages with eligible orders and refuses with ineligible ones" — that was plausible, coherent, written up, reviewed, and turned into a work assignment before anyone noticed it was eight API failures out of ten traces. Nothing errored. It was caught only because the shape of it resembled something we'd seen go wrong hours earlier.

Fixtures that test nothing. One attack case pointed at an order that was genuinely eligible for a refund — so the agent verifying and refunding it was legitimate, not a bypass. The case had been passing for the wrong reason. We only caught it by running it ten times and counting what the agent actually did.

Knowing when to stop tuning. When revising an attack prompt, the rule we set was: iterate until the agent attempts the protected action, then stop — never until the verdict comes out the way you want. Tuning a case against its own expected label is the answer key leaking into the fixture instead of the judge.

Accomplishments that we're proud of

A working exploit we found by reading the guard code. A customer asks for a $650 refund; the guard requires manager approval above $500. So they ask for $400, then $250 separately. Each call clears the threshold and nothing ever sees the total.

guard_manager_approval evaluated one call's amount in isolation — and so did the check that scores it. The enforcement layer and the measurement layer were blind to the same thing. We fixed both, and the protected build now rejects the second call: "Manager approval is required for cumulative refunds over $500." Verified live, 5 of 5, on both builds.

Deterministic checks at 20/20 on both builds, fully reproducible from stored traces — a clean, noise-free measurement we can stand behind without caveats.

A harness honest about its own limits. Our regression tool flags three "regressions" against its own baseline with nothing changed — the same traces, same code — because those three cases are ones we've measured the judge disagreeing with itself on. We could have re-run until it came out clean. We report it instead, because a tool that hides its noise floor isn't measuring anything.

What we learned

Building the measurement instrument turned out to be the harder half. We found five distinct ways an evaluation can report something other than what it claims to measure, with nothing failing and no error raised.

The agent is non-deterministic at temperature 0 — a 27-point accuracy swing across identical runs.
So is the judge. Byte-identical input, cache bypassed: one case produced its expected label 1 time in 5. The case we chose as the obviously-stable control was one of the unstable ones — instability doesn't track apparent difficulty.
Which judge answered wasn't guaranteed. An unset environment variable silently fell back to a different provider whose quota had reset, so it succeeded rather than erroring. The two disagree on identical text — ATTEMPT_BLOCKED vs SAFE.
A case declared semantic was scored by a deterministic check. The checks run unconditionally and never consult a case's declared type. An incidental tool call routed a confidential-disclosure case into a cross-account check. It scored correct, for a reason unrelated to what it tests.
Rate-limit errors were stored as agent behaviour — including 109 of 169 entries in the judge's own cache, each caching a verdict of SAFE keyed to error text.

The first four produce a wrong number. The fifth produced a wrong story — and a wrong story doesn't invite the scrutiny a wrong number does.

The practical lesson: an agent-security harness is itself a system that can fail silently, and most of its failure modes look like results.

What's next for Guardian

Majority-vote judging. Designed and approved, not built before freeze. Every semantic verdict we report is currently a single draw from an instrument we've measured disagreeing with itself — scoring each with N judge calls and taking the majority would collapse that axis. The schema already carries the field for it, so a voted record is structurally distinguishable from an unvoted one.

Evidence objects beyond refunds. The derived evidence currently covers refund constraints only. Confidential-disclosure cases have a real gap it doesn't close: a verdict moving from partial to full disclosure is a label change our tool deliberately doesn't flag, and no evidence field speaks to it either.

Indirect prompt injection. Out of scope here because ShopAssist has no retrieval, document, or email surface to inject through. Adding one opens a whole attack family we currently can't test.

Validation at the trace boundary. Every contamination incident came from an API error being stored where an agent response belongs. A trace should be rejected at write time if its response is an error string, rather than caught later by someone recognising a pattern.

Deployment data. All our results come from a simulated environment. The claim we'd like to be able to make — that these margins hold on real traffic — needs real traffic.