Default page Gate feature
Changes which show Base SHA and Head SHA
Recall feature

Gauntlet: Gate + Recall

Inspiration

Every check is green. The PR gets merged. And every customer on a phone can no longer buy anything.

That's not a hypothetical. It's the first thing Gauntlet caught.

A pull request added a friendly floating Help button to a checkout page. Lint passed. Unit tests passed. Code review looked fine. On a 375-pixel phone screen, that Help button sat exactly on top of "Place order." Every tap meant to buy something opened a help widget instead.

No test caught it, because nobody had written a test for a small screen. That's the real problem with CI in 2026:

CI only checks what a human remembered to check. AI is now writing a large share of our code, and nobody is writing the tests for it.

Then we asked the scarier question. AI agents now approve refunds, answer customers, and read internal knowledge bases. Suppose a bad article slipped in — one that told a support agent "ownership was already verified upstream." Weeks later someone notices. Which of the agent's past decisions did that article actually change?

Nobody can answer that today. Teams either reverse everything in a panic or reverse nothing and hope. We built Gauntlet to answer both questions with evidence instead of guesses.

What it does

Gauntlet is an adversarial tester that reads what changed and proves what broke. It has two modes.

🛡️ Gate: catch it before merge

1. Read — Maps every changed line to what it affects: a page, a component, a CSS breakpoint, an AI agent's tool.

2. Attack — An AI planner invents the test the PR deserves: "open checkout at 375px and try to tap Place order."

3. Freeze — The plan is fingerprinted with SHA-256 before anything runs, so nobody can move the goalposts.

4. Prove — Each attack runs 2× on the new code and 2× on the old code in isolated cloud sandboxes, in a real browser.

5. Judge — A fixed oracle checks hard evidence (which element actually received the tap) and ignores the model's opinion.

6. Report — A verdict lands on the PR as a GitHub check: confirmed_regression, introduced by this PR.

7. Fix — A proposed patch waits for a human to approve its exact fingerprint, then gets re-attacked to prove it works.

Here's what the Help-button PR looked like:

Gate run · "Add a floating Help button"
  Planner  : attack below and at the 900px breakpoint
  Plan hash: frozen before any sandbox existed
  Head (PR): ✗ ✗   tap landed on Help, not Place order
  Base     : ✓ ✓   tap reached Place order
  Verdict  : confirmed_regression · introduced by this PR

On an unrelated change, it said passed. No false alarm.

It doesn't stop at UI. When a PR let the support assistant issue refunds and accidentally deleted the ownership check, Gauntlet switched strategies by itself: it played an attacker in a chat with the bot, asked for a refund on someone else's order, and checked the ledger, not the chat. On the PR, money went to the wrong person. On main, the tool refused.

🔍 Recall: measure the damage after release

Point Recall at the bad knowledge-base article, and it:

Traces the blast radius through recorded history, including second-order exposure: article → saved memory note → a decision made days later.
Replays each decision twice: once with everything pinned (it must reproduce the original, or we don't trust it), and once with only that article removed.
Sorts the truth:

🔴 tainted — The article changed the decision. (3 in the demo, one through memory)
🟢 cleared — Exposed, but the decision would have been the same. (2 in the demo)
⚪ not_replayable — A critical input was never logged, so we refuse to guess. (1 in the demo)

A separate audit checked 36 refunds against the ledger and found 3 real violations. Everything goes into a human review queue. Nothing is reversed automatically.

How we built it

The pipeline: PR diff → Impact map → Planner (AWS Bedrock) → Validator → Frozen plan → Daytona sandboxes (base + head) → Anchor cloud browser → Write-once evidence → Trusted oracle → Verdict → GitHub check run + Dashboard + Voice briefing.

The core rule: the AI proposes, the oracle decides. The planner can only output typed steps — navigate, resize, inspect, click — built from registered pages, buttons, and test users. No AI-written code ever runs. Repository text and web pages are treated as untrusted data.

A verdict you can verify by hand. Gauntlet only calls something a regression introduced by the PR when it sees at least 2 failures on the PR build and 0 failures on the main build. One failure is never enough. If main fails too, it's a pre-existing bug — not blamed on the PR.

Counterfactuals with a sanity check. For a past decision, Gauntlet first replays it with all original inputs pinned — it must reproduce the original outcome, or the result is inconclusive. Only if the control passes does it replay again with the suspect source removed. If removing it changes the outcome, the decision is tainted. Missing data is never reported as "clear."

Evidence that can't be faked. Every screenshot and measurement is stored write-once with a hash the store computes itself, bound to the exact run, attempt, commit, and step. The oracle re-reads the bytes. Swap or alter one and the check fails closed.

Every sponsor does real work:

AWS Bedrock (GPT-6 Luna) — The planner brain: reads the diff and invents the attack
Daytona — Spins up identical, isolated copies of the app for "before" and "after"
Anchor Browser — The cloud browser that actually taps, scrolls, and screenshots
Arize AX — Traces every planner call, worker step, and oracle check
SurrealDB — The event log and lineage graph behind Recall
SurrealDB Agent Memory — Remembers verified findings so the next plan is smarter, without leaking hindsight
ElevenLabs — Speaks a briefing of the findings; it can read results but can never approve anything
GitHub — Signed webhooks start a run; the verdict comes back as a check on the PR

Stack: Python 3.12 · FastAPI · Pydantic v2 · hexagonal architecture with swappable ports · Next.js 15 · Tailwind · shadcn/ui · React Flow · Recharts · Playwright · ~900 automated tests plus a browser rehearsal of the full demo in CI.

What actually ran live:

A real GitHub pull request fired a signed webhook → live Daytona sandboxes → live Anchor browser → head failed 2/2, main passed 2/2 → confirmed_regression → check run posted back to GitHub.
The live GPT-6 Luna planner on AWS Bedrock found the same bug blind, from the diff alone, by testing both sides of the changed breakpoint.
Every screen carries a provenance badge showing exactly what ran live and what didn't. We'd rather show less than overclaim.

Challenges we ran into

"Just let the AI test it" doesn't work. Our first AI plans used made-up selectors that nothing could verify. We had to teach the planner each oracle's evidence rules and reject any attack that didn't fit them.
The AI sometimes missed it. One early live run only tested desktop and passed. A single general rule — "test both sides of every changed condition" — made run after run catch the bug.
A screenshot isn't proof. We had to prove which element received the tap, on both versions, and store it in a way nobody could tamper with.
A new model, a new format. GPT-6 Luna on Bedrock rejects the temperature setting and thinks out loud before answering, so every first call failed until we taught our adapter to handle reasoning models.
Fair comparisons are hard. "Before" and "after" must start from byte-identical state, or the comparison is meaningless. Any mismatch is reported as an environment error, never a bug.
Cloud resources cost money. Sandboxes and browser sessions must always be cleaned up, even when a run is cancelled halfway through.

Accomplishments that we're proud of

🎯 Caught a real mobile-checkout regression that every normal CI check missed, and proved the PR caused it.
🤖 The AI found the bug on its own, with no hint about what was wrong.
🧘 No false alarms: an unrelated change correctly passed.
💸 Caught an AI agent sending money to the wrong person by checking the ledger instead of believing the chat.
🧭 Separated real harm from noise after a bad article shipped: 3 tainted, 2 cleared, 1 honestly marked "can't tell."
🔒 Safety enforced in code, not in a slide: frozen plans, write-once evidence, single-use human approvals, and no auto-merge, ever.

What we learned

Let AI be creative and let rules be strict. An AI that proposes plus a deterministic oracle that decides gave us imagination without hallucinated verdicts.
"It's broken" is half an answer. "This PR broke it" requires testing the old code too, and that comparison is what makes a result worth acting on.
"I don't know" builds trust. Saying inconclusive out loud made every confident answer more believable.
Build offline first. Every integration has a fake twin, so three of us could build in parallel and the demo never depended on a single API being up.

What's next for Gauntlet

🔌 One-click install on any repository through the GitHub App
🧪 More attackers: multi-step workflows and race conditions
📊 A held-out benchmark on unseen codebases to measure misses and false positives publicly
🔐 Locked-down network egress for the cloud browser, and team roles before connecting real customer systems

Every PR has to survive the Gauntlet.