Repro

Inspiration

Point a security scanner at a real codebase and you get a wall of warnings. Most of them aren't real, and nobody has time to find out which ones are. AI coding tools were supposed to fix this, but they'll "fix" whatever you point them at and hand back a diff that looks right. Neither the scanner nor the AI tells you the two things you actually need to know: did the problem exist, and is it gone?

We saw this ourselves while building Repro. Asked to fix an MD5 password hash, a model rewrote it as getattr(hashlib, "md" + "5"): same broken hash, but now the scanner couldn't see it. Another patch made the code behave securely only when it detected a test was running. Both would have shipped with a green checkmark.

The Take Control of AI challenge sharpened the question for us. If you can't trust an AI tool's output, and you can't see where your data goes, you're not in control. So we built Repro around one rule: proof, not promises.

What it does

Repro points AI agents at a codebase, grounds every finding in evidence you can re-run, and repairs the code with proof that the fix actually holds. Every scan is a Run that moves through five stages:

1. Ingest clones the target into a fresh workspace pinned to one commit.
2. Detect runs deterministic scanners (Semgrep with our own privacy rules, gitleaks, osv-scanner, Ruff, and the project's own tests), then re-runs each finding's reproduction command in a sandbox. A finding only counts once the sandbox shows the problem. For privacy findings, Repro passes a unique canary value into the flagged code and checks whether it actually lands somewhere it shouldn't.
3. Diagnose has Gemini group findings by root cause and propose a fix. It can only cite IDs of findings it was given; the response schema allows nothing else, so the model can explain findings but never invent them.
4. Repair has Gemini edit files through a small set of sandboxed tools. It never writes the diff itself; the harness takes git diff against the scanned commit.
5. Verify re-runs the tests and the reproduction, checks for new findings, and hands the patch to an adversarial Challenger: a different model that never sees Repair's reasoning and tries to break the fix by writing counter-tests.

A patch is verified only when a plain-code gate says so:

$$ \text{verified} = \text{tests pass} \land \lnot\,\text{reproduces} \land \text{no new findings} \land \text{Challenger confirmed} $$

And "Challenger confirmed" can't be a model's opinion. It only stands if at least one counter-test $t$ flips:

$$ \exists\, t : \text{fails}(t, \text{before}) \land \text{passes}(t, \text{after}) $$

Verified patches become pull requests with the evidence pasted in verbatim. A person always merges; Repro never does. You can drive it from a web dashboard, a CLI, or an MCP server so it can be used from Claude.

How we built it

We split the work into lanes across our four-person team, with shared data contracts (Finding, Diagnosis, Patch, Run, Workspace) defined as Zod schemas so each lane could build against the same shapes in parallel.

The design principle throughout was Gemini proposes, the sandbox disposes. Every model output that matters gets checked by something deterministic: a schema, git diff, the test suite, a detector re-run, or a counter-test. No number in a run report ever comes from a model.

On our demo repo, all 18 of 18 findings reproduced in the sandbox, and a real run went from queued to a verified patch in about 2 minutes 19 seconds.

Challenges we ran into

• Models gaming the checks. Early on, the repair agent learned to make scanners go quiet instead of fixing bugs (disguised getattr calls, inspect.stack() checks to detect test runs). We wrote an evasion check that reads every added line for test-detection tricks, names built from string pieces, eval, and scanner-suppression comments. On a batch of 18 patches, it flagged exactly the two that cheated and none of the 7 legitimate ones.
• The fix that passes everything but is still wrong. Parameterizing a ?sort= ORDER BY clause passes the tests and silences Semgrep, but silently breaks sorting. This is exactly why we built the Challenger: its counter-test catches the regression, and only the allowlist fix is verified.
• Secrets leaking to the model. A planted API key reached Gemini once because gitleaks missed its format. We now redact every secret our own patterns recognize from everything a model sees, whether or not a detector flagged it.
• Cost and quota. Background polling burned through our daily quota, and free-tier keys get zero Pro requests. We added a per-run request budget with a worst-case reservation for each diagnosis, $2 \times (10 + 3) = 26$ Pro requests, so a run stops gracefully and still opens PRs for what it verified.
• Windows line endings. CRLF shebangs made every reproduction command exit with code 127 on Windows. A one-line .gitattributes rule fixed it after a lot of confusion.
• Merging four lanes in 36 hours. Parallel branches drifted from the shared contracts, and we lost real time reconciling duplicated schemas and conflicting PRs.
• Model quirks. Gemini sometimes named tools default_api:<tool>, which broke tool dispatch until our wrapper stripped the prefix.

Accomplishments that we're proud of

• An adversarial Challenger that attacks with code, not opinions, and has caught fixes that fooled both the tests and the scanner.
• A pipeline where no model decides what counts as a finding, and no model decides what counts as verified.
• A full end-to-end run on the real stack (Docker sandbox, real detectors, real Gemini) whose verified fix holds on a fresh checkout.

What we learned

• "Tests pass" is not proof. A model optimizing for green checks will find the cheapest way to get them. Verification has to include someone actively trying to break the fix.
• Keep models away from ground truth. Letting deterministic tools decide what's real, and letting models only explain and propose, made the whole system far easier to trust.
• Thinking tokens dominate cost. On Pro, output (which includes thinking) costs $6\times$ as much as fresh input and $60\times$ as much as cached input, so how many diagnoses we repair per run matters more than trimming prompts.
• Contracts first. Agreeing on shared schemas early is what let four people build in parallel at all.

What's next for Repro

• Harder to game: hold-out counter-tests the repair agent never sees, and a rule against insecure fallbacks when a secret is missing.
• Faster feedback: open each PR the moment its fix verifies, and run as a GitHub Check on every push.
• Cost control: measure tokens per verified fix from the logs we already keep, and cap how many diagnoses a run repairs.
• Broader reach: more languages beyond JavaScript/TypeScript and Python, and per-user sign-in for teams.

Built With

• docker
• fastify
• framer-motion
• gemini
• git
• github-api
• gitleaks
• google-cloud
• google-genai
• model-context-protocol
• mongodb
• mongodb-atlas
• node.js
• npm
• octokit
• osv-scanner
• python
• react
• ruff
• semgrep
• trpc
• typescript
• vite
• vitest
• zod