Popper

Inspiration

While my partner and I built with AI coding agents, fixes arrived faster than we could review them. Each one came like a sealed scroll marked solved, but the most important piece was often missing: proof.

We saw tests turn green after a fix, only to discover that they also passed before it. The agent had written convincing code and a convincing test, yet the original bug had never been caught.

We were using AI for speed while remaining responsible for every merge. That put us inside the exact problem Popper solves. As coding agents grow faster, every development team will face the same challenge: how can we verify their work without manually rebuilding it ourselves?

What it does

Every pull request makes a claim: this bug is fixed, this behavior is safer, this edge case now works.

Popper turns that claim into adversarial tests designed to break it. It runs each test against the code before and after the pull request.

Useful evidence should look like this:

$$ \text{Fail Before} + \text{Pass After} = \text{Evidence of a Fix} $$

A test that passes on both versions is marked inconclusive because it never detected the original problem.

Popper also compares the executed results with CodeRabbit’s independent review. It shows reviewers the claim, test evidence, disagreements, and a clear recommendation before the merge. A human always makes the final decision.

How we built it

We built Popper as a Next.js and TypeScript application.

Fireworks reads the pull request, extracts its behavioral claim, and generates tests designed to falsify it. Daytona runs those tests safely inside isolated sandboxes against both versions of the code.

CodeRabbit provides an independent static review. Popper keeps that opinion separate from the evidence produced by executed tests, then highlights where they agree or conflict.

Braintrust records the pipeline’s reasoning and execution, while CopilotKit lets reviewers ask questions about the results. We also built a replay system so a previously verified run can be shown instantly if a live service becomes unavailable.

Challenges we ran into

Our greatest challenge was making sure Popper did not create the same false confidence it was designed to catch.

Generated tests sometimes checked the wrong behavior. Some passed against both code versions. Others failed because a sandbox timed out rather than because the code was broken.

We had to define strict boundaries between three things: an executed result, an AI opinion, and an infrastructure failure. If Popper cannot gather trustworthy evidence, it keeps the gate closed and explains why.

We also worked through malformed model responses, changing model catalogs, sandbox limits, and slow review tools. Each failure helped us make the pipeline more careful and honest.

Accomplishments that we're proud of

We built a complete verification pipeline that turns a pull request’s claim into adversarial tests, executes them against real code, compares the results with an independent review, and explains the verdict to a human.

We are especially proud that Popper recognizes meaningless green tests. It checks whether each test could detect the original bug before treating it as evidence.

We built Popper while actively using AI coding agents ourselves. We experienced the problem as both builders seeking speed and reviewers responsible for the final code. That experience guided every major product decision.

What we learned

We learned that passing after a fix is only half the story. A test must first prove that it can detect the bug.

We learned that disagreement is valuable. When static review approves a change but executed tests break it, Popper has found exactly where a human should look.

Most importantly, we learned that trustworthy AI needs honest uncertainty. Saying “we could not verify this” is far safer than presenting weak evidence as confidence.

What's next for Popper

Next, we want Popper to connect directly to GitHub pull requests, support more languages and testing frameworks, and develop specialized adversarial strategies for different types of code changes.

We also want teams to track which claims, repositories, and coding agents create the greatest verification risk over time.

My partner and I built Popper because we needed a way to trust the speed AI gave us. As coding agents write more of the world’s software, Popper can become the gate that makes sure their claims survive contact with reality.