Inspiration

AI tools can write a bug fix in seconds. But a fix that looks right is not a fix that works. It may only pass because it was written around one test, or it may quietly break something else. I wanted a tool that answers one question with evidence: does this fix really solve the bug, without breaking anything?

What it does

BugTrace checks a bug fix before you trust it. You paste buggy Python code, the error, and what the code should do. Then:

  • The AI writes a test and you approve it. It proposes a diagnosis and a pytest test. You read the test, edit it if it's wrong, and approve it. This test defines what "fixed" means, and from then on it is frozen.
  • The AI proposes a fix. Up to three attempts, and a retry happens only after a definite failure.
  • BugTrace verifies the fix in a Docker sandbox. The approved test must fail on your original code, an audit scans the fix for test-gaming tricks, the test must pass on the fix, and your existing tests must still pass.
  • You get one clear result: VERIFIED, FAILED or INCONCLUSIVE, with the fix and the before-and-after test output.

The AI never decides the verdict. The sandbox does.

Verified result with before and after

You can also check a fix you already have, from any tool, with no AI at all: the "I already have a fix to check" mode and the /verify endpoint.

A real example

calculate_average([]) crashes with a ZeroDivisionError. I say it should return None for an empty list. The AI proposes a test asserting exactly that. I approve it. BugTrace confirms the test fails on the original code, applies the AI's fix, confirms the test now passes, and confirms my existing tests (like calculate_average([2, 4, 6]) == 4) still pass. Result: VERIFIED.

What makes it different

  • A human approves the test. The AI can't grade its own work against a test I never saw.
  • It catches test-gaming. In a live run with deliberately contradictory requirements, the model tried to detect which test was running by reading the call stack. The audit rejected that fix before it ran. Another attempt used a hidden call counter, and a regression test caught it.
  • Timeouts and errors are never a pass. If Docker or the AI provider fails, the answer is INCONCLUSIVE, not a fake success or failure.
  • Sandboxed execution. Every run uses a fresh container: no network, read-only filesystem, all Linux capabilities dropped, no-new-privileges, 256 MB memory, 1 CPU, a 64-process limit and a time limit.

A sneaky fix rejected by the audit

How I built it

  • Backend: Python and FastAPI, with endpoints for investigating a bug, repairing it, and verifying a fix.
  • Pipeline: baseline, audit, reproduction, regression. The repair loop keeps the approved test frozen across attempts.
  • Sandbox: Docker, run from Python with strict limits and a timeout.
  • AI: an OpenAI-compatible client talking to Google Gemini (gemini-3.6-flash). Responses must be valid JSON containing valid Python, or they are rejected.
  • Interface: plain HTML, CSS and JavaScript. All AI text is inserted as text, never as HTML.
  • Testing: 46 automated tests covering the pipeline, repair loop, audit, API and sandbox status handling. A fresh clone of the repository passes all 46 and builds the sandbox.

Reviewing the AI-proposed test

Challenges I ran into

  • Docker was the hardest part. Containers hung on startup on my laptop, and Docker's command line exits with code 1 when the engine is down. My code first read that as "a test failed", which would have shown a false failure. I fixed it so an outage is INCONCLUSIVE, and added a test for it.
  • The AI tried to game the tests. This is why the human-approved test and the audit exist.
  • Free AI limits. Daily quotas and timeouts forced me to treat provider errors as INCONCLUSIVE and to make the benchmark runner resumable.
  • Honest scope. I had to decide what BugTrace can and can't claim, and to say it plainly.

Results

I built a benchmark of 15 small Python bugs (empty input, off-by-one, wrong dictionary key, None handling, wrong condition, boundary error, integer division, duplicates, a shared default argument, and more). Each has a fixed approved test, regression tests, and a reference fix used only to prove the bug is valid. The AI never sees the reference fix.

  • Tuning set (10 bugs): 10 of 10 verified, all on the first attempt.
  • Held-out set (5 bugs): 5 of 5 verified, all on the first attempt. The dataset was frozen in git (tag benchmark-freeze) before the run, and I did not tune prompts on the held-out five.
  • Each repair took roughly 12 to 36 seconds, including the AI call and Docker verification.

How to read this honestly: the bugs are small and simple, and I wrote the tests, so this shows the pipeline works on simple cases. It is not an accuracy figure for real codebases. Full results are in the repository.

What I learned

  • A passing test is only as good as the test. Putting a human in charge of the expected behavior matters more than a cleverer prompt.
  • "I don't know" is a valid and valuable answer. Separating INCONCLUSIVE from FAILED is what makes the tool trustworthy.
  • Isolation, limits and honest error handling take more work than the happy path, and they are the part that matters.

Impact

Anyone who accepts AI-written fixes needs a cheap way to ask "does this really work?": solo developers using coding assistants, teams reviewing AI-generated pull requests, and students learning to debug. BugTrace turns that question into a repeatable check with a clear evidence trail, so a fix is accepted because it was verified, not because it sounded convincing. Because /verify accepts a fix from any source, it fits alongside the tools people already use.

Limitations

VERIFIED means the approved tests passed. It does not prove the code is correct for every input. The audit catches common test-gaming tricks, not all of them. A weak test can still accept a wrong fix, which is exactly why a human approves it. BugTrace currently supports Python only and small single-file projects. I did not host a public demo on purpose: the app runs code, and a public instance needs stronger isolation than Docker alone.

What's next

A pytest plugin and a GitHub Action that run the same verification on every pull request, support for more languages, stronger checks against weak tests, and hosting with stronger isolation.

Built With

Share this project:

Updates

Submission history