Traceback

An AI agent that fixes a bug, then proves its own tests are strong enough to trust the fix.

Comment

Inspiration

AI coding agents love to say "fixed!" the moment the tests pass. But passing tests only prove something if the tests are good. A weak test suite will happily pass a wrong fix. The hackathon's problem statement was "Done Means Done," and I wanted an agent that's honest about what it can actually prove.

What it does

Traceback takes a one-line goal (for example, "Make sure orders of exactly 100 dollars receive the 10% discount"), turns it into a checkable definition of done, repairs the bug in an isolated copy of the project, and then audits its own work:

  • Defines done: writes a new test that must fail on the original code first, so it proves the bug is real
  • Repairs: changes only the files it's allowed to touch
  • Audits the tests: deliberately breaks the lines it just changed (swapping comparisons, shifting numbers by 1, removing conditions) and checks whether the tests notice
  • Closes the gaps: for any fault the tests missed, it writes a stronger test, kept only if it catches that fault. These are clearly labeled "agent-written, flagged for review"

It never says "fixed." Every run ends with an honest, bounded status: Verified within scope, Verification gap remains, Blocked, Attempt limit reached, Stopped or Failed.

How I built it

  • A plain Python controller runs the whole investigation as a state machine. The AI models only propose. The controller makes every decision based on real command output
  • Three agents: a planner and a repair agent on NVIDIA Nemotron 3 Super 120B, and a gap-closer on Nemotron 3 Nano 30B, all through Crusoe's OpenAI-compatible API
  • Mutation testing written from scratch, run with pytest
  • A Flask web interface in plain HTML, CSS and JavaScript (no framework), plus a command-line version
  • Every run saves its full evidence, and each run is capped at 12 model calls

Challenges I ran into

  • Weak tests that let wrong fixes pass. That's the whole problem, and it's why I built the mutation audit
  • Agents going in circles. I added a loop guard: if two repair attempts fail the same way, it re-plans once, then stops instead of burning calls
  • Messy model replies. Agents reply with a small JSON decision plus at most one code block, so code never has to live inside JSON, and malformed replies get one corrective follow-up
  • Runs crashing mid-way. Accepted results are saved as they happen, so if the server is killed, it resumes from the first unfinished step without redoing model work. I tested this by killing it mid-run on purpose

Accomplishments that I'm proud of

In the demo run, the repair made all 9 tests pass, and a normal agent would have stopped there. Traceback's audit then found that 1 of 4 deliberate faults (changing 100 to 99) slipped through the tests, so it wrote a new test for a $99 order that closed the gap. The final status was "Verified within scope," in about 4 model calls and under a minute, with the original project untouched.

What I learned

"Tests pass" and "the fix is right" are different claims, and the gap between them is where AI agents quietly fail. Keeping the AI in a proposing role, with plain code making the decisions, made the whole system far more trustworthy.

What's next for Traceback

  • Support for real projects beyond the demo fixture
  • Applying verified patches back to the original project
  • Watch mode, ticket queues and parallel agents

Built With

Share this project:

Updates