FixedYet

FixedYet runs a bug report at the reported version and on main, in a sandbox, and tells you whether the bug is real, whether it is fixed yet, and, if still open, proves a fix.

Comment

Inspiration

Most bug reports get answered from memory, from the docs, or from a model that never ran the code. A maintainer needs two answers first: does the bug exist at the version the reporter used, and is it already fixed on main? I wanted an agent that answers both by running code, and lets the sandbox, not the model, decide.

What it does

FixedYet takes a GitHub issue URL. An agent on NVIDIA Nemotron 3 Super reads the thread, builds a sandbox checkpoint of the project at the reported version, and writes a minimal pytest test that asserts the correct behaviour. The same test then runs twice at the reported version and twice on main, in forks of the checkpoint. A fixed rule table, not the model, turns the four results into a verdict: fixed on main, open bug, not reproduced, or inconsistent, with a confidence level. Tavily finds duplicates, upstream reports and the release that shipped the fix. Then it acts on the verdict. For an open bug, fix mode asks Nemotron for the smallest source change and accepts it only when, in the sandbox, the test fails twice without it and passes twice with it and the project's own suite has no new failure. For a bug fixed on main, backport mode cherry-picks the fix onto the reported version's maintenance branch and proves it the same way. Either can open a draft pull request, labelled as AI-assisted. It runs as a CLI, a web demo and an MCP server.

How I built it

  • Python 3.12+, asyncio, Pydantic, httpx; a hand-written tool-calling loop on the stock OpenAI client pointed at Nebius Token Factory (no agent framework)
  • NVIDIA Nemotron 3 Super for test writing, root cause and fixes; Nemotron 3 Nano for intake; both on Nebius Token Factory
  • A sandbox interface built around checkpoints and forks, with two backends: Nebius Sandboxes, which the live demo runs on, and Docker, which runs FixedYet locally and in CI and ran the earlier evaluation
  • Tavily Search and Extract for duplicates, upstream reports and changelogs
  • An MCP server with four tools: reproduce_issue, check_fixed, propose_backport and propose_fix (the last two are dry runs that never open a pull request)
  • A FastAPI web demo with live progress, an all-reports page, judge access with its own quota, and a maintainer mode for backports, fixes and draft pull requests
  • A SWE-bench evaluation harness; a spend ledger with a hard total budget, because Token Factory has no spending cap
  • About 1,780 unit tests plus Docker integration tests, GitHub Actions CI

Challenges I ran into

  • The model must never decide the verdict: the sandbox does, from exit codes, so a false "reproduction" cannot slip through.
  • Old versions of projects are hard to install; the environment builder handles extras, dependency groups, other Pythons' pin files and a conftest that cannot load.
  • Nemotron quirks: tool names with reasoning leaked into them, tool calls written as text, and long fix answers that spent the whole token limit on reasoning. Each needed its own handling.
  • Measuring honestly: the same 15 issues give 11 to 13 reproductions from run to run, so every run is published, the six runs on intermediate agents and every miss included. A first "verified" fix for a Black bug passed all of Black's tests and was still wrong; a hand check caught it, and the agent now asserts every failing input an issue names.

Accomplishments that I'm proud of

  • Real verdicts on real issues: pallets/jinja#2244 is an open bug (high confidence, still open upstream), and pallets/click#3384 is fixed on main, with the fixing commit and the release it shipped in found through Tavily.
  • Fix mode on the live demo: for jinja#2244 it proposed and verified a one-line change on the first attempt (911 tests pass with and without it). It is the same check the jinja maintainer asked for in the open upstream pull request, so no duplicate pull request was opened.
  • A verified backport: FixedYet reproduced pydantic/pydantic#13991 and verified the backport of the fix onto v2.14-fixes (6,250 tests, no new failure). After a human review it went upstream as pydantic/pydantic#14043; the maintainer closed it unmerged, as they do the backports themselves before a patch release.
  • A fix for an open bug in Black (about 42,000 stars), psf/black#5578, ready for review: FixedYet reproduced psf/black#5468 on main and Nemotron's fix was verified in the sandbox; after a review that moved it to the literal PEP 440 form and added test cases, Black's own CI commands pass (596 tests, and 83 in the Jupyter run).
  • On 15 SWE-bench Verified issues, the final agent reproduced 13, 12 and 11 in three full runs (14 of the 15 at least once), at about $0.03 and 80 seconds per issue; every miss was a wrong model-written test.
  • On ten recent real issues (none from SWE-bench, opened July to October 2026), rerun live on the demo with the final agent: 8 of 10 verdicts correct, and the other 2 flagged at low confidence with the reason; none wrong at high confidence.
  • Nebius Sandboxes, written before beta access came, ran unchanged on the day it did. A full SWE-bench run there reproduced 12 of the 15 issues (the Docker runs: 13, 12 and 11), and the demo moved onto it: a project's setup takes 13 seconds there, against about a minute in Docker on the demo's 1 GB VM.
  • Two more upstream pull requests start from FixedYet's reproductions and fixes: RDFLib/rdflib#3603, merged on 24 October, and nedbat/cog#98, open. They were completed with Claude's help after FixedYet's first version, and the rdflib change needed a different approach after review: FixedYet's first fix also changed SPARQL MIN and MAX.

What I learned

  • A two-sided test is a much stronger answer than a model's opinion, and a fix is only as good as the test that checks it: the other callers of a changed function now appear in every fix report, and the test asserts every input the issue names.
  • "Passes on main" is not "fixed on main" when the release came from another branch: pyparsing 3.3.3 carried a regression master never had. FixedYet now checks git history before it says fixed.
  • The bigger model earns its price, and its reasoning matters. With Nemotron 3 Nano as the agent, 14 of 30 SWE-bench runs reproduced, against 36 of 45 with Super, at a third of the cost; Nano spent its turns searching and often wrote no test. Turning Super's reasoning off saved no money, since input tokens are most of the bill, and gave more wrong verdicts. So Nano reads the issue and Super writes the test.

What's next

  • Raise the reproduction rate (better test writing), use the SWE-bench images preloaded in Nebius Sandboxes for the evaluation, a GitHub App mode, and more languages.

Try it

  1. Open https://fixedyet.benlevin.fyi/ . Nothing to install and no sign-up.
  2. Read a saved report first: "See a finished one first" above the input box opens the pallets/jinja#2244 report: an open bug, then the fix FixedYet verified in the sandbox, then the two-sided result. Every saved report is at https://fixedyet.benlevin.fyi/reports.
  3. Run one live: paste a GitHub issue URL and press Run. You see each stage live; a run takes 1 to 4 minutes, and the demo runs one issue at a time. The public demo allows 5 runs a day.
  4. Scope: pure-Python projects tested with pytest. Issues from repositories in other languages, pull request URLs and numbers with no issue behind them are turned away before a run is counted. The "Try one" links under the input fill in issues known to work.
  5. Run it locally: follow the Quickstart in the README. You need a Nebius Token Factory API key and Docker (the default local sandbox); a Tavily key is optional.
  6. MCP: after the local install, claude mcp add fixedyet -- /path/to/fixedyet/.venv/bin/fixedyet mcp, then ask Claude Code whether an issue is fixed yet, or to propose a fix (a dry run; it never opens a pull request).

Built With

  • docker
  • fastapi
  • github-actions
  • mcp
  • nebius-sandboxes
  • nebius-token-factory
  • nvidia-nemotron
  • pytest
  • python
  • tavily
Share this project:

Updates

Submission history