fixhunt

Inspiration

Linters such as ruff, Biome, ESLint and clang-tidy apply thousands of autofixes every day, and developers accept them because each one is meant to be a refactoring: different code, same behaviour. When a fix breaks that promise, nothing fails loudly. The program simply does something else. We wanted a tool for the people who maintain these fixes that tests the promise by running code, not by asking a model whether the code looks equivalent.

What it does

fixhunt has two modes.

Hunt a rule. Nemotron 3 Super reads the rule's documentation, lists the ways the rewrite could change behaviour and writes small deterministic programs around them. Each program is run, linted, fixed, syntax-checked, run again and re-linted in a fresh Token Factory Sandbox with no network. Automated checks decide whether something went wrong, with no model involved: the tool crashed, the fixed program does not parse, the output or exit status changed, the fix is not a fixpoint, or a breaking fix was labelled safe.

Scan a repository. Each rule's fix is applied to a copy of a real project, one rule at a time, and the project's own command (for example its test suite) runs before and after. A fix that changes the normalized output or exit status is a finding.

Every finding is reduced to a minimal reproducer, confirmed by re-running it, given an advisory grade by Nemotron 3 Ultra against the rule's documentation, searched upstream on GitHub and through Tavily, and written up as an evidence sheet with reproduction commands and a regression fixture. Fixes that made no difference in a scan are combined, run once more and offered as a patch. Nothing is posted anywhere automatically.

Two real ruff bugs came out of it, both investigated and reported by hand:

• FURB103, found by hunting the rule: flag and f.write(x) becomes an unconditional Path(p).write_text(x), so the file is written even when the condition is false (astral-sh/ruff#29452).
• FURB129, first found by scanning a personal Python project: for line in fp.readlines() becomes for line in fp, so code in the loop body that reads or seeks the same file now moves the outer loop's position, and a passing test fails (astral-sh/ruff#29640).

How we built it

• Bun and TypeScript, with a streaming dashboard served by the same process.
• Linters are JSON specs in tools/*.json (version, rule lookup, lint, fix, syntax check, run, enabled rules, catalog). Biome, ruff, ESLint and clang-tidy ship; adding a linter needs no pipeline code.
• Nebius Token Factory, through its OpenAI-compatible endpoint:
  • nvidia/nemotron-3-super-120b-a12b: rule analysis, program generation, upstream search queries, explanations.
  • nvidia/Nemotron-3-Ultra-550b-a55b: advisory grading and duplicate triage.
  • nvidia/Nemotron-3_5-Lightning: proposes minimization steps, which the checks accept or reject.
• Token Factory Sandboxes run every program and every repository command: a fresh VM from node:20-slim, python:3.12-slim or gcc:14, about $0.001 and one to two seconds per run.
• Tavily and the GitHub API for the upstream search.
• Hosted on Render from a Dockerfile, with access codes and a daily spend cap gating live hunts and scans.
• A typical hunt costs one to two cents in model calls and a four-finding scan about two. Every call is logged.

Challenges we ran into

• Keeping the model out of the verdict. Models are good at proposing hazards and bad at deciding equivalence, so every pass or fail decision is made by a check on real program output, and the judge's grade is labelled advisory.
• Non-determinism. Programs run twice per version and outputs are normalized (timings, paths, positions) so a finding is a real behaviour change, not noise.
• Minimizing findings in real repositories down to a 1-minimal set of hunks that still applies with git apply and reproduces the difference.
• Measuring the judge honestly: Ultra's grades agree with hand labels on 87% of 89 findings (bun src/cli.ts judge-eval).
• Model output is not always valid code. Generated programs that do not parse are rejected and fed back to the next round rather than counted.

Accomplishments that we're proud of

• Two real ruff bugs reported upstream, each with a minimal reproducer produced by the tool.
• A pipeline that names no linter in code: four very different tools run from JSON specs.
• A hosted dashboard where judges can run a live hunt and watch every stage stream in.

What we learned

• A behavioural oracle is cheap now. A fresh VM per program at a tenth of a cent made "run it and see" the default check, which is a different design from anything we would have built on a shared machine.
• The two ways of finding bugs find different bugs. Hunting a rule found FURB103 within a few generated programs; FURB129 only showed up when a real project's test suite ran before and after the fix. Neither mode would have found the other's bug.
• Explanations are the least trustworthy output. The model's account of why FURB129 breaks was wrong about control flow even though the finding itself was right, so the evidence sheet now leads with outputs and the explanation is marked as a draft.

What's next for fixhunt

• More tool specs (for example rubocop and golangci-lint fixers).
• Running scans in CI when a project upgrades its linter.
• Hardening program generation against malformed model output.

Built With

• biome
• bun
• clang-tidy
• docker
• eslint
• github-api
• nebius-token-factory
• nvidia-nemotron
• render
• ruff
• tavily
• token-factory-sandboxes
• typescript