ForgeBench

💡 Inspiration

Every model launch ships a benchmark number. Nemotron 3 Super launched with 60.47% on SWE-bench Verified — measured by the vendor, on the full set, with an internal harness nobody outside NVIDIA can inspect. We wanted the other kind of number: an independent one, produced on the stack a real customer actually uses (Nebius Token Factory's public API + the standard open-source mini-SWE-agent), with the receipts published — every trajectory, every patch, every cent of API spend.

🛠 What it does

ForgeBench is a one-command evaluation harness for agentic coding models:

• A Nemotron 3 Super agent (mini-SWE-agent + LiteLLM → Token Factory) with a 75-step budget per issue and a spend watchdog.
• A seeded, reproducible 50-instance subset of SWE-bench Verified spanning 12 repositories, run with the agent inside the official SWE-bench Docker environments.
• A scoring pipeline that folds official SWE-bench harness verdicts (FAIL_TO_PASS) back into a public results dashboard.

Headline run: 29 of 50 issues resolved (58%) by the official harness — 29 of the 43 patches that apply. 44 patches were submitted, 6 runs hit the step limit, 3 patches did not apply. Total API cost $59.49 (~$1.19 per issue). Full trajectories and the prediction file are in the repo.

Our first pass ran the agent in a local checkout instead of the official images and scored 22/50 (44%) on the same subset at $108.73. Environment fidelity moved 7 instances (+10 resolved, −3 lost; McNemar p = 0.09) — more than any prompt change we tried. Two things changed between the runs, not one: the environment, and dropping the never-invoked read_docs block from the prompt, so read the +14 points as an upper bound on the environment effect.

Costs are tokens x price, not billed amounts (the API reports no per-request cost for this model). We calibrated the price against our own invoices: 70.98M input + 0.917M output billed ~$181.82, i.e. $2.45 in / $8.00 out, which reproduces the bill to within 0.3%. An earlier draft of this page said $7.23 for the headline run because it trusted a hardcoded $0.30/$0.80 that was ~8x too low — our mistake, corrected here.

🚧 Challenges we ran into

• We ran the whole 50 first in a local checkout instead of the official instance images. The score was real but measured the wrong thing: 28 of 50 trajectories contain ModuleNotFoundError, so the agent was patching code it could not test. Re-running the same instances inside the official Docker environments moved the score from 22/50 to 29/50 and halved the cost.
• The first eval box had 7.5 GB of disk, so the official Docker scoring harness could not run there. We shipped an honestly-labelled submission rate first, then rescored the same patches on a machine with Docker.
• The official harness is Linux-only in practice — swebench 4.x imports the POSIX-only resource module, and on a Windows host it writes eval.sh / patch.diff with CRLF endings, which silently corrupts every run inside the Linux containers. Our first "unresolved" verdict was exactly that bug; the fix was to run the harness itself in a container.
• mini-SWE-agent's builtin 60 s per-command timeout kills the whole instance — it raises TimeoutExpired before saving the trajectory, so the run is lost and scores as an empty patch. It cost us 7 of 50 instances (5 of which resolve when retried with a sane timeout).
• Mid-run, one agent executed rm -rf /tmp/* and destroyed the shared checkout for 42 subsequent runs. The fix — per-instance isolated checkouts outside shared temp space — is now part of the harness design.
• We offered a documentation-retrieval tool (keyless Tavily) and the agent never bootstrapped it, let alone called it, in 50 out of 50 runs. On inspection the premise was wrong too: SWE-bench pins each repo to an old commit, so current web docs are the wrong oracle (the probe returns requests.readthedocs.io/en/master for a 2016 tree). We removed the tool instead of claiming it.

🏆 Accomplishments we're proud of

• A complete, reproducible eval for ~$60 end to end, plus an official FAIL_TO_PASS score (29/50) that anyone can re-derive from the committed predictions file.
• Radical auditability: anyone can clone the repo and inspect or rescore every single run — including the worse 44% one we did not lead with.
• The harness survived repeated infrastructure restarts without losing a single completed instance (resume-by-prediction).

📚 What we learned

• Environment fidelity beats prompt engineering. Giving the agent the real SWE-bench environments (+14 points, cost halved) did more than a documentation tool (0/50 usage) or a step-budget warning (no measurable effect in an A/B on the 4 near-miss runs).
• Where the agent does submit, it is usually right: 29 of 43 applied patches pass FAIL_TO_PASS and PASS_TO_PASS. 3 more submissions diffed a .bak backup file instead of the source.
• Scaffold and serving choices are part of the measured system: a vendor score is a property of model + harness + infrastructure, not the model alone. Our 50-instance, 75-step, mini-SWE-agent number is not comparable to a vendor's full-set number, and we do not present it as such.
• Agents need sandboxes that assume the agent itself is the threat.

🔮 What's next

• Scale N before claiming anything: 29/50 is a 95% CI of roughly 44–71%, and the local-vs-docker gap (p = 0.09) needs a larger sample to settle.
• Push environment fidelity further: the 6 remaining step-limited runs still burned their budget without a patch, and pylint (0/3) and seaborn (0/2) resisted every configuration we tried.
• Same harness on SWE-rebench (decontaminated) to quantify memorization, and a Nano-vs-Super cost/performance comparison.

Built With

• litellm
• mini-swe-agent
• nebius-token-factory
• nvidia-nemotron
• python
• swe-bench
• tavily