Stepswitch

Each step of a coding task on the right Nemotron model on Token Factory: 21 of 21 tasks passed, like Ultra alone, at 18% of its dollars. Your tests decide what is done; every number re-runs.

Comment

The story in 30 seconds

Give a coding agent 21 small tasks (bug fixes, small features, refactors, a little numerical code) and run every step on Nemotron 3 Ultra, the largest Nemotron model. All 21 pass, for US$ 1.16. Yet most of those calls do not need the largest model: find a file, read it, write a small edit, run the tests.

Stepswitch gives each step to the Nemotron model that fits it on Token Factory. Lightning finds and reads the files, Super writes the code and runs your tests, Ultra plans and reviews, and gpt-oss-120b gives a second opinion. Ultra makes 42 of the 261 calls, and all 21 tasks still pass, for US$ 0.21: 81.6% fewer dollars. When a smaller model's change fails your tests twice, Ultra takes the task over; the demo replays a run where that happened. And each number here can be re-checked with one command, without an API key, because each is backed by a claim with a test.

Who it is for: developers and teams who run coding agents and pay by the token, and want the large model's judgement where it matters without paying the large model's price for every step.

The numbers

On the two mini bench suites (21 tasks: small bug fixes, features, refactors and numerical tasks in Python), the Nemotron ladder and Nemotron 3 Ultra alone ran interleaved on Token Factory, one task run per task, with the same checks (format, lint, the project's tests, numerical golden checks for the science tasks). From the evidence of claim C-029 (claims/evidence/C-029.json):

  • Task runs passed: 21 of 21 on both sides.
  • US dollars for all 21 task runs: 0.21 for the ladder, 1.16 for Ultra alone.
  • US dollars per task run: 0.0101 against 0.0551.
  • Tokens per task run, cached included: 47,156 against 53,067.
  • Uncached tokens per task run: 22,878 against 50,787.
  • Minutes per task run: 0.75 against 0.68.

  • Escalations: 0 on both sides.

  • The ladder spent 81.6% fewer dollars per task run. Its share of Ultra's dollars was 18.4%, with a 95% interval over tasks (cluster bootstrap) of 13.8% to 23.6%.

  • Per suite: general 0.0106 against 0.0522 dollars per task run, science 0.0089 against 0.0626.

  • Where the ladder's dollars went: Super 80.0% (151 calls), Ultra 16.0% (42 calls), Lightning 2.8% (47 calls), the second reviewer 1.2% (21 calls).

  • Reviews (claim C-030): gpt-oss-120b agreed with Ultra's verdict on 20 of the 21 changes both reviewed. Every reviewed change had already passed its tests (reviews run after the checks), so this is agreement on passing changes.

  • The ladder took a little longer per task run than Ultra alone (0.75 against 0.68 minutes).

Why you can trust these numbers

  • Each one has a test. Every number in the README is rendered from the evidence of a claim in claims/claims.yaml; a number typed by hand fails CI. uv run ssw prove --claim C-029 --claim C-030 re-checks these from the recorded runs, with no key and nothing billed, and CI re-runs the claim tests on Linux, macOS and Windows on every push.
  • Each run is published, failed and calibration runs included, with a record of every step, and so is the whole bill (below).
  • We tried to break our own result. A second pair of runs on a later commit, registered and pushed as claim C-032 before its first task run (ADR 0047), again passed 21 of 21 task runs on both sides, and the ladder's dollars came to 18.0% of Ultra's (18.4% above). Its 95% interval reached 27.7%, above the 25% we had set in advance: on one task Super's edit replies stopped at the output cap four times. So C-032 is withdrawn (ADR 0048), and "under a quarter" holds on the runs above only. Both pairs put the ladder at about 18% of Ultra's dollars.

Inspiration

A coding agent makes dozens of model calls per task, and most of them are routine: find a file, read it, condense a test log, make a small edit. Agents still send every call to one model, usually the largest, and report a pass rate without the bill. We wanted an agent that picks the model per step, proves the change with the project's own tests, has the change checked by a second model family, and publishes only numbers that anyone can re-run.

What it does

  • Per-step routing. Each step of a task has a type: plan, navigate, read, edit, run, triage, review, summarize. A policy file maps step types to models. The Nemotron ladder on Token Factory:
    • nvidia/Nemotron-3_5-Lightning navigates, reads, summarizes and writes short text;
    • nvidia/nemotron-3-super-120b-a12b edits, runs the tests and triages failures;
    • nvidia/Nemotron-3-Ultra-550b-a55b plans, reviews, and takes the work over when the gates fail twice on Super;
    • openai/gpt-oss-120b gives a second opinion on each of Ultra's reviews.
  • Your tests decide what counts as done. After a change, the project's own checks run, which we call gates: format, lint, types and tests (impacted tests first, then the full suite; science projects add numerical golden checks and protected constants). A change that fails goes back with the failure report; after two failures on one model, the task escalates up the ladder.
  • Dollars as a first-class budget. Prices per model come from Token Factory's model list (GET /v1/models?verbose=true, which includes pricing). The ledger records each call's model, tokens, cached tokens and dollars; --budget-usd stops a task at a dollar cap and --budget-usd-total stops a bench run.
  • Claims you can re-run. Each public claim has a test. ssw prove runs the tests, writes the evidence and renders the README's numbers from it; the README lint fails on any number typed by hand. The C-029 numbers above are regenerated from the committed runs with uv run ssw prove --claim C-029.
  • A demo you can audit. The demo page replays the recorded Token Factory runs side by side, step by step: the model of each step, its tokens, its dollars, the gate results and the reviews.

How we built it

  • Python 3.11, uv, Typer (ssw), Pydantic v2, SQLite in WAL mode, httpx; a React dashboard for local use; the public demo is a static page built from the committed runs and published by a GitHub Actions workflow.
  • A compatibility layer speaks the chat-completions, responses, messages and generate-content formats. Token Factory is a provider preset (registry/providers/tokenfactory.toml): base URL, bearer key from NEBIUS_API_KEY, the model IDs, list prices and the reasoning efforts each model accepts. We probed each model with ssw probe (tool calls, parallel calls, JSON Schema, streaming, usage, reasoning fields, prompt cache, context) and recorded the results in probes/ and docs/COMPATIBILITY.md.
  • Escalation-aware policy rules (when = { step_type = [...], escalations = 0 }) and a second-opinion review from another model family.
  • The bench runs contestants interleaved on the same tasks under a dollar cap, writes each step to steps.jsonl, and records the policy and prices in run.json, so every number in this file traces back to committed files.

Challenges we ran into

  • Thinking that effort does not bound. In reviews, Ultra used its whole 4,000-token output cap on thinking at low and at medium effort and never wrote a verdict; at none it answered in 12 tokens. The ladder now reviews without thinking.
  • Truncated thinking in the answer. When the output cap cuts a reply mid-thought, the unfinished thinking comes back in content. A summary step on Lightning with a 300-token cap returned its thinking as the summary. Summaries now run at effort none.
  • Different usage and reasoning fields per model, and a reviewer model that rejects effort none: the codec reads both reasoning field names and top-level cache counters, and the registry records which efforts each model accepts so the router asks for the nearest one.
  • Windows file locks and long paths during bench runs (the bench copies a repository per task run): retries on refused copies and deletes, and long paths turned on in the bench's git config.
  • Sandboxes answered with every permission false for our project (beta access not granted), so the sandboxed gate backend stayed a design (sandboxes.md).
  • Git and 260-character paths on Windows. The first replication attempt passed every task run, but its step records showed the protected gate ending with an error on every science task run of Ultra alone. Ultra's long contestant name put the task worktree's git objects past 260 characters, and the gates' git queries did not set core.longpaths. We fixed it, kept those runs in the bill, and made the replication again (ADR 0047).

Accomplishments that we're proud of

  • The same pass rate as Ultra alone on both suites at 18.4% of its dollars per task run, from runs anyone can inspect step by step.
  • A second reviewer from another family that disagreed with Ultra once in 21 changes, on a design question in a change that passed its tests, which is the kind of call a second opinion is for.
  • Every number in the README rendered from evidence, and claims that fail are narrowed or withdrawn in the open (claims/claims.yaml, docs/decisions/).
  • The whole bench bill for this submission is recorded run by run, failed and calibration runs included (below).
  • A replication registered and pushed before its runs, and reported though it missed its bound (ADR 0047, ADR 0048).

What we learned

  • Most calls do not need the largest model. Lightning ran the summaries with no cut-off calls, as Ultra did, at 6.8% of Ultra's dollars per call, over 21 calls each.
  • The money is in the edit loop. Super made 151 of the ladder's 261 calls and took 80.0% of its dollars. The next saving is there, not in planning or review.
  • Prompt caching matters as much as model size. Over these runs Super's input came from the prompt cache in 61.7% of tokens and Lightning's in 51.4%, Ultra's in 3.6%. The ladder sent fewer than half of Ultra alone's uncached tokens per task run.
  • Small models fail differently. On navigate steps 10 of Lightning's 71 tool calls returned errors, against none of Ultra's 61: it guessed absolute paths such as /repo/todo/model.py, read the tool's message ("paths are relative to the repository root") and corrected itself. No task failed because of it. Clear tool errors are part of the routing design.
  • The ladder's smallest model alone is not the low-cost choice it looks like. Lightning alone, on the same 21 tasks in separate runs, passed 19 (both failures ran out of steps at 40). It cost US$ 0.0071 per task run, 70% of the ladder's 0.0101, but used 130,369 tokens per task run: 2.8 times the ladder's 47,156 and 2.5 times Ultra alone's 53,067. On the science suite it cost more per task run than the ladder (0.0100 against 0.0089) and took 1.24 minutes against 0.65. Of the three, the ladder used the fewest tokens per task run.
  • Tool messages decide how many turns go to waste. After the tool fixes (container paths, cd <dir> &&, edits sent as JSON strings, read's line prefixes in an edit's old text), the ladder alone on the same 21 tasks had 9 of 215 tool calls refused, against 19 of 211 in C-029's runs; Lightning's went from 11 to none. It passed 20 of 21: in the failure, the models wrote read's line prefixes into a file's new text, which the tools now strip as well.
  • One task can move an interval. With 21 tasks and one run each, one task on which Super's output was cut off four times cost the ladder 77.6% of Ultra's dollars on that task (15.4% in C-029's runs), and the second pair's interval reached 27.7% (23.6% for the first).
  • Moving the edit loop down a rung saves little and costs a task (exploratory). With edit, run and triage starting on Lightning, Super after the first escalation and Ultra after a second (bench/policies/tokenfactory-lightning-first.toml, one run per task on 2026-10-29), 20 of 21 passed for US$ 0.0082 per task run: 81% of the ladder's 0.0101 in C-029's runs, 76% of its 0.0108 in C-032's, which ran nearer the current code. It used 125,013 tokens per task run against the ladder's 47,156, took 1.00 minutes against 0.75, and 57 of Lightning's 314 tool calls were refused. In the failure, Lightning's change failed the tests twice and the task reached Super at step 36 of its 40, so Super had four steps left. A cheap first rung spends the step budget the next rung needs. The ladder with Super editing stays the default.

What's next

  • Gates in Token Factory Sandboxes, with the local gates as fallback, and claim C-031: the same verdicts as local gates on every task run (sandboxes.md).
  • A ladder that learns from these step records which step types each model handles, per repository, and the Nano tier for the shortest steps.
  • A step budget per rung, so that an escalation arrives with steps to spend (the Lightning-first run above lost its one task that way).
  • The ladder on SWE-bench Verified Mini, which so far ran on a local model only.
  • More tasks per suite before the next claim is registered, so that one task cannot move its interval past the bound. More seeds would not do it: pooled, our two pairs of runs put the interval at 13.8% to 24.1%, because the spread is between tasks.
  • A live button on the demo, behind a rate-limited function that holds the key (DEPLOY.md).

Beyond Token Factory: SWE-bench Verified Mini on a local model

Before the hackathon work, Stepswitch and mini-SWE-agent each ran once on SWE-bench Verified Mini with the same local open-weight model (gpt-oss:120b), graded by the official harness (claims C-026 to C-028). Stepswitch resolved 24 of 49 instances and mini-SWE-agent 7. Most of the gap is robustness: the model server ended 24 of mini-SWE-agent's runs and 1 of Stepswitch's. On the instances where no server error ended either run, Stepswitch resolved 29.2% and mini-SWE-agent 25.0%. Stepswitch used 32.3% fewer tokens per instance, counting cached tokens.

The bill

Every hosted bench run of this submission, failed and calibration runs included, cost US$ 5.65 by Stepswitch's ledger at list prices (20 bench runs on 2026-10-28 and 2026-10-29). The Token Factory console showed US$ 6.06 of consumption, tax included, for every call on the account, which also covers model probes, direct API checks, each bench run's canary calls and two test runs outside the bench. The run-by-run table is in SUBMISSION.md, and hackathon/bill.json records the console reading.

Feedback on Token Factory, Sandboxes and Nemotron

Seven sections of measured findings and asks (reasoning effort and output caps, truncated thinking, usage fields, prompt caching, tool calling, the bill and its posting lag, docs): hackathon/FEEDBACK.md.

What changed during the submission period

The submission period opened on 2026-08-26. Stepswitch's first commit, e833c99, is dated 2026-10-16, so the whole project was built during the period (COMPLIANCE.md has the log). The Token Factory and Nemotron work for this submission is the commits after de1cf95 on the hackathon branch:

  • hosted provider presets with per-model prices and reasoning efforts, dollar budgets per task and per bench run (ADR 0044);
  • escalation-aware policy rules and a second-opinion review from another model family;
  • the Token Factory preset, probes of the Nemotron models and the reviewer, and the Nemotron ladder policy;
  • bench support for hosted runs (interleaving, dollar caps, step records) and fixes from the pilots;
  • claims C-029 and C-030 with their evidence, the demo page, and this package;
  • ssw policy check, which shows the worker each step type goes to, before and after an escalation, and flags rules that match no configured worker, without calling a model;
  • tool fixes drawn from the refused tool calls in the recorded runs: container-style absolute paths, cd <dir> && in run, edits sent as a JSON string with a stray bracket, and old text copied from read output with its row prefixes, measured in the tool-fix runs; then read's prefixes stripped from new text too, and a test selector that repeats the pytest command;
  • the long-path fix for the gates' git queries on Windows, and claim C-032, a replication of C-029 registered before its runs and withdrawn when it missed its bound (ADR 0047, 0048);
  • an exploratory run with the edit loop starting on Lightning, and the demo page in tabs.

Built With

Share this project:

Updates

Submission history