Every LLM call gets a receipt before January 14, 2027. That day OpenAI shuts down GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini (deprecations page), and 37 of 50 popular Python LLM repositories still name them in their tests (static scan of the most-starred active Python repositories a GitHub search for "openai" returns; the selection rule was pushed before the scan). Model Moving Day is an agent that finds the LLM call sites in a repository and gives each one a receipt: the smallest NVIDIA Nemotron on Nebius Token Factory that the repository's own tests accept, run 5 times per model in Token Factory Sandboxes and priced against the next bigger Nemotron, or the reason the tests can't vouch for it. It writes each move as a branch that the same tests confirm.

First census, in 3 libraries and tools: 26 LLM call sites found. 9 were set aside before the ladder (6 Responses API sites, which our proxy does not route, and 3 embedding sites, whose model is fixed); at 8 of the other 17 the repositories' own tests can decide: they notice a wrong answer, and a live model can pass them. 4 got a decision, in 2 libraries. instructor's sync retry funnel moved down to Lightning ($0.046 per 1,000 calls, against $0.157 on Super). Its async twin stayed on Super: Lightning passed 22 of 23 tests, but a nested-model test failed in every run, so that call costs about 3x OpenAI's replacement model priced on the same tokens. instructor's llm_validator and llm's chat call moved to Nano, the llm one with a remedy from a list written before the runs. The coverage gate we wrote in our internal notes before the runs (3 repositories, 5 independent decisions; not in the pushed manifest) was not met. Every cell, failures included, is in CENSUS.md.

On October 28 we ran llm's call site again, from the ladder to the confirmed branch, and recorded the terminal: the same decision, and each of its 68 model calls carries its Token Factory response id (evidence/rerun-2026-10-29/).

The January 14, 2027 date is checked at run time, not copied once. mmd notices asks Tavily to find OpenAI's own deprecation notice for each retiring model (a search limited to OpenAI's domains, then an extract of the pages it found), and code decides: the page must be on an official OpenAI host (forum threads that the domain filter lets through are rejected), a table row must name the model's exact snapshot IDs in its model column, and that row's date must be January 14, 2027. The recorded live run on October 30 (5 Tavily calls, 9 credits) confirmed 4 of 4 models from the "2026-10-14: GPT-4 series model deprecations" section of developers.openai.com. Without a key the command replays those recorded answers; with TAVILY_API_KEY it calls Tavily live. Search's indexed copy of the page and the extract of the bare URL both stopped at the September 11 entry, so the command checks every extracted copy and records how fresh each one is.

Demo (replay of the census, no model calls): https://model-moving-day.retailbox-automation.com/?r=devpost-nebius-c1 Code (MIT): https://github.com/retailbox-automation/model-moving-day Video (2:56): https://youtu.be/5K6B22zD5bs

Check it in two minutes

Claim Where to check
Each model choice comes from the repository's own tests demo: click a row for the flip tests and runs per model; census/MANIFEST.json → flip_tests
Rules pushed before the decision runs `git show 4f78a82:census/MANIFEST.json \
3 of 4 decided sites on Lightning or Nano; retry.py:523 needs Super CENSUS.md "Which model does what"; demo board
Receipt against the next bigger Nemotron demo receipt panel; CENSUS.md "Every cell" fallback rows
The agent writes the move; the patched code passes the same tests evidence/census/{instructor,llm}/patch/; confirm runs 5/5
The retirement date is OpenAI's own, checked at run time with Tavily mmd notices; evidence/notices/2026-10-30/NOTICES.md (row, section and Tavily request id per model); demo date strip
Failures are published demo "What did not work"; CENSUS.md paper-qa runs and "Found, not measured"

How this differs from routing each step to a model by price: the target repository's own frozen tests choose the model per call site, every decision comes with a receipt against the next bigger Nemotron measured on the same tests, and the output is a migration branch confirmed by re-running those tests.

What it does

For each target repository, pinned to a commit, the agent:

  1. installs the repository into a Token Factory Sandboxes image;
  2. runs the repository's own tests three times through a recording proxy: with a working model, with an empty answer, and with a well-formed wrong answer. A tracer tags every request with its call site and the test that sent it. A call site counts as covered only by tests that pass the first run and fail the other two (we call these flip tests);
  3. writes a manifest (pinned commits, flip tests, hashes of the frozen files, ladder rules, models, prices, the allowed fixes, budgets) and pushes it before the first ladder job;
  4. climbs a ladder per call site, Nano, then Lightning, then Super, then Ultra, with 5 runs per cell, at least 4 of which must pass, and keeps the smallest model that passes;
  5. writes the migration as a local branch of the target: an opt-in helper (NEMOTRON_MOVE=1) with the measured model and request parameters per call site. A run of the patched code confirms it.

Why

OpenAI retires GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini on 2027-01-14, and the tests in our three target projects still name them. Moving a project to another model usually means guessing one model for every call. We wanted each call site's own tests to make that choice and show the cost of each step.

How it works

  • Ladder models on Token Factory: Nemotron Nano (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B), Nemotron 3.5 Lightning, Nemotron 3 Super and Nemotron 3 Ultra. Nano and Lightning cost the same per token ($0.06 in / $0.24 out per 1M), Super $0.30 / $0.90, Ultra $1 / $3.
  • The brain is a Nemotron model that diagnoses red cells and writes the patch: Super for repositories under about 200K tokens of source, Ultra above. Diagnoses come from a proxy rule on the call log. The brain then picks one fix from a list written into the manifest, or none, and a fix counts only if the request really changed.
  • Token Factory Sandboxes run every ladder and confirm test run. A billing check on 2026-10-24 showed no Sandboxes charge (Sandboxes are in beta; Usage data up to 12:11 UTC that day).
  • paper-qa's embedding calls go to Qwen3-Embedding-8B on every rung.
  • Targets: 567-labs/instructor (6cd771c, MIT), simonw/llm (aebaca2, Apache-2.0), Future-House/paper-qa (57e89f7, Apache-2.0, core install only).
  • Tavily at run time: search (limited to OpenAI's domains) and extract find OpenAI's deprecation notice for each retiring model; code checks the host, the exact model IDs and the date (model_moving_day/notices.py).

Results (test traffic, not production traffic)

  • 26 LLM call sites found in 3 repositories. 9 were set aside before the ladder (6 Responses API sites, which our proxy does not route, and 3 embedding sites, whose model is fixed); at 8 of the other 17 the repositories' own tests can decide (they notice a wrong answer and a live model can pass them), and those 8 went into the ladder. The other 18 are listed with a reason each.
  • 4 call sites got a decision, in 2 repositories (instructor and llm). 3 of them moved to Lightning or Nano.
  • Counted per call site, that is 4 independent decisions, 3 of them on Lightning or Nano. If the sync and async faces of instructor's retry funnel count as one site (at the larger of their two models), it is 3 decisions, 2 on Lightning or Nano.
Repository Call site What the call does Flip tests Model after the census $/1k calls after / before
instructor instructor/v2/core/retry.py:224 sync retry funnel for structured output 8 Lightning $0.046 / $0.052: about the same
instructor instructor/v2/core/retry.py:523 async face of the same funnel 23 Super $0.170 / $0.058: 2.9x more
instructor instructor/v2/validation/llm_validators.py:34 an LLM judges a field value 2 Nano no saving (see below)
llm llm/default_plugins/openai_models.py:1494 non-streaming chat call with tools 1 Nano, reasoning off $0.030 / $0.052
paper-qa 4 sites evidence summaries, citation parsing, the answer, the post prompt 5, 5, 5, 1 no decision (infrastructure) -

"Before" is a price calculation, not a run: the same calls at the prompt and visible output token counts Nemotron produced, priced at gpt-6-luna ($0.10 in / $0.50 out per 1M). That is the model OpenAI names as the replacement for the models these tests use. We rechecked OpenAI's live pricing and deprecation pages on 2026-10-24. gpt-6-luna was never called, and we assume its pass rate is 1.0.

Site by site:

  • retry.py:523: on Lightning, 22 of the 23 tests passed in every run, but test_nested_model_extraction failed 0/5, so the site went up to Super, which costs 2.9x the base here. Moving to Nemotron does not always save money.
  • openai_models.py:1494: Nano failed 0/5 because the answer started with a newline ('\nYES' == 'YES'). A proxy rule flags this as leading_whitespace, and the fix list allows only "reasoning off" for it. The brain (Super) chose it from three options; its system prompt says that reasoning can add a leading newline. The change reached 15 of 15 requests, and Nano then passed 5/5. The estimate is about 40 % below the base ($0.030 against $0.052 per 1,000 calls), from one test and 15 calls.
  • llm_validators.py:34: Nano passed, but Token Factory reports no reasoning tokens for Nano, so the base counts Nano's hidden reasoning as visible output and comes out too high. Sized by Lightning's visible output at the same site, the base is about $0.09 per 1,000 successful calls against Nano's $0.128. Nano is not cheaper there, and the measured Lightning fallback ($0.073) is cheaper than Nano.

Migration branches: instructor, 3 sites moved (branch mmd/move-to-nemotron, 2318ca4), confirmed 5/5 on all three sites; llm, 1 site moved (a90319e), confirmed 5/5. Both branches live only in local clones. Nothing was pushed upstream and no fork was created.

Spend: $0.52 of model tokens for the decision ladders. All C1 spend in our ledger came to about $0.90 of model tokens as of October 24 (the recorded re-run of October 28 added $0.0124), plus $0.03 for one lost job and up to $1 unknown for another.

Each decision run used a manifest that was pushed before the run started. The ladder refuses to run when the manifest is not on the remote branch or does not match the code, the evidence or the clones. The manifest has eight amendments (the eighth, logging only, came after the census and before the recorded re-run), each recording the previous version's SHA-256.

What didn't work, and limitations

  • The coverage gate was not met: decisions in 2 of 3 repositories, and 3 or 4 independent decisions depending on how instructor's sync and async faces are counted. We wrote the gate and the pairing rule on 2026-10-24 in internal notes before the ladder runs, but they are not in the pushed manifest. Counting a pair at its larger model was added to the report script after the runs.
  • paper-qa has no decision. Token Factory returned 503/500 on Nano calls in its runs (89, 136 and 21 error responses in the three runs). Our Sandboxes client lost two jobs when its status polls timed out. One Semantic Scholar 429 counted as a model failure. The third run used a newer harness and a changed execution (sequential runs, a $0.66 token budget), introduced after two failed runs.
  • Each decision rests on few tests: 8, 23, 2 and 1. The two Nano decisions rest on 3 tests between them, and retry.py:523 went to Super because of one test.
  • llm :1494 rests on one test whose passing side is llm's recorded cassette of gpt-4o-mini; Super failed that test live. Only llm's non-streaming branch moved. The streaming branch (llm's default), the async branches and the Responses API sites were not measured.
  • The '\nYES' fix was provided for in advance: a proxy rule gives the diagnosis, the fix list was pre-registered, and the brain's prompt hints at it.
  • retry.py:523 on Super passed 5/5 on its re-run. Its first batch was set aside for 503s, and 4 of those 5 runs had failed underneath.
  • The instructor patch took four attempts. In the third, the brain wrote a wrong import and every confirm run failed at test collection.
  • The 5 runs of one cell are not independent (same sandbox, in parallel, often temperature 0), so a cell cannot tell a model that passes 80 % of the time from one that always passes.
  • instructor and llm are libraries. A decision covers the library's own test prompts, not the schemas of applications built on them.
  • The repositories were picked by hand and the sites are few, so the share on Lightning or Nano is not an estimate of how often LLM calls can move.
  • Tokenizers differ by up to about 33 % (Nano counted 409 prompt tokens per call where Lightning counted 545 on the same test), so the 11 % gap at retry.py:224 is within noise.
  • 22 runs in the headline's decision runs (48 in all decision runs) were set aside as infrastructure and re-run. 4 sites have no decision because of infrastructure.
  • The broken answers in the controls are synthetic, the proxy re-synthesizes streaming, and it normalizes requests during the ladder. The patched code is confirmed without normalization.
  • The platform spike plans were committed locally before their runs but reached GitHub only afterwards; the server-side timestamps in the raw replies show the runs started after those commits.

What's synthetic and what's real

  • Real: the target repositories and their own tests at pinned commits, frozen by hash; live calls to Nemotron Nano, Lightning, Super and Ultra and to Qwen3-Embedding-8B on Token Factory; the Sandboxes runs; the brain's fix choice and patch edits; the patched branches and their confirm runs.
  • Synthetic or assumed: the empty and wrong answers in the controls; the "before" dollars and the base pass rate of 1.0; the streaming events the proxy re-synthesizes; the per-rung request normalization during the ladder.

What's next

  • Reach the gate we set: a third repository with a decision and at least 5 independent decided sites.
  • Measure what the census could not reach: the Responses API sites (our proxy does not route /v1/responses) and llm's streaming branch, whose tests assert the old model's exact sentences.
  • Run the ladder on an application's own prompts rather than only on a library's tests.
  • Extend the run-time notice check to other vendors (Anthropic, Google, Mistral): each needs its official domains, its own list of retiring model IDs with their dates, and a reader for its deprecations page layout, and for Google's and Mistral's SDKs the scanner also needs new call-site rules (Anthropic's messages.create and litellm calls are already found).

Built With

  • cloudflare-pages
  • nebius-token-factory
  • nemotron-lightning
  • nemotron-nano
  • nemotron-super
  • nemotron-ultra
  • nvidia-nemotron
  • pytest
  • python
  • qwen3-embedding
  • tavily
  • token-factory-sandboxes
Share this project:

Updates

Submission history