Inspiration

Teams default to the biggest model because checking whether a smaller one is good enough is real work: you need a held-out set, a grader, repeated runs, and statistics that tell noise from a real gap. Most never do it, so they pay Ultra prices for Nano work. At a million receipts a month, our squeezed receipts setup is the difference between about $3,700 and about $90 a month.

What it does

  1. You pick a preset or bring your own task: an instruction plus 8 to 200 example rows. Rows without an answer get one from Ultra.
  2. A grader is set up. Presets ship a checker; for a custom task, Ultra writes one, and it is tested before use (correct answers must score about 1.0, deliberately wrong ones low). Checker code runs in Nebius Sandboxes with networking off.
  3. The rows are split three ways: tuning (the optimiser sees these), validation (only used to choose between prompt candidates) and held-out (scored once).
  4. Good Enough climbs down the ladder, cheapest first: Nemotron 3.5 Lightning, Nemotron 3 Nano, Nemotron 3 Super with thinking off, then Lightning with thinking on. On each rung, Ultra rewrites the prompt from tuning rows only, the best rewrite is picked on validation, and only that prompt touches the held-out rows.
  5. The verdict is a paired bootstrap comparison with Ultra on the same rows. The first rung that provably matches wins. If none does, it checks whether simply turning Ultra's thinking off is enough. Otherwise: keep Ultra.
  6. You get the model id, the thinking flag, the final prompt, a ready-to-paste Python snippet for Token Factory, the held-out score with its CI, cost per 1k calls and p50 latency.

The UI streams the run live: a cost ladder lights up rung by rung, rejected rungs turn orange, the winner turns green with its savings multiple. Finished runs can be replayed at any speed.

How we built it

  • Models: NVIDIA Nemotron 3.5 Lightning, Nemotron 3 Nano 30B, Nemotron 3 Super 120B and Nemotron 3 Ultra 550B, all on Nebius Token Factory through its OpenAI-compatible API. Ultra also writes reference answers, checkers and prompt rewrites. Thinking is toggled per call with chat_template_kwargs.enable_thinking. Live prices come from /v1/models?verbose=true, so cost per 1k calls is measured, not estimated.
  • Grading: Nebius Sandboxes, networking disabled, so model-written checker code never runs on the web server. The checker and hidden answers are sealed once into a checkpoint image; every batch is graded in a disposable branch of that checkpoint, so every model is marked against byte-identical answers (same image id, same SHA-256), and a branch starts faster than a fresh instance (1.4 to 2.1 s vs 2.4 s).
  • Statistics: 95% bootstrap CIs over rows, paired comparisons on the same rows, a 5-point non-inferiority rule, and a cached 3-run Ultra reference per task (Ultra itself moves up to 3 points between runs at temperature 0).
  • App: Python, FastAPI with server-sent events, a single static page, Docker, deployed on Render. Hard spend caps per run and per server.
  • Fine-tune fallback: when no Nemotron rung matches, Good Enough can launch a Token Factory fine-tune job on a small open model trained on Ultra's answers (trained end to end in testing; serving the result is pending custom-weights access).

Challenges

  • The answer-key problem. How do you prove a cheaper setup is good enough without already knowing the right answers? Checking an answer is easier than writing one, so Good Enough needs only a check: a known label, a test the answer must pass (code), or, when there is neither, Ultra's own answer, tested before use. Three guard rails make "as good as Ultra" mean something: the sealed answer key in Sandboxes, Ultra's own run-to-run wobble measured 3 times, and the guess check (if a fixed answer scores like Ultra, there is no verdict).
  • Cheaper levers that were not real. Token Factory lists cached prompt tokens 5 to 8x cheaper; across 15,000+ Nemotron calls we measured zero cache hits, so we claim no caching saving. We found no published Batch API discount and no time-of-day pricing, so we claim none.

  • Noise. In our first runs, on small splits, Ultra scored anywhere from 70% to 77.5% on the same task. Our first 3-point rule passed nothing, not even Ultra against itself with thinking toggled. We moved to bigger held-out splits, a cached 3-run reference, paired bootstrap CIs and a 5-point rule, and wrote down why 5 (RESULTS.md).

  • Overfitting. Prompt rewrites overfit the rows they were tuned on (Lightning on receipts: 88.5% on validation, 82.3% held out). Selecting on validation and scoring held-out once made that visible and kept it out of the verdict.

  • Fine-tuning is a next-day step. Our end-to-end job (LoRA on Qwen3-0.6B) sat in the queue for 4 h 19 min before 7.5 minutes of training. So Good Enough exhausts prompting first.

Accomplishments

  • A tool that says "keep Ultra" when that is the truth. One of three tasks did.
  • A measured, repeatable 36x saving on a real document task, 42x after the squeeze, with the evidence attached.
  • The same tool on photos, video and code, including one honest "keep thinking on" (video).
  • A "keep Ultra" task (contracts) turned into a proven 2.5x saving by the escalation chain, with no extra model calls.
  • The thinking-mode finding, which a single lucky run would never have shown.

What we learned

Measure, don't assume. Bigger and "thinking" are not automatically better on narrow tasks, and the run-to-run noise of a frontier model is large enough that single-run comparisons mislead.

What's next

  • Re-check verdicts on fresh production traffic on a schedule (for example as a Nebius Serverless AI Job) and alert when a downshifted task drifts.
  • Let custom tasks upload their own photos for the photo check (today only the receipts preset ships them), and serve the fine-tune fallback once custom weights are available.
  • Larger held-out sets, which tighten the CI and let the tolerance shrink below 5 points.

Limitations (honest)

  • A checker defines what "as good" means. Good Enough now catches a checker that a fixed guess passes, but anything a checker doesn't grade (e.g. my judge bot's written reasons) can silently disappear in a cheaper prompt.

  • Cheaper rungs are scored once inside a ladder run; only the headline Nano verdict was re-run 3 times.

  • Public datasets (Banking77, CORD v2, CUAD, Oxford-IIIT Pet, KTH, MBPP, UCI SMS), not production traffic. The photo, video and code lever results come from measurement scripts on one dataset each; the code preset has no cached Ultra reference yet. The Super VL result is one dataset (receipts), and the in-app photo check is one pass per run.

  • The fine-tune job ran end to end but its weights could not be served, so its quality is unmeasured.

Built With

  • bootstrap-statistics
  • docker
  • fastapi
  • nebius-sandboxes
  • nebius-token-factory
  • nemotron-super-vl
  • nvidia-nemotron
  • openai-python-sdk
  • python
  • render
  • server-sent-events
Share this project:

Updates

Submission history