Ratchet: rent intelligence today, own the adapter tomorrow. License-clean distillation with NVIDIA Nemotron on Nebius Token Factory.

Ratchet in 30 seconds

  • The problem: apps pay a big AI model for every request, even when it's the same kind of question for the thousandth time.
  • The fix: change one line, the base URL. A small Nemotron student answers when it's sure; when it isn't, Nemotron 3 Ultra on Nebius Token Factory answers, and that answer is saved with its license as a lesson.
  • The guard: each night a new LoRA adapter trains on those lessons, and it serves only after passing a held-out exam at 95% precision.
  • The proof: on exams it never saw, Nano 4B went from 12.8% to 39.0% on text-to-SQL and from 62.7% to 90.7% on tool calling, for about $17 on record (the video's opening quotes an earlier estimate of about $23). The whole loop ran on Nebius, and the gate turned away an adapter that wasn't ready.
  • Where it stands today: on the live site the student is the hosted Nemotron 3 Nano 30B, not my adapter. Serving my own adapter means keeping a GPU endpoint up (about a 7-minute cold start, paid while idle), which only pays off at high volume. Per-token LoRA serving would make it a switch, and that's what's next.

Inspiration

Every AI app pays a frontier model, again and again, to answer the same kinds of questions. That is rented intelligence: you pay for every token, forever, and nothing it learns from your traffic ever stays with you.

The fix is old and obvious: teach a small model to take over the repetitive calls. NVIDIA's own researchers make the case in Small Language Models are the Future of Agentic AI, and they lay out the steps to get there. In practice it is a months-long manual project, and if your big model is closed, its terms forbid training on what it writes, so you cannot even begin.

I wanted it to be a switch, not a project. Point your app at a gateway. Let a big open model do the work while a small one quietly learns from it. Hand traffic over only when the small one has proven it is ready, and take it back the moment it slips.

That is Ratchet. Like the tool, it only turns one way: forward, one checked step at a time.

How Ratchet works: a request passes the gateway's license check and redaction; Nano 4B with your adapter answers when it is sure, Nemotron 3 Ultra on Nebius Token Factory answers when it is unsure, and Ultra's answer becomes a lesson for Night School, an exam and the golden gate at 95% precision before any release.

What it does

Ratchet is a drop-in, OpenAI-compatible gateway. Your app changes one line:

client = OpenAI(
    base_url="https://your-ratchet-gateway/v1",   # was: your model provider's URL
    api_key=KEY,
)

A real request to the live gateway: the response headers say who answered (x-ratchet-served-by) and what it cost.

Behind that one line, a full distillation loop runs by itself:

  • The small model answers first. NVIDIA Nemotron 3 Nano 4B with a LoRA adapter, only when it is confident.
  • The big model covers the rest. When the student is unsure, Nemotron 3 Ultra on Nebius Token Factory answers instead. Your users never get a guess.
  • Every answer becomes a lesson. Ultra's answers are stored with their teacher and license, then become training data for the next adapter. Random audits keep measuring how often the student is really right.
  • New adapters have to earn their place. A held-out exam, then shadow, then a canary at 10%, 50% and 100%. If quality slips, it rolls back on its own.

Built on NVIDIA Nemotron and Nebius: every model call on the live site runs on Nebius Token Factory, the adapter was retrained on Nebius AI Cloud, and the student was served from a Nebius Serverless Endpoint in a measured test. On the live site today, Ultra teaches and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, a quarter of them re-checked by Ultra.

On top of the gateway sits Scout, a five-agent research app that shows the idea under real load. A planner splits the question, a searcher calls Tavily, triage and reader agents rank and extract, and a writer returns a cited answer. Triage and reading are short, narrow, repetitive calls: exactly what a small model should take over. Every Scout call flows through Ratchet, so you can see who answered and what it cost. A typical run: 18 to 20 model calls, three searches, about two cents. Replay three recorded runs.

Try it in 90 seconds, free, no sign-up: send a real request on the live page, or start at the judges page, where every number links to its evidence.

How we built it

The gateway is Python and FastAPI, with a ledger that records every request: who answered, what it cost, and, for anything that becomes training data, which teacher and license it came from. (Source on GitHub, Apache-2.0.)

  • License-clean by construction. Closed-model teachers are refused in code.
  • Private by default. Emails, phone numbers, cards and IDs are redacted before anything is stored.
  • Provable. Every adapter ships with a bill of materials listing exactly what it learned from, signed with Ed25519 and verifiable with one command (the signed Night School BOM, the public key).

The gateway's policy, from its own code: an open teacher (Nemotron 3 Ultra, OpenMDW-1.1) is allowed to teach and stored with its license; a closed-terms teacher is refused and never stored; emails, keys and phone numbers are redacted before anything is stored.

The routing decision is the heart of it. The student samples a few answers, and their agreement and token probabilities give a confidence score \( c \in [0, 1] \). On a calibration half of the held-out data, Ratchet picks the lowest threshold that holds precision at the target:

$$ \tau^* = \min { \tau : \mathrm{precision}(c \ge \tau) \ge 0.95 } $$

At runtime the student answers only when \( c \ge \tau^* \). The other half of the data, which the threshold never saw, measures how much traffic that really covers. In production, audits keep scoring the student against Ultra, and a canary rolls back the moment the upper end of its Wilson 95% interval for audited precision falls below the floor:

$$ \mathrm{Wilson}_{95\%}^{\mathrm{upper}}(\text{audited precision}) < 0.92 \;\Rightarrow\; \text{roll back} $$

NVIDIA Nemotron does the heavy lifting.

  • Nemotron 3 Ultra on Nebius Token Factory is the teacher. It wrote every training example: 4,322 text-to-SQL rows (each query executed in a sandbox and kept only if it worked) and 2,940 tool-calling rows, for about $12 in labels. Its OpenMDW-1.1 license puts no restrictions on using its outputs, and that is what makes training on them clean.
  • Nemotron 3 Super is the judge, and the hosted Nemotron 3 Nano 30B takes a random, audited 40% share of eligible live traffic.
  • Nemotron 3 Nano 4B is the student, fine-tuned with QLoRA (4-bit, rank 16, two epochs) using PEFT. The published adapters were trained on a borrowed GPU.

Then I ran the loop on Nebius AI Cloud for real. Every run is evidenced, dated and recomputable:

  • Retrained (27 Oct). The round-2 recipe ran as a Serverless Job on a spot L40S, with checkpoints in Object Storage, for about $2.16. It scored 40.7% and 34.5% against 39.0% and 33.0% before. Same recipe, new cloud, same result. (Evidence, adapter on Hugging Face.)
  • Served (27 and 28 Oct). The student ran on a Nebius Serverless Endpoint, with vLLM loading both adapters at runtime. A cold start took about 7.6 minutes, a restart 7.1, and with 16 users at once the median answer took 1.6 s for SQL and 1.2 s for tool calling. (Evidence.)
  • Night School, end to end (28 Oct). The gateway ran on a Nebius CPU VM, a spot L40S job trained a fresh adapter (loss 0.15 to 0.08), and the adapter hot-loaded into a Nebius endpoint. Then the golden gate rejected it: 81.8% precision against the 95% target. It never went live. That is the system working exactly as designed. (Evidence.)
  • Checked against the price list (29 Oct). Our per-token prices match the list Token Factory publishes, and a live request costs about $0.0002. (Evidence.)

Where everything runs

Everything else:

Challenges we ran into

  • Exams took six minutes per question. The model's generation code recomputed the whole prompt for every new token. I wrote a decoder that fixes the cache path: 45 times faster, identical answers.
  • The Mamba-hybrid Nano fought back. Its Mamba kernels read some weights directly, so those layers cannot go 4-bit, and PEFT's usual "all-linear" setting breaks on them, so I picked the target modules by hand. The first step spends about 18 minutes compiling kernels. The borrowed GPU could vanish at any moment, so one 66-minute run took 15 attempts before I moved to a detached, cost-capped job.
  • Nebius taught me resilience. Spot GPUs were reclaimed mid-training, so jobs resume from Object Storage checkpoints. The bucket mount refused in-place writes, so a finished adapter failed to save: I recovered the weights from a temp file, and the job now saves locally and copies to the bucket. The compiled kernels are cached in the bucket, so later jobs skip a 20-minute build.
  • My first scorer lied to me. It compared SQL text. The real scorer runs both queries on the same database and compares the actual rows.
  • My first exam was too kind. The same model wrote the exam and the training data, so I added a second exam from a different author, set in ten industries the training data never touched.
  • The hardest call was being honest. The adapter tripled the untrained model's score, yet at 95% precision the gate trusts it with almost nothing. I chose to show that instead of hiding it.

Accomplishments that we're proud of

Real adapters, strict exams they never saw, and every answer public, so anyone can recompute every score.

Text-to-SQL results

Text-to-SQL, exact rows only: the untrained Nano 4B scores 12.8% and 12.5% on the two exams, the teacher Nemotron 3 Ultra 25.6% and 15.0%, and Nano 4B with the adapter 39.0% and 33.0%. The same recipe retrained on Nebius AI Cloud scores 40.7% and 34.5%. (Every answer.)

Tool-calling results

Tool calling: on the hard exam (150 questions, 20 tools never seen in training) the adapter goes from 62.7% to 90.7%, against 91.3% for Ultra. On the public BFCL benchmark (our 300 questions) it goes from 77.3% to 80.3%, against 88.3% for Ultra.

  • A 4B model within a point of a frontier teacher, on tools it had never seen. On the hard exam the adapter fixed 42 questions and broke none.
  • Three times the untrained model on SQL, on both exams. Ultra mostly loses points by returning extra columns, so the honest reading is that the student learned the job, not that it is smarter.
  • The full loop ran on Nebius: train, serve, gate, and the gate said "not yet". At a 95% bar the SQL adapter answers 1 of 86 questions alone and sends the rest to Ultra. A gateway that says not yet instead of quietly serving bad answers is exactly what production needs.
  • Radically checkable: Apache-2.0 code, all three adapters, the training data, the exams, every single answer, a signed bill of materials, and one verify command per claim in docs/VERIFY.md.

What we learned

  • Accuracy is not the question; coverage at a guaranteed precision is. A student three times better than where it started can still safely take very little traffic. Measuring that honestly is most of the work.
  • The teacher's license is the product. An open teacher turns distillation from a legal risk into a feature.
  • Hosted inference is the easy part. Token Factory made the teacher side almost boring, in the best way. Proving a small model is ready is where the real engineering lives.
  • A gate that says no is a feature. The first Night School run on Nebius trained cleanly and was still rejected. That is trust, enforced in code.

What's next for Ratchet

  • Night School every night on live traffic. The live gateway connects to a Nebius student endpoint, and the student starts taking real calls the moment it clears the gate.
  • Close the gaps the exams found: dates, free-text arguments, and knowing when not to call a tool.
  • Per-token LoRA serving, so even modest-traffic apps profit from their own adapter.
  • More tasks, each with its own exam, so any team can see exactly which of its calls a small model can safely take over, and watch its bill ratchet down.

The Ratchet wheel clicks forward with each lesson: it only turns forward. Rent intelligence today, own the adapter tomorrow. Thank you for judging. Signed Harsh Bhanushali, builder of Ratchet.

Built With

  • astro
  • cloudflare-workers
  • docker
  • fastapi
  • github-actions
  • hugging-face
  • lora
  • nebius-ai-cloud
  • nebius-token-factory
  • nemotron-3-nano
  • nemotron-3-super
  • nemotron-3-ultra
  • nvidia-nemotron
  • openai-api
  • peft
  • python
  • pytorch
  • qlora
  • react
  • sqlite
  • tailwindcss
  • tavily
  • transformers
  • typescript
  • vllm
Share this project:

Updates

Submission history