Ratchet

• GIF

  ▎ Change one line in your app. A small Nemotron answers first, and Nemotron 3 Ultra steps in when it's unsure.

• GIF

  ▎ The small model answers when it's sure, Ultra covers the rest, and every Ultra answer teaches the next adapter.

• GIF

  ▎ Text-to-SQL on questions it never saw: 12.8% untrained to 39.0% with the adapter, scored strictly.

• GIF

  ▎ Tool calling on 20 unseen tools: 62.7% to 90.7%, close to Ultra's 91.3%. A smaller gain on BFCL, below Ultra.

• 

  ▎ Every part has one label. Teacher calls and Tavily searches are real, results are measured, the loop is Simulation.

• 

  ▎ Try it in 90 seconds, free and with no sign-up: Scout, the release loop (Simulation) and every exam answer.

Inspiration

Most of what an app sends to a big model is repetitive. The same kinds of questions, the same kinds of answers, all day, and every call is billed at frontier prices. Everyone knows the fix: teach a small model to take over those calls. NVIDIA's own researchers make the case in Small Language Models are the Future of Agentic AI, and they lay out the steps to get there.

In practice, though, it's slow manual work. And if your big model is a closed one, its terms forbid training on what it writes, so you can't even start.

I kept thinking it should be a switch, not a project. Point your app at a gateway. Let a big open model do the work while a small one quietly learns from it. Hand traffic over only when the small one has proven it's ready, and take it back the moment it slips. That's Ratchet. Like the tool, it only moves one way: forward, one checked step at a time.

What it does

Ratchet is a drop-in gateway that speaks the OpenAI API. Your app changes one line:

client = OpenAI(
    base_url="https://your-ratchet-gateway/v1",   # was: your model provider's URL
    api_key=KEY,
)

Behind that one line:

• The small model answers first. A Nemotron 3 Nano 4B with a LoRA adapter answers, but only when it's confident.
• The big model covers the rest. When the small one isn't sure, Nemotron 3 Ultra on Nebius Token Factory answers instead.
• Every answer becomes a lesson. Ultra's answers are saved with their teacher and license, and they become training data for the next adapter. Random audits keep checking how often the small model is really right.
• New adapters have to earn it. Each one has to pass a held-out exam, then a shadow run, then a canary at 10%, 50% and 100%. If it slips, it rolls back by itself. In the demo, this nightly loop runs on mock models and is labelled Simulation.

On top of the gateway I built Scout, a five-agent research app. A planner breaks the question down. A searcher calls Tavily. Triage and reader agents rank the sources and pull out what matters, and a writer produces a cited answer. Triage and reading are short, narrow, repetitive calls, exactly the kind a small model should take over. Planning and writing stay with Ultra. Every Scout call goes through Ratchet, so each one shows which model answered and what it cost. A typical run makes 18 to 20 Ultra calls and three Tavily searches for two or three cents.

How we built it

The gateway is Python and FastAPI. Next to it sits a ledger that records every request: who answered, what it cost, and, for training data, which teacher and license it came from. Closed-model teachers are refused in code. Personal data (emails, phone numbers, cards, IDs) is redacted before anything is stored. Every trained adapter ships with a signed bill of materials listing exactly what it learned from.

The routing decision is the heart of it. The student samples a few answers, and their agreement and token probabilities give a confidence score c between 0 and 1. On a calibration half of the held-out data, I pick the lowest threshold that keeps precision at the target:

τ* = the lowest τ where precision(c ≥ τ) ≥ 0.95

At runtime, the student answers only if c ≥ τ*. The other half of the data, which the threshold never saw, measures how much traffic that actually covers. In production, audits keep scoring the student against Ultra, and a canary rolls back when its audited precision is clearly below the floor:

95% Wilson upper bound of audited precision < 0.92   →   roll back

Nemotron does the heavy lifting:

• Nemotron 3 Ultra on Nebius Token Factory is the teacher. It wrote all the training data: 4,322 text-to-SQL examples, each query run in a sandbox and kept only if it worked, plus 2,940 tool-calling examples. Labelling cost about $2.87 for the first SQL round and $9.18 for the second. Ultra is also Scout's planner and writer. Its license, OpenMDW-1.1, puts no restrictions on using its outputs, and that's what makes training on them clean.
• Nemotron 3 Nano 4B is the student. I fine-tuned it with QLoRA (4-bit, rank 16, two epochs) on a borrowed GPU.
• Nebius AI Cloud is where the full loop is headed. A vLLM Serverless Endpoint for the student that loads adapters at runtime, nightly training as a Serverless Job, a VM and Object Storage: all scripted and tested offline, ready to run with make nebius-plan and make nebius-up.

The rest: a React web app with a story page, Scout, a live dashboard and a page just for judges, plus a docs site. About 1,900 tests run with no GPU and no network. CI recomputes every published number from the saved answers on each push, so a result can't quietly drift. The adapters, the training data and every exam answer are on Hugging Face.

Challenges we ran into

• Answering exams was painfully slow. At first it took about six minutes per question. The model's own generation code recomputed the whole prompt for every new token. I wrote a decoder that fixes the cache path, and it runs about 45 times faster with identical answers.
• The Mamba-hybrid Nano fought back during training. The Mamba kernels read some weights directly, so those layers can't be quantized to 4-bit. PEFT's usual "all-linear" setting breaks on the Mamba layers, so I had to pick target modules by hand. The first training step spends around 18 minutes compiling kernels. And the borrowed GPU could vanish at any moment, so one 66-minute run took 15 attempts before I moved to a detached, cost-capped job.
• My first scorer lied to me. It compared SQL text instead of results. I replaced it with one that runs both queries on the same database and compares the actual rows.
• My first exam was too easy on the model. The same model had written both the exam and the training data. So I added a second exam from a different author, set in ten industries the training data never touched.
• Being honest about "not ready yet." The numbers looked great next to the untrained model, but the gate said the student could answer almost none of the SQL questions on its own at 95%. I had to decide to show that, not hide it.

Accomplishments that we're proud of

Two real adapters, tested on exams they never saw, scored strictly, with every answer published so anyone can recompute the scores.

• On the hard tool-calling exam the adapter fixed 42 questions and broke none, and it landed within a point of Ultra, on tools it had never seen.
• On SQL, the adapter scores about three times the untrained model on both exams. Ultra loses points there mostly by returning extra columns, so the honest reading is that the small model learned the job, not that it's smarter.
• On BFCL it's a small gain and still below Ultra. About half of the misses are questions where the right move is to call no tool at all.
• At a 95% precision bar, the SQL adapter answers 1 of 86 questions on its own and sends the rest to Ultra. A gateway that says "not yet" instead of quietly serving bad answers is exactly what I wanted to build.
• Everything is open: the code under Apache-2.0, both adapters, the training data, the exams and every answer. docs/VERIFY.md has one command per claim, and the exam explorer shows every single answer.

What we learned

• Accuracy alone isn't the question. What matters is how much of your traffic the small model can take while staying right. That can be very little even when it's three times better than where it started. Measuring it honestly is most of the work.
• Where the teacher's labels come from matters as much as the model. An open license on the teacher is what turns distillation from a legal risk into a feature.
• Hosted inference is the easy part. Token Factory made the teacher side almost boring, in the best way. Teaching a small model, and proving it's ready, is where the real engineering is.
• Strict exams keep you honest. Every time I made the scoring stricter, the story got less flattering and more useful.

What's next for Ratchet

• Run the whole loop on Nebius: the student on a Serverless Endpoint and nightly training as a Serverless Job, both already scripted.
• Fix the weak spots the exams found: dates, free-text arguments, and knowing when not to call a tool.
• Per-token LoRA serving, so a small fine-tuned model pays off even for apps with modest traffic.
• More tasks, each with its own exam, so a team can see exactly which of its calls a small model can safely take over.

Built With

• astro
• cloudflare-d1
• cloudflare-pages
• cloudflare-workers
• docker
• fastapi
• github-actions
• hugging-face
• lora
• nebius
• nebius-token-factory
• nemotron-3-nano
• nemotron-3-ultra
• nvidia-nemotron
• peft
• pytest
• python
• qlora
• react
• sqlite
• tailwindcss
• tavily
• transformers
• typescript
• vite