Distillery

Autonomous distillation on Token Factory: Nemotron builds verified data, fine-tunes a small student, and a deterministic gate evaluates if it passes the threshold of teacher/base model.

Comment

Distillery

Change one base URL, and Distillery swaps your big model for a small, certified specialist that hands back anything it can't guarantee.

Inspiration

Plenty of teams pay a large model to do the same narrow job millions of times. Shrinking that job into a small model is well understood. Trusting the small model is the hard part. Most distillation demos stop at "it scored well on a set we wrote."

We wanted an agent that does the whole loop, from making the training data to deciding whether the result is good enough, and that is allowed to say no. No when it decides whether to ship the small model. And no again on every single request, when the small model isn't sure.

What it does

Distillery is an autonomous agent that teaches a small model one narrow job (today: turning questions into SQL) by learning from a big teacher model, then checks the result honestly.

  • It writes the training questions and runs every answer before it counts. A lesson that doesn't execute correctly never reaches the student.
  • It locks a final exam away, opens it once, and lets plain code (no AI judge) decide PROMOTE or REJECT.
  • It puts the small model behind an OpenAI-compatible gateway. The student answers only where its error rate is certified. Everything else goes to the teacher, with the reason attached.
  • If traffic stops looking like what was certified, a drift guard rolls everything back to the teacher on its own.
  • Every claim has a receipt: its value, the file that backs it, and hashes. Claims we haven't measured say pending, with no number.

How we built it

The loop. Questions and reference answers come from a task pack, and the answers are run in sandboxes. A big Nemotron model writes lessons; only lessons that execute correctly are kept. A small Qwen model is fine-tuned on them. A planner model looks at what the student gets wrong and asks for targeted lessons, and the loop repeats. Each round is a branch of the sandbox, so you can see exactly what changed.

The honesty rules. We wrote the pass/fail thresholds down before we saw results. The final exam can only be read by one module, and a test checks that nothing else touches it. The gate is plain Python. The router's threshold is certified on sealed data, and it refuses to certify unless the pre-registration commit is part of the history.

The pieces.

  • Models on Nebius Token Factory: Nemotron 3 Ultra (planner), Super (teacher), Nano (triage and paraphrases), Qwen3-1.7B with LoRA (student), Qwen3-Embedding-8B (router signal).
  • Sandboxes for running SQL and branching experiments, and Token Factory for fine-tuning.
  • A FastAPI backend with a SQLite store, a resumable, supervised run loop, and the gateway.
  • A React and TypeScript site, with every number read from the API, hand-built charts, and accessibility checks in the tests.
  • Docker, deployed as one service on Render.

Challenges we ran into

  • Our first fine-tunes learned nothing. The provider's defaults gave only 3, 6 and 9 training steps, with no warning, and the loss still dipped slightly, so nothing looked wrong. We pinned the hyperparameters, refused any run below a 300-step floor, and ran a pilot to pick a learning rate. Then we proved the pipeline could fit data: 64 of 64 training prompts right, against 17 of 64 for the base model.
  • The student failed the honest test. It looked great on templated questions and was no better than the base model on naturally phrased ones. We published both rejections and built the certified router because of them.
  • A model we could train was hard to serve. Qwen3-1.7B wasn't on the serverless API, and custom weights on dedicated endpoints were in beta. So the student runs on sandbox CPU, which takes about 39 seconds per answer.
  • Unattended runs on flaky infrastructure. A local DNS outage burned 5 of 6 restarts in about 25 minutes. We made restarts outage-aware, and fine-tune jobs are now adopted after a crash instead of bought twice.

Accomplishments we're proud of

  • A student that matches its teacher on the sealed exam. On 300 held-out questions: student 92.3%, teacher 90.0%, untrained base 25.7%. The gate said PROMOTE, and the whole live run cost $4.26.
  • We published our rejections. On 86 natural questions the student scored 40.7% against 38.4% for the base model (REJECT). After retraining on natural phrasing it reached 57.1% against 29.6%, better, but still short of the bar (REJECT).
  • A certified router. On a sealed test of 490 questions, the student answered 240 and was wrong on 10 (4.2%, with a 95% upper bound of 7.0%, inside the 8% we promised). Overall accuracy was 95.1% against 93.5% for the teacher alone.
  • Recovery from faults. In a chaos run with 8 injected faults, the agent finished with one fine-tune job billed.
  • One base-URL change. An unchanged agent works against the gateway.

What we learned

  • Phrasing in the training set decides everything. A student that matches the teacher on templated questions is indistinguishable from the base model on natural ones.
  • The useful product is a router that knows when not to answer, with a certificate behind it, not a smaller model taken on faith.
  • "Cheap" is not "few tokens." In the main run the small triage model made 684 calls and used 126,538 input and 753,151 output tokens.
  • Writing the rules first changes what you're willing to publish.
  • Check the provider's defaults. An implicit setting was the root of our worst bug.
  • A test set the teacher helped write flatters the teacher. So we read the base and student numbers, not the teacher's.

What's next

  • Measure the real cost per 1,000 requests on a proper serving endpoint.
  • Widen the certificate beyond templated traffic, and test on a question set written by humans.
  • More task packs than SQL, and streaming and tool-calling in the gateway.

Honest limits

The win is in-distribution: it holds on templated questions, and on natural phrasing the router mostly hands the question back to the teacher, which is safe but saves little. The database is a made-up toy, so this says little about real-world SQL. Results come from one run and one seed each, on small sets. Student answers are slow, and the hosted demo sleeps when idle. The "natural" question sets were written by an AI agent, not by people.

Disclosure: an earlier AWS-based prototype (August 2026) never completed an end-to-end run. Distillery is a from-scratch rebuild started on September 29, 2026, on Nebius, and no code was carried over.

Built With

Share this project:

Updates

Submission history