Apprentice

After takeover: captured traffic, the feedback your app reports falling 0.95 to 0.55, and a retrain offered once enough new gold lands.
Your expert verifies rows with one keystroke. Their verdicts become the gold set the model trains on and the bar it has to beat.
The eval gate: the base 4B scored 42.50 and was refused. Fine-tuned on verified rows it hit 89.17, cleared the 79.58 bar, and took over.
Apprentice learns your task from the frontier model, then only takes traffic after it beats your bar on data a human verified.
GEPA rewrites the prompt against your verified rows. Held-out score 72.92 to 79.58. You keep the rewritten prompt as an artifact.

Inspiration

Fine-tuning made small models cheap. Knowing when to trust one is now the expensive part.

Swapping a frontier model for a small fine-tuned one is a two-hour job. Convincing an engineering lead that quality won't quietly fall over on the 3% of inputs nobody looked at is the part that never happens. So teams keep paying frontier prices for extraction and classification work a 4B model could do in its sleep and they aren't being irrational. They're being asked to bet on a number nobody produced.

So we built the number.

What it does

Apprentice watches a task your app already runs on a frontier model, then earns the job:

Capture : two lines of SDK, and real production traffic starts landing.
Verify : rows become gold (a human checked it) or silver (deterministic checks passed). Everything else stays raw and never counts.
Optimize : DSPy GEPA rewrites the prompt against the verified set.
Train : a small model is fine-tuned on gold rows only.
Eval gate : every candidate is scored on held-out gold. It is promoted, or it is refused. There is no third option and no human override.
Watch for drift : after takeover, the console charts captured traffic and the feedback your app reports, and offers a retrain only when enough new gold has arrived.

The refuse path is the product. A model that fails the gate doesn't ship, and you keep paying the model that works. Everything else exists so we can say no with a straight face.

How we built it

Codex is the implementing engineer on the backend, not an autocomplete. The workflow is written into the repo's AGENTS.md and followed every time: a bounded contract (the exact unit, the files to mirror, the guardrails, the gate commands), work isolated in a git worktree, Codex implements and runs mypy/ruff/pytest itself, then a mandatory human review reads the diff before anything reaches main. Codex does not commit. 152 Codex sessions.

Codex runs on gpt-5.6-sol. GPT-5.6 also runs inside the product: the GEPA optimizer's student and reflection models are configurable, and this submission's run uses gpt-5.6-luna for both.

The feature built during the window is the drift panel the surface that catches a model going quietly wrong in production.

Challenges we ran into

The review kept catching what green tests could not. The retrain endpoint counted gold and silver rows as training data. Training reads gold only. Every test passed, because the tests encoded the same wrong contract I had handed to Codex. Two smaller bugs rode along: the cutoff used queue time instead of the moment the worker snapshots rows, and the eligibility flag ignored the minimum-rows gate so the panel would have offered a retrain the API rejects with a 400. A dead end, one click from the user.

Then a worse one, and it was ours. The run report showed a cost of $0.796 next to token counts worth about $0.05. A DSPy cache hit is replayed from disk it never reaches the provider and costs nothing but the replayed entry still carries the original call's cost, and we were summing all of them. We had been billing for calls that never happened, on a panel whose entire job is to be trusted about money. Fixed; reported cost now matches the token math to the cent.

And the thesis, demonstrated on us. Our public benchmark script first scored 72.92 instead of 100. GEPA rewrites a prompt from the feedback the metric returns, and ours only said "field names or values differ" so GEPA invented a separate key for each document type (po_number, receipt_number) while every gold answer uses invoice_id. It wrote a long, confident, carefully-reasoned, wrong prompt and was sure of itself. Naming the missing keys in the feedback took it to 100.00.

A model optimizing against a vague signal converges on something plausible and wrong. That is exactly why the eval gate scores against human-verified data instead of a model's opinion of its own work.

Accomplishments that we're proud of

The eval gate refusing a bad model is a first-class demo beat, not a footnote.

And every number is real or labeled projected. When we found our own cost figure inflating — making us look more expensive than we are we still stopped and fixed it before shipping.

What we learned

Depth of agent collaboration is a review discipline, not a prompting trick. Codex ships production code when the brief carries the contract and a human reads the diff against the codebase instead of against the test output. Green gates prove the code ran. They do not prove it is right.

What's next for Apprentice

Design partners: three teams through a real migration, structured extraction first, eval-gated end to end. Then the fine-tuned models get published with their benchmarks so anyone can check our arithmetic.

Reproduce everything: https://github.com/singhabhishekkk/apprentice-build-week drift-demo/ runs in one command. benchmark/ reproduces the GPT-5.6 numbers with nothing of ours in the loop: baseline 57.87 → 100.00 on a 12-row held-out split, $0.047, 67s.

Codex Session ID: 019f5eb6-27b4-7e00-af1b-04285e89a907