Inspiration
Track 01 caps entries at 50 million parameters, including the token embeddings and the output head. I entered it to learn how these systems actually work, which set one rule for the whole project: nothing goes in the repository unless I understand why it is there.
That rule turned out to be the interesting constraint, not the parameter cap. The cap is fixed and everyone has it. What I actually had to spend was compute, and I did not have much. So every decision had to be justified in accuracy per GPU-hour, not accuracy per parameter. That single framing produced almost everything below.
What it does
Kestrel-49M is a general-purpose language model with 49,482,752 parameters, which is 517,248 under the cap, trained from random initialisation on a 6-billion-token corpus built for it, with its own 16k BPE tokenizer.
Against GPT-2-124M, a model 2.5× its size:
| Benchmark | Kestrel-49M | GPT-2-124M |
|---|---|---|
ARC-Easy (acc_norm) |
0.4217 | 0.3948 |
HellaSwag (acc_norm) |
0.3001 | 0.3114 |
PIQA (acc_norm) |
0.5958 | 0.6251 |
WinoGrande (acc) |
0.5233 | 0.5162 |
| WikiText-103 word ppl | 40.35 | 37.37 |
It wins ARC-Easy, loses HellaSwag, PIQA and perplexity. WinoGrande it does not win. See below, because that distinction is the point.
All of it cost 2.08 H100-hours and $8.22. The entire project, including a seven-shape architecture sweep and a five-variant annealing study, was 3.65 rented GPU-hours, which cost about $14.
How I built it
The architecture was measured, not adopted. The published advice for sub-1B models is "go deep
and thin." I tested it: seven shapes, every one refitted to the same parameter budget, identical
data, schedule and seed. Deep-and-thin lost, not on quality, but on cost. It needs 64% more
GPU-hours than the winner for a worse loss, because depth means more sequential kernel launches and
more stored activations, which caps the batch size. The winner was wide-shallow, 640 × 9.
The hardware decision came from a benchmark, not a spec sheet. Everything was developed on one 8 GB laptop GPU. When the throughput measurement put the 6B-token run at 24.8 hours there against 2.1 hours on a rented H100, I rented one for two hours. Both numbers are measured; neither is projected from the other.
Evaluation had to be reproducible before any number was reported. Scoring runs in fp32 with tf32 disabled, because bf16 batch regrouping moved PIQA by 0.0017, which is enough to flip a claim. Three consecutive scoring runs give byte-identical output.
The baseline is scored here, not quoted. GPT-2 runs through the identical padding, precision, batching and rolling-window code as Kestrel. Running one model through my harness and quoting the other from a paper would fold "the two harnesses disagree" into the headline gap.
Training data is decontaminated against all five scored sets, and not just perplexity. A 16-token order-sensitive n-gram index over HellaSwag, ARC-Easy, PIQA, WinoGrande and WikiText-103 dropped 943 documents before tokenization.
The experiment that half-worked
Late on, the organizers confirmed synthetic training data was permitted. Looking at where Kestrel loses, namely HellaSwag and PIQA, both want the knowledge nobody writes down because it is too obvious: that a wet cup slips, that you preheat before baking. That is a gap in the corpus, not in the model, which makes it the kind of gap targeted data can close.
So I generated 7,000 passages (6.3M tokens) with Adaption Labs and annealed on them.
First, contamination. A model asked for "PIQA-style examples" has read PIQA and can reproduce items from memory, and n-gram filtering does not catch a reworded leak. So the prompts target the underlying skill in prose and never mention the benchmarks. Then I ran the project's own decontamination index over the output. Zero of 7,000 rows matched, while 94% of PIQA's most common content words appeared. Same subject matter, none of its items.
Then I swept the dose, and this is the actual result:
| synthetic share | HellaSwag | WikiText ppl |
|---|---|---|
| 0% (submitted) | 0.3001 | 40.35 |
| 4% | 0.3018 | 41.54 |
| 20% | 0.3107 | 42.63 |
| 50% | 0.3061 | 43.97 |
HellaSwag is an inverted U; perplexity is monotone. The capability peaks near 20% and falls back by 50%, while word perplexity degrades at every step with no turning point. The synthetic data does not add capability for free. It buys one benchmark by spending another, and past the optimum it stops buying while still spending. At 50% the model was simply memorising: validation loss rose from the first evaluation onward while training loss fell to 2.32.
At the optimum, HellaSwag goes from 0.3001 to 0.3107 against GPT-2's 0.3114. This means 94% of the gap to a model 2.5× the size is closed.
And PIQA never moved. Not at any dose, every p above 0.68. The physical corpus was 4,000 of the
7,000 rows, which was the larger investment, and it produced nothing. That asymmetry is the most interesting
part of the result and I do not have a confident explanation for it.
What I learned
That the error bars everyone quotes are the wrong ones. I had written "gaps that survive their error bars" into my own README. They don't: under unpaired standard errors, none of the four gaps against GPT-2 clear 2σ. But both models answer the same items, so item difficulty cancels if you pair them, and a paired bootstrap resolves three of the four cleanly. WinoGrande's interval straddles zero, so Kestrel's nominal lead there is noise, not a narrow win. I took the claim back out.
That with twenty comparisons, one "significant" result is what noise hands you for free. The annealing sweep is 20 paired comparisons; after a Benjamini and Hochberg correction, two survive. Both are HellaSwag, from independent runs.
That the submitted model should be the one from before the annealing. The annealed version wins HellaSwag and loses perplexity, and perplexity is scored too. It is a trade, not an upgrade, so I kept the checkpoint whose reproducibility I can demonstrate.
Challenges
- A parameter-count section that described the wrong model. The compliance section printed output for a config I had abandoned, which was 49,366,144 against the real 49,482,752. Found by re-running the script instead of trusting the pasted output.
- A stored evaluation result that would not reproduce. Tracked down to a stale bf16-era artifact rather than a code regression, via three independent lines of evidence, and regenerated.
- A flag that silently delivered 40% of what it claimed.
--synth-frac 0.5was mixing 19.9% synthetic, because the dataset permutes blocks across the whole shard pool while the run stops at its token budget. The runs looked perfectly healthy while doing it. Found by checking the achieved ratio on the volume rather than trusting the flag. - A factual claim about GPT-2 that was backwards. I had written that WikiText-103 comes from GPT-2's training distribution. The GPT-2 paper says the opposite: Wikipedia was removed from WebText. Wikipedia is 18% of my corpus, so Kestrel holds the in-domain advantage on that metric and loses it anyway, to 2.5× the parameters.
Every one of those was caught by checking rather than assuming, which is the same habit the results section is built on.
What's in the repository
Everything, with the script that produced it: the model, trainer, Muon implementation, data pipeline, decontamination index, evaluation harness, both sweep drivers, 115 passing tests, and the raw JSON behind every number in this write-up.
Log in or sign up for Devpost to join the conversation.