Inspiration
Track 01's rules cap every model at 50 million parameters, and one clause changes everything: the cap includes the token embeddings and the output head, not just the transformer itself. The moment I did the arithmetic on GPT-2's standard 50,257-token vocabulary, I found that the embedding table alone eats 25.7M parameters. That's 54% of the entire budget, gone before a single attention layer exists.
That felt like the actual competition. Not "can you train a small model," but "given that vocabulary size quietly determines how much of your budget survives to become reasoning capacity, what size should you pick?" I couldn't find anyone who had measured this properly under a hard parameter cap, so I decided to find out myself.
What it does
Parsimony is a small language model trained completely from scratch (no pretrained weights, no distillation), built around a parameter-matched, compute-matched sweep across vocabulary sizes. Instead of picking a vocabulary size by convention, it treats vocabulary size as a capacity allocation decision and measures the tradeoff directly.
The headline finding: with GPT-2's 50,257-token vocabulary, the 50M budget buys just 7 transformer layers. Drop to a 16,384-token vocabulary and the same budget buys 13 layers instead, and it turns out 13 layers is close to the sweet spot. Across a sweep of vocabulary sizes, both compression (bits-per-byte) and reasoning accuracy peaked at the same vocabulary, 16,384 tokens, not the smallest option and not the largest.
The final flagship model (48,872,576 parameters, right under the 50M cap) trained for 294.9M tokens on a Kaggle T4 GPU, reaching 0.9416 bits-per-byte and 99% on the held-out reasoning check. It was then run through the standard, required benchmarks (HellaSwag, ARC-Easy, PIQA, WinoGrande, WikiText-103) via lm-evaluation-harness, scoring close to chance, which is exactly what you'd expect from a 49M-parameter model with zero pretraining, and the README says so plainly rather than dressing it up.
There's also a live interactive demo where you can drag the vocabulary size yourself and watch the parameter budget reallocate between embeddings and transformer depth in real time.
Live demo: https://1234620.github.io/parsimony/ Repo: https://github.com/1234620/parsimony
How we built it
Data mixture. 55% TinyStories (fluent, simple English that's learnable at this scale), 25% FineWeb-Edu (clean world knowledge without web noise), and 20% procedurally generated worked-reasoning traces (step-by-step arithmetic and transitive logic, built over a 56,990-word lexicon with Zipf-distributed sampling).
Architecture. Deliberately conventional: RMSNorm (pre-norm), RoPE for positions, SwiGLU feed-forward, grouped-query attention with QK-norm, and tied input/output embeddings. The point was to keep the architecture boring so the vocabulary allocation was the only variable actually under study.
The metric. Everything is reported in bits-per-byte, not perplexity, because perplexity isn't valid to compare across tokenizers:
$$BPB = \frac{\text{total_NLL_nats} / \ln(2)}{\text{total_bytes}}$$
A larger vocabulary predicts fewer, easier tokens and its perplexity improves without actually modeling the text any better. Normalizing by raw bytes makes the tokenizer drop out of the comparison entirely.
The experiment design. Every vocabulary configuration is re-solved to sit at roughly the same parameter count (parameter-matched) and trained for however many steps it takes to consume the same FLOPs as every other configuration (compute-matched), not the same number of steps, since matching steps alone would unfairly favor shallow, cheap-per-step models.
Training. A CPU pilot on synthetic data validated the pipeline cheaply, then the real GPU sweep ran four vocabulary sizes on a Kaggle T4 at a 12M-parameter budget, and the winning vocabulary (16,384) was carried to the full 50M-parameter flagship run.
Evaluation. Parsimony is a plain PyTorch nn.Module, not a Hugging Face PreTrainedModel, so it doesn't fit lm-evaluation-harness's built-in wrappers. I wrote an adapter implementing the harness's three-method model interface (loglikelihood, loglikelihood_rolling, generate_until) directly against the model, verified against an independent brute-force computation, so all five benchmark tasks run through the harness's own scoring code rather than a reimplementation of it.
Frontend. The interactive allocator demo is vanilla JS with ApexCharts for the sweep visualizations and GSAP for the reveal animations, deployed as a static page on GitHub Pages.
Challenges we ran into
Kaggle's 12-hour session cap. The flagship training run hit the hard session limit at step 10,000 of 12,000 (exit code 137). The training loop already checkpoints and resumes, so the checkpoint carried into a fresh Kaggle session and finished the remaining 2,000 steps cleanly. A sampling bug that would have quietly rigged the whole study. The reasoning-trace generator originally sampled words uniformly from its lexicon. Uniform sampling starves a large BPE vocabulary of the frequent word types it exists to capture, which would have biased the entire vocabulary comparison. Switching to Zipf-distributed sampling fixed it before it could corrupt any results. GPT-2's own vocabulary (50,257) didn't finish inside the compute budget during the GPU sweep, so the honest reading of the result is "16,384 is the best vocabulary we measured," not "16,384 is globally optimal." That's stated directly in the write-up rather than glossed over. A tokenizer mismatch bug caught during evaluation. The flagship checkpoint's output directory ships all five swept tokenizers side by side. An early version of the harness adapter picked "the alphabetically first tokenizer file" to auto-detect vocabulary, which happened to be correct by luck of alphabetical sort, not by design. Caught and fixed by matching the tokenizer's actual vocab size against the checkpoint's config before it could silently score the model with the wrong vocabulary. Perplexity almost made it into the results table. Early on it was tempting to report perplexity since it's the standard metric, but it isn't comparable across tokenizers. Catching this early is why the whole project reports bits-per-byte instead.
Accomplishments that we're proud of
Vocabulary size is not a preprocessing detail when your parameter budget includes the embedding table. It's a first-class architectural decision with a real, measurable optimum, and getting that measurement right required matching both parameters and compute, choosing a tokenizer-invariant metric, and catching sampling bugs that would have quietly favored one outcome over another. It was also a good reminder that a small, cheap pilot experiment can point in the wrong direction entirely, and it's worth the extra compute to check a real finding at real scale before trusting it.
What we learned
Vocabulary size is not a preprocessing detail when your parameter budget includes the embedding table. It's a first-class architectural decision with a real, measurable optimum, and getting that measurement right required matching both parameters and compute, choosing a tokenizer-invariant metric, and catching sampling bugs that would have quietly favored one outcome over another. It was also a good reminder that a small, cheap pilot experiment can point in the wrong direction entirely, and it's worth the extra compute to check a real finding at real scale before trusting it.
What's next for Parsimony: A 50M-Parameter Budget Allocation Study
Extend the sweep to GPT-2's full 50,257-token vocabulary, which didn't finish inside the compute budget this time, to see whether the small downturn observed at 32,768 tokens continues or reverses. Run a proper two-dimensional sweep over both vocabulary size and model width (d_model), since only one width was tested at each stage this time. Re-run the byte-normalized, forced-choice reasoning rescoring (used to de-confound the CPU pilot's results) at full GPU scale, since some of 16,384's reasoning edge could still be a free-generation artifact rather than pure capability. Try the same allocation methodology at a larger parameter cap to see whether the interior optimum shifts to a larger vocabulary as the budget grows.
Log in or sign up for Devpost to join the conversation.