TinyBench-LM

Training Curve
Architecture Diagram
Benchmark Results

Inspiration

Getting a transformer to train is not hard anymore. Reference implementations are everywhere, and you can have one emitting tokens in an afternoon. So a 50-million-parameter cap is not really a coding challenge — it is a budgeting challenge. Every decision has to be defended out of a fixed account: depth or width, vocabulary size (every entry costs d_model parameters, twice if you don't tie the output head), how many tokens to spend, and what those tokens should be.

The second thing that pushed us was quieter. Small-model results are extremely easy to fake — and usually by accident. Train on web text you didn't filter and you have almost certainly swallowed HellaSwag and ARC items. Warm-start from a pilot checkpoint and your "from scratch" claim is gone. Run twenty configs and submit the best benchmark score and you have tuned on the test set. None of those leave a visible trace. Nobody can catch them from the outside, which means the only person who can keep the result honest is the person who built it.

So we decided the provenance would be a deliverable, equal in weight to the weights.

What it does

TinyBench-LM is an English causal language model with 49,658,368 trainable parameters — 341,632 under the Track 01 cap, counting embeddings and the tied output head. It continues English text. It is a base model, not an instruction-tuned chatbot: give it a prefix and it writes the next tokens.

It was trained from random initialization on 1,000,079,360 tokens on a single consumer GPU (RTX 4070 SUPER, 12 GB) in 4 hours 22 minutes. No pretrained weights, no distillation, no teacher model, no synthetic corpus, and no hosted API anywhere in the training, evaluation, or inference path.

A judge can run it in five commands on a laptop with no GPU: build the Docker image, download the ~199 MB model (every asset size- and SHA-256-checked against a release manifest), print the parameter count, and generate. The whole flow is CPU-only and needs no API key and no corpus download.

How we built it

The model. 14 layers, d_model 512, 8 query heads with 4 key/value heads (grouped-query attention), SwiGLU feed-forward of width 1,504, RoPE (theta 10,000), RMSNorm, no biases, tied input/output embeddings, 1,024-token context, 12,288-token byte-level BPE vocabulary, attention through PyTorch SDPA.

The budget drove every one of those. Tying the embeddings frees ~6.3M parameters (12,288 × 512) to spend on depth instead. The small vocabulary is a deliberate trade — fewer embedding parameters and uint16 token storage, paid for with longer sequences per unit of text. GQA cuts KV and projection cost without the quality drop of full multi-query.