ShoestringLM

• 

  WikiText-103 PPL 628.7 to 28.1 (v3 4k) and 25.5 (v3 8k); ARC-Easy +4.1 pts vs v1 - $0 spend

• 

  Public repo: full from-scratch stack - model, tokenizer, training loop, param-count check, eval harness

• 

  Zero-shot results across all five checkpoints - same eval code and same WikiText-103 slice throughout

Inspiration

Training a serious language model is supposed to cost millions of dollars and a datacenter. Track 01 asks for a foundational model under 50M parameters, trained from scratch. We asked the more honest version of that question: how good a model can one builder train from zero - no pretrained weights, no budget - on free-tier GPUs?

What it does

ShoestringLM is a 30.7M-parameter decoder-only transformer, written from scratch and trained from scratch, small enough to train on a free Kaggle T4 and readable enough to grade in an afternoon. It generates text, and it is evaluated zero-shot on the four Track 01 benchmarks (HellaSwag, ARC-Easy, PIQA, WinoGrande) plus WikiText-103 perplexity - all with one command: bash scripts/run_eval.sh.

How we built it

• Model: decoder-only transformer in plain PyTorch - RMSNorm, rotary position embeddings, GELU MLP, tied input/output embeddings. 30,686,592 trainable parameters (cap: 50,000,000), verified by python3 -m shoestring.count_params.
• Tokenizer: byte-level BPE, 8,192 tokens, trained from the training corpus itself - no borrowed tokenizer.
• Training: AdamW + cosine schedule, gradient clipping, validation logging, and an automatic hardware/time/FLOPs report written on every run.
• Data: v1 trained 20,000 steps (327.7M tokens) on the public TinyStories corpus. v2 and v3 continued from the same checkpoint on disjoint samples of FineWeb-Edu (general web text) at lower LR.
• Hardware: Kaggle's free GPU T4 x2 sessions only. Total spend across every run: $0, roughly 12.5 GPU-hours.

Results - all measured, zero-shot, same eval code for every checkpoint

Headline checkpoint: v3 at 4,000 steps. Versus v1: ARC-Easy +4.09 pts, WinoGrande +1.19 pts, HellaSwag +0.25 pts, PIQA -0.81 pts, perplexity 628.71 -> 28.06. No checkpoint wins every metric, and the multiple-choice scores remain near or below task baselines on these out-of-domain benchmarks - we make no competitive-performance claim. The cost story is the claim: 30.7M parameters, about 12.5 GPU-hours, $0.

Challenges we ran into

• The first full run's benchmark stage failed on an HF export bug - 20,000 training steps with no scores. We fixed the export in the repo, then built a separate public eval-only Kaggle notebook so every published number has a public, rerunnable provenance link.
• Fitting long training into Kaggle's session limits: checkpointing, resume logic, and a compute report on every run so nothing was lost when sessions ended.
• Continued training on web data (FineWeb-Edu) raised a contamination question we cannot fully rule out: the web sample was not audited for overlap with benchmark questions, so v2/v3 scores carry that caveat instead of a contamination-free claim.

Accomplishments that we're proud of

• A complete, readable from-scratch stack: model, tokenizer, training loop, parameter-count checker, and eval harness in one repo.
• Every number in the table is measured with the same eval code and the same WikiText-103 slice (21,977 scored tokens) - apples to apples across five checkpoints.
• Perplexity 628.71 -> 25.51 across the v1-to-v3 continuation, for $0.

What we learned

Small from-scratch models gain far more from data breadth than from more steps on narrow data: continuing from TinyStories onto general web text moved perplexity by an order of magnitude, while multiple-choice accuracy on out-of-domain benchmarks barely moved. Honest evals beat flattering ones.

What's next for ShoestringLM

A final banked checkpoint (v3 at 4k or 8k steps), published eval notebooks for every reported number, and possibly a v4 on a larger FineWeb-Edu sample.

AI assistance disclosure

Per the hackathon's AI policy: an AI coding assistant (Instinct, running Claude) wrote most of the code and documentation, with the author directing architecture, constraints, and review. The author owns the design decisions: model size and shape, tied embeddings, dataset choices, the shoestring compute budget, and all submission content.

Built With

• kaggle
• lm-evaluation-harness
• python
• pytorch