Anu-39M

## Inspiration

The brief was simple on paper — build a 50M parameter model from scratch — but
I didn't want to just fine-tune a known recipe or chase a crowded niche like a
"coding model." I went through a few directions (an on-device scam-SMS
classifier, a Google Edge Gallery-style device-action model) before realizing
the actual judging criteria — HellaSwag, ARC-Easy, PIQA, WinoGrande, and
WikiText-103 perplexity — meant I needed a competent *general* small language
model, not a narrow task model. That reframing shaped everything after.

## What it does

A decoder-only transformer, trained fully from scratch, under the 50M
parameter cap (39.66M params). RoPE instead of learned position
embeddings, tied input/output embeddings, and a 16K-token custom vocabulary —
all chosen to spend the parameter budget on actual reasoning capacity instead
of lookup tables.

## How I built it

Training data was a deliberate mix: the WikiText-103 **train** split (since
perplexity is scored on a held-out WikiText-103 slice, training on the
disjoint train split targets that metric directly, with zero leakage),
TinyStories for coherent narrative text, and a FineWeb-edu sample for broader
general-domain coverage. Everything ran on a single AWS g4dn.xlarge (T4,
16GB VRAM) — mixed precision, gradient accumulation, and a cosine LR schedule.

## Challenges I ran into

Compute was the real constraint the whole way. I burned time early chasing
GCP GPU quota before realizing new projects get hard-gated without billing
history, and switched back to the AWS instance I already had working. Closer
to the deadline I had to match the LR schedule to my actual time budget
instead of a schedule designed for a much longer run, while keeping the final
run as a single from-scratch training run.

## What I learned

At this scale and token budget, perplexity moves fast but multiple-choice
reasoning benchmarks (HellaSwag, WinoGrande especially) need far more tokens
than a single-GPU hackathon budget allows before they clear chance-level — that's
a known small-model scaling pattern, not a bug in the pipeline. PIQA showed real
signal earlier than the others, which tracks with it needing less world
knowledge.

## Results

| Metric | Score |
|---|---:|
| WikiText-103 Perplexity | 38.608 |
| HellaSwag | 26.78% |
| ARC-Easy | 33.25% |
| PIQA | 56.15% |
| WinoGrande | 49.88% |

## What's next

More training tokens and a broader science/reasoning-format slice in the data
mix (e.g. SciQ) would likely move ARC-Easy and HellaSwag the most — both were
compute-limited here, not architecture-limited.