Inspiration
Most "build an LLM" projects either fine-tune someone else's model or train an English-only toy. I wanted to see what one person could build from scratch on free GPUs by owning every layer: tokenizer, data pipeline, model, training, eval. English-first tooling also has a real cost for other languages. GPT-2's tokenizer spends about 50% more tokens on Spanish and French than on English, so I picked EN/ES/FR.
What it does
Trilingua is a 70.55M-parameter language model, trained from scratch, that writes and continues text in English, Spanish or French. A single language-tag token (<|en|>, <|es|>, <|fr|>) picks the output language. A short SFT stage adds a "write about X" mode. You can run it from a CLI, a Gradio demo, or Google Colab. The weights are on Hugging Face (aladewealth/trilingua).
How I built it
- Tokenizer: 32,768-token byte-level BPE trained on a balanced EN/ES/FR sample
- Data: 1.3B tokens streamed into uint16 shards on a free CPU notebook. English is 650M from FineWeb-Edu; Spanish and French are 325M each from FineWeb2-HQ.
- Model: PyTorch decoder, 10 layers, d_model 640, 10 heads, SwiGLU (1728), RoPE, RMSNorm, tied embeddings, KV cache, 1024 context
- Training: 2,385 steps × 524,288 tokens = 1.25B tokens on Kaggle's free 2× T4, using DDP, fp16 + GradScaler, torch.compile, and bitwise-exact resume across sessions
- SFT: about 9k Aya rows plus 4.5k continuation examples, 832 steps, about 11 min
- Eval: per-language perplexity, bits per byte against 6 open models on identical text, and fastText language-ID of generations
- The code was written with an AI coding agent. I ran the training, debugged the runs, fixed the fp16 and resume issues, and ran every evaluation.
Challenges we ran into
- bf16 on T4: the T4 reports bf16 as supported, but it's emulated. Session 1 ran at about 8k tok/s (3% MFU) and covered only 615 of 2,385 steps in 11 hours. Switching to fp16 + GradScaler + compile got about 57k tok/s (about 22% MFU) and finished in about 4h 45m.
- Session limits: training had to survive Kaggle's 12-hour cap without losing data order or optimizer state. Resuming from the bf16 checkpoint also needed a GradScaler fix.
- Fair comparison: the models use different tokenizers, so I compared bits per byte instead of perplexity.
- SFT language drift: after SFT, language-ID fell to 0.87 / 0.20 / 0.47 (EN/ES/FR), down from 0.98 for the base model.
Accomplishments that we're proud of
- Tokenizer fertility: 1.417 tokens per word in Spanish and 1.515 in French, against GPT-2's 2.099 and 2.159. That's 32% and 30% fewer tokens, at a 7% cost on English.
- Bits per byte, averaged over EN/ES/FR: 1.060 for Trilingua (71M), against 1.186 for Pythia-160M, 1.204 for SmolLM2-135M, 1.375 for Pythia-70M and 1.630 for GPT-2. BLOOM-560M (0.894) and Qwen2.5-0.5B (0.903) are lower.
- English is its weakest language: 1.105 bpb, behind Pythia-160M, SmolLM2 and GPT-2.
- Caveat: the test docs are held out but come from the same FineWeb distribution Trilingua trained on, so it has an in-domain advantage.
- The base model holds the requested language 98% of the time. Resume and DDP are verified, and 46 tests pass.
- Total pretraining compute: about 15.8 GPU-hours on free hardware.
What I learned
- Tokenizer fertility matters a lot at small scale. GPT-2 loses badly on ES/FR largely because of its tokenizer.
- Check hardware claims yourself. bf16 on a T4 cost about 11 hours.
- A small, English-heavy SFT set can undo language control that pretraining got right.
- Fair evals take work: same text, same window, and bits per byte.
What's next for Trilingua
- A language-balanced SFT set with in-language instructions, to fix the ES/FR drift
- More English data and an out-of-domain benchmark
- Document-masked attention and experiments with language sampling temperature
- More languages: Portuguese, Italian, Yoruba
- A quantized build for phones
- Trying a frozen multilingual encoder and decoder with a trained middle stack, in the style of Meta's concept models