Inspiration

Most "build an LLM" projects either fine-tune someone else's model or train an English-only toy. I wanted to see what one person could build from scratch on free GPUs by owning every layer: tokenizer, data pipeline, model, training, eval. English-first tooling also has a real cost for other languages. GPT-2's tokenizer spends about 50% more tokens on Spanish and French than on English, so I picked EN/ES/FR.

What it does

Trilingua is a 70.55M-parameter language model, trained from scratch, that writes and continues text in English, Spanish or French. A single language-tag token (<|en|>, <|es|>, <|fr|>) picks the output language. A short SFT stage adds a "write about X" mode. You can run it from a CLI, a Gradio demo, or Google Colab. The weights are on Hugging Face (aladewealth/trilingua).

How I built it

  • Tokenizer: 32,768-token byte-level BPE trained on a balanced EN/ES/FR sample
  • Data: 1.3B tokens streamed into uint16 shards on a free CPU notebook. English is 650M from FineWeb-Edu; Spanish and French are 325M each from FineWeb2-HQ.
  • Model: PyTorch decoder, 10 layers, d_model 640, 10 heads, SwiGLU (1728), RoPE, RMSNorm, tied embeddings, KV cache, 1024 context
  • Training: 2,385 steps × 524,288 tokens = 1.25B tokens on Kaggle's free 2× T4, using DDP, fp16 + GradScaler, torch.compile, and bitwise-exact resume across sessions
  • SFT: about 9k Aya rows plus 4.5k continuation examples, 832 steps, about 11 min
  • Eval: per-language perplexity, bits per byte against 6 open models on identical text, and fastText language-ID of generations
  • The code was written with an AI coding agent. I ran the training, debugged the runs, fixed the fp16 and resume issues, and ran every evaluation.

Challenges we ran into

  • bf16 on T4: the T4 reports bf16 as supported, but it's emulated. Session 1 ran at about 8k tok/s (3% MFU) and covered only 615 of 2,385 steps in 11 hours. Switching to fp16 + GradScaler + compile got about 57k tok/s (about 22% MFU) and finished in about 4h 45m.
  • Session limits: training had to survive Kaggle's 12-hour cap without losing data order or optimizer state. Resuming from the bf16 checkpoint also needed a GradScaler fix.
  • Fair comparison: the models use different tokenizers, so I compared bits per byte instead of perplexity.
  • SFT language drift: after SFT, language-ID fell to 0.87 / 0.20 / 0.47 (EN/ES/FR), down from 0.98 for the base model.

Accomplishments that we're proud of

  • Tokenizer fertility: 1.417 tokens per word in Spanish and 1.515 in French, against GPT-2's 2.099 and 2.159. That's 32% and 30% fewer tokens, at a 7% cost on English.
  • Bits per byte, averaged over EN/ES/FR: 1.060 for Trilingua (71M), against 1.186 for Pythia-160M, 1.204 for SmolLM2-135M, 1.375 for Pythia-70M and 1.630 for GPT-2. BLOOM-560M (0.894) and Qwen2.5-0.5B (0.903) are lower.
  • English is its weakest language: 1.105 bpb, behind Pythia-160M, SmolLM2 and GPT-2.
  • Caveat: the test docs are held out but come from the same FineWeb distribution Trilingua trained on, so it has an in-domain advantage.
  • The base model holds the requested language 98% of the time. Resume and DDP are verified, and 46 tests pass.
  • Total pretraining compute: about 15.8 GPU-hours on free hardware.

What I learned

  • Tokenizer fertility matters a lot at small scale. GPT-2 loses badly on ES/FR largely because of its tokenizer.
  • Check hardware claims yourself. bf16 on a T4 cost about 11 hours.
  • A small, English-heavy SFT set can undo language control that pretraining got right.
  • Fair evals take work: same text, same window, and bits per byte.

What's next for Trilingua

  • A language-balanced SFT set with in-language instructions, to fix the ES/FR drift
  • More English data and an out-of-domain benchmark
  • Document-masked attention and experiments with language sampling temperature
  • More languages: Portuguese, Italian, Yoruba
  • A quantized build for phones
  • Trying a frozen multilingual encoder and decoder with a trained middle stack, in the style of Meta's concept models

Built With

  • aya
  • datasets
  • fasttext
  • fineweb-edu
  • fineweb2-hq
  • github
  • google-colab
  • gradio
  • hub
  • hugging-face-(tokenizers
  • kaggle-(2x-t4)
  • matplotlib
  • numpy
  • pytest
  • python
  • pytorch
  • safetensors
  • transformers)
Share this project:

Updates

Submission history