-
-
Training v validation loss over 8,000 steps. Train loss fell near 0.04 val loss rose to about 8.93. The 49M model overfit the synthetic mix.
-
Live sample from latest.pt on a Colab T4. Continues chat and calculator templates from the training mix.
-
GIBC Track 01 count: 49,297,408 trainable parameters, including embeddings and the tied head. Under the 50,000,000 cap.
Inspiration
GIBC Track 01 asks for a language model trained from scratch with at most 50 million parameters. I wanted to see how far a small decoder-only Transformer could get on my own synthetic mix, using only a free Colab T4, with no pretrained weights.
What it does
It is a 49,297,408-parameter language model. It predicts the next token, like a tiny GPT. After 8,000 steps it can continue chat-style text and calculator/tool templates from the training data. It is not a chatbot with world knowledge. Train loss fell to about 0.04 while validation loss rose to about 8.93, so it overfit the mix.
How we built it
Architecture: 12 layers, embedding size 512, 8 heads, SwiGLU inner size 1536, context 512, vocab 16,384. RMSNorm, RoPE, QK-Norm, tied embeddings, no bias.
I trained a byte-level BPE tokenizer, packed a slice of my synthetic corpus into train.bin / val.bin, and ran a PyTorch training loop on Google Colab (Tesla T4, fp16). Effective batch was 65,536 tokens per step. Checkpoints went to Google Drive (latest.pt, step_08000.pt). Sampling uses the trained weights plus a ByteLevel decoder.
Cursor helped write training and eval code. I ran the training, recorded the demo, and take responsibility for the results.
Challenges we ran into
Colab disconnected more than once, so training had to resume from latest.pt. train.bin went missing from Drive after a session, which blocked resume until I re-uploaded it. Hugging Face dataset names for the official eval broke (wikitext is now Salesforce/wikitext). The model memorized templates instead of generalizing. One sampler printed Ġ artifacts until the ByteLevel decoder was set.
Accomplishments that we're proud of
Finished a from-scratch run under the 50 million cap. Saved a full 8,000-step checkpoint. Have a parameter-count script, a loss plot, and a live sample on GPU. The writeup is honest about overfitting instead of hiding it.
What we learned
A 49M model on repetitive synthetic text will drive train loss to near zero and still get worse on validation. "From scratch" means the weights, not reinventing PyTorch. Checkpointing to Drive matters more than a pretty first run. Eval dataset IDs change, so the harness has to use current namespaced repos.
What's next for From-Scratch 49M Language Model
Finish the GIBC eval (WikiText-103 perplexity, HellaSwag, ARC-Easy, PIQA, WinoGrande), put code and the parameter report on a public GitHub repo, and keep the numbers in the README. I am not training a 70B model for this track.
Log in or sign up for Devpost to join the conversation.