Inspiration
We were inspired by the challenge of building a capable language model under a strict 50-million-parameter limit. We wanted to understand how much a small model could learn through careful data preparation, efficient training, and focused adaptation.
SLM_v2.1 grew from that goal: build our own model from scratch, measure its strengths and limitations, and make the process reproducible.
What it does
SLM_v2.1 is a 46.35-million-parameter English language model that generates text continuations and scores multiple-choice answers.
We evaluated it on tasks covering science questions, everyday physical reasoning, plausible event completion, and pronoun resolution. The project also includes a command-line demo, Hugging Face-compatible checkpoints, and documented training and evaluation pipelines.
Its current capabilities center on text completion and benchmark question answering.
How we built it
We built the project in four stages:
Designed a compact transformer. We used a Llama-style decoder with 12 layers, a hidden width of 512, and a 1,024-token context window. Tied input and output embeddings helped keep the model below the parameter limit. We also trained our own 16,384-token byte-level BPE tokenizer.
Prepared a diverse corpus. We assembled 22.45 billion unique tokens from DCLM, FineWeb-Edu, Wikipedia, StackExchange, Project Gutenberg, and a targeted web-text subset. We applied exact and near-duplicate removal, benchmark-overlap filtering, and source-domain exclusions.
Pretrained from random initialization. The V2 base model learned from approximately 45 billion next-token targets on one NVIDIA H100 NVL. Using PyTorch, mixed precision,
torch.compile, and a warmup-stable-decay learning-rate schedule, the main run completed in 31.9 training hours. We used no external pretrained weights or distillation.Adapted our own model into V2.1. We fine-tuned V2 on 79,425 training-split questions from HellaSwag, ARC-Easy, ARC-Challenge, PIQA, and WinoGrande. The objective combined multiple-choice learning, correct-answer language modelling, and replay of pretraining text. The submitted run completed three epochs in about 20.5 minutes.
We recorded phase objectives, data identities, checkpoint hashes, execution events, and evaluation decisions throughout the process.
Challenges we ran into
The main challenge was balancing specialization with general language modelling. Task tuning improved benchmark accuracy, but it also increased Wikipedia perplexity. Replay helped preserve language modelling performance, although it did not eliminate the trade-off.
Data integrity required substantial work. We needed to remove duplicates and benchmark overlap while maintaining a diverse corpus. Matching our question formatter to the evaluation harness was also essential for consistent training and scoring.
Long training runs brought engineering problems, including a validation-labelling crash and GPU memory limits during experiments. Checkpoint recovery, an automatic resume supervisor, and recorded failure reports helped us continue while preserving the experiment history.
Accomplishments that we're proud of
We completed the full path from corpus preparation and tokenizer training to a scratch-trained model and a task-tuned submission—all within 46,346,752 parameters.
Our training optimizations increased throughput from approximately 137,000 to 395,000 tokens per second, about 2.9× the earlier pipeline.
Under our pinned evaluation protocol, the submitted model achieved:
| Metric | V2 base | SLM_v2.1 |
|---|---|---|
| HellaSwag accuracy | 28.85% | 39.15% |
| ARC-Easy accuracy | 47.39% | 54.12% |
| PIQA accuracy | 60.77% | 65.34% |
| WinoGrande accuracy | 51.46% | 52.41% |
| Four-task mean accuracy | 47.12% | 52.76% |
| WikiText-103 perplexity ↓ | 23.85 | 25.46 |
Evaluation disclosure: V2.1 was fine-tuned on benchmark training splits, with detected evaluation-overlapping items removed; evaluation splits were excluded from training. These are task-tuned results and should be interpreted alongside the pretraining-only V2 base. We adopted V2.1 after observing its official scores, overriding our pre-registered Wikipedia retention guard. Its WikiText-103 perplexity is 6.8% higher than V2 base.
We are also proud of preserving the evidence behind our decisions, including unsuccessful experiments.
What we learned
We learned that lower language-model loss and higher benchmark accuracy measure different aspects of a model. Improving one does not guarantee improvement in the other.
At this scale, data composition, training efficiency, and task formatting matter greatly. We also learned to evaluate each candidate against its immediate parent, account for uncertainty, and distinguish development-based selection from decisions made after observing official results.
Reproducibility became part of the project itself: the checkpoint matters, but so do the records explaining how it was produced and selected.
What's next for SLM_v2.1
Our next priority is to evaluate generalization on fresh tasks beyond the benchmarks used for tuning.
We also want to measure CPU inference speed and memory use, explore quantization for smaller deployments, and make the demo easier to use. Future training experiments will target a better balance between task accuracy and general language modelling, using new hypotheses, bounded compute budgets, and evaluation rules established before execution.
Log in or sign up for Devpost to join the conversation.