J3 | Devpost

1B-Token Training Data Mixture
Zero-Shot Benchmark Evaluation
J3 Stage 3 Training Loss

Inspiration

Large language models are usually discussed at scales of billions or even hundreds of billions of parameters. J3 started from the opposite question: how much language-model capability can we extract from a strict 50-million-parameter budget, using hardware that a student can realistically access?

We also wanted the experiment to be inspectable rather than presenting only a final benchmark table. Training a small model involves many choices — architecture, tokenizer, data composition, decontamination, optimization, checkpoint selection, and evaluation — and each of those choices can make a result difficult to reproduce if the evidence is not preserved.

J3 therefore became both a compact language model and an auditable training project.

What it does

J3 is a decoder-only language model with 49,001,408 trainable parameters, trained from scratch without pretrained initialization, distillation, or a hosted inference API.

The selected competition model was trained on a materialized 1-billion-token dataset and evaluated zero-shot on the required GIBC V2 benchmark suite.

Its final benchmark results are:

HellaSwag: 27.25% raw accuracy / 28.08% normalized accuracy
ARC-Easy: 40.70% raw accuracy
PIQA: 59.68% raw accuracy
WinoGrande: 51.62% accuracy
WikiText-103: 96.12 word perplexity

The repository includes not only the model implementation and checkpoint, but also the tokenizer, data manifests, decontamination records, training metrics, environment snapshots, evaluation adapter, benchmark output, and release hashes needed to inspect how the final model was produced.

How we built it

J3 is implemented in native PyTorch as a 32-layer decoder-only Transformer with model width 384.

Its architecture uses:

grouped-query attention with 6 query heads and 2 key/value heads
64-dimensional attention heads
1,216-dimensional MLP layers
RMSNorm with Q/K normalization
rotary positional embeddings
tied input/output embeddings
xIELU activations
a rank-12 residual output head
a 16,384-token byte-level BPE vocabulary

The final training stream contains exactly 1 billion materialized tokens drawn from a mixture of FinePDF text, FineWeb-Edu, Ultra-FineWeb, Cosmopedia V2, and WikiText-103.

The pipeline applies normalization, filtering, deduplication, deterministic mixing, and benchmark decontamination. Public benchmark examples are checked using normalized 12-word shingles before training.

The selected Stage 3 run used BF16 mixed precision, AdamW, gradient clipping, cosine learning-rate decay, deterministic seeds, and an effective batch size of 131,072 tokens.

Training was performed on a single NVIDIA GeForce RTX 4060 Laptop GPU. The selected run processed 1,000,079,360 tokens in 7,630 optimizer steps and took 24,595.263 seconds.

For evaluation, we integrated the native J3 model directly with lm-evaluation-harness. The submitted results use a full zero-shot evaluation rather than smoke-test or manually selected subsets.

Challenges we ran into

The first constraint was architectural: every embedding, attention layer, MLP, normalization parameter, and output component had to fit under the competition's 50-million-parameter limit. This forced us to treat parameter allocation as a real design problem rather than simply scaling a standard architecture down uniformly.

The second challenge was training under commodity hardware constraints. An RTX 4060 Laptop GPU provides limited memory, so batch sizing, BF16 execution, gradient accumulation, checkpointing, and data streaming all had to work reliably within that environment.

A third challenge was experimental discipline. We explored later continuation and multiple-choice post-training experiments, including runs that were unsuccessful or produced results we did not consider appropriate for the final submission. Rather than silently replacing the experiment history, we retained those records and explicitly selected the Stage 3 1B-token model as the competition release.

Finally, reproducibility required more than saving a checkpoint. We had to bind together the tokenizer, data manifest, configuration, environment, training logs, benchmark protocol, and final tensor so that the published numbers referred to one unambiguous model.

Accomplishments that we're proud of

The main accomplishment is completing an end-to-end foundational language-model training pipeline under the competition constraint using a single consumer laptop GPU.

J3 remains below the 50M parameter ceiling while completing a full 1B-token training run and the complete required benchmark suite.

We are also proud of the audit trail around the model. The repository includes deterministic data manifests, tokenizer and checkpoint hashes, raw training records, benchmark metadata, experiment history, and a verification script instead of asking judges to trust manually copied numbers.

The final checkpoint is published separately as a GitHub Release and cryptographically bound to the repository through SHA-256.

What we learned

The project reinforced that small-model development is not simply a miniature version of large-model training.

Under a strict parameter and hardware budget, architecture allocation, vocabulary size, data quality, batching strategy, and evaluation methodology become tightly coupled. Small changes can consume a meaningful fraction of the entire resource budget.

We also learned that experiment management matters almost as much as the final training run. Keeping rejected experiments visible while clearly defining one submission boundary made it much easier to determine which measurements actually belonged to the final model.

What's next for J3

J3's competition release is intentionally frozen at the Stage 3 1B-token model so that its benchmark results remain reproducible.

Future work could study more compute-efficient parameter allocation, alternative data mixtures, longer-context training, improved post-training methods, and stronger reasoning performance while preserving the same small-model and auditable-training philosophy.

Most importantly, the repository provides a reproducible baseline for asking a broader question: how much useful capability can be obtained from language models small enough to train and study on ordinary consumer hardware?