Pocket Titan

Github Repo
model.py code
HuggingFace
Training Curve
System Summary
Benchmark Results
Demo Generation Output
Demo Model Loaded
Verification
GIF
Our Thumbnail

Inspiration

Most language models today are trained on large GPU clusters. We wanted to see how much of the real pipeline we could build, correctly, on an ordinary gaming laptop.

Track 01 gave us a hard limit: at most 50 million trainable parameters, trained from random weights. Our hardware gave us another one: one NVIDIA RTX 3050 Laptop GPU with 6 GB of VRAM.

Limits like these make every choice a trade-off. A bigger vocabulary means fewer layers. An untied output head alone would break the parameter cap. Memory caps the batch size, and the deadline caps how many tokens we can train on. So we set two goals:

Get as much language-modelling ability as possible out of that budget.
Be able to prove every number we report.

What it does

GIBC-43M is a 42,968,576-parameter decoder-only language model. We built every part of it for this project: the tokenizer, the data pipeline, the training loop, checkpointing, the Hugging Face export and the evaluation. No pretrained weights, fine-tuning or distillation were used.


Unique trainable parameters	42,968,576 (86% of the 50M cap)
Training data	1,000,013,824 prediction targets from FineWeb-Edu
Hardware	1 × NVIDIA RTX 3050 6 GB Laptop GPU (55 W), Ryzen 5 6600H, 16 GB RAM
Training time	15.76 h of training steps (15.87 h wall clock)
Approximate compute	about 2.6 × 10^17 FLOPs (6·N·D estimate)
Throughput	17,630 prediction targets/s (run average)
Peak GPU memory	3,167 MiB allocated, out of 6 GB
Stability	0 skipped steps, 0 NaN/Inf values, 0 restarts
WikiText-103 test token perplexity	46.245

The trained model is exported in standard Hugging Face Llama format, so anyone can load it with transformers. A terminal demo (scripts/demo.py) checks the checkpoint's SHA-256 hash against the recorded results and then generates text locally. No external model or API is involved.

How we built it

1. Spending the parameter budget

The model is a Llama-style decoder: 9 layers, hidden size 512, 8 attention heads (head size 64), a SwiGLU MLP of width 1,536, RoPE positions, RMSNorm, no biases, and a 512-token context. The output head is tied to the token embedding, so the two share one tensor.

$$ N = Vd + L\,(4d^2 + 3\,d\,d_{ff} + 2d) + d = 12{,}288{,}000 + 9 \times 3{,}408{,}896 + 512 = 42{,}968{,}576 $$

Tying was required, not optional. Without it the model would have 55,256,576 parameters, which is over the cap.
A 24,000-token vocabulary keeps the embedding at 12.3M parameters (29% of the budget). A GPT-2-sized 50k vocabulary would have used about 25.7M on the embedding alone, leaving much less for the layers that do the thinking.
We checked the count four ways: by formula, on the live model, on the exported Hugging Face model after a fresh reload, and in the training log.

2. Data: FineWeb-Edu, without downloading 28 GB

Source: FineWeb-Edu sample-10BT, pinned to one exact dataset revision.
Range reads: we used HTTP range reads to fetch only the parquet row groups we needed. The 28.5 GB sample was never downloaded.
Safe to restart: we built a 100 MB pilot first, then 27 hash-verified chunks that continue from the exact row where the pilot stopped.
Wikipedia filter: we dropped 9,857 documents from wikipedia.org to reduce overlap with WikiText. This is not full decontamination, and we don't claim it is.
Duplicate audit: we removed 87 validation documents whose exact text also appears in the training data.
Locked token files: the result is 1,100,000,632 training tokens and 10,753,165 validation tokens in read-only files. Their SHA-256 hashes are checked at every start and every resume.

3. Our own tokenizer

Byte-level BPE with exactly 24,000 entries, trained on our own pilot text (20,765 documents, 99 MB). It needs no "unknown" token, and every string round-trips exactly.
One special token, <|endoftext|> (id 0), placed between documents.
A hidden trap we fixed: web pages sometimes contain the literal text <|endoftext|>. By default, the tokenizer library turns that text into the special token. We force it to be encoded as plain bytes, and the Hugging Face export keeps the same rule.
Compression: about 4.52 bytes of text per token.

4. A training system in plain PyTorch

Setup: bf16 mixed precision with fp32 master weights; fused AdamW (β = 0.9/0.95, weight decay 0.1, gradient clipping at 1.0); learning rate warmed up for 256 steps to 1e-3, then cosine decay to 1e-4.
Measured, not guessed: we benchmarked micro-batch sizes 1 to 8 on the real GPU (11,972 up to 17,730 tokens/s). We picked 8 × 512 tokens × 4 accumulation steps, which is 16,384 targets per update. bf16 ran within 3% of fp16, so we used bf16 and skipped the loss scaler. A 120-second sustained test checked for thermal slowdown.
Learning rate chosen by a rule written down in advance: we ran two 1,024-step trials (6e-4 and 1e-3). 1e-3 won on validation loss (4.8853 vs 4.9451).
One-pass sampler: a seeded shuffle of non-overlapping 513-token windows, so every target is predicted at most once and no data repeats.
Exact resume: checkpoints store the model, optimizer, schedule position, sampler cursor and every random-number state, and are written atomically. We tested that a stopped-and-resumed run matches an uninterrupted one.
218 automated tests cover parameter count, weight tying, causality, resume, the sampler, export parity and more.

5. Evaluation we can trust

Before running any benchmark, we exported the checkpoint to LlamaForCausalLM and reloaded it in a fresh, offline process. There we confirmed that:

the parameter count and the weight tie survived the export;
all 84 weight tensors are identical to the source;
the tokenizer matches on 305 test texts;
CPU logits match exactly, and CUDA logits match within 4.0e-5;
lm-evaluation-harness's own log-likelihoods agree with ours within 3.8e-6.

Only after these checks passed did we run lm-evaluation-harness 0.4.13: 0-shot, on the full evaluation sets, once. We fixed the WikiText-103 method in code before the final model existed, and computed only that one version.

Results

Benchmarks (lm-evaluation-harness 0.4.13, 0-shot, full sets, final checkpoint):

Benchmark	Examples	acc	acc_norm	Random chance
HellaSwag	10,042	0.2711	0.2806	0.2500
ARC-Easy	2,376	0.4280	0.3927	0.2502
PIQA	1,838	0.5974	0.5860	0.5000
WinoGrande	1,267	0.5067	n/a	0.5000

Language modelling:

WikiText-103 test token perplexity: 46.245. This scores 303,524 tokens with our tokenizer, using rolling 512-token windows with stride 256. Each token is scored exactly once.
Internal FineWeb-Edu validation: loss fell from 10.17 to 3.457 (perplexity 31.74).

$$ \text{PPL} = \exp!\left(\frac{1}{N}\sum_{i=1}^{N} -\log p(x_i \mid x_{<i})\right), \qquad N = 303{,}524 $$

Training curve

Benchmark results

Efficiency:

Steady speed: 1.0B targets in 15.76 hours at 17.6k targets/s. Every logged 50-step window stayed between 17,511 and 17,718 targets/s, so the laptop never slowed down.
Memory to spare: peak memory was 3.1 GiB of 6 GB, so we didn't need activation checkpointing.
Well-balanced training length: about 23 training tokens per parameter, close to the ~20 that compute-optimal scaling research (Chinchilla) suggests.
Fast evaluation: the full evaluation suite (four benchmarks plus WikiText-103) runs in about 11 minutes on the same laptop.
Cool and stable: GPU temperature stayed between 53 and 75 °C under a 55 W power cap.

Live demo: our demo video runs python scripts/demo.py --fixed. It verifies the checkpoint hash, loads the model in 2.7 s, and generates 60 tokens per prompt in about one second on the laptop GPU.

Challenges we ran into

One shot at training. A 16-hour run inside a ~50-hour hackathon leaves no room for a second attempt. So we built and tested exact-resume and hash checks before the main run. The run then finished with zero skipped steps, zero NaNs and zero restarts.
The parameter cap vs. the vocabulary. An untied head would have meant 55.3M parameters. Tying the head and choosing a 24k vocabulary solved it, with 7.0M parameters to spare.
A bug in our own speed benchmark. Our first timer stopped before the optimizer step had actually finished on the GPU. We caught it, re-measured with proper GPU synchronization (17,685 vs 17,730 tokens/s), and only then locked the training config.
Loss-scaler skips counted as real steps. In fp16, a skipped update was being counted as an optimizer step, which would have shifted the learning-rate schedule. We fixed it, even though we ended up using bf16.
A tokenizer setting that doesn't save. The "treat <|endoftext|> as plain text" flag isn't stored in tokenizer.json. We had to carry it into the Hugging Face export by hand and add tests for it.
An unclear WikiText-103 protocol. The exact method wasn't specified, so we chose one, documented every detail (dataset revision, windows, stride, units), and locked it in code before training finished.
Windows and OneDrive. OneDrive file locks and sync load can get in the way of huge data files, so the code refuses to write big artifacts inside the repo or a OneDrive folder.

Accomplishments that we're proud of

A complete language-model pipeline on a laptop, from raw web text to a public Hugging Face model.
61,036 optimizer steps with zero failures.
Every number traces to a committed file in results/, with hashes, dataset revisions and the exact commands.
Clearly above chance where it counts for a model this size: +17.8 points on ARC-Easy and +9.7 points on PIQA, with 43M parameters and 1B tokens.

What we learned

Small models learn grammar before facts. Sampled text sounds fluent but is often wrong. For example, the model says water boils at a lower temperature at high altitude "because it is too cold".
Measure, don't guess. The batch-size sweep showed that speed gains flatten above micro-batch 6, which we would not have known otherwise.
Always report the chance baseline. WinoGrande at 0.507 is basically chance, and saying so plainly is more useful than hiding it.
Verification gates are cheap insurance. The export parity checks took minutes and removed any doubt about the benchmark numbers.
Write the rules down before you see the results. Fixing the learning-rate rule and the WikiText method in advance kept us honest.

Limitations

Weak generation: greedy decoding loops, and sampled text is often factually wrong.
Near-chance scores: WinoGrande is at chance and HellaSwag is only slightly above it.
Limited experiments: we did one production run with no seed repeats, and tried only two learning rates.
Partial filtering: the Wikipedia URL filter is not full benchmark decontamination.
Tokenizer-specific perplexity: the WikiText-103 number is per token under our 24k tokenizer, so it can't be compared directly with word-level perplexities.
No tuning or safety checks: there is no instruction tuning and no safety evaluation.

What's next for GIBC-43M

Faster generation: add a KV cache (the current generator recomputes the full context at every step).
Better use of the budget: try grouped-query attention or a smaller vocabulary to move more parameters into depth.
Measure variance: run seed repeats to see how much the scores move between runs.
More training: train on more tokens and compare against this run.
Easier to try: add a small instruction-tuning stage and a hosted demo on a Hugging Face Space, and pick an open license.

AI assistance (disclosure)

We used AI tools for development only. None of them produced the model's weights, training data, tokenizer or results.

Claude Code (Claude Opus 5.5): the main implementation agent. It wrote the code, tests, configs and docs, and ran the commands.
ChatGPT: planning, stage specifications and review.
Codex / GPT "Astra": an independent review and compliance audit.
Context7 MCP: documentation lookup.

A human set the requirements, made the decisions and reviewed every stage. Full details are in docs/AI_ASSISTANCE.md.