ANKA-50M

complete architectural diagram

Inspiration

Large language models are usually improved by increasing parameter count, training data, and compute. ANKA-50M began with a different question:

How much useful language capability can we obtain from fewer than 50 million parameters and a single GPU?

I built ANKA-50M as an independent research project to explore whether recurrent computation, learned self-correction, and sparse specialization could improve the efficiency of a very small language model. The goal was not to imitate a large model at a smaller scale, but to design an architecture specifically around a strict parameter and compute budget.

What it does

ANKA-50M RMW3 is a 48,944,657-parameter English foundational language model trained from scratch on 5 billion tokens.

It can:

Complete English text
Generate short explanations and structured passages
Continue scientific, educational, and narrative prompts
Perform zero-shot multiple-choice language benchmarks
Run locally on systems with sufficient CPU or GPU memory
Serve as a compact base model for later instruction tuning

ANKA is a base completion model rather than a chat-aligned assistant. No pretrained model, distillation target, or external model initialization was used.

How I built it

The model uses a decoder-only Transformer foundation with:

512-dimensional hidden representations
13 physical Transformer blocks
Two recurrent passes, producing 26 effective block applications
Grouped-Query Attention with 8 query heads and 2 key-value heads
RMSNorm, RoPE, SwiGLU, and tied input/output embeddings
A 16,384-token tokenizer
A 1,024-token context window

The architecture adds three main ideas.

ANKA Reflect

Reflect compares representations produced across recurrent passes and applies a learned low-rank correction.

Instead of treating the second pass as a simple repetition, Reflect allows each shared block to modify its state based on what changed between passes. This gives the model a lightweight form of iterative refinement without duplicating the full Transformer stack.

In a 4,096-token ablation sample:

Full model loss: 2.6290
Reflect disabled: 2.8961
Loss degradation when disabled: +0.2671
Every one of the 13 Reflect units produced a positive individual ablation contribution on the measured sample

ANKA Morph

Morph is a sparse transformation system composed of four stages. Each stage contains four low-rank experts and a learned router that selects a transformation for each token.

This gives tokens access to specialized transformations while keeping the total parameter count below 50 million.

In the same ablation sample:

Full model loss: 2.6290
Morph disabled: 2.7969
Loss degradation when disabled: +0.1678
All four Morph stages produced positive individual contributions
Shifting router-to-expert assignments increased loss by 0.0266, showing that the learned routing decisions were meaningful

ANKA WorldPairs

WorldPairs is a structured data component containing paired relational examples involving order, position, causality, negation, state changes, and physical relationships.

The final 5-billion-token corpus contained 200 million WorldPairs tokens, representing 4% of training data. These examples were trained through the same next-token prediction objective as the rest of the corpus.

Training

ANKA-50M was trained from scratch using:

Training tokens: 5,000,000,000
Hardware: One NVIDIA A100-SXM4 80GB
Precision: BF16
Batch size: 64
Sequence length: 1,024
Training steps: 76,294
Training time: 42,409 seconds, approximately 11 hours 47 minutes
Average end-to-end throughput: 117,899 tokens per second
Training efficiency: approximately 424 million tokens per GPU-hour
Best internal validation loss: 2.3344

The complete training pipeline, architecture implementation, experiment notebooks, parameter verification, evaluation scripts, and architecture audits are publicly available.

Results

The final base checkpoint was evaluated with zero-shot settings.

Benchmark	Metric	Result
HellaSwag	Accuracy normalized	28.46%
ARC-Easy	Accuracy normalized	42.26%
PIQA	Accuracy normalized	56.80%
WinoGrande	Accuracy	51.46%
WikiText-103	Perplexity	25.97

ANKA was designed primarily as an architecture and efficiency experiment. Its strongest result is not that it replaces much larger models, but that a model below 50 million parameters can combine recurrent depth, learned correction, and sparse specialization while remaining fast enough to train from scratch on one GPU.

Challenges

The strict 50-million-parameter limit affected every design decision. Increasing vocabulary size, hidden width, expert capacity, or normalization parameters meant reducing capacity somewhere else.

The most difficult challenges were:

Designing meaningful architectural additions without exceeding the parameter limit
Preventing sparse expert routing from collapsing
Making recurrent weight sharing improve capability rather than simply repeating computation
Building and tokenizing a multi-billion-token corpus
Keeping the data pipeline fast enough to feed a small model on an A100
Measuring architecture contributions separately from ordinary validation loss
Recovering safely from interrupted Colab sessions and GPU-memory failures
Packaging custom model code and weights into reproducible public releases

Several early Reflect and Morph versions failed to outperform the control architecture. I redesigned both systems, tested them independently, combined them only after ablation studies, and verified their contribution in the final RMW3 checkpoint.

What I learned

This project taught me that small-model development is not simply large-model development with smaller dimensions.

At this scale:

Tokenization consumes a meaningful part of the parameter budget
Kernel-launch and memory overhead can matter as much as raw compute
Parameter sharing must be paired with mechanisms that distinguish repeated computation
Sparse routing requires direct monitoring and ablation
Validation loss alone is insufficient for judging architectural changes
Data quality and composition can be more important than adding another layer
Reproducibility, throughput, and deployment are part of the model design

The most important lesson was that architectural ideas should be validated by disabling them and measuring what is lost—not only by observing that the complete model trains.

Accomplishments

I am proud that ANKA-50M:

Was designed and trained independently from scratch
Fits under the competition’s 50-million-parameter limit
Processes 5 billion training tokens in under 12 hours on one A100
Includes reproducible parameter-count and benchmark scripts
Provides public model weights, tokenizer, inference code, and training notebooks
Demonstrates measurable contributions from Reflect and Morph
Can be downloaded and run without relying on a hosted proprietary API