HelixDepth

training_in_gpu
codebase
cover
architecture
benchmark
training cost and memory

The question behind HelixDepth

My laptop has 16 GB of RAM and no GPU. I’m an 18-year-old student in the Philippines, and this was my first solo hackathon and my first time training a language model from scratch. With $15 set aside for cloud GPU time, I had to make every experiment count.

I wanted to test a simple question: could a small language model process text more deeply by reusing the same learned blocks, instead of storing a completely new set of blocks for every stage?

What I built

HelixDepth is a language model that learns by predicting the next piece of text. I compared it with a conventional Transformer baseline.

The baseline has six independent Transformer blocks and processes text through them once. HelixDepth stores six blocks too, but reuses them across twelve processing stages. A small conditioning network creates stage-specific, low-rank adjustments, so the reused blocks can behave differently at different stages.

Both models were trained from random initialization. They used the same tokenizer, training data, and 299,982,848 next-token targets each. The baseline has 40,968,320 trainable parameters; HelixDepth has 41,184,432—about 0.53% more. That makes them approximately parameter-matched, though HelixDepth does more computation.

What the evidence says

On a fixed WikiText-103 test slice, HelixDepth reached 85.42 perplexity, compared with 86.51 for the baseline—a 1.26% reduction. Lower perplexity means the model was less uncertain about the next token on that text.

The other results were mixed: HelixDepth tied HellaSwag, scored lower on ARC-Easy, and scored higher on PIQA and WinoGrande. Its recorded training time was about 2.1 times longer, and it used more GPU memory.

So this experiment does not show that HelixDepth is faster, cheaper to train, or broadly better at reasoning. It does show a modest improvement in text prediction under this matched-data, approximately matched-parameter comparison. There was one training run per model, so the result needs more seeds and targeted ablations before we can tell which part of the design helped.

Why I think it matters

The value of HelixDepth is the question it makes testable: when compute is limited, can carefully reused weights make a small model more capable of processing a sequence? The answer here is promising in one measure, mixed across benchmarks, and not yet a claim of efficiency.

I built and checked the training, recovery, and evaluation pipeline, then released the code, results, and trained HelixDepth model so other people can inspect and reproduce the work. I used Codex for implementation and debugging; I’m responsible for understanding the experiment and reporting what the evidence does—and doesn’t—show.

This project took an idea I had studied in theory and turned it into a real experiment I could run, measure, and share. HelixDepth is my first step toward learning how to ask better questions about efficient language models.