Tracking front-end
Exclusive-GPU guard.
Correctness is a wall, not a knob.
RTX 4070 Trajectory.
Tracking front-end

Ratchet — TikTok TechJam 2026, GPU Kernel Track

Inspiration

The challenge sounds simple: write a fast GPU kernel for a transformer layer. In practice, the problem is much narrower. The submission is graded against a pinned four-layer transformer reference across 14 announced shape configurations, all with a fixed tolerance.

Because every shape is known in advance, we realised that this is less about finding one universally fast kernel and more about dispatch: deciding which implementation should run under which conditions.

The differences between configurations make this especially important. Config 2, with a batch size of 1, spends much of its time on CPU dispatch overhead while running a relatively small GPU kernel. Config 6, with 1.28 million tokens, pushes its FFN to roughly 99–100% of measured DRAM bandwidth. A change that helps reduce bandwidth pressure is essentially irrelevant in the first regime, while it has little room to help in the second.

There is therefore no single kernel that wins everywhere. The real challenge is to build a small set of mechanisms and determine when each one should be used.

We also wanted to avoid simply memorising the 14 announced configurations. Every dispatch decision in Ratchet is therefore based on something we have actually measured from the device, rather than a hard-coded configuration ID or shape constant. In other words, we wanted to build a dispatch system that generalises from measurements rather than an answer key.

Going into the project, we also believed that the biggest obstacle to automating the expert workflow of “propose → build → benchmark → decide” would not be code generation, but trustworthy measurement.

That turned out to be correct. On this submission, nearly two-thirds of the final two days went into building and validating the measurement system rather than writing kernels.

What it does

Ratchet is built around two components: an evolutionary search loop that proposes, builds, and evaluates candidate kernels, and a measurement harness designed to be deliberately difficult to fool.

At the centre is an orchestrator that owns the single GPU lock. It selects both a candidate to expand and an idea to test using two separate Thompson samplers, then dispatches the work to a subagent.

Each result first goes through a cheap screening stage: a 30-second run across four configurations. This stage is advisory only. A candidate only enters the authoritative ledger after surviving the full sweep, with correctness checked before timing and measured in the same process using the harness's locked tolerance (atol 2e-3, rtol 2e-2, OR-combined).

The dispatch predicates themselves are based only on measured device properties such as bandwidth, occupancy, and launch counts. They never use configuration IDs or hard-coded shape constants. We also enforce this structurally: a test walks the compiled bytecode (co_consts) rather than relying on what the source code claims to do.

The current frontier, v43_replicated_tile, is generation 43 of a lineage that traces back through v26. It achieves a 3.46× geometric-mean speedup against torch.compile(mode="max-autotune") with TF32 across the 13 runnable configurations, while passing correctness on 13 of 14 configurations with 0 failed elements.

The remaining configuration is unusual: the reference implementation itself cannot run on existing hardware because it would require materialising an 18.63 TiB tensor. We still compute our result and verify it in two independent ways, but report it as a feasibility result rather than claiming a speedup against a denominator that cannot actually be measured.

How we built it

The search loop.

Ratchet uses four roles around a single scarce resource: one GPU.

An orchestrator owns the GPU lock, experiment ledger, and every merge. Researchers read papers, kernel repositories, and changelogs and turn their findings into scored proposals, but never touch the GPU. Expanders take a candidate–idea pair into an isolated git worktree and develop it independently. Finally, a verifier is responsible for attacking claims: can a check actually fail, was the subject ever built, and is there a positive control?

This separation keeps exploration from interfering with measurement and makes it possible to question the results independently of the code that produced them.

Two Thompson samplers, deliberately different.

We use one sampler to decide which candidate to expand and another to decide which idea to try.

For candidates, Thompson sampling operates on clade metaproductivity. A node's Beta posterior incorporates the outcomes of its entire descendant subtree. This means that a mediocre-looking candidate can still rank highly if it consistently produces strong descendants.

Getting the definition of “success” right took three attempts. We measured each version rather than assuming it would work. Initially, defining success as speedup > 1.0 against eager marked 88% of rows as successful and provided very little discrimination. Switching to the compiled baseline made the criterion even less useful. What finally worked was requiring improvement over the nearest ancestor as well.

The second sampler chooses which idea to test. Its Beta prior is based on the idea's rubric quality score, with its strength determined by inverted entropy. This gives uncertain but interesting ideas a weak prior that can change substantially after one measurement, while safer ideas begin with stronger priors.

Git is the evolutionary tree.

Each candidate is represented by a commit, while lineage is represented by git ancestry over a declared-parent graph. There is no separate tree database, so reproducing any result is as simple as git checkout <sha>.

This approach also exposed one of our own bugs. For 18 generations, candidate branches had been created from the tip of the trunk rather than from their actual parents. As a result, raw git ancestry appeared to show a single straight line even though the registry contained a branching evolutionary structure.

We now draw the declared lineage in the dashboard rather than relying on git ancestry. The disagreement between the two turned out to be the actual finding.

Visibility

All of these are accessible through the front-end which tracks performance as improvements are implemented. This shows the exact ideas implemented and results of each iteration.

Based on this, users are free to propose to the agent to follow a certain track. This way human expertise can still be leveraged.

Beyond this submission.

Although Ratchet was built for this challenge, the measurement infrastructure is designed to support a broader research workflow.

The system maintains an append-only, hash-chained experiment archive that can be selectively used to generate a paper without allowing negative results to disappear. A human-injection queue (IDEA / CONSTRAINT / PRIORITY / LITERATURE / REDIRECT) allows researchers to steer the search without manually editing experimental evidence.

We also use per-experiment git worktree lanes so that multiple people or hypotheses can be developed in parallel without contaminating each other's measurements. Finally, a fail-closed vendor-qualification path is being developed for non-CUDA backends such as XPU and HIP. If a backend has not actually been qualified on hardware, Ratchet records a formal no-run event rather than fabricating a result.

Challenges we ran into

The recurring challenge throughout the project was surprisingly simple: our instrumentation kept lying to us.

And it did so in ways that were both specific and expensive.

Two processes on one GPU inflated a baseline by 4.1×. Host spill over PCIe was responsible, and we only discovered it after enforcing exclusive GPU access.
Our timing protocol inverted two signs. Roughly 600 ledger rows had been measured with isolated arms: baseline first, candidate second. The device had become hotter between measurements, introducing a systematic bias. The official harness instead interleaves the two arms specifically to cancel this effect. Once we matched its protocol, two configurations flipped from “worse” to “better.”
A tuner was effectively ranking a table of ties. Our CUDA event timer had a 1.024 µs quantum, while the kernels being measured took only 1–13 µs. For 18 generations, the tuner had been selecting between measurements that were exact multiples of the timer's resolution without us realising it.
Noise on the GPU was one-sided. This created a counterintuitive problem: taking the minimum across repeated sweeps systematically favours the arm that happened to avoid contamination. As a result, two sweeps disagreeing can actually be the expected behaviour, even when the true performance margin is 28%. Our first attempt at fixing this — requiring two sweeps to agree — produced the wrong answer in 5 of 10 fresh processes. The eventual solution was to establish a floor for each arm once, rather than repeatedly voting between contaminated measurements.
Byte-identical code could still differ by 7.1%. At one point, we were debating a supposed 46% performance gap. After replicating the measurement properly, both arms came out at 52.22 µs, to the hundredth of a microsecond. The replication took less than two minutes.

Correctness created another major constraint. bf16 alone can fail our accuracy gate because of its representation: with only 8 mantissa bits, the spacing near 1.0 is roughly 0.008, even without any implementation bug. In another case, a residual-stream modification measured 1.4× faster but failed correctness on 11 of 13 configurations. Both had to be rejected regardless of how attractive their performance numbers looked.

The most subtle lesson was that where a performance win occurs matters more than how large the win is.

Under our capped scoring objective, a real and replicated 1.345× improvement on an already-saturated configuration contributes exactly 0.0000 to the score. The same improvement on a configuration just below the ceiling can account for an entire generation's score increase.

The reverse is also true: a regression on a capped configuration is equally invisible. This allowed a genuine 1.6× performance defect to survive two complete sweeps before we caught it.

Accomplishments that we're proud of

3.46× geometric-mean speedup against torch.compile(max-autotune)+TF32 — not eager — across 13 runnable configurations, while achieving 13 of 14 configurations with 0 failed elements on correctness. The results sit at 62.5–74.1% of the absolute tolerance budget rather than barely passing the threshold.
We chose to report the honest baseline. The same codebase achieves 13.7× against eager, and we report that number too, but only alongside the compiled-baseline result. An earlier version of this project had reported 7.2× where the fair baseline gave only 1.69×. That experience was a major reason we built the measurement discipline that Ratchet uses today.
We measured the noise floor instead of assuming it. We initially assumed around 3% measurement noise. Under byte-identical code and exclusive GPU access, the actual figure was closer to ±7%. That changed which performance improvements we considered genuine.
Seven previously accepted findings fell apart under better measurement during the final two days. Two inverted their original sign and two required us to withdraw published numbers. None was caught by the search loop itself. Every one was discovered through replication, a control arm, or simply asking what a measurement depended on that we had never varied.
Config 14 was handled honestly. Rather than report a speedup ratio with no valid denominator, we independently verified our result using both a causal-prefix theorem against the unmodified reference and a blocked fp64 evaluation of the reference's own arithmetic. We therefore report it as a feasibility result.
Every result is preserved. Accepted, rejected, and inconclusive measurements are stored in an append-only ledger keyed to (commit_sha, config_id). Nothing is quietly removed just because it makes the final story less convenient.

What we learned

The most repeated lesson throughout the project was the question:

“What does this depend on that we never varied?”

That question went 7-for-7. It exposed dependencies on padding ratio, the eager-versus-compiled baseline, dtype, input scale, allocation context, process contention, and the causal flag.

None of these seven issues were discovered by the automated search loop. Every one came from a human or the verifier deliberately asking the question.

We also learned that profiling is not a supporting detail that comes after the “real” engineering. It is a major part of the engineering itself. Almost every dramatic early result eventually turned out to be a measurement artefact — an unwarmed run, shared-GPU contention, or a timer coarser than the kernel being measured — rather than a genuine performance effect.

Another important lesson was to treat “no result,” “negative result,” and “the reference itself cannot run this configuration” as first-class outcomes. Recording these results permanently, rather than quietly dropping them or forcing them into a speedup ratio, is what makes the rest of the dataset trustworthy.

Finally, we learned that the mathematically largest performance improvement is not necessarily the improvement that matters. Under a capped scoring objective, the win that moves the score can be very different from the largest raw speedup. Optimising for the latter without considering the former wastes effort on saturated configurations — and can even allow real regressions on those same configurations to go unnoticed.

What's next for Ratchet

Finish the current frontier. Land the candidate branches that were still being measured when we last snapshotted the frontier, rather than shipping while a sweep is still in progress.
Expand beyond CUDA. Extend the fail-closed vendor-qualification path to a real non-CUDA backend, such as XPU or HIP, on actual hardware. This would close the loop that our current design-only lane leaves open.
Improve lineage representation. Resolve the remaining tension between our declared-parent lineage graph and genuine two-parent recombination, which git can naturally represent through merge commits.
Improve tuner resolution without losing measurement robustness. Now that we understand the event-quantum table-of-ties problem, we can increase the tuner's resolution. We will pair this with floor reduction on every measurement so that a sharper instrument does not reintroduce the one-sided contamination problem we worked so hard to eliminate.