gpu kernels

11.44× geometric-mean speedup over the 13 runnable published shapes, every output inside the atol 0.002 / rtol 0.02 gate, organiser's benchmark file untouched.

The finding

The baseline is launch-bound: ten of the 13 shapes take about 2 ms regardless of batch size, width or head count, across two orders of magnitude in FLOPs. That is 132 kernel launches per forward; the first lever is removing launches, not arithmetic.

What we built

We classified every shape as launch-, bandwidth- or compute-bound before the first GPU run and committed the prediction. A five-rule dispatch table routes each shape to one of four kernel families: CUDA Graphs plus torch.compile for launch-bound shapes, an fp16 GEMM path where compute dominates, SDPA flash attention where attention dominates, and a batch-chunked forward for shape 14. Speedups run from 1.99× (compute-bound, d=1024) to 28.26× (batch 1).

Shape 14 (B=32, S=100,000) cannot run in the baseline: it OOMs at the first projection on any 32 GB GPU. We report it as OOM and time our kernel against our own validated chunked reference (75×, outside the geomean).

How we know the numbers are real

On a rented multi-tenant GPU, every speedup carries its spread over three process runs and sits beside an identity null: a do-nothing kernel timed the same way, which should read 1.00× and doesn't quite. Two free controls are published for every shape: the organiser's --compile-baseline, and SDPA alone, which gets 2.3 to 3× on most shapes.

Why it matters

Batch 1 is the interactive-serving case; S=100,000 is long context. Each row is priced in tokens/s and dollars per billion tokens. The method (measure, classify the regime, dispatch by table) transfers beyond these 14 shapes.

Cost and limits

One RTX 5090, about a dollar an hour, 36 hours. Pinned versions, no runtime autotuning. Caveats: the large multipliers partly price away launch overhead rather than arithmetic; shape 14's denominator is ours; bf16 fails the gate against a TF32 reference.

Tools

Development tools: Claude Code, git, uv, Vast.ai. APIs: Anthropic Claude via Claude Code, which ran the build as an autonomous loop: workload analysis, regime prediction, kernel writing and gating. Lane briefs and reports are in briefs/ (tech report §8). Libraries: PyTorch 2.11 (cu128), Triton 3.6, CUDA 12.8, CUDA Graphs, SDPA, Pillow/OpenCV. Datasets: none; inputs are the organiser's synthetic tensors at seed 1234.

Built With

• claude