OK: the Odyssey of a Kernel

OK: The Odyssey of a Kernel
Architecture Graph

Inspiration

Every AI company pays senior engineers to make one matrix multiply 20% faster. It is the highest-paid, least scalable work in the industry — and most of it is not insight. It is toil: designing sweeps, building scaffolding to run them, re-verifying every edge case after every change, and maintaining the map of what was tried, which is the first thing abandoned under deadline.

Two external results convinced us this toil can be delegated. KernelBench shows how hard the problem is: frontier models beat the PyTorch baseline on well under 20% of kernel tasks. MIT HAN Lab's team took 1st, 2nd and 3rd across the NVIDIA tracks of the MLSys 2026 FlashInfer contest with a fully agent-driven workflow — and their ablation found that the harness (a plan/execute/verify loop with recording discipline), not model capability, did most of the work.

So we did not set out to write a kernel. We set out to build the ship that finds kernels — and named it after Homer's "man of many turns." The kernel is the cargo; the map is the point.

What it does

Odyssey is an autonomous agent framework for GPU kernel optimization. At its core is one loop the agent runs end to end:

Plan the attempt;
Implement the candidate;
Measure through the organizer's unmodified evaluator, the only judge;
Profile the next move comes from the timeline, not intuition;
Record every measurement into an append-only ledger; every candidate, rejected branches included, onto the search map;
Review adversarial, fresh-context, before the next round.

Five iterations per direction, then move on. Three phase prompts steer the loop — correct first, then fast where the profile says, then specialized per shape — with research skills for kernel techniques and profiler reading. The agent takes the toil; the human keeps the helm: directions, targets, judgment.

The deliverable is always twofold: the kernel, and the full map of the search — a benchmark ledger where every row carries the evaluator's md5, the git sha and the exact flags, plus a candidate DAG in which even dead branches have a burial record.

How we built it

The framework is deliberately not code. It is what the agent reads — three phase prompts, four research skills, a rules file — and what it is forced to write: benchmark.csv, solutions.jsonl, profiler reports. The only Python per task is a thin verify.py that patches a candidate into the organizer's vendored, unmodified benchmark and reads the verdict off its own main() and exit code.

The kernels grew as an evolution tree: a fused-QKV body → manual CUDA-graph capture → the same body compiled inside the candidate (torch.compile reduce-overhead, masks built in-graph) → a handwritten Triton online-softmax flash-attention kernel with TF32 tensor-core dots → a hybrid that traces that Triton kernel into a compiled body so Inductor fuses the LayerNorm/FFN glue around it. A dispatch layer routes each shape to whichever candidate won it under a conservative admission rule — correct on every case in the group, dense and padded, worst-case speedup above 1.05× — otherwise it falls back to the baseline path.

Stack: PyTorch 2.8 / CUDA 12.8, Triton for the custom kernels, torch.compile + CUDA Graphs + SDPA, Nsight Systems (nsys) and torch.profiler for evidence, matplotlib for the roofline. Dev tools: Claude Code sessions in tmux driving the loop, the humanize plan/execute/review plugin, git, uv. Hardware: one rented RTX 6000 Ada on RunPod, one day. Data & assets: the organizer's benchmark script (vendored, md5-pinned) and its 14 official shapes, plus derived smoke/dev/stress shape sets; vendored KernelWiki and ncu-report skills (MIT HAN Lab, MIT-licensed). No external APIs — everything runs locally on the GPU box.

Challenges we ran into

The judge's own compiler fails the judge. The obvious opponent, torch.compile max-autotune, drifts 0.0053 (fp32) and 0.0625 (bf16) from its own reference — past the benchmark's own tolerance. The agent re-anchored on the fastest numerically admissible configuration, and later recruited that opponent as a candidate, making "never behind the compiler" a table property rather than a hope.

Measurements lie by default. Running the sweep in one process tripped dynamo's recompile limit, silently degrading "compiled" baselines to eager — our first opponent numbers were fiction. The fix (fresh process per case) is now protocol, and the trap is recorded as a lesson. Same day: a 192-thread host made a 0.47 ms CPU case take 1.8 s until thread-pinned.

No hardware counters on a rented pod. ncu is denied (ERR_NVGPUCTRPERM), so all profiling evidence is nsys timelines and torch.profiler tables — named as such — and the roofline is drawn against ceilings measured on the card (TF32 GEMM 70.7 TFLOPS, IEEE 30.3, copy 803 GB/s), never spec sheets.

A shape the benchmark itself cannot run. At S=100,000 the reference must materialize \( O(B \cdot H \cdot S^2) \) attention scores — about 12.8 TB. No hardware runs it. We built a chunked reference, proved it bit-exact against the unmodified script at S=2048, then verified our kernel at S=100,000 element-by-element: labeled off-script, never claimed as an official pass.

Accomplishments that we're proud of

Never slower than the baseline. Anywhere. 13/13 official shapes at the zero-bad-element tolerance — worst case 1.23×, best 12.3×, and 16.1× where attention dominates.
Faster than its own compiler — ahead of the strongest numerically admissible torch.compile configuration on every measured lane.
The impossible shape, served: S=100,000 in 23.2 s, zero bad elements out of 3.28 billion, judged by the script's own comparator.
Zero error where it counts: bf16/fp16 lanes ship bit-exact (max_abs = 0) at ~2.5×, on the one axis where no compiled path is numerically legal at all.
At the wall, and able to prove it: the GEMM-bound shape lands at 77% of the card's measured TF32 roof.
Behind the numbers: ~400 recorded measurements on a 19-node search map, every wrong turn included.

What we learned

One benchmark, four problems. The official grid hides four regimes — launch-bound (116 kernel launches per forward), attention-bound, GEMM-bound, and memory-impossible. No single kernel wins all four; a dispatch policy does.
Verify your judge before you race it. The strongest baseline was not a legal one — a fact the agent had to measure to discover.
On the launch-bound end, the win is fewer kernels, not faster kernels.
Speed/accuracy trades must be named candidates, never flags — IEEE fp32 runs at 30 TFLOPS where TF32 runs at 71 on this card; calibration, not opinion, decides which ships.
The discipline reproduces discoveries. In our ablation, a harness-run arm with no skills independently rediscovered the inadmissible-compiler finding in 12 minutes; a bare arm got fast numbers but shipped a fallback it never tested. The harness converts effort into ceiling, coverage, and findings that reproduce.

What's next

A second operator through the same three phases, on a different card — the scaffold (new_workspace.sh --from) already carries the evaluator and candidates across while leaving the evidence behind, because a number measured on one card is not evidence about another. Then: a box with hardware counters to upgrade the roofline from analytic to counted, and the input_scale axis swept rather than smoke-tested. The design bar stays what it was on day one: could a platform team take this harness and run it on their own operators tomorrow?