-
Poster
Inspiration
Every "autoresearch" tool released this year works one kernel at a time. But, when I measured where a 4B model actually loses its tokens per second at decode, it was not inside the kernels. Native Transformers spends 17 to 22 ms per step on an H100 against a 2.4 ms memory-bandwidth floor, and that time is launch overhead, Python dispatch and activations bouncing through HBM between operations. The Dryft challenge (make Qwen3 4B decode faster without changing the output) was the right place to try out whether an agent loop searches the structure of the whole decode step, and can it be trusted to do so under a correctness gate that fails on one wrong token?
What it does
neoKernel is an autoresearch harness for LLM inference engines, and the Qwen3-4B engine it produced.
The judge replicates Dryft's evaluation on Modal: a fresh engine process per workload, one warmup of the exact shape, timed samples with the clock in a separate process, then a teacher-forced replay of the emitted tokens through untouched native Qwen to apply the 2.0-logit rule at every position. Correctness runs on an L4, throughput on an H100, so an experiment costs cents and most die before they reach the H100.
The guard exists because two 2026 studies caught agents cheating on benchmarks like this one . Agents can write only under engine/ and the judge is a separate process they cannot import. Prompts are fresh random ids they never see and linting refuses time, CUDA events, sys.modules and anything not importable in Dryft's sandbox; and any result faster than the weight floor is flagged as a suspected hack.
The loop takes whole-file proposals from a model on Baseten's Model API, runs them through guard, a repair stage (the model sees its own unit-test failures and gets two turns to fix them), the judge, and a git keep-or-revert with tags, backups and a resumable transaction journal.
The engine is what came out: . Official Dryft score: 932 tokens per second against the starter's 217.3, every token identical to greedy.
How we built it
Solo, in the order the harness demanded.
Friday night I started with the judge, calibrated against the official run of the unchanged starter per-step time was slower because Modal's host core is slower and native decode is single-threaded Python, so gates became relative to native measured in the same container.
Saturday the engine, through the same funnel, by AI coding agents pointed at one item at a time: static cache and graph capture first , then the hand-rolled step that replaced Transformers' sixty-kernel layer with a dozen.
Sunday morning was mostly the agent loop, GLM 5.2 then Kimi on Baseten, with a repair stage added after the first ten one-shot proposals produced zero keeps. Then, moving onto the more advanced models like Astra and Fable 5.1
No human wrote engine code. The human built the judge, wrote the playbook, and chose which model to point at which item. The log shows the rest.
Challenges we ran into
The correctness gate is the whole problem. Reordering arithmetic is fine; moving a bf16 cast is not, and the reference rounds the normalized value before the weight multiply. Every one of the cheap model's one-shot failures was a numerics failure of that kind.
The local judge disagreed with Dryft twice, in opposite directions: a change that scored +9 percent locally scored −4 percent on the board, and one the judge reverted on speed scored a new best on the board. The correctness verdict never disagreed. So the judge became a correctness gate and a coarse filter, Dryft became the speed judge, and the keep rule gained a no-shape-may-regress clause. That is an evaluator-fidelity finding, with numbers, and it is on the page.
Random-token prompts contain no repeated n-grams, so prompt-lookup speculative decoding cannot be measured locally at all. Modal mounted results/ with the app and a lock file inside it killed initialization for an evening. The Dryft CLI got a non-JSON 403 from a proxy. Codex's sandbox refused to send repository context to Baseten, so the loop had to run from a plain shell. Each of these is in the log.
Accomplishments that we're proud of
A 3.9 × engine on the leaderboard, token-identical to greedy, built entirely by agents through a judge that replicates the real one. A repair stage that turned a 0-for-10 model into one that landed a structural change on the first try. A guard and a judge that caught a wrong result and a gamed-shape result before either touched the board.
What we learned
The evaluator is the product. Agents are only as good as the thing that decides what stays, and this weekend the thing that decided was off by ±10 percent on speed and exactly right on correctness; knowing which is which is worth more than any single kernel.
Cheap models fail architectural moves and can land bounded ones if they see their own test failures. Strong coding agents with a shell land the rewrites. Numeric knobs belong to a sweep. That is what InferenceBench and AutoKernel measured, and our log reproduces it on a harder gate.
Tolerance is a budget. Two logits of slack per position is a resource every fused kernel spends, and nobody measures it. That is the next thing the harness will report per kernel.
What's next for neoKernel
Corpus-derived prompts so speculative decoding can be measured locally; wider hidden-shape guesses and a keep threshold set from the measured local-versus-Dryft error; per-shape configuration so a change that wins at batch 1 and loses at batch 32 is chosen per shape; a numerics localizer that names the first layer where a failed candidate diverges; a merge queue for parallel workers; and the model comparison the harness was built for.
Log in or sign up for Devpost to join the conversation.