Why I built it
Gemma 4 26B A4B is sparse, but its expert banks still have to live somewhere. Stock llama.cpp can keep routed-expert work on CPU, which fits but creates a bottleneck, or move whole layers to CUDA, which is too coarse for this 16 GB memory budget.
ExpertFlow is a placement compiler for quantized mixture-of-experts models. It profiles the routed-expert work separately and places only the expert banks from the highest-value layers on CUDA.
What it does
The verified Q6 deployment uses complete 128-expert static banks for layers [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 20]. Packed Q6 experts are copied once into persistent CUDA storage with identity mapping. The shipped configuration has no reactive cache, eviction, predictor, repacking, or per-token expert transfer.
The release includes a small CLI. Judges can reproduce the headline result without the 22.9 GB model or a GPU: uv sync --frozen uv run expertflow demo --replay
The replay hash-checks the committed evidence and reconstructs the measured stock and ExpertFlow results. A compatible Windows/NVIDIA system can also inspect the runtime, generate a deployment manifest, run live inference, benchmark it, or start an OpenAI-compatible local server.
Measured result
On a 16 GB RTX 5060 Ti using Gemma 4 26B A4B IT Q6_K:
ExpertFlow: 28.13 decode TPS Strongest fair stock Q6 configuration: 22.967 decode TPS Improvement: 22.48% Peak process-owned VRAM: 10,966.801 MiB Matched protocol: ten 512-token runs
A separate five-repetition, four-slot server test completed 20/20 requests at 35.6699 aggregate generated TPS versus 24.5231 stock. That is a concurrent server-throughput result, not the single-stream headline protocol.
How it works
ExpertFlow reads measured routing and backend-placement evidence, ranks complete expert banks by CPU relief per byte of VRAM, and emits a deployment manifest. The runtime establishes CUDA placement before graph construction, so the selected packed expert operands remain on CUDA throughout execution.
Predictive caching looked attractive early, but a bounded simulation using measured routing and transfer data found no configuration worth shipping on this GPU. The measured answer was full static residency, so that is what the product uses.
Quality and limitations
I kept the claims deliberately narrow:
MMLU moved from 49/100 to 50/100. The perplexity point estimate improved by 2.92%, but the 95% upper bound was +2.25%; the strict +1% non-inferiority gate was not met. Four-slot outputs were not fully deterministic across repetitions. A 262,144-token context was allocated with 675.418 MiB reserve, but the bounded run processed 417 tokens; this is not a filled-context claim. Predictive caching was simulated and rejected. It is not a live-runtime claim and is not shipped. Live acceleration is verified only on the documented Windows 11/NVIDIA setup. Evidence replay is portable.
How I used Codex and GPT-5.6
Codex helped build the project across isolated worktrees: runtime instrumentation, correctness tests, placement experiments, benchmark harnesses, the CLI, release packaging, and reproducibility checks. I set the scientific gates and product direction, and rejected approaches when the evidence did not support them.
GPT-5.6 helped scope bounded experiments, diagnose failed cache and placement paths, keep an append-only decision ledger, interpret measurements without overstating them, and turn the final verified runtime result into a judge-friendly product workflow.
Log in or sign up for Devpost to join the conversation.