ExpertFlow

An optimA placement compiler that runs high-quality quantized MoE models faster on memory-constrained GPUs.al placement compiler for MoE models.

Comment

ExpertFlow

ExpertFlow is a developer tool. The fastest judge path does not require the model, CUDA, or an NVIDIA GPU.

Overview

Inspiration

I wanted to run the high-quality Q6 version of Gemma 4 26B A4B on a 16 GB GPU without routed-expert computation becoming trapped on the CPU.

Mixture-of-experts models are sparse, but their expert banks still consume significant memory. Stock layer-offload controls are too coarse: a layer can appear GPU-offloaded while its expert matrix multiplications still execute on the CPU. That hidden boundary became the real problem ExpertFlow set out to solve.

The original idea was predictive expert caching. Testing showed that the more fundamental opportunity was placement—deciding which complete expert banks should live on the GPU before execution begins.

What It Does

ExpertFlow is a placement compiler for quantized MoE models.

It profiles the model and hardware, identifies where routed-expert work actually executes, measures the CPU relief provided by each expert bank, accounts for its complete packed size and a safe VRAM reserve, and emits a hardware-specific deployment plan.

For the verified Gemma 4 26B A4B Q6_K configuration, ExpertFlow places complete 128-expert banks from twelve selected layers on CUDA. The selected packed experts remain resident throughout execution, with no eviction, repacking, prediction, or per-token expert transfer.

On an RTX 5060 Ti 16 GB, ten matched 512-token runs measured:

  • ExpertFlow: 28.13 decode TPS
  • Strongest fair stock configuration: 22.967 decode TPS
  • Improvement: 22.48%
  • Peak process-owned VRAM: 10,966.801 MiB

The project also provides a CLI for evidence replay, hardware inspection, placement optimization, local inference, benchmarking, and OpenAI-compatible serving.

Technical Details & Development

How I Built It

ExpertFlow combines a pinned llama.cpp C++/CUDA runtime with a Python CLI, benchmark harnesses, deployment manifests, evidence validation, and an offline judge replay.

GPT-5.6 was involved throughout the entire ideation and project progression. It helped convert the original cache concept into a sequence of bounded experiments, interpret results, establish correctness and performance gates, and reshape the product when measurements contradicted the initial direction.

Codex with GPT-5.6-sol managed the engineering workflow end to end:

  • Creating and protecting isolated Git worktrees
  • Investigating llama.cpp’s real MoE execution path
  • Instrumenting router and expert execution
  • Writing tests before narrow runtime experiments
  • Collecting routing traces, timings, VRAM measurements, and hashes
  • Implementing observer, cache, predictor, and placement prototypes
  • Diagnosing crashes, parity failures, and performance regressions
  • Running deterministic correctness and cleanup checks
  • Maintaining the append-only engineering ledger
  • Packaging the CLI, replay, dashboard, build scripts, and release archive
  • Handling the continuous testing, refinement, and release polish

I selected the problem, set the scientific gates, approved scope changes, and made the final product decisions. Codex handled the implementation and verification loop needed to make those decisions from evidence rather than intuition.

How I Used Codex and GPT-5.6

  • Codex: Helped build the project across isolated worktrees: runtime instrumentation, correctness tests, placement experiments, benchmark harnesses, the CLI, release packaging, and reproducibility checks. I set the scientific gates and product direction, and rejected approaches when the evidence did not support them.
  • GPT-5.6: Helped scope bounded experiments, diagnose failed cache and placement paths, keep an append-only decision ledger, interpret measurements without overstating them, and turn the final verified runtime result into a judge-friendly product workflow.

Challenges I Ran Into

The hardest challenge was discovering that reported GPU layer placement did not necessarily describe where routed-expert operations actually executed.

Several technically plausible approaches failed under measurement:

  • The first routing observer changed graph behavior and had to be quarantined.
  • Reactive LRU caching preserved exactness but synchronous transfers made it slower.
  • Temporal and next-layer predictors produced useful routing structure, but useful prefetches were not consistently ready soon enough.
  • An early whole-layer placement approach crossed an architectural boundary that would have required broad scheduler changes.
  • Increasing cache capacity reduced misses without producing an end-to-end speedup.

The key challenge was resisting the urge to protect the original cache idea. Every experiment had explicit stop conditions, and failed approaches were preserved as evidence. That process eventually exposed a narrower, faster, and more maintainable solution: compile complete expert-bank placement before graph construction.

Key Achievements & Lessons

Accomplishments That I'm Proud Of

The biggest accomplishment is turning a low-level runtime investigation into a reproducible developer product rather than leaving it as a benchmark patch.

I am especially proud that ExpertFlow:

  • Runs the high-quality Q6 model faster than the strongest comparable stock configuration on the same 16 GB GPU
  • Achieves a measured 22.48% decode TPS improvement
  • Uses complete packed Q6 expert banks without per-token repacking
  • Produces a hardware-specific deployment manifest instead of hard-coding a universal layer list
  • Preserves failed experiments and decision evidence
  • Includes a hash-verified, GPU-free judge replay
  • Provides a one-command live matched TPS demonstration
  • Packages the pinned llama.cpp source patch series and complete clean-build path
  • Includes an OpenAI-compatible local serving workflow
  • Keeps every headline result tied to machine-readable evidence

The release is also fully tested and reproducible from the public repository.

What I Learned

The most important lesson was that an MoE performance problem that looks like a routing or caching problem can actually be a placement problem.

I also learned that reducing cache misses does not automatically improve throughput. Transfers, synchronization, remapping, and bookkeeping can erase the theoretical gain. The simplest-looking runtime policy may win once the complete system cost is measured.

More broadly, I learned the value of explicit stop conditions. Codex made it practical to explore several difficult runtime ideas without allowing failed branches to destabilize the working product. That made failure useful: each rejected experiment narrowed the search space until the placement-compiler direction became clear.

What's Next for ExpertFlow

The next step is to generalize the placement compiler beyond the verified Gemma Q6 deployment.

Planned directions include:

  • Profiling additional quantized MoE architectures
  • Automatically discovering compatible expert bundles and placement boundaries
  • Generating deployment plans for different GPU memory budgets
  • Validating additional NVIDIA GPUs and Linux CUDA environments
  • Expanding the placement objective beyond latency to throughput and context capacity
  • Comparing full expert-bank placement against other memory-allocation strategies
  • Adding more deployment profiles and server integrations
  • Exploring dynamic loading again only on hardware where measured transfer and overlap characteristics justify it

The long-term goal is for developers to provide a model and a hardware target, then let ExpertFlow measure, compile, validate, and emit the best defensible placement plan.

Built With

Share this project:

Updates