Stillwater

Stillwater turns routed MoE experts into a storage-backed execution backend for Hugging Face Transformers, with exact native parity and configurable memory residency across multiple model families.

Comment

Stillwater

Stillwater is an experimental storage-backed inference runtime for sparse Mixture-of-Experts language models. It separates a model’s total weight capacity from the amount of RAM required to execute it.

Inspiration

Stillwater began with a slightly unhinged hardware question:

Could a hobbyist build a model-specific “LLM ASIC” and have it fabricated by a service such as PCBWay?

The answer was, realistically, no. Modern inference accelerators require custom silicon, enormous non-recurring engineering costs, advanced packaging, specialized memory systems, and a full compiler/runtime stack.

But the investigation exposed a more practical question: could ordinary hardware imitate part of that architecture in software?

Model weights are immutable during inference. In a Mixture-of-Experts model, each token activates only a small routed subset of the total expert pool. Keeping every expert weight resident in expensive RAM therefore seemed wasteful.

That became the central Stillwater hypothesis:

Store immutable expert weights in dense indexed banks, keep the shared model scaffold resident, and bring only routed experts to compute when they are required.

The longer-term goal is a low-cost, single-user inference appliance for schools, libraries, rural environments, and other places where cloud inference or large GPU systems may be impractical.

What existed before Build Week

Before Build Week, I had a proof of concept built around an experimental Hugging Face Transformers fork. It could tile expert tensors, materialize several models into bank files, and execute selected expert layers from storage.

It demonstrated that the basic idea worked, but it was model-specific, difficult to validate, and nowhere near a usable inference system.

What I built during Build Week

During Build Week, I turned that proof of concept into a substantially more general runtime and deployment stack.

Stillwater now includes:

  • Dense, append-only expert-bank formats
  • A durable SQLite record catalog
  • Read-only memory-mapped expert execution
  • Configurable drop, bounded-residency, and retain policies
  • Family-specific adapters for different MoE tensor layouts
  • BF16, grouped, packed, and FP8 expert formats
  • Exact checkpoint reconstruction tests
  • Output-ID, logits, routing, and execution-trace parity validation
  • Memory, page-fault, cache, PSS, and swap telemetry
  • Sealed dependency profiles for reproducible model loading
  • Fail-closed preparation and evaluation campaigns
  • An OpenAI-compatible chat-completions server
  • Streaming responses for OpenWebUI
  • A live machine and model-memory dashboard

The work expanded across Qwen3.5-397B-A17B, Qwen3.6-35B-A3B, Tencent Hy3, GLM-5.2, Kimi-K2.7-Code, and DeepSeek-V4-Pro, in addition to the earlier Mixtral and OLMoE experiments.

The shared runtime handles storage, mappings, record lookup, residency, telemetry, and lifecycle management. Explicit family adapters handle the parts that genuinely differ: grouped tensors, packed projections, FP8 scales, shared experts, auxiliary layers, and checkpoint-specific loading contracts.

How it works

Stillwater keeps relatively small shared components resident:

  • Embeddings
  • Attention layers
  • Routers
  • Normalization layers
  • Shared experts
  • The language-model head
  • KV-cache state

The much larger routed expert pool is externalized into immutable bank files.

When a router selects an expert, Stillwater:

  1. Looks up its record in SQLite.
  2. Opens or reuses the relevant memory-mapped bank view.
  3. Constructs tensor views over the required byte ranges.
  4. Executes the expert operation.
  5. Retains or releases the mapping according to the configured memory policy.

The canonical runtime does not maintain a decoded tensor cache. It relies on memory mapping, Linux page-cache behavior, and bounded logical residency instead of copying the entire expert pool into process-owned RAM.

The result is not magic: insufficient RAM can still produce page-cache churn, and CPU inference can be extremely slow. What changes is that the full checkpoint no longer has to fit in physical memory before inference can begin.

Results

Stillwater achieved exact semantic parity in the finalized campaigns, including reconstructed tensor bytes, generated token IDs, logits, routing decisions, and execution traces.

On Qwen3.6-35B-A3B, a warm-cache three-repetition benchmark found that a 4 GiB routed-expert residency budget delivered throughput within 0.17% of full retention while reducing mean peak proportional-set size by 59.87%.

At the larger end, Stillwater successfully executed GLM-5.2, a checkpoint of roughly 1.5 TB, using only a small fraction of that amount as resident memory.

Kimi-K2.7-Code and DeepSeek-V4-Pro required packed and FP8-aware loading paths. These runs were correctness-complete but extremely slow on CPU, which is itself an important result: storage-backed execution makes otherwise impossible models executable, but it does not erase storage bandwidth or compute limits.

For the live demonstration, I deployed Qwen3.6-35B-A3B on a CPU-only DigitalOcean instance with approximately 15.6 GiB of RAM. The model is served through an OpenAI-compatible endpoint and used from OpenWebUI while a second browser window displays live CPU, memory, proportional-set-size, file-backed residency, I/O, and process telemetry.

The cloud demo is intentionally not presented as a speed benchmark. It demonstrates that a checkpoint substantially larger than the machine’s RAM can remain usable as a real chat service.

How I used OpenAI

GPT-5.6 and Codex substantially accelerated the project.

I used GPT-5.6 for architecture review, experiment design, failure analysis, campaign interpretation, deployment planning, and deciding where validation requirements were producing meaningful evidence versus unnecessary blockers.

I used Codex to implement and refactor runtime components, add model-family adapters, build campaign and evaluation infrastructure, generate regression tests, audit provenance, and produce evidence reports.

The major architectural and experimental decisions remained mine: the residency boundary, canonical bank format, validation standard, model-family strategy, benchmark methodology, and the choice to prioritize exact correctness over merely plausible generated text.

Challenges

The hardest problem was not simply reading tensors from disk. It was proving that the resulting model was still the same model.

Different families represent experts in radically different ways:

  • Separate projection tensors
  • Grouped expert tensors
  • Packed gate/up projections
  • Quantized weights with external scale objects
  • Shared experts and routed experts
  • Auxiliary prediction layers
  • Model-specific tokenizer and dependency requirements

Native Transformers loading reports also do not always use the same semantic namespace as checkpoint objects. DeepSeek, for example, intentionally maps 244 externally owned weight-and-scale objects into 122 aggregate model parameters. Treating that as a literal one-to-one mismatch would incorrectly reject a valid load.

Other challenges included:

  • Multi-hundred-gigabyte and terabyte-scale checkpoints
  • Multi-hour preparation and evaluation runs
  • Linux page-cache behavior that can make “free,” “available,” and process-resident memory appear contradictory
  • Separating historical producer identity from current runtime compatibility
  • Preserving reproducibility without making every harmless code change invalidate prepared artifacts
  • Supporting OpenAI-compatible streaming and OpenWebUI behavior
  • Deploying the complete runtime, banks, model scaffold, dashboard, Docker frontend, and Nginx routing to a constrained cloud instance

Build Week repeatedly forced a tradeoff between perfect forensic completeness and actually shipping a working system. The final evidence policy preserves exact correctness and immutable campaign artifacts while allowing bounded finalization rather than repeatedly rehashing hundreds of gigabytes.

What I learned

The biggest lesson was that sparse computation does not automatically imply sparse I/O. Router locality, repeated expert access, page-cache behavior, admission policy, and storage latency matter enormously.

I also learned that there is no honest universal MoE adapter. The correct abstraction is a common runtime with explicit family-specific loading and execution contracts.

Most importantly, Stillwater demonstrated that model capacity and resident-memory capacity can be decoupled. It is not yet fast enough to replace conventional GPU inference, especially for the largest models, but it turns some previously impossible executions into slow, measurable, and reproducible ones.

That creates a practical path toward storage-aware inference appliances, aggressive quantization, learned prefetching, more intelligent residency policies, and eventually hardware designed around the same architecture.

Built With

Share this project:

Updates