profile_router_moe

Swappable MoE algorithm for LLM and multi lora serving. Plug, unplug, or upgrade specialized domain experts and loras on the fly with zero retraining and zero output degradation.

Comment

Inspiration

I spent the past year staring at large language models. But while most of my peers were focusing on fine-tuning prompt templates or writing wrapper scripts around API dumps, I kept getting pulled toward the underlying engineering of open-source Mixture-of-Experts (MoE) runtimes like Mixtral, Qwen, and DeepSeek.

When you look closely at standard MoE architectures, they rely on a trained router matrix \( W_r \in \mathbb{R}^{d \times K} \) to route hidden states \( x \) across \( K \) expert networks:

$$ \text{Router}(x) = \text{Softmax}(\text{TopK}(W_r \cdot x, k)) $$

On paper, this is mathematically elegant. But in practice, it creates a massive operational bottleneck: \( W_r \) is tightly coupled to those exact \( K \) experts during backpropagation. If you want to replace Expert 3 (say, an outdated coding expert) with a freshly fine-tuned model trained on Rust, you can't just plug it in. The router logits collapse, output distributions shift, and the model suffers catastrophic interference. Your only option is to retrain the whole router matrix — or re-tune the entire model.

I kept asking myself: Why can't an MoE model work like a motherboard? Why can't we hot-swap domain experts in production the same way we plug a peripheral into a PC?

What it does

Profile-MoE is a modular, zero-backprop routing infrastructure for LLMs that decouples the router from the expert weights.

Instead of retraining a global routing weight matrix whenever an expert is updated, Profile-MoE evaluates the semantic intent profile of an incoming prompt against calibrated, \( D \)-dimensional profile vectors representing each expert.

Swapping an expert's underlying model — same domain slot, upgraded implementation — is where the hard guarantee lives: across every configuration tested, untouched experts showed exactly 0.00% collateral change in prediction quality or routing behavior. Not "small" — zero, to five decimal places, every run.

Adding a brand-new domain the system has never seen is a different, harder operation, and this writeup isn't going to pretend otherwise — it's the problem that took up most of the project, not something glossed over. More on that below, because the failure mode and the fix are the most interesting part of this whole project.

How we built it

The root problem wasn't the experts — it was forcing a single weight matrix \( W_r \) to memorize both what the input wants and who the experts are. The fix: decouple input profiling from expert execution completely.

#1. Input Profiling. A lightweight classifier head projects the input hidden state \( x \) into a normalized coordinate profile space \( \phi(x) \in \mathbb{R}^D \):

$$ \phi(x) = \text{Softmax}(W_\phi \cdot x) $$

\( W_\phi \) has real, calibrated parameters — this is not a claim of a parameter-free system. What's parameter-free is the routing step itself, below.

#2. Static Cosine Matching. The router evaluates cosine similarity between the input profile \( \phi(x) \) and each expert's calibrated static profile \( p_i \in \mathbb{R}^D \):

$$ \text{Sim}(\phi(x), p_i) = \frac{\phi(x) \cdot p_i}{\Vert\phi(x)\Vert_2 \cdot \Vert p_i\Vert_2} $$

No gradient, no backprop, no z-loss term needed — similarity is bounded in \( [-1, 1] \) by construction, so there's no unconstrained-logit collapse to guard against the way there is in a standard trained gate.

#3. Weighted Top-K Blending. Selected top-\( k \) experts blend via temperature-scaled softmax:

$$ g_i = \frac{\exp(\text{Sim}(\phi(x), p_i) / \tau)}{\sum_{j \in \text{TopK}} \exp(\text{Sim}(\phi(x), p_j) / \tau)} $$

Because expert profiles live in a plain in-memory array, swapping Expert \( A \) for Expert \( A' \) is just an index update:

expert_profiles[0] = A_prime_profile

Nothing else in the router changes.

Challenges we ran into

Adding a genuinely new domain broke things, and the first fix that got tried was the obvious one — and it's now provable, mathematically, why it couldn't have worked.

The first instinct was temperature scaling: soften \( \tau \) for ambiguous inputs so blending is gentler near a decision boundary. It helped — measurably, about 30% error reduction on the affected cases. But instrumenting which expert actually won top-1 before and after revealed something unexpected: temperature never changed the winner. Not once, across every test. That's not a tuning failure — it's a property of softmax: \( \arg\max \) is invariant to temperature. Scaling \( \tau \) changes how confidently the blend favors whichever expert already won; it can't change who wins.

That sent the search toward the actual mechanism. Retraining the profiler jointly across old and new domains does two things: it lets the new domain contaminate unrelated inputs, and — this was the surprising part — it can shift decision boundaries between two domains that were never touched at all, purely as a side effect of retraining the classifier as a whole. One live case flipped its routing decision with essentially zero measured affinity for the new domain — the damage came from an old boundary moving, not the new domain leaking in.

The real fix: freeze the base profiler permanently, and add each new domain through its own independently-trained detector, gated by a threshold calibrated on held-out data so the false-positive rate is a known, chosen number instead of an accident. This is the standard fix pattern from open-set recognition research (rejecting low-confidence inputs rather than forcing every input through a shared decision), applied to a routing context where, as far as could be found, nobody had characterized this specific failure mode before.

One more honest finding, because it was surprising too: the assumption going in was that a bigger, more expressive gate would resolve the genuinely hard boundary cases. Testing a 326x larger gate against the original showed no measurable improvement — same accuracy on the hard cases, same overall performance. The ambiguity turned out to be a property of the data (genuinely overlapping domains), not something more parameters could fix. Knowing that changed where the rest of the project's time went.

Accomplishments that we're proud of

  • Proven zero-interference swapping. Not approximately zero — 0.00% collateral change on untouched experts, verified across multiple configurations and dimension counts.
  • A mathematical proof, not just an observation, for why the obvious fix fails. Temperature scaling can't correct a wrong top-1 decision — that's provable from how softmax works, and it held in every test rather than being asserted.
  • A working fix that actually re-selects the correct expert, not just one that reduces error by blending around a wrong answer — recovering the correct routing decision in the large majority of tested failure cases, with the remaining cases traced to genuinely ambiguous inputs rather than a fixable bug.
  • A quantified, budgetable cost model for onboarding several new domains at once — the false-capture risk compounds across simultaneous additions in a precise, predictable way (a direct multiple-comparisons effect), so it can be planned for instead of discovered in production.

What we learned

The interesting failure wasn't in swapping — that worked cleanly from early on. It was in addition, and the lesson was that a fix which lowers error isn't the same as a fix that's correct. Confirming that adaptive temperature never changed which expert won, even though it clearly helped the numbers, was the moment that mattered most in this project — it's the difference between patching a symptom and understanding a mechanism.

It also turned out that this specific failure — adding a class shifts decision boundaries for classes never touched — isn't new. It's catastrophic forgetting, a 35-year-old, well-documented phenomenon in continual learning. The contribution here wasn't discovering it; it was recognizing that a training-free MoE router inherits it too, and borrowing the right fix (open-set rejection) instead of reinventing something weaker from scratch.

What's next for profile_router_moe

  • vLLM integration — propose profile_router_moe as a native, pluggable routing policy inside open-source inference frameworks.
  • Scheduled recalibration, not continuous auto-tuning. The original plan was for profiles to update continuously from production feedback — but that's just a slower-motion version of the exact joint-retraining problem that breaks addition in the first place. The safer design is periodic, discrete recalibration snapshots, versioned and tested against the isolation guarantee before they ship, not live updates.
  • Testing the addition-isolation fix against real semantic embeddings, not just the lexical (TF-IDF) representations available offline — the mechanism should generalize, but that's not yet verified, and it's worth saying so rather than assuming it.
  • Multi-LoRA swapping — extending the coordinate profiling layer to route across thousands of lightweight adapters serving multiple tenants on one shared base model.

Built With

Share this project:

Updates

Submission history