La caverne aux 40 voleurs

Inspiration

We started out believing the obvious thing: more experts must mean better answers. Our first Mixture-of-Experts orchestrator on Qwen Cloud fanned every question out to a panel and fused the results.

Then we benchmarked it, and it regressed. On a pure reasoning task ("eventual vs strong consistency for a shopping cart"), the naive MoE scored 13.33/20 while a single qwen-turbo agent was competitive. Fusion was diluting the thesis: blending four expert voices into one answer destroyed the sharp argument that made the answer good.

That failure became the whole project. The interesting problem was never "build a MoE" — it was knowing when not to be one.

What it does

La Caverne aux 40 Voleurs is an adaptive MoE orchestrator built natively on Qwen Cloud / Alibaba Cloud (DashScope). A "Genie" classifies each query before routing it, then decides how much machinery it deserves:

• Analytical, mono-domain question → delegate to one strong expert, k=1, no fusion at all. The MoE deliberately switches itself off.
• Constructive, multi-domain question → route to top-k experts by cosine(query, expert) × historical performance, then fuse the fragments with an editor-in-chief pass (qwen-max) that picks one base answer and integrates the rest only where they fix errors or fill gaps — one narrator, not a patchwork.

Measured result. Final configuration, 3 independent runs, 40 pairwise rounds: 29 wins / 6 losses / 5 ties — an 82.9% win rate on decided rounds, mean quality gain +21.9% (range +18.9% to +23.6%) over a single qwen-turbo agent.

Longest run (5 reps, 20 rounds):

The consistency row is the story: the case where the naive MoE collapsed (13.33) is where the adaptive one flies (19.50, unanimous).

Where we still lose: on secure login we score 10.80 against the single agent's 14.30. We publish that case rather than dropping it.

Beyond routing, the product has its own identity — Le Camp: L'Embûche (an adversarial strategist audits the gang and recruits a breakthrough expert), Le Conciliabule (an MoE recruiter forms a squad from a natural-language mission, then runs a recruitment debate), and Les Sceaux (a generative SVG sigil for every Genie).

How we built it

• Backend: native Node.js ESM, zero heavy dependencies, port 8787. SSE streaming so the orchestrator's reasoning is observable live in L'Observatoire — which agent was picked, why, cost, latency, confidence.
• Frontend: React 18 + TypeScript + Vite, 11 panels, self-hosted Monaco editor.
• Qwen family orchestration: qwen-turbo (router), qwen-plus (experts), qwen-coder-plus (code), qwen-max (fusion + judge), text-embedding-v3 (1024-d routing + KB), z-image-turbo (portraits), wan2.1-t2v-plus (video).
• Architecture: 81 functional layers (L0–L80) across 12 pillars, each proven implemented — a real file exporting a real symbol — by a reproducible audit script. Behaviour is covered separately by 25/25 green tests, plus typecheck and build.
• Benchmark harness: scripts/bench-hard.mjs runs randomized pairwise A/B with a double judge (qwen-max + qwen-plus), counting a win only when both judges agree, on a 4-criteria rubric.

Challenges we ran into

The naive MoE regressed. Fusion diluted analytical answers. Solved with the adaptive delegation gate — the core idea of the project.

The LLM judge is noisy — much noisier than we expected. The same qwen-turbo baseline scored between 13.30 and 16.00 across runs on identical cases: a 2.70-point swing, larger than the improvement we were trying to measure. Any headline built on comparing absolute means between two runs would have been meaningless. That is why we score by randomized pairwise A/B with double-judge consensus, report a range rather than a single flattering number, and commit every benchmark run to bench/ — including the early one that gained only +4.17%.

Our own fix didn't work. We built a mode veto (L80): the orchestrator re-runs the best single expert alone, has a judge compare it against the fusion, and rejects its own fused answer if the single expert wins. It was designed to rescue exactly the weak secure login case. We benchmarked it and it made things worse — global quality fell 15.21 → 14.08 (−7.40%) and login collapsed to 5.67/20. It ships opt-in and off by default, and we claim no benefit from it.

Strict provider compliance. The codebase carried legacy references to a non-permitted provider; all of it was removed. The project now runs exclusively on Qwen Cloud / AI Studio / Alibaba Cloud / Ollama Cloud.

Accomplishments that we're proud of

• Turning a measured failure into the architecture: the v1 regression on consistency is what produced the adaptive gate, and the flip from 13.33 to 19.50 (5/5 unanimous) is the proof it worked.
• A benchmark we would defend in front of a hostile reviewer: randomized pairwise, double-judge consensus, ties counted honestly, judge variance disclosed rather than buried.
• Publishing the runs that don't flatter us — the +4.17% run, the secure login loss, and the negative veto result are all in the repository. We would rather ship a true modest number than an impressive one we can't defend.
• 81/81 layers proven implemented by a reproducible script instead of asserted in a checklist.

What we learned

That building a Mixture-of-Experts is not the hard part. Knowing when not to be one is.

And a second lesson we didn't expect: when your measuring instrument is an LLM, measuring honestly is harder than building the thing you're measuring. A 2.70-point baseline swing taught us more about methodology than any gain we recorded.

What's next for La caverne aux 40 voleurs

• Diagnose the veto regression. We expected it to rescue the weak cases; it degraded them. Understanding why the veto path damages single-expert answers is our first task after the hackathon — the failing run is committed for anyone who wants to look.
• Fix secure login. It is the one case where the adaptive MoE still loses, and we know it.
• Grow the roster. To be precise about the name: "40 Voleurs" is the Ali Baba reference in the title, not a headcount. The benchmarked Champion roster holds 5 specialized thieves with top-k = 3 selected per query. The orchestration layer is roster-size-agnostic — the thesis we defend is "know when to use only one", not "we have the most experts".
• Finish the Alibaba Cloud deployment. The backend is containerized (Dockerfile) with a one-shot ECS provisioning script (scripts/deploy-ecs.sh) that installs Node, configures a systemd service and exposes /api/health. We ran out of hackathon time to stand the instance up; the path is scripted and reproducible in the repo.

Built With

• alibaba-cloud
• claude
• dashscope
• fable5
• node.js
• qwen
• react
• typescript