QuorumQA — MagiAchiral

Three cheap Qwen solvers vote; a Skeptic, tool-using Verifier, and flagship Judge escalate only on the 37.8% that split, closing most of the gap to the flagship at 11% lower cost.

Comment

Inspiration

Plain majority voting is the default trick for making cheap LLMs punch above their weight: sample the same model a few times, take the most common answer. But majority voting doesn't reason, it just counts. On genuinely hard "Google-proof" science questions, a confident 2-vote plurality can beat a correct 1-vote minority even when that minority is far more certain, because self-consistency throws away exactly the information (confidence, reasoning quality, contested claims) that would let a smarter process catch it. We wanted an agent society that argues about a hard question the way three specialists actually would, and only pays for a senior second opinion when they can't agree, rather than polling everyone every time or paying for the expert every time.

What it does

QuorumQA answers graduate-level GPQA-Diamond science questions ("Google-proof": unsearchable, expert-checked) with a society of Qwen agents in escalating tiers. Three Solver agents (qwen3.6-flash) answer independently and in parallel, each through a different assigned reasoning lens. If they agree, that's the final answer, no further cost: this is the common case, and it's what makes the economics work. If they split, the society escalates. A Skeptic (qwen3.6-flash) attacks the plurality answer's weakest inferential step and must name the specific step it disputes. A Verifier (qwen3.6-flash) extracts checkable numeric and factual claims and grounds every one of them through a real MCP tool server, never asserting a number from memory. A Judge (qwen3.7-max, the same flagship model used as our baseline) reads the full transcript and rules by weighing arguments, never by counting votes. The Judge can and does overturn 2-vs-1 pluralities, and every ruling records any unresolved dissent verbatim on a rendered Verdict Card.

On the full 90-question GPQA-Diamond set (complete run, zero dropped questions), QuorumQA reaches 78.9% accuracy at $0.0213 per question, versus 58.9% for the identical cheap models run as a plain self-consistency ensemble at $0.0093, versus 84.4% for the single flagship agent answering everything at $0.0240. The flagship still wins outright on raw accuracy by 5.5 points, and we say so everywhere we report this. The claim we actually make is narrower and fully checkable: routing that same flagship model to only the 37.8% of questions that genuinely need it beats paying for it on every question, while closing about 78% of the accuracy gap, and it shows its reasoning while doing it. The Judge overturned the solver panel's plurality 14 times and was right 11 of those times (78.6%), direct evidence that adjudication by argument beats majority vote on the exact cases where majority vote fails. We also publish the number that doesn't flatter us: 58.8% of escalations only re-confirm the panel, the honest cost of catching the 11 it fixed.

How we built it

The core loop is a plain async orchestrator in Python: three Solver calls fan out in parallel against the DashScope OpenAI-compatible endpoint, and only on disagreement do we fan out the Skeptic, Verifier, and Judge calls. The Verifier's tools are a genuine MCP server (Model Context Protocol), not a bespoke function-calling shim, exposing a constant lookup and a sandboxed calculator so every numeric claim in a deliberation is machine-checked, not recalled. Every transcript, tool call, and ruling is persisted as a rendered Verdict Card and pushed to Alibaba Cloud OSS from an Alibaba Cloud ECS instance running the pipeline. The public site, magiachiral.com, replays real recorded deliberations from the frozen n=90 benchmark run: the hero section is a scroll-scrubbed film of a genuine 2-1 split the Judge overturns, the scoreboard publishes the full comparison including the numbers that don't flatter us, and a filterable gallery exposes all 33 selected cases with every transcript, tool call, and ruling. We also kept a public build log on the site itself, dated from real commit history, including the attempts we reverted and the architecture experiments we tried and deliberately did not ship.

Challenges we ran into

Two of the harder bugs were both "worked, then silently didn't." After rotating our Alibaba Cloud OSS and HuggingFace credentials, both started failing with no crash, just quiet unauthenticated requests. The root cause was a leftover angle-bracket placeholder character from an earlier manual paste, repeated across three separate credential values, found only by actually attempting the real authenticated call rather than by reading the .env file and assuming it looked right. Separately, our site was built, deployed, and confirmed live by an external fetch, yet completely unreachable from our own development network. The entire hosting zone turned out to be blocked at the ISP level on that specific connection, confirmed by testing unrelated domains that loaded fine, and fixed by redeploying the identical build to a domain on our own Cloudflare account. On the modeling side, we tried swapping in a qwen3.7-plus solver seat for genuine model diversity, and it was the weakest solver in the panel and the source of every JSON-parse failure: reverted the same day.

Accomplishments that we're proud of

The frozen, complete, zero-dropped 90-question benchmark run is the one we're proudest of, because every number in it is independently reproducible from the committed harness and the dataset's public answer key: nothing here is self-graded. We're also proud that when we went looking for architecture improvements after that run, we found and honestly reported both a real positive result (a single reasoning-enabled solver seat gated behind a cheap doubt-check matches or beats the flagship at two independent held-out samples) and a real negative one (scaling that same change to all three solver seats makes results worse, not better, because it removes the productive disagreement that triggers escalation in the first place), and we chose not to ship either into the submitted number, because a genuine held-out claim needs a third, still-unseen sample before it's a claim we'd stand behind publicly.

What we learned

The most surprising empirical finding was mechanistic, not just statistical: the value in our architecture wasn't making agents individually smarter, it was preserving genuine disagreement between differently-calibrated agents so that disagreement could trigger the tribunal where most of the actual error-correction happens. Making every solver seat "smarter" (thinking-enabled) made them agree with each other more often, which meant fewer questions ever reached the Judge, which made results worse despite costing more. We also learned, the hard way twice, that a credential which looks correct on a file read can still be silently broken, and that a deployment can be confirmed live from outside your network while being completely invisible from inside it. In both cases the fix was insisting on a real end-to-end check over a plausible-looking file or log.

What's next for QuorumQA / MagiAchiral

The engine and the n=90 benchmark are real and independently reproducible today. Vote cheap, escalate on disagreement is a general pattern for any contested-judgment pipeline: claims triage, content-moderation appeals, medical second-opinion routing. GPQA was our proving ground because its public answer key makes every claim auditable. Next is a pilot customer in one of those verticals, a validated price point, and vertical-specific reasoning lenses, packaged around the same audit-artifact format and escalation-integrity guarantees rather than around the orchestration algorithm itself, since the algorithm is not the moat: a well-resourced lab will always out-engineer it. The moat is the audit trail and each customer's own accumulated calibration data becoming what reviewers and regulators come to trust.

Built With

Share this project:

Updates