Quorum

Inspiration

Systematic reviews are one of the places where AI should be useful but also has to be measured carefully. Reviewers often screen hundreds or thousands of titles and abstracts to decide which papers are relevant, and small mistakes can affect the evidence base. I wanted to build something that did not just “look agentic”, but actually tested whether an agent society can outperform a simpler baseline on a public benchmark with gold labels.

Quorum was inspired by one question: when does multi-agent deliberation actually help, and when does it add noise?

What it does

Quorum is an adjudicated Qwen agent society for systematic-review abstract screening.

Given a title and abstract, Quorum uses role-specialised Qwen agents to review the record from different perspectives: advocate, methods, and population. A router applies explicit escalation rules. Clear records go through a cheaper fast path, while contested records are sent to an audited adjudicator.

The project compares four systems:

• A: Baseline single-agent path
• B: Naive 3-vote ensemble
• C: Full Quorum with skeptic
• D: Quorum minus skeptic

On the primary van_de_Schoot_2018 holdout, Quorum D doubled precision compared with the baseline: 0.875 vs 0.419, while reducing reviewer workload by 57%. It also disclosed the recall trade-off honestly: 0.875 vs 0.969.

I also tested a second out-of-domain dataset, Hall_2012, under frozen rules. There, the baseline was already precise, so Quorum did not beat it. That became an important boundary condition: agent societies help when there is precision headroom, but they are not universally better.

How we built it

Quorum is built with Qwen models on Alibaba Cloud, Python, FastAPI, Function Compute, and Tablestore.

The local evaluation pipeline uses public SYNERGY systematic-review data. I created a benchmark comparison between the baseline, naive vote, full Quorum, and Quorum without the skeptic. The deployed proof runs a thin API on Alibaba Function Compute, calls Qwen/DashScope, stores the decision in Tablestore, and exposes an auditable /trace/{record_id} readback.

The public dashboard is display-only. It does not spend tokens or expose a live screening playground. It shows the precision-recall trade-off map, the ablation findings, cost routing, audit-trail examples, the Alibaba cloud-proof case, and the Hall_2012 generalization mechanism table.

A big part of the build was probe-first engineering. I added probes for dashboard safety, bundle scope, deployment configuration, local serving, Tablestore readback, benchmark integrity, label leakage, secret leakage, and Function Compute packaging.

Challenges we ran into

The hardest challenge was criteria contamination. Early on, I discovered that the criteria file I was using was not the true eligibility criteria. It contained noisy concept tags from OpenAlex rather than the actual review inclusion criteria. Two runs were therefore answering the wrong question. I fixed this by recovering the verbatim eligibility criteria from the SYNERGY metadata, rebuilding the criteria module, and adding a five-positive sanity check so the same problem could not silently happen again.

Deployment also took real debugging. I had to solve environment variable issues, Function Compute permission problems, an HTTP trigger mismatch, and a runtime issue where the deployed function started with Python 3.7 instead of the Python 3.10 layer. The final deployment now runs through Alibaba Function Compute with the correct trigger and runtime.

The most important research challenge was that the results were not a simple “more agents are better” story. The full society with an adversarial skeptic performed worse than the skeptic-less version. Instead of hiding that, I turned it into an ablation finding and then tested it across a second dataset.

Accomplishments that we're proud of

I am proud that Quorum is measured rather than just demonstrated.

On the primary holdout:

• Baseline precision: 0.419
• Quorum D precision: 0.875
• Precision gain: 2.1x
• Reviewer workload reduction: 57%
• F2 for deployed D path: 0.875 vs 0.767 baseline
• Recall trade-off disclosed: 0.875 vs 0.969

On the Hall_2012 generalization run, the baseline won, but that made the project more honest. Hall had only 9 baseline false positives versus 43 on the primary dataset, meaning there was little over-inclusion for deliberation to prune. That gave a practical rule: deploy adjudicated screening when the baseline over-includes; skip it when baseline precision is already high.

I am also proud of the skeptic analysis. Across both datasets, C and D disagreed on 14 records. The skeptic helped on 2, hurt on 11, and 1 had neither arm correct. That is a small, descriptive analysis, not a universal claim, but it supports the main lesson: adversarial agents need measurement because they can inject noise.

What we learned

The biggest lesson is that agent societies are not automatically better than single agents. They need routing, measurement, ablation, and failure analysis.

Quorum helped when the baseline over-included. It did not help when the baseline was already precise. The skeptic sounded like a responsible design choice, but in the measured traces it usually made the system worse. That changed the project from “multi-agent systems are better” into a more useful finding: multi-agent systems are conditional tools, not magic upgrades.

I also learned that evaluation integrity matters as much as implementation. Pre-registration, frozen rules, paired comparisons, trace-only analysis, and honest cost reporting made the project much stronger.

What's next for Quorum

The next direction is to turn Quorum into a decision-support tool for evidence-synthesis teams. I would like to add more systematic-review datasets, test more domains, and build a calibration layer that decides when to use the full society and when to stay with a cheaper baseline.

The product version would not sell “more agents”. It would sell measured screening workflows: fast-path for clear records, adjudication for contested records, transparent audit trails, and evidence that tells teams when the extra deliberation is worth the cost.

Built With

• agent
• agents
• alibaba
• benchmarking
• cloud
• compute
• dashscope
• evaluation
• fastapi
• function
• multi-agent
• qwen
• reviews
• screening
• society
• systematic
• tablestore