Before a treatment reaches a clinical guideline, a trained team spends months finding every trial, reading them all, extracting the numbers, and pooling them statistically. Evidence synthesis is some of the highest-stakes, most labor-intensive knowledge work that exists — and it's a natural question whether AI agents can do it.
Published meta-analyses document every decision the experts made, so each stage — screening, extraction, risk-of-bias judgment, meta-analysis — can be re-posed as an independent agentic task over the same papers and graded deterministically against what the experts actually decided.
This is exactly the shape a verifiable-reward RL environment needs: files in, JSON out, deterministic verifiers, no judge model touching the reward — and structurally long-horizon, since one instance stages more full-text papers than fit in any context window.
What it does
synthesis-bench measures whether an AI agent can do medical evidence synthesis: given the full-text papers behind a published meta-analysis, extract arm-level results (means, SDs, Ns, events) and judge risk of bias, graded by deterministic code against the published expert decisions.
The open data version covers 13 reviews / 40 papers / 194 extraction facts plus 10 reviews / 35 papers / 178 risk-of-bias domain judgments, built on the public MetaPsy databases — with every gold value pinned to upstream release SHAs.
Grading is recall-only and forgiveness-asymmetric by design: honest omission costs little, fabricated numbers cost ~10× more — the failure mode that actually matters in medicine.
Frontier models score roughly 27–50% on extraction. Months of expert work, and the frontier is at half — that's a benchmark with headroom.
How we built it
One continuous six-day Codex session: 154 asks from me, 7,221 commands, 881 patches, 385 subagents spawned by Codex. I drove strategy (task fit, grading philosophy, scope decisions); Codex drove execution.
Codex built the acquisition pipeline that resolved DOIs, downloaded PDFs, and OCR'd 753 papers with zero failures — after self-diagnosing that its first pass used the wrong downloader and re-ordering the routes itself.
Gold data came from the researchers themselves, and 194/230 rows were verified to exist in the context supplied to the AI agent.
Challenges we ran into
Authoring defensible gold was the whole fight: the frozen instruction demanded exhaustive extraction while the gold held one hand-picked fact per paper and the grader punished correct extras — a real contract contradiction we resolved by exhaustive row adjudication plus flipping the grader to recall-only.
"Model failures" kept turning out to be curation bugs: the panic review's recall-zero traced to gold that was internally inconsistent with the review context every model correctly followed. Four broken reviews were excluded; the benchmark got more honest, not easier.
Scope contamination in long agent sessions is real: a contaminated continuation silently discarded six human-approved facts and rewrote durable goal files. Codex caught it, removed the false approvals, and restored 34 frozen provenance files byte-for-byte.
LLM auditors hallucinate confidently — one "corrected" a right answer (PSWQ→PSQW); only a deterministic source comparison caught it. Every LLM output in the pipeline now passes through a deterministic gate.
Due to copyright restrictions, publishing meant rebuilding both repos into fresh clean histories with leak scans and ownership guards — and correctly diagnosing a flaky pre-push test as a race rather than bypassing the gate.
Accomplishments that we're proud of
A fully reproducible public benchmark: anyone can clone, rebuild gold byte-for-byte from pinned public upstream data, and watch the oracle prove every instance grades exactly 1.0 — no credentials, no trust required.
Deterministic, degenerate-case-proof graders — no judge model anywhere in the reward path.
Real results with a real story: 117-cell Luna/Terra/Sol sweeps showing the frontier at 27–50%, and that more reasoning effort doesn't reliably buy more reward — one variant peaks at low effort.
What we learned
Put determinism everywhere you can and use the model only where you must: the winning architecture was "deterministic script + workflow for the nondeterministic LM parts," after we caught ourselves generating invariant fields with a model.
Never trust a single LLM pass on ground truth. Draft→audit helps, but only a deterministic comparison against source data reliably catches confident hallucinations.
When a benchmark scores zero, suspect the gold before the model. Our worst "failures" were curation inconsistencies, and fixing them was the most valuable error analysis we did.
Human-drives-strategy / agent-drives-execution genuinely works at six-day scale, if the human owns scope decisions and the agent owns catching its own drift.
What's next for Synthesis Bench
Evangelize Synthesis Bench and the importance of high quality evidence synthesis for medicine and public health
Extend to the remaining pipeline stages — full-text screening and the meta-analysis itself — so an agent can be measured end-to-end, from search results to pooled effect.
Log in or sign up for Devpost to join the conversation.