🧬 The Problem
A CRISPR screen switches off ten thousand genes one at a time and hands a lab a ranked list of thousands of hits. Somebody picks three to chase. Each one costs about a year of a researcher's time and six figures — and over half of preclinical research doesn't reproduce, an estimated $28B a year in the US alone.
Nothing currently tells that person how much of their ranking is real.
📏 What denali Found
We scored all 50 MSigDB Hallmark pathways against 9,837 CRISPRi knockdowns from a published Perturb-seq screen (Replogle et al., Cell 2022 — not our data), then asked how much of the ranking you could predict knowing nothing about what any gene does.
Pathway size alone explains 46.5%.
Bigger sets return more hits regardless of function — the way a raw crime count always ranks big cities as the most dangerous. A six-feature model that never looks at biology explains 56–75%. Run against seven other groups' published supplementary tables — CRISPR knockout, CRISPRi/a, organoid, primary human T cell, bulk RNA-seq — 36–88% of each ranking is set construction. Across 1,272 published screens from BioGRID ORCS the field's median is 0.224.
🔧 What It Does
pip install -e packages/denali-audit
denali audit my_results.csv
denali rerank my_results.csv --top 10
It reads the table your enrichment software already produced. Ten formats auto-detect — g:Profiler, DAVID, clusterProfiler, Enrichr/GSEApy, fgsea, GSEA desktop, MAGeCK, drugZ, BAGEL — because asking a biologist to rename columns is the reason a check like this never gets run. You get:
- A verdict — CONFOUNDED, PARTIALLY CONFOUNDED, or NOT SIZE-DOMINATED
- A percentile against 1,272 published screens. Not "your R² is 0.62" but "your ranking is more size-confounded than 91% of published screens." On our own screen it says we're worse than nine in ten. The tool says that about us.
- The correction —
rerankapplies it and shows what leaves your top N
On our own screen, three of the top ten hold and seven do not. HALLMARK_MYC_TARGETS_V1 — the largest set at 194 measured members, ranked first, most hits — falls to twenty-fourth.
🚫 What It Refuses To Do
It never nominates. The output says so in as many words: "Not a candidate list. This says which entries were carried by size, not which to chase." A tool that turned a confound estimate into a shortlist would be committing the exact error it exists to detect.
Guide-pair concordance came back at −0.019 — two independent reagents against the same gene disagree — so no novel gene is named anywhere, and a test fails the build if a gene symbol appears near verdict language.
🧠 Core Architecture
The model orchestrates. Frozen code decides what is true.
A freeze boundary is the load-bearing idea. results/frozen/ is written once and read by everything downstream; nothing after it recomputes. The predictor was fit on 50 scored pathways, serialised and hashed — and only then were ten held-out pathways scored, with the scorer verifying that hash on load and aborting on mismatch. Scoring them earlier would have let the model see its own test set.
Two edges encode claims rather than data flow:
- Modal points into the frozen interface — it re-runs the 50-pathway sweep across 10 containers in 133s and reproduces all 50 identically. It verifies the result without being allowed to produce it, and a test asserts that.
- The VIF edge points *outward* — our two dominant features turn out to be the two terms of CAMERA's variance inflation factor (Wu & Smyth 2012), recovered from data rather than fitted to.
🛡️ What Keeps Us Honest
Eleven evaluations. Seven came back negative. All eleven are reported.
| # | evaluation | result |
|---|---|---|
| 2 | does the obvious quality filter work? | 20 of 50 sets fail it and return hits anyway |
| 3 | does the predictor generalise? | held-out balanced accuracy 0.4375, zero true positives — published unrefit |
| 5 | second cell line? | pre-registered at R² ≥ 0.25, returned 0.276 — clears by 0.026, and we say so |
| 6 | but it replicated? | 26% of the cross-screen agreement is set size |
| 7 | looser annotations? | our own power rule fired — no verdict issued |
| 10 | does our headline describe the field? | across 1,272 screens the median is 0.224 — ours is atypical |
Every threshold was written down, hashed and committed before the value it decides. 471 automated invariants plus 27 cross-surface checks fail the build if a number in the prose stops matching the frozen file behind it. They caught us five times — including a coefficient published with the wrong sign, and a hash in our own limitations file that matched nothing.
🔬 Reproducibility
Judges can verify with no download, no API key, no network, no account:
make judge-check
It runs the invariants over the frozen interface, checks the packaged tool still computes what the paper published (exactly 0.4649, or the build fails), runs the tool on a g:Profiler-shaped export, and applies the correction to show what leaves the top ten.
The headline was also independently recomputed by a second implementation that never read the first — from the method spec alone, no scipy, hand-rolled Benjamini-Hochberg.
🧩 What Was Hard
Four guards passed while testing nothing. One was gated on data a clean clone doesn't have. One was keyed to a commit a rebase had erased. One matched markup that had been rewritten. One was the counter itself. All four passed — because a skipped check and a passing check look identical in the output. They're now content-addressed, and the mechanism that caught two of them is the suite counting itself.
A question broke the headline and we kept the question. Three of six "measurability" features are properties of the gene set, not of our measurement. Split apart: measurement alone gives adj R² 0.152; set construction alone gives 0.697. The number stood; the word measurement did not.
Our own quality filter was wrong 20 times out of 50. The held-out program fails our own filter and ranks 11th of 50 with 773 hits. We would have discarded our best result.
The MCP server worked from exactly one directory — the one we always demoed from. Found by starting it from /tmp the way a stranger's client does.
📚 Citations
Replogle et al., Cell 2022 (Perturb-seq, CC BY 4.0) · DepMap 24Q4 Chronos (CC BY 4.0) · MSigDB v2026.1.Hs · Wu & Smyth, NAR 2012, doi:10.1093/nar/gks461 · Freedman, Cockburn & Simcoe, PLOS Biology 2015 · BioGRID ORCS 2.0.18, doi:10.1002/pro.3978
Log in or sign up for Devpost to join the conversation.