-
-
Catch the statistical errors in a research paper before the reviewers do. The AI reads; exact math renders every verdict.
-
The integrity report: a score, the impact, and which single number to fix first, all from exact math you can check by hand.
-
Every finding in plain English with the exact calculation shown: recomputed p-values, impossible means, and df-vs-N clashes.
-
A real Qwen tool-calling agent: it decides what to check, calls each deterministic tool itself, then writes a plain verdict.
-
Impact at scale: 26 real papers screened in one pass, about three hours of hand-checking done in under six minutes.
-
The whole design: Qwen reads the messy paper; deterministic math renders every verdict, so nothing can be hallucinated.
-
Six provable checks: p-values, GRIM, GRIMMER, df-vs-N, effect size, and claim-vs-evidence, each grounded in exact math.
-
Beyond detection: Rigor points to the number to fix, shows how sure it read the paper, and screens a whole submission queue.
-
How it compares: Rigor does what statcheck and closed publisher tools cannot, all of it, on any paper, for the author.
-
Honest limitations and roadmap: Rigor checks only what it can prove, and names what it deliberately leaves out of scope.
Inspiration
We kept running into the same uncomfortable fact. Studies that scanned tens of thousands of published papers found that about half of them carry at least one statistical number that does not add up, and roughly one in eight has an error big enough to change the paper's conclusion. Most of it is not fraud. It is tired people making small mistakes: a p-value typed wrong, an average that cannot exist for the sample size, a result called significant that, when you actually check it, is not.
These slip through because reviewers read for the ideas and the argument, not the arithmetic. Almost nobody sits down and recomputes a p-value by hand. And the tools that can do it are expensive, closed, and sold to journals, so they only run after a paper is submitted. The one person who would benefit most, the author about to hit submit, gets nothing. We built Rigor for them.
What it does
You paste a paper, or drop in a PDF, and Rigor checks whether its statistics actually add up. In seconds it:
- Recomputes every p-value from its test statistic and flags the ones that disagree, especially the ones that flip a result from significant to not significant.
- Runs the GRIM and GRIMMER tests to catch means and standard deviations that are arithmetically impossible for the sample size.
- Cross-checks the degrees of freedom against the reported sample size.
- Recomputes a reported effect size (Cohen's d) from its t-statistic.
- Reads the paper's claims and flags where the words overstate the numbers.
Then it does two things no other tool does. It tells you which single reported number is most likely the mistake, by finding the one correction that resolves the most errors and proving it by re-running the checks. And it puts your paper in context, comparing it to the published field baseline and showing how much time it just saved you against checking by hand.
Every finding is explained in plain English with the exact calculation shown, so you can verify it yourself. You dismiss any false positive and export a clean report. The same engine runs as a website, a command-line tool, a batch tool for a whole submission queue, a GitHub Action, and an MCP server other AI agents can call. There is also a real Qwen agent that decides what to check, calls the tools itself, and writes a plain-language verdict.
How we built it
The whole design is one split. Qwen, on Alibaba Cloud Model Studio, does the one thing only a strong language model can: read messy, free-form scientific prose and pull the statistics out as clean structured data. We use Qwen's native function calling for this, so it returns a proper typed form instead of text we have to guess at.
Everything after that is exact math. The checks are deterministic Python, using SciPy distributions and plain arithmetic, so every verdict is provable and cannot be hallucinated. The model reads; the math judges.
On top of that core:
- A tool-calling agent runs Qwen in a multi-turn loop over the same deterministic checks.
- Because reading is the only uncertain step, extraction can run several times and keep only the numbers the runs agree on, reporting a live agreement score.
- The backend is FastAPI with a clean static frontend, containerized with Docker and deployed on Alibaba Cloud ECS in Singapore, with a live status endpoint.
- It is covered by 58 tests, a 530-case benchmark that runs offline with no API key, and continuous integration.
Challenges we ran into
- Proving it is not just an AI wrapper. We solved this in the architecture: the model never produces a verdict, only the deterministic engine does. You can turn the AI off entirely and the math still scores 100 percent on 530 cases.
- Messy real PDFs. On long papers the model can misread a table or a citation. We added multi-run reconciliation, a confidence score, and a human review step, and we are honest that single-pass flags are leads to verify, not verdicts.
- False positives on continuous data. An early version wrongly applied GRIM to physical measurements. We tightened it so those checks only fire on whole-number rating scales.
- Running at scale found real bugs. Screening 26 papers in one pass surfaced a crash on a null field and a timeout too short for long PDFs. Both are fixed and covered by tests.
Accomplishments that we're proud of
- 100 percent precision and recall on a 530-case deterministic benchmark, offline and reproducible.
- Six provable checks, including GRIMMER, which almost no tool implements.
- Error localization: pointing to the single number most likely at fault, which no other integrity tool does.
- A verified real catch: on a published geology paper, Rigor flagged a correlation built on only four data points that recomputes to non-significant, confirmed against the paper's own methods.
- A real corpus result: 26 published papers screened in one pass, about three hours of hand-checking done in under six minutes, and the paper it ranked worst was the one we had verified.
- A genuine Qwen tool-calling agent, an MCP server, and a live deployment on Alibaba Cloud, all behind a clean, honest interface.
What we learned
Use the model for the one thing only it can do, making sense of human writing, and let ordinary code own anything that has to be correct. That single division is what turns an LLM demo into a tool people can actually trust. We also learned that for an integrity tool, honesty beats coverage. Naming what it cannot do, and only ever showing numbers we can cite or compute, made the whole thing more credible, not less.
What's next for RIGOR
More provable checks on the same deterministic core: recompute the test statistic itself from reported means, SDs, and Ns; more test types and one-tailed detection; confidence-interval consistency; and sample-size consistency across a whole paper. We also want to scale the validation to a large corpus of retracted and corrected papers, and wire the batch tool into a submission portal so screening runs on upload.
Some things are deliberately out of scope, because they are different problems that need different tools: detecting fabricated raw data, image manipulation, and whether the chosen method was appropriate. We name these rather than pretend to cover them. A tool that claims to catch every error is either dishonest or an unreliable oracle, and Rigor stays provable.
Log in or sign up for Devpost to join the conversation.