Bean

Researchers hand-trace survival curves from papers, point by point. Bean reads them in seconds, rebuilds the patient data, checks itself, and compares drugs never tested head to head.

Comment

Inspiration

Science moves forward by comparing results across studies, and that work is slow. A systematic review takes on average 67 weeks from start to publication. Pulling the data out of a single study, with a second reviewer checking, takes about 107 minutes. Even then, methodology reviews have found data-extraction error rates as high as 63%.

In medicine, the main result of a trial is usually a Kaplan-Meier survival curve: a staircase plot showing how many people were still event-free over time, with a small "number at risk" table under it. Hidden behind that picture is the data that matters: one row per person, with when the event happened or when they left the study. Trials almost never share it.

Researchers who need that data for meta-analyses, health technology assessments or new trial designs rebuild it by hand. They click hundreds of points along each curve in a digitizer, type in the at-risk table, and run a reconstruction script. Published tools describe this work as "time-consuming, expensive, and error-prone", and nobody checks the result against the paper.

A second problem got us hooked. Doctors often need to choose between two new drugs that were never tested against each other, only each against the same older treatment in separate trials. There is a sound statistical way to compare them, but it needs good data from both trials first. We wanted a tool that goes from "a figure in a paper" to "a defensible comparison" in seconds of reading instead of hours of hand-tracing.

What it does

Bean turns a published survival curve back into individual-level data and checks its own work.

  1. Bring in a figure. Upload a screenshot or phone photo, or paste a paper link (DOI, PubMed, PMC or a journal URL) and type something like "Figure 3C". For open-access papers, Bean finds the paper, downloads the figure, reads its caption and fills in the trial name. You drag a box around the panel you want.
  2. Read it automatically. Computer vision finds the axes, reads the tick numbers, separates each treatment arm by color, traces the curves, and reads the number-at-risk table. It also detects what the curve measures (overall vs progression-free survival) and any hazard ratio printed on the figure. This takes about 5 to 8 seconds.
  3. Check the numbers. Everything it read is shown in an editable table, with warnings when something looks off.
  4. Rebuild the patients. Using the Guyot et al. (2012) algorithm, it reconstructs one row per patient and computes the hazard ratio, confidence interval, p-value and medians.
  5. Verify. It redraws the curves from the rebuilt patients and measures how well they match the original figure. It also compares its hazard ratio with the paper's and flags anything more than 10% off.
  6. Compare treatments that never met. Pick two studies that tested different new treatments against the same old one, and it runs an anchored indirect comparison (Bucher method). It refuses unsafe comparisons, such as mixing outcomes (overall survival vs progression-free survival), comparing different control treatments, or comparing a drug with itself.
  7. Ask Mr. Bean. A coffee-bean assistant powered by Gemini answers questions about using the site, the computer vision and the statistics, and it can see the numbers on your screen to explain your own results.

It works on any Kaplan-Meier figure with an at-risk table. We validated it on oncology trials, where these comparisons are made most often.

How we built it

Backend: Python with FastAPI. Frontend: a single page in plain HTML, CSS and JavaScript. Hosting: one Docker container on Hugging Face Spaces. No AI model reads the figures; every number comes from classical computer vision and OCR, so each step can be inspected.

Reading the figure

  • Perspective correction: for phone photos, we find the page's four corners and apply a homography to flatten it.
  • Plot box: we detect long dark line segments and look for the "L" where the x and y axes meet, then choose the tightest box holding most of the colored curve pixels.
  • Axis calibration: RapidOCR (an open-source ONNX OCR engine running locally) reads the tick labels. We fit a robust linear map from pixels to values, constrained so time increases to the right and survival decreases upward. If time 0 does not line up with the y-axis, we correct the shift and warn the user.
  • Curve separation: plot pixels are converted to Lab color space and clustered with k-means, shades of the same hue are merged, and the legend is masked out. Each cluster is traced column by column as a continuous step function.
  • At-risk table: we split the area under the plot into text rows, read each row (with small tiles so single digits are not missed), handle formats like "84 (4)", and keep the longest non-increasing sequence, because the number of people at risk can only go down.
  • Matching rows to curves: by label color when the table is colored. For black-and-white tables we use label positions plus a consistency rule: the fraction still at risk can never exceed the survival curve,

$$\frac{n(t)}{n(0)} \le S(t).$$

Rebuilding the data

Between consecutive at-risk times, the Guyot algorithm estimates how many people were censored and how many had the event, so that the rebuilt Kaplan-Meier estimate

$$\hat S(t) = \prod_{t_i \le t} \left(1 - \frac{d_i}{n_i}\right)$$

reproduces both the traced curve and the published at-risk counts. We fit a Cox proportional hazards model (lifelines) to the rebuilt patients to get the hazard ratio and its standard error.

Comparing two new treatments

If trial 1 compared drug A with control C and trial 2 compared drug B with the same C:

$$\log \mathrm{HR}{AB} = \log \mathrm{HR}{AC} - \log \mathrm{HR}{BC}, \qquad \mathrm{SE}{AB} = \sqrt{\mathrm{SE}{AC}^2 + \mathrm{SE}{BC}^2}$$

This keeps each trial's randomization intact. For several studies of the same drug, we also offer DerSimonian-Laird random-effects pooling with a forest plot and the $$I^2$$ heterogeneity statistic.

Papers from a link: Europe PMC, PubMed and PubMed Central APIs find the paper (including from Elsevier article codes in Lancet and Annals of Oncology links) and its full-text XML. We parse the figure list and captions, take the exact image address from the PMC article page, and read the caption to learn what each panel measures.

Mr. Bean: a FastAPI endpoint calls the Gemini API with a written knowledge base about the tool plus a snapshot of the user's current results. The API key stays on the server as a secret.

Challenges we ran into

  • The wrong box. Our first plot detector locked onto the outer frame of the figure. Detecting the "L" corner of the real axes fixed it.
  • A missing 7. OCR kept skipping lone single digits in the at-risk table. Reading the row in small overlapping tiles fixed it, so nobody has to type numbers in by hand.
  • Swapped arms. In black-and-white tables, the reader sometimes matched the wrong row to the wrong curve. The $$n(t)/n(0) \le S(t)$$ rule now catches and fixes this.
  • Off by one tick. One figure's x-axis was read one tick to the right, which silently shifted every number. We now anchor time 0 at the y-axis and warn when we correct it.
  • Speed. The first version took up to 3.5 minutes per figure because the OCR upscaled every crop. A smaller detector and lazy reading brought it down to about 5 seconds.
  • Apples and oranges. We caught ourselves comparing one trial's overall survival with another's progression-free survival. The tool now detects the outcome and blocks those comparisons.
  • Journal figures are tiny. One panel of a published figure can be about 600 pixels wide, too small to read. Enlarging small crops about 2x made them readable, and going further made accuracy worse.
  • Image addresses with random codes. PubMed Central now stores figures under unpredictable addresses, so we read the exact address from the article page.
  • OCR typos in drug names. "Docetaxel" was read as "Doxetaxel", which would have blocked a valid comparison. Names are now snapped to the spelling in the paper's title.
  • A test that was too easy. Our first LLM comparison used a famous trial, and Gemini answered perfectly. It had memorized the paper. We built a second test on never-published simulated figures to measure reading rather than recall.
  • Deploying. The cloud build installed a newer OCR release that changed its internals. We pinned the exact versions we validated, so the live site gives the same numbers as our tests.

Accomplishments that we're proud of

Accuracy against the hazard ratios reported in the papers:

  • CheckMate 017 (nivolumab vs docetaxel): ours 0.591, paper 0.59, 0.2% off
  • OAK, squamous (atezolizumab vs docetaxel): ours 0.743, paper 0.73, 1.8% off
  • OAK, fetched from a paper link, with no manual typing: ours 0.73, paper 0.73, 0.6% off
  • ORIENT-3 (progression-free survival): ours 0.484, paper 0.52, 6.9% off
  • S-1 trial: 10.5% off, and the tool flagged it itself as needing a check
  • 20 simulated trials with known answers: mean hazard ratio error 2.1% (median 1.3%), every at-risk table read correctly, and the true answer inside our 95% interval 20 out of 20 times
  • 8 brand-new figures made for the LLM test: mean error 1.3%

From two figures, the tool compared nivolumab and atezolizumab, two drugs never tested head to head: HR 0.79 (95% CI 0.53 to 1.20). It leans toward nivolumab, and the tool correctly says the data cannot confirm a difference.

We are also proud that it is honest by design. It shows every number it read, measures its own error, and refuses comparisons that would be scientifically invalid.

What we learned

  • Reading charts is a measurement problem, not a recognition problem. An LLM can describe a chart, but rebuilding data needs exact pixel-to-value mapping, and you need to show the user where every number came from.
  • LLMs recall; they don't measure. With the answers hidden on the CheckMate 017 figure, Gemini returned the paper's exact hazard ratio and confidence interval 5 out of 5 times, including a confidence interval that cannot be read from the picture. Researchers need a tool for the papers that aren't famous.
  • Small rules beat big models. "Counts never go up", "time starts at the axis" and "a fraction at risk can't exceed survival" fixed more errors than any amount of tuning.
  • Statistics has guardrails for a reason. We learned why indirect comparisons must share a control arm, why outcomes can't be mixed, and why a confidence interval that touches 1 means "we can't tell".
  • Verification is a feature. Comparing our hazard ratio with the paper's turned a black box into something a researcher can trust.
  • We are not the first to automate this. Tools like SurvdigitizeR and KM-GPT also digitize survival curves. Bean's difference is that no LLM touches the numbers, it starts from a paper link, it checks itself against the paper, and it carries the result all the way to a guarded drug comparison.

What's next for Bean

  • Population checks: read both papers' abstracts and warn when the trials enrolled different patients (for example squamous vs all lung cancer), so the comparison stays fair.
  • More than two arms and more than two trials: extend from a pairwise comparison to a full network meta-analysis.
  • Beyond oncology: generic outcome labels and validation on cardiology, transplant and engineering reliability curves.
  • Batch mode: point it at a systematic review's list of papers and rebuild every survival figure at once.
  • Export: download the rebuilt data as CSV and ready-to-run R or Python code, so results drop straight into existing workflows.

Sources

  • Borah R, et al. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 2017.
  • Li T, et al. A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews. J Clin Epidemiol 2019;115:77-89.
  • Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol 2017;17:152.
  • Zhang JZ, et al. SurvdigitizeR: an algorithm for automated survival curve digitization. BMC Med Res Methodol 2024;24:147.
  • Guyot P, et al. Enhanced secondary analysis of survival data. BMC Med Res Methodol 2012;12:9.
  • Bucher HC, et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J Clin Epidemiol 1997;50:683-691.

Built With

Share this project:

Updates

Submission history