Inspiration

I work in biomedical imaging, and I have faced this problem myself and watched colleagues across labs face it too. One dataset passes through many hands: one person acquires the images, another pre-processes them, another segments the cells, another trains a model, and someone writes the paper. At terabyte scale nobody can check all of it by hand. Files go missing, get overwritten, or end up copied into both the training and the test set, and months later nobody can say which raw files stand behind a figure.

The obvious answer today would be an AI assistant, but we cannot use one. Our data is unpublished, and much of it comes from patient samples or human clinical studies, so it cannot be uploaded to a cloud service. Most biologists are also not programmers, so command-line data tools do not help them either.

The published record shows how common this is. Of 1,792 papers stating data were available on request, 6.8% supplied them (Gabelica et al., J Clin Epidemiol 2022). A cancer-biology replication project could not obtain data for 68% of experiments (Errington et al., eLife 2021). The odds that a dataset still exists fall 17% per year (Vines et al., Curr Biol 2014).

What it does

A scientist asks in plain language, and the Skill answers on the lab NAS, where the data already lives. One principle runs through every mode: the lab's own documents say what should exist, and the NAS proves what does exist.

  • Run Check: reads the acquisition protocol and plate map, works out how many files should exist, and compares that with the run folder. It reports missing images, timepoint gaps, cut-short files, duplicates and unexpected files, each by name.
  • Stage Coverage: checks that every raw file has its processed image and mask, and flags results that are older than their raw file.
  • Change Check: finds raw files modified after acquisition, and compares a folder with an earlier saved record.
  • Split Integrity: finds identical files, repeated names, and the same well or subject in both training and test sets.
  • Figure Trace: reads a manuscript and links each figure to the data it names, and flags figures with no traceable data.
  • Departure Handover: finds a leaving member's files that exist nowhere else, and copies approved items to the lab archive with checksum proof.
  • Audit log: every check can be saved as a dated record.

It is read-only by default. It never deletes, moves or edits research files.

How we built it

One Uliya Skill: a SKILL.md plus nine reference rule files, with no external code or services. It coordinates the NAS's built-in experts:

  • document-agent reads protocols, plate maps and manuscripts, quoting the source of every value.
  • file-agent lists files, runs the duplicate scan, copies approved files and calculates checksums.
  • knowledge-agent saves reports after approval.

Every later step uses only the real results of earlier steps. We also built a synthetic test kit with an answer key and scorer: a 576-file plate with 27 planted faults, and a handover scenario with 21 items.

Challenges we ran into

  • Working within what the NAS does reliably. Looking at images one by one is slow and unproven on scientific formats, so the Skill relies on file names, sizes, times, checksums and documents.
  • Never guessing. A file that could not be read is reported as unreadable, not missing. File names that cannot be matched are reported as unmatched.
  • Keeping it usable for non-programmers. Every mode starts from a plain request and the documents a lab already has.

Accomplishments that we're proud of

  • All six modes pass the official static checker with 0 errors and 0 warnings.
  • Every finding cites a file name or a quoted line from the lab's own document.
  • Approved copies are proven by checksum.
  • A scored benchmark for Run Check and Departure Handover is ready for real-NAS testing.

What we learned

Scientists do not want AI that guesses. They want a fast answer they can audit: is my data all here, is it unchanged, and can I trace it? Keeping the AI to reading documents, and leaving the counting and comparing to the NAS, is what makes the answer trustworthy.

What's next for Scientific Research Lab Data Provenance

  • Build test data with planted faults for Stage Coverage, Change Check, Split Integrity and Figure Trace.
  • Run the benchmarks on an iDX6011 Pro, and on public image set BBBC039v1 (200 nuclei images with masks and official splits, released under CC0).
  • Save a checksum record at intake so a later check can prove files are unchanged.
  • Add a sequencing delivery check against the core facility's checksum list.

Built With

  • document-agent
  • file-agent
  • knowledge-agent
  • python
  • ugreen-ai-nas
  • uliya
Share this project:

Updates

Submission history