Inspiration

A 13th-century Byzantine pharmacy text tells the physician to take alyssos. Which plant is that? Modern readings of names like this rest on dictionaries and glossaries that disagree, and a wrong identification sends lab work after the wrong species.

In 2024, Lardos et al. replaced dictionary lookups with a systematic test. They compared each botanical detail in Dioscorides’ description with modern flora descriptions of every suggested candidate. That covered 50 historical names, 61 Dioscorides plants, 141 candidate records and 1,213 expert comparisons, all done by hand. Each candidate got one label (Strong, Moderate or Weak), with no measure of how fragile that label is, and no link to the chemistry that depends on it.

Their alyssos example shows why this matters. Fibigia clypeata fits the ancient description strongly but has no modern medicinal uses in the sources checked. Scutellaria galericulata has the uses but fits weakly. Choosing by modern use would pick the worse botanical match.

What it does

Materia Incerta takes a historical plant name and returns a ranked, uncertainty-aware identity card. It is built for ethnopharmacology researchers deciding which historical plants deserve lab work.

Name-masked scoring: an LLM compares each ancient detail with each candidate’s modern description, with every Greek, Latin and common name, synonym and spelling variant replaced by [PLANT]. To claim a match, it must quote both texts word for word; otherwise the answer counts as absent. The paper’s formula: scores are combined exactly as Lardos et al. do, into Strong, Moderate or Weak. The pipeline first recomputes the paper’s own labels from its expert scores and stops if they don’t match. Stability: the feature categories are resampled 1,000 times to show how often each candidate keeps its label and its first place. Compound ledger: each candidate is joined to LOTUS, and every compound is marked robust (found in all candidates with data), ID-dependent, or no data. Every compound is a computational, unvalidated hypothesis. A three-page site: a name index, a plant case page (ranking, stability bars, evidence matrix, ledger) and a benchmark-and-limits page. Hazard labels stay visible on toxic genera such as Aristolochia. How we built it

Architecture: Lardos data + WCVP → load and check → hide names and score with the LLM (answers saved) → paper formula and 1,000 resamples → LOTUS compound ledger → static site.

Pipeline: Python, pandas, NumPy and Pydantic, with every table and model answer validated. There’s one module per stage, joined on lemma tags, Dioscorides IDs, WCVP IDs and InChIKeys, never on plant names. AI: Nebius Token Factory through its OpenAI-compatible API, tested with Qwen/Qwen3-235B-A22B-Instruct-2507 using JSON-schema output. Every answer is saved, so one command rebuilds every number offline with no API key. Frontend: Next.js 16 static export, React 19, TypeScript in strict mode, Tailwind CSS 4. There’s no backend, login or live model call. Data: Lardos et al. (2024) research data, the World Checklist of Vascular Plants (WCVP) and LOTUS. Testing and tooling: pytest (88 tests), Playwright for browser checks, and The Hackathon Playbook MCP for pitch guidance.

Challenges we ran into

An ambiguous rule in the paper. It doesn’t say how a score between 4 and 5 is labelled. Rather than guess, the code stops on those values, and the reproduction check reports any published ones. Masking that was too eager. The name filter matched a genus stem (“Modelia” → “model”) inside our own prompt instructions. We moved the check to the parts of the prompt that change between plants, so a real name still can’t slip through. Stale and malformed model answers. A test caught old model scores surviving after a saved answer went missing. We also made sure one malformed answer is reported and kept for review instead of crashing every rebuild. A moving model catalogue. The model we first planned to use was retired from Nebius, and it was unclear whether schema-constrained output was supported. The adapter now falls back to plain JSON mode automatically, and every answer is still validated. Copyright. The Dioscorides translation and the regional floras reach us only inside the research files, so their text stays off the site until the licences are confirmed. Getting the data. Our build environment couldn’t reach the research data host, so the step that loads the Lardos files is still waiting on the files.

Accomplishments that we’re proud of

We turned a hand-made method into a checked pipeline. It reproduces the paper’s labels before any AI runs, so automation can’t quietly change the method. 84 live Nebius calls, every answer schema-valid. We scored one test name with 49 calls in about 41 seconds and another with 32 calls in 25 seconds. A rebuild with no key took every answer from the saved copies. Uncertainty carried into the chemistry. Each compound says whether it holds across candidates or depends on which plant is meant. That is the question a lab actually needs answered. An honest evaluation design. The 10 tuning and 40 held-out names are assigned once by historical name, and the bars are fixed before the held-out run. Results are reported as agreement with published expert scoring, never as accuracy, and failures are published alongside successes. 88 passing tests. They cover score thresholds at their boundaries, that no plant name reaches a prompt, the evidence rule, identical output for the same seed, the no-data compound case, and a full offline rebuild.

What we learned

Reproduce the method before automating it. Checking against the paper’s own labels is what makes the LLM layer trustworthy. Masking needs a gate as well as a filter. Replacing names isn’t enough; you also need to check every prompt and refuse to send one that still holds a name, including transliterations such as khamaimelon vs Chamaemelum. Requiring a quote stops invented matches. Making the model quote both texts word for word is a simple, checkable defence. Missing data is not absence. A compound missing from LOTUS may simply be unstudied, so it needs its own “no data” label. Saving every model answer makes the project reproducible. Judges and collaborators can rebuild every number without calling the model.

What’s next for Materia Incerta

Load the real benchmark. Bring in the Lardos research files, confirm their licences, and reproduce the paper’s labels for five names, then all 50. Run the held-out evaluation. Fix the model and the pass marks, tune prompts on the 10 tuning names, then score the 40 held-out names once. Publish the agreement figures, the false-match rate, the memorisation gap (names hidden vs shown) and every disagreement. Polish the demo cases. Finish alyssos, khamaimelon and aristolokhia end to end on real data. Search beyond the suggested candidates. Score every species in a genus or region, starting with whether the method surfaces Equisetum palustre, Agrimonia repens and Hyoscyamus reticulatus, which the authors flagged as possible missed matches. Extend to other texts and languages, including Arabic and Latin pharmacy texts.

Built With

Share this project:

Updates