KAPPA — Knowledge-guided AI for Phyto-Pharmacogenomics

Scaffold-held-out benchmark: Random Forest ROC-AUC 0.782 vs logistic 0.703, at 82.6% active test prevalence.
Duplicate-structure sensitivity: ranking is stable (ROC 0.774-0.825); the threshold-0.5 point is not.
Disconnected PGx component: verified CPIC CYP2C19-clopidogrel guideline, no patient genotype.
KAPPA: evidence-separated phyto-pharmacogenomics with curated PTGS2 assays and an explicit refusal rule.
In-domain: curcumin 15.0% uncalibrated, flagged as an exact training-structure match (retrospective).
The refusal rule in code: sucrose similarity 0.193 < 0.3236 reference, so the triage value is withheld.
Provenance-labelled ethnobotany: one attributed 2012 report, limitations attached, coverage stated.

Inspiration

Natural-product evidence is scattered across four kinds of record that look interchangeable on a slide and are not: a plant occurrence statement, a traditional-use report, a measured biochemical assay, and a computational model output. Joining them is easy. Joining them without losing the difference between them is the hard part.

The failure mode we wanted to fix is not missing data, it is missing evidence contracts. An occurrence gets read as efficacy. An uncalibrated model score gets read as validation. A pharmacogenomic association gets read as plant-specific. The field's own resources document this problem: LOTUS reports scarce structure–organism interlinkages, NPASS deliberately keeps measured activity separate from similarity-estimated activity, and a 2026 compound-centric framework already links plant occurrence, taxonomy, scaffolds and ChEMBL evidence.

That last point shaped the project. We are not claiming to build the first integrated natural-product platform, because that platform already exists in the literature. We built the narrower thing that is checkable in code: an evidence-type contract with an explicit refusal rule, and a reproducible benchmark that reports its own weaknesses.

What it does

KAPPA is a runnable research workbench, FastAPI backend, single-file frontend, RDKit and scikit-learn underneath that keeps five evidence axes separate and traceable: botanical occurrence, a qualified traditional-use report, measured human PTGS2 (COX-2) assay data, uncalibrated model estimates, and pharmacogenomic guideline evidence.

Curates human PTGS2 IC50 records from ChEMBL with pre-declared filters, instead of using whatever the database returns. Benchmarks a class-weighted Random Forest against a logistic baseline on a Bemis–Murcko scaffold-group holdout, so near-identical analogues cannot leak across the split. Refuses to rank. Outside a training-derived similarity reference range, the triage value is null in the API and "Not rankable · probability withheld" in the interface with the similarity value and the reason shown, not hidden. Validates ethnobotanical records fail-closed: authors, year, title, journal and URL/DOI required, limitations required, and any record that asserts a compound linkage is rejected. Demonstrates a verified, independent CPIC CYP2C19–clopidogrel guideline in a graph component deliberately disconnected from the phytochemical model. Exports everything, curated CSV, metrics, a model card, a provenance register, an integrity audit — with no composite "candidate score" anywhere. It is a research prototype. It makes no efficacy, safety, clinical or patient-response claim.

How we built it

Data 7,482 ChEMBL human PTGS2 IC50/nM records → exact-relation, nM-unit, single-protein, human-species, quality-flag and structure-validity filters → exact canonical-structure duplicate collapse (lowest IC50) → 3,164 structures: 2,505 at IC50 ≤ 1 µM and 659 at ≥ 10 µM. The 1–10 µM interval is omitted, not relabelled inactive — we would rather lose 497 ambiguous labels than teach a model that "not potent enough to call" means "inactive".

Model and evaluation Seven RDKit descriptors plus a 512-bit Morgan radius-2 fingerprint. Fixed stratified scaffold-group split: 2,462 training / 702 test structures, zero shared Bemis–Murcko scaffolds. Everything is regenerated by one command (python -m src.run_pipeline), including training-only 3-fold scaffold CV, four-point hyperparameter sensitivity, calibration diagnostics, permutation importance, a training-derived applicability reference, and two internal stress tests.

The refusal rule Similarity is a diagnostic, never a reward. The threshold (0.3236) is the 5th percentile of maximum training similarity for training-only scaffold out-of-fold compounds empirical, relative, and explicitly not a validated domain boundary. Sucrose scores 0.193, below it, so the interface returns "Not rankable · probability withheld" while an in-domain molecule still receives its number in the same panel.

Provenance, enforced The single ethnobotanical record is not prose in a README. It passes a fail-closed validator, is served at /api/ethnobotany with a coverage statement "limited demonstration corpus, not a systematic ethnobotanical survey" and is rendered with taxon and record ID so a judge can trace it. Nothing in the app links that practice to curcumin or to PTGS2, because the source does not support that link.

Data sources and attribution ChEMBL (target CHEMBL230) under CC BY-SA 3.0; CPIC/ClinPGx (pair 110091, guideline 100411) under CC0 1.0 with attribution and current-version caveats; NCBI ClinVar VCV000016897 for variant identity only; UniProt P35354 under CC BY 4.0; Wikidata Q312266 / Q42562 (P703) under CC0; and one secondary ethnobotanical report (Velayudhan et al. 2012, Indian Journal of Traditional Knowledge 11(4):607–614), cited rather than reproduced, whose reuse licence we did not independently verify. The ClinVar record's cached "Likely benign; other" label concerns a not-specified disease trait; a disease-association statement, not a drug-metaboliser function call. Full register: https://github.com/AdemolaTolulope/kappa-ethnohack2026/blob/main/docs/provenance_register.md

Challenges we ran into

A model that always answered 0.980. Every "Predict" returned the same number. The cause was not the model: the first UI load silently overwrote the SMILES box with the auto-selected training compound, so we were scoring the same molecule every time. We only found it by hand-computing the probability (0.1504) and comparing it to the interface. The fix keeps the typed input and flags exact training matches as retrospective.

Metrics that would not reproduce. An earlier draft quoted ROC-AUC 0.832 / PR-AUC 0.966. We re-ran seven protocols and random split, all five scaffold folds, no deduplication, grey-band relabelling, pooled out-of-fold, and a 12-seed sweep of single splits and none reproduces that pair. It sits inside the spread of one arbitrary scaffold split (ROC-AUC 0.804–0.880 across seeds), which is the most consistent explanation: one favourable, non-pre-registered split. We withdrew the pair and kept the audited 0.782 / 0.940.

A composite score that should not exist. An earlier build averaged model output, similarity, occurrence and pharmacogenomics into a single "candidate score". It was uninterpretable and it made an arbitrary weighting look like a scientific result. We deleted it; there is a test asserting no ranking value is returned outside the empirical range.

Ethnobotany that was prose, not code. A literature record sat in a JSON file with nothing enforcing its provenance. Claims that live only in documentation decay; now a validator refuses to load a record that omits attribution or asserts a compound link.

Accomplishments that we're proud of

A refusal that a judge can trigger. Type sucrose: similarity 0.193 < 0.3236 → no triage value, with the reason on screen. Type an in-domain molecule: you get 0.150, labelled uncalibrated, flagged as an exact training match. The gate is behaviour, not a slogan. Numbers withdrawn in public. We documented the 0.832/0.966 discrepancy, showed the seven protocols that fail to reproduce it, and left the audited numbers standing instead of quietly reporting the flattering pair. A sensitivity analysis most weekend projects skip. Of 4,327 unique eligible structures, 371 carry more than one measurement and 32 carry both an active and an inactive one. Three aggregation rules under one protocol: ROC-AUC 0.774 / 0.782 / 0.825, balanced accuracy 0.671 / 0.637 / 0.729. Ranking is moderately stable; the threshold-0.5 operating point is not so we make no triage-accuracy claim at a fixed cutoff. Reproducibility that is verified, not asserted. 22 tests, and a 16 MB bundle that was extracted into an empty directory where pytest -q passed and the pipeline reproduced the stored metrics to 1.1e-16. Same result from a fresh anonymous clone of the public repository. A pharmacogenomics demo that stays in its lane. A real CPIC guideline, filtered to the intended phenotype and ACS/PCI context, in a graph component with no edge to the phytochemical pilot and no patient genotype.

What we learned

Grouped splitting is the whole ballgame. A random split on the same curated data inflates ROC-AUC from 0.782 to 0.927, because 223 scaffolds appear on both sides. Any natural-product ML claim without scaffold grouping is probably optimistic. PR-AUC is meaningless without prevalence. At 82.6% active, the no-skill baseline is 0.826. Quoting PR-AUC 0.94 without that sentence is misleading, so the sentence travels with the number everywhere in our artefacts. Refusals are more persuasive than predictions. The most credible thing the interface does is decline to answer. Provenance is cheap in prose and expensive in code — which is exactly why it belongs in code. Metrics chosen after seeing the evaluation are not evidence. We had to learn that on our own repository, not in the abstract. Limitations we state up front No external validation: both stress tests (publication-held-out ROC-AUC 0.724; temporal-held-out 0.602) are internal ChEMBL re-splits. Probabilities are uncalibrated, and the 0.3236 similarity threshold is an empirical reference, not a domain guarantee. One target, one assay family human PTGS2 biochemical IC50 only. Ethnobotany is one qualified secondary report, not a corpus, and the underlying Malayalam source was not independently accessed. The PGx component is a curated-evidence demonstration, not a KAPPA-computed prediction. The app runs locally; we claim no public deployment.

What's next for KAPPA — Knowledge-guided AI for Phyto-Pharmacogenomics

Turn the contract into coverage, and the prototype into something a judge can click:

A real ethnobotanical corpus under the existing validator licence-checked sources with locality and community attribution, added only with permission, never scraped into a claim. An independent assay source for PTGS2 to test transportability honestly, with publication- and structure-overlap checks before any score is reported. Calibration fitted inside training folds, reported on the holdout, so a probability can eventually be described as a risk instead of a ranking device. Expert review of the applicability boundary, replacing our 5th-percentile heuristic with a defensible, documented domain rule. A containerised deployment so a judge can paste a SMILES string into a URL instead of running a command. Prospective experimental follow-up for the shortlist the triage actually produces — the only thing that would turn any of this into biological evidence. Source code, tests, deck and audit: https://github.com/AdemolaTolulope/kappa-ethnohack2026