HonestG Genome Firewall

Bacterial resistance testing takes a day or two. We read the genome and answer in seconds. The difference is what it does when unsure. It provably refuses to guess.

Comment

Inspiration

Antibiotic resistance is linked to more than 4.7 million deaths a year, and the standard susceptibility test takes one to three days. The genome holds much of the answer already. The missing piece is a model a clinician could actually trust to read it. When I looked at how AMR models are usually scored, I found the catch: bacterial genomes come in near-identical clonal clusters, so a random train/test split lets a model memorize clones and score beautifully on genomes it has effectively already seen. On my data that trick reports an AUROC of 0.96 for carbapenems. The honest number, with whole lineages held out, is 0.64. Genome Firewall is built to show that gap instead of hiding it, and to know when to say "I don't know."

What it does

Give it an assembled genome (K. pneumoniae, with a live cross-species test on E. coli). For each antibiotic it returns ALLOW (likely to work), BLOCK (likely to fail), or REVIEW (a no-call when the evidence is too weak), each with a calibrated confidence, the exact genes behind it, and a plain-language rationale. It is decision support, never a diagnosis, and every result keeps a "confirm with standard lab testing" banner.

The core promise is a no-answer guarantee with two checkable parts. It never answers below a calibrated per-drug confidence threshold (\tau_d): it emits ALLOW or BLOCK only when (c \ge \tau_d), and REVIEW otherwise. A verifier confirms zero answers below the line across all 240 predictions. And when it does answer, a Clopper-Pearson bound on the (k) errors among (n) answered cases certifies a 95% error ceiling per drug: ceftriaxone at most 9.9%, ciprofloxacin 15.5%, gentamicin 20.0%, and meropenem 25.9% while abstaining on 29% of cases.

How I built it

Simple by design, so you can read it end to end:

FASTA -> AMRFinderPlus (Docker) -> gene / mutation features
      -> per-drug class-balanced logistic regression, tuned threshold
      -> bounded selective no-call -> deterministic intrinsic-resistance gate
      -> Streamlit console

Stack: Python, scikit-learn, Streamlit, Plotly, AMRFinderPlus via the ncbi/amr Docker image, and OpenAI gpt-5-mini for the rationale with an offline deterministic fallback. Data is real: 302 K. pneumoniae genomes from BV-BRC with lab-measured phenotypes and MLST types.

Three choices carry the thesis. Homology-grouped splits: whole MLST lineages are held out, so no clone ever leaks across the split. Evidence categories: a known resistance gene vs a statistical association vs no signal, with a deterministic post-check that stops the rationale from ever calling a merely-correlated feature causal. And a gate where biology overrules the model: K. pneumoniae is intrinsically ampicillin-resistant, so the rule returns BLOCK even when the genes might suggest otherwise.

Challenges I ran into

Almost nothing that worked on toy data survived the real run. Silent data leakage produced a first result that was beautiful and wrong, and getting to the real 0.64 meant trusting the ugly number. Numeric genome IDs parsed as floats and quietly broke every join. Platt calibration compressed the separable drugs and my first no-call band abstained on 100% of cases, fixed with class-balanced regression, a tuned threshold, and a bounded quantile cutoff. My causal guardrail was paraphrasable, so the rationale could imply causation without the banned phrase. And three independent code audits found nine real bugs, two of which would have broken the honesty thesis. The hardest part was not technical. It was the discipline to lead with the 0.64 instead of burying it.

What I learned

The honest number is a feature, not a confession: leading with meropenem's 0.64 next to the leaky 0.96 became the most persuasive thing in the project. Abstention is a first-class output, not a dodge. Generalization is what tells you what a model really learned: acquired-gene resistance transferred almost intact from K. pneumoniae to E. coli (ceftriaxone 0.96 to 0.93, ciprofloxacin 0.96 to 0.97), while the species-specific gate deliberately did not, which is the honest boundary. And a model that explains itself has to be stopped from lying convincingly, because the most dangerous output is a fluent causal sentence about a feature that is only correlated.

Built With

Share this project:

Updates