MuFFLe++

A breast cancer patient file that keeps every test's evidence, labeled observed, inferred or hypothesis; a Nemotron agent on Nebius Token Factory writes the case note; NVIDIA FLARE trains the model.

Comment

Inspiration

A breast cancer changes over time, and the evidence about one patient is scattered: a stage recorded at diagnosis, a pathology report, slides, a breast MRI, a sequencing panel, expression and methylation profiles, tumor markers, the treatments given and when. Each test has its own specialist and its own software, and no hospital has enough patients to learn from alone. A clinician needs that evidence in one place, with what was measured kept apart from what a model guessed and what is only a lead from the literature.

Our long-term goal (docs/VISION.md) is a multimodal, longitudinal cancer intelligence platform: it connects specialized models, public datasets and each patient's record over time into one evidence-linked picture of the disease, and learns across institutions without pooling patient data. It is research and decision support for clinicians, never a diagnosis or a prescription. MuFFLe++ is its first milestone, for breast cancer.

What it does

MuFFLe++ is a clinician application. It opens on a patient list; a patient file opens on an overview of what every test says; each test opens a workbench built for its own data; a timeline keeps every version of the report. Every finding carries one of three labels: observed (measured or recorded), model inference (a prediction, with its uncertainty) or hypothesis (literature and evidence leads). Every module says whether it is implemented, experimentally validated or planned, and every threshold names its source; a value nobody has validated is marked placeholder in orange.

The files. SYNTHETIC-001, a synthetic patient we wrote by hand (not a real person, not taken from any dataset), carries a 40-month record: diagnosis, adjuvant therapy, a bone recurrence, tumor sequencing (PIK3CA H1047R), CDK4/6 therapy, a liver progression and alpelisib. The other 40 files are real, public TCGA-BRCA patients the risk model never trained on, each assigned to one of three simulated hospitals.

Patients. A Start here card opens the longitudinal example (SYNTHETIC-001) or a case with its own slide and sequencing (Case 299); pinned and recently opened files sit above the list. Search by patient, gene (PIK3CA), subtype, stage or workbench; filter by PAM50 subtype, mutated gene, stage, open alerts and what the file holds (slides, imaging, documents). Search every file looks through every file's findings at once (cards, attention items, variants, documented facts, methylation calls, hypotheses) in plain words, such as "HER2 disagree" or "stage III PIK3CA", and opens the finding; it is a text lookup, not a model. A four-step walk-through on the first visit, a "?" guide and a tooltip on every control say how to read it; breadcrumbs name the open workbench ("Tests › Labs › CA 15-3, month 11") under a sticky patient header, and the address (?patient&view&test&pointer&version) reopens the same screen on the same finding, so a link lands on the evidence.

The patient file. It opens on finding cards from every workbench (receptors recorded and documented side by side, PAM50 subtype, drivers with their best CIViC level, the slide picture, imaging lesions, the risk, the top evolution hypothesis). Each card has Why? (source test, method and version, threshold source, trust) and an Open that names where it lands ("Open · Sequencing, month 0") inside its workbench. One attention queue gathers alerts, tests that disagree (seven cross-test rules, such as ERBB2 copy number against HER2 status, each with its cited basis) and every workbench's "look here" items, ordered by consequence × trust, with keyboard triage, review marks and a Next step line. An action row on every view holds what a file can do: Attention queue, Import a test, Compare versions, Export this report, Your marks, Pin this file, Methods and validation, How to read this. Where this patient sits places the patient's scores among the 40 TCGA files, and Similar patients lists the five nearest TCGA files on eight recorded and derived fields (equal weights, marked placeholder): lookups over what the files hold, never a prognosis. Tests opens on a launcher, a tile per test with what it holds and its look-here count (a dated series is one tile with a pill per month). Every report exports as one page. Between the cards and the queue, the report's cockpit sets three columns side by side: the risk model, the driver mutations with their CIViC evidence ladder, and what to look at next; every finding there shows its source, method, version and evidence item on hover and opens its workbench (or civicdb.org) on click. The app has its own clinical light theme (IBM Plex Sans and Mono, compact controls, AA colours, every chip keeping its word so colour is never the only signal), and the sidebar's About this system shows the system's status: the release, each simulated site's held-out C-index, the planner and writer models, the case-note spend against the hard cap, and the code's version.

Eight workbenches, each with its own viewer, measurements, look-here queue, quality checks, "what this test cannot see", a model and validation card, and export:

  • Pathology (five TCGA cases, eight of their own diagnostic sections): a slide map on the patient's own scan, a see-through colour layer in seven views (tissue, zones, risk, flags, certainty, quality, immune), with search inside the slide in plain words, a section report and a pan-and-zoom viewer down to single nuclei one click away; a tissue map from a breast tissue head trained on BCSS on a frozen open encoder (GigaPath-Flash; tiles unlike its training tissue are not scored); the invasive front, its zones and the direction of invasion per arc (a hypothesis about the section); StarDist nuclei, breakaway (budding) candidates in the ITBCC counting field and lymphocytes by TILs zone; a second opinion on lymphocytes from the WSInfer TIL model, whose disagreements with the head go to the queue. Seven tile encoders were compared with the same head on the same held-out slides (accuracy: GigaPath-Flash 0.919, QuiltNet-B-32 0.877), and the sections were re-scored with the winner.
  • Expression, methylation and protein (TCGA cases): PAM50 subtype with its margin, Hallmark pathway scores, ten RPPA protein pathways, promoter methylation of six genes, an immune and stromal picture read against the slide's lymphocytes, and receptor consistency (IHC against RNA, copy number and protein).
  • Genomics: variants with detection, interpretation and predicted effect kept apart (Ensembl VEP, ClinVar, AlphaMissense), the CIViC evidence ladder with resistance items, the copy-number landscape with ABSOLUTE purity and ploidy, mutation burden and allele fractions, a cited pathway map and variants over time.
  • Radiology: the synthetic patient's imaging reports read as text; for five TCGA cases with public TCIA studies, a scan map on the study's own image (see-through layers: lesions with their MedSAM contours, kinetics by curve type, enhancement, change, density; a slice slider inside the map; search in plain words; click a square for its full story), with a cine, MedSAM contours of rule-seeded candidates, size, volume and contrast kinetics; mammography with mammoscreen and MONAI density; change over time on a labeled public example pair (none of the bundled patients has two study dates).
  • Clinical documents: pathology reports (TCGA-Reports) and imported documents; breast pathology facts read by rules and shown at their source sentence, documented kept apart from inferred, conflicts side by side; an open oncology named-entity model's reading drawn as a second layer (model inference) and set beside the rules field by field, measured on the same held-out reports. It mostly marks where a field is named ("ESTROGEN", "MARGINS") rather than its result, so the rules stay the documented reading and the model is a second opinion. Scanned PDFs are read to text by a page model in the local pipeline.
  • Labs: any analyte, with adult reference intervals cited from the Mayo Clinic Laboratories test catalog, the lab report's own interval winning, CTCAE grades, marker kinetics and the EGTM confirmation rule for tumor-marker rises.
  • ctDNA monitor: allele fractions over time, the ESR1 rule (PADA-1, SERENA-6) as evidence for review and a clonal hematopoiesis flag, on a synthetic series for SYNTHETIC-001 only (no open serial breast ctDNA data exists).
  • Evolution: what to watch next from a cited table of 18 context → escape-route rows (37 papers, every PMID checked), clonal order from allele fractions shown as alternatives, and recruiting trials for the patient's observed genes from ClinicalTrials.gov. Hypotheses for clinician review, never a regimen.

Import tests: lab CSVs, reports as text or PDF, VCF and MAF sequencing results, imaging reports, slide images and DICOM files (header only), clinician notes. Each import is checked row by row, adds a report version and re-runs the alert rules. Only public or synthetic data is imported.

Timeline: the record on one month axis, a lane per kind (report versions, treatments, imaging, labs, ctDNA, sequencing, pathology, clinic visits, imports), SOS alerts marked where they were raised; each report version is marked by how the risk moved since the one before. Time travel: click any point, or a row of the versions table (newest first), and the whole file opens as it stood at the report version that point created. Compare two versions comes first on the page: the risk delta as a chip, red or green only when it goes beyond the spread of the five models (otherwise "within the model's uncertainty"), why the risk moved one change at a time, then the findings, tests, attention items and note sentences that were added, changed or removed, each opening its evidence as of its version. A delta is the difference between two snapshots of the record, not a forecast, and the page says so.

Feedback: every review mark (agree, disagree, second look, reviewed) becomes a ledger record naming the model or rule behind its finding, with its version and evidence label; the page shows disagreement rates per model or rule with a 95% interval, the disagreed findings as a queue, and exports and imports the ledger. Marks stay in the browser session and nothing is trained on them: learning from clinicians needs consent, ethics approval and data agreements first, so retraining is a labeled roadmap card.

The case note (Nemotron on Nebius Token Factory). For a TCGA file, a case agent writes a tumor-board note: Nemotron 3 Super plans and calls our tools (what was recorded, the risk and percentile, each data layer's effect, top features, CIViC evidence for the observed drivers, a Tavily literature search); Nemotron 3 Ultra writes the note, each bullet labeled observed, model inference or hypothesis; a plain-code grounding check flags any number, gene or link no tool returned, and the evidence chain ties each sentence to its tool, field and value. The hosted app runs the agent live: Write the note with Nemotron makes these calls to Nebius Token Factory and Tavily at run time, five live notes per browser session, every call charged against the server's spending cap before it is sent ($3.00 on the hosted app, under the code's hard $25.50). Nine saved notes in the release show the agent's output without a wait.

Learning across hospitals (NVIDIA FLARE). The risk model is trained across three simulated hospitals (TCGA-BRCA tissue source sites) without moving patient data: each site trains locally and only weights travel, with NVIDIA FLARE's FedAvg. A hospital that never measured a data layer still takes part (the layer is masked).

Methods and validation: a card per model and rule with its status, numbers, sources and known failures; federated training; data sources and licenses; every known failure in one table and every placeholder in another.

Only derived numbers (risk, percentile, contributions) and gene and mutation names go to Token Factory and Tavily: never a patient's omics values, and nothing from MSK-CHORD (neither its data nor the output of a model trained on it, which includes SYNTHETIC-001's risk scores).

How we built it

  • Risk model (PyTorch): an encoder per data layer (copy number, RNA, methylation, protein, clinical), masked additive fusion and a Cox (DeepSurv) head, trained with modality dropout that never drops clinical data. Clinical-anchored residual: the model learns a clinical risk first; the omic layers may correct it by an amount chosen on hospitals it wasn't trained at.
  • Federated training: NVIDIA FLARE's FedAvgRecipe in the simulator, three simulated hospitals of 355, 355 and 354 patients; leave one hospital out, five reported seeds, tuning on five other seeds, always against clinical-only, one-hospital and pooled baselines.
  • Case agent: OpenAI-compatible tool calling to Nemotron 3 Super and Ultra on Nebius Token Factory; the tools run our models locally; every call goes through a hard spending cap that reserves its worst-case cost first. Without a key the same tools run in order and a template writes the note.
  • Patient timeline: gradient boosting trained on MSK-CHORD (5,368 breast cancer patients; CC BY-NC-ND 4.0, local only) re-predicts the 12-month risk of progression or death at each new event; it scores the synthetic patient.
  • Workbench pipelines run offline (in WSL on a Windows PC, CPU only) and ship derived numbers and small images; the hosted app runs no model. Slides: OpenSlide tiling at 0.5 µm/px, GigaPath-Flash embeddings, a logistic tissue head trained on BCSS, StarDist, WSInfer TIL, Deep Zoom pyramids served from a GitHub release to an OpenSeadragon viewer. Radiology: TCIA downloads, a seed rule, MedSAM box prompts, kinetics, a rigid alignment by mutual information written with SciPy. Omics: PAM50 centroids (genefu), MSigDB Hallmark, RPPA pathways (Akbani et al. 2014), ESTIMATE gene lists. Genomics: ASCAT3, ABSOLUTE, Ensembl VEP, AlphaMissense. Documents: regular expressions and an open zero-shot oncology named-entity model, both measured against TCGA's clinical fields; a page model reads scans to text. Evolution: a hand-curated table with PubMed-checked citations and ClinicalTrials.gov API v2.
  • App: Streamlit and Altair on Streamlit Community Cloud, from a small release bundle (model weights and 40 held-out cases' processed inputs) plus the slide tiles on a second release. The 41 files are built once per bundle, and each workbench's summary and queue once per file and version (a report shown again costs under 0.6 ms instead of 45–267 ms; whole screens about 5% faster). A memory pass held the server under the hosted app's 690 MB: a ten-file walk ends at 612 MB instead of 673–701 MB, and each extra reader adds about 5 MB instead of 33. Search across the files is a plain index of their findings, built on the first search.

Models and their licenses. Used in the app (open licenses, results precomputed offline): GigaPath-Flash (Apache-2.0), StarDist 2D_versatile_he (BSD-3-Clause), WSInfer TIL InceptionV4 (CC BY 4.0), MedSAM ViT-B (Apache-2.0), mammoscreen (Apache-2.0), MONAI breast density (Apache-2.0), AlphaMissense predictions (CC BY 4.0), OpenMed ZeroShot NER Oncology 220M (Apache-2.0; its spans ship for the bundled reports), Granite-Docling 258M (Apache-2.0; scans, local pipeline only), our own breast tissue head (trained on BCSS, CC0) and risk model (MIT, TCGA open data), and NVIDIA Nemotron 3 Super and Ultra (NVIDIA open model license) on Token Factory. Local only: models with non-commercial or no-derivatives terms (UNI2-h and CONCH, CC BY-NC-ND 4.0; GPFM, whose official repository is CC BY-NC-ND 4.0; SPIDER-breast, CC BY-NC 4.0; PanNuke-trained cell models) and the MSK-CHORD timeline models may be measured on our machine, but nothing from them enters the repository, the hosted app, a release or an LLM call (enforced in code for UNI2-h and CONCH). Every dataset, model and service, with its license and what leaves the machine: docs/registry.md.

Challenges we ran into

  • Omics underperformed clinical data. Copy number alone ranked patients worse than chance across hospitals (0.485), a sign of site and batch effects. The clinical-anchored residual, with its weight chosen on held-out hospitals, closed most of the gap to clinical-only (pooled 0.700, federated 0.693, against 0.702); no fusion variant beats clinical data alone.
  • Agent reliability. Notes dropped the clinical layer, Ultra's reasoning ran past the token limit, the planner stopped before the literature step. Fixes: the grounding check with one rewrite, Super writing the note when Ultra is cut off, one reminder turn.
  • Pathology without a GPU. Whole-slide encoders run at a fraction of a tile per second on a CPU. We scored the slides' regions of interest with a small open encoder (QuiltNet-B-32 first, then GigaPath-Flash, Prov-GigaPath distilled to a ViT-S/16), kept the viewer's tiles on a release (661 MB that never enter the repository) and wrote down every cutoff carried over from colon tissue as a placeholder.
  • A better encoder brought a new failure. GigaPath-Flash raised the tissue head from 87.7% to 91.9% on the held-out slides, but on one section it read cautery-crushed margin tissue, a class neither head was trained on, as carcinoma. Tiles unlike anything the head was trained on are now left unscored (a cutoff calibrated on the training slides, flagging 0.3% of held-out tiles), and what that rule misses is a stated known failure on that section's page. The re-score also found that three sections scored through a window had been read from the wrong part of the slide (tissue 3–5 mm away, or glass) while the viewer drew them over the right part; they are now read at their place.
  • Licenses that change the design. The strongest pathology encoders are CC BY-NC-ND; we kept them local only and built on MIT and Apache-2.0 models. AlphaMissense's table header and its README disagree; we follow the licensor's current statement and ship only the 21 scores we use. MSK-CHORD allows local research but no redistribution, so its models and their outputs stay local and the timeline patient is synthetic.
  • Records are incomplete and messy. A patient with no mutation call may not have been sequenced (unknown, not wild type). OCR in pathology reports slid a form's answers one row, so a HER2 "positive" belonged to PR; the rules now refuse a status that the report's own IHC score argues against.
  • Two patients for change over time. None of the bundled patients has imaging on two dates, so change over time runs on a public example pair, labeled as such, with a local check that says where the alignment fails.

Accomplishments that we're proud of

  • A clinician's path from a patient list to the evidence: an overview with one attention queue, and eight workbenches each at the depth of a specialist's tool, not a summary widget.
  • Honest labels everywhere: observed, model inference, hypothesis, synthetic, simulated, placeholder; a Methods page that lists every known failure and every placeholder.
  • Measured where we could: PAM50 agrees with TCGA's published calls on 94.7% of 950 tumors (κ 0.92; the centering was chosen after a first run, so not an independent test); the tissue head reaches 91.9% held-out accuracy on three patient-disjoint slides; the second-opinion lymphocyte model reaches AUROC 0.955 on those slides (an upper bound: it was trained on TCGA); when a report states a fact, the document rules read it right 91–98% of the time for seven of ten fields on 481 held-out reports. The numbers appendix lists each with its caveat.
  • Federated training across simulated hospitals comes within 0.01 of pooling the data (0.693 ± 0.017 against 0.700 ± 0.012, pooled site-stratified C-index over five seeds) and beats one hospital alone (0.649 ± 0.032), with the clinical-only baseline (0.702 ± 0.004) reported even though it wins. The app's demo model scores a held-out C-index of 0.728 (a different quantity: one trained model, the mean over the three hospitals' held-out patients).
  • Four QA rounds in a headless browser at two widths; the last walked 762 screens after the final UI round, every expander opened, with no exception, error box or cut text.
  • Every claim in a case note traces to a tool call, and the app shows the chain.

What we learned

  • A fair split matters more than a clever model: whole held-out hospitals took away the omics' apparent edge.
  • Small and large Nemotron models work well together: Super for quick structured tool calls, Ultra for the written synthesis; and an LLM in medicine needs a check that doesn't trust it.
  • A placeholder that looks validated is worse than a missing feature, so the app colors every placeholder and lists them in one place.
  • Depth per test beats breadth: a clinician trusts a tissue map they can zoom into and check, not a score.

What's next for MuFFLe++

  • A pathologist's and a radiologist's review of the scored slides and studies, and calibrated cutoffs in place of the placeholders.
  • Stronger pathology encoders on a GPU (open-license ones in the app), and validation of StarDist and MedSAM on breast.
  • Inferred mutation status with calibrated uncertainty, and evolutionary hypotheses from repeated sequencing and real serial ctDNA.
  • Learning from clinicians' review marks, the Feedback page's roadmap: re-calibrating placeholder cutoffs against disagreed findings and retraining the slide head on tiles a pathologist confirms, validated on held-out patients before use.
  • Real contributing institutions with consent, ethics approval and data agreements first, and secure aggregation or differential privacy in the federation.

Built With

Share this project:

Updates

Submission history