OncoloHealth

OncoloHealth is a one-stop shop for cancer AI research: find models and datasets, check if they can be trusted, and run experiments, all in one place for researchers and anyone curious.

Comment 1

Inspiration

This one is personal for us. A close friend's mother went through breast cancer, and seeing what she and her family went through stayed with us. We wanted to build something that could one day help people like her.

One of our teammates also works with medical researchers, and kept asking what would actually help them. Good cancer AI models get published all the time, but reusing one for a new question means starting over: locate the weights, read the license, match the preprocessing, supply the right input and work out what the output actually supports. Small labs rebuild that stack for every paper, and when a step gets skipped, results look better than they really are. We built OncoloHealth so a research question leads straight to a real model result, with a record of exactly how it was produced.

What it does

OncoloHealth is a cancer research workbench that pairs a conversational assistant with local imaging specialists, a model and dataset catalog, and reproducible experiment records. It runs on one PC.

Attach a scan in the chat, pick a compatible task, and watch the assistant call an installed specialist live. The app returns real class scores, segmentation masks or detection boxes, followed by an explanation grounded in that output. Reference labels and masks are stored separately and compared only after inference, and disagreements stay visible.

Today it runs 10 specialist variants across seven working paths:

  • bone X-ray classification
  • breast ultrasound classification
  • lung CT patch classification
  • full-scan lung nodule detection: top-box reference IoU 0.163 on an already-inspected publisher-validation demonstration
  • brain MRI tumor segmentation: whole-tumor Dice 0.897, tumor-core 0.835 and enhancing-tumor 0.832 on one potentially overlapping demonstration, not an independent reserved test
  • pathology patch classification
  • skin-image classification

The catalog holds 14 model records and 18 dataset records, drawn from the 94 public cancer datasets we mapped, with installed artifacts clearly separated from resources that are only cataloged. Every saved run keeps the input identity, model and code identities, preprocessing, output, timing and any failure. Unsupported tasks stay visibly unavailable instead of getting an invented prediction.

How we built it

The app is a Python backend with a browser interface: streaming tool steps, file attachments, saved conversations and result cards. The codebase includes an Anthropic provider integration; the current PC app permits only local coordinator endpoints and runs Google's Gemma 4 26B A4B with our own workflow LoRA adapter.

We fine-tuned that adapter on one GPU of an NPS DGX GB300 supercomputer through Run.ai, using 83 workflow examples, 21 development examples and 42 optimizer steps. Then we converted the base model to Q4_K_M GGUF and served it with llama.cpp, CUDA and CPU offload on a desktop RTX 5070 Ti with 16 GB of VRAM, with a verified 16,384-token context. An earlier catalog-workflow demonstration measured 38–59 generated tokens per second; this is not the latency of the slower imaging tasks. Everything interactive, including the specialists, runs on the PC, independent of the cluster.

The imaging specialists use restored MONAI segmentation and detection models, frozen ResNet-50 features with task heads we fitted locally, and explicit simple baselines. PyTorch, timm, Transformers, scikit-learn, NumPy and NiBabel handle training, preprocessing and inference. Claude Research helped us map the public dataset and model landscape behind the catalog, and Claude Code and OpenAI Codex helped us build and test the software.

Challenges we ran into

The hardest part was completing the whole path. A catalog entry, downloaded weights or an uploaded preview doesn't prove a model actually works. We had to restore preprocessing and trained heads, validate input formats, keep answer keys separate from model inputs, and verify every result through the app itself.

The supercomputer was its own adventure: a brand-new model family on brand-new GPUs, and jobs that kept crashing, one right after saving its checkpoint. So we made every checkpoint earn our trust. We reloaded the saved adapter in a fresh process, checked all 230 saved tensors, and reproduced reference outputs before using it. When a newer candidate improved some isolated checks but finished fewer complete workflows, we kept the one that works.

We also hit context overflow, local memory pressure and model-generated factual mistakes. We answered with bounded workers, cancellation, and tools that read both models' measured evidence before computing any comparison.

Accomplishments that we're proud of

  • Real file-to-result workflows run locally, end to end, including brain MRI segmentation and full-scan lung CT detection.
  • Our fine-tuned local coordinator passed 20 of 22 fixed application-workflow checks, including real MRI and CT tool execution. Two grounding failures remain documented; this is workflow success, not prediction accuracy.
  • On already-inspected sample-level test splits, our ResNet pathology head reached 80.8% accuracy versus 51.0% for the pixel baseline across 7,180 tissue patches. Skin classification reached 59.2% versus 51.5% across 2,005 images.
  • For ultrasound we reported an honest tradeoff instead of a headline: accuracy rose from 76.3% to 77.6% on 156 images, while AUROC moved from 0.833 to 0.830 and malignant recall held at 66.7%.
  • Our audit of BTXRD found 21 exact duplicate image pairs among 3,746 images, seven of them crossing the authors' published split, and no patient IDs at all.

What we learned

The source of the data matters enormously. In the exploratory RAD-DINO tumor-vs-normal experiment, ROC-AUC was 0.846 on the image-stratified test and 0.616 on the source-center test. Those source centers were already inspected, and missing patient IDs prevent a patient-independent interpretation.

Running a model, agreeing with one reference label, measured test performance and independent clinical validation are different kinds of evidence, and a useful tool has to make that difference obvious. We also learned that you don't always need a bigger model. Fixing input contracts, evidence retrieval, context handling and honest error reporting made the models we have dramatically more useful.

What's next for OncoloHealth

Our long-term goal is to turn OncoloHealth into a company that helps researchers move faster toward better ways to prevent and cure cancer. Next, we're:

  • training and evaluating main5, whose workflow curriculum is prepared but untrained, and requiring it to pass reload and complete-workflow checks before replacing the current coordinator
  • bringing in stronger specialists and comparing them fairly
  • opening the catalog to contributions from every lab
  • sitting down with cancer imaging researchers to measure how much time it saves them

After that, we'll work with dataset maintainers on patient grouping and independent test sets, and expand into whole mammography, whole-slide pathology and full DICOM studies as verified models and data allow.

OncoloHealth is a research prototype. These demonstrations and development comparisons do not establish clinical diagnostic or cancer-detection performance.

Built on the research community

We credit Google Gemma, llama.cpp/ggml, MONAI, PyTorch, timm, scikit-learn, MedMNIST, BTXRD, BraTS/MSD and TCIA/LUNA16 contributors, and NPS computing infrastructure. Cataloged resources from other institutions are credited in their records; inclusion is not an endorsement or a claim that every resource is installed.

Built With

Share this project:

Updates

Submission history