LUAD AI

Grad-CAM flags the tumor mass; PC dims 4 and 2 are the dominant genomic drivers of the Stage III/IV prediction

This project was reviewed & approved by Dr. Tshering Lachenpa, MD

Inspiration

Lung adenocarcinoma (LUAD) is the most common form of lung cancer, yet staging accuracy at early presentation remains a critical bottleneck — late-stage detection dramatically worsens survival odds. We were inspired by the growing body of research showing that CT imaging and bulk RNA-seq capture complementary biological signals. No single modality tells the full story, so we set out to build a system that fuses both.

What it does

LUAD AI takes a CT scan, RNA-seq gene expression data, or both, and classifies lung adenocarcinoma as early-stage (I/II) or late-stage (III/IV). It runs in three modes — Imaging Only, Genomic Only, or Multimodal — and produces interpretable outputs including Grad-CAM saliency maps highlighting the CT regions driving the prediction and gradient×input attribution across the top genomic PCA dimensions. A Gemini-powered clinical summary contextualizes the result with the patient's profile.

How we built it

We trained a LINEAR fusion model combining a ResNet-18 backbone pretrained on RadiologyNET (frozen, 512-dim CT embeddings) with ComBat batch-corrected RNA-seq features reduced to 480 PCA dimensions (99.9% variance). The fusion head is a lightweight MLP: Linear(992→64) → ReLU → Dropout(0.3) → Linear(64→1). Training used 5-seed × 5-fold cohort-stratified cross-validation across TCGA, CPTAC, and Stanford datasets (520 patients total). The demo app is built in Streamlit with Git LFS hosting model weights, and the showcase site uses Three.js and GSAP. Deployed on Streamlit Community Cloud and Cloudflare Pages.

Challenges we ran into

Batch effects across three independent cohorts were the biggest hurdle — ComBat normalization was essential to make the genomic features comparable. CT imaging data required careful preprocessing to extract 2D axial slices from DICOM volumes. We also found that multimodal AUC (0.618) came in lower than genomic-only (0.636), which forced us to think carefully about when fusion helps vs. hurts — ultimately framing it as a scientific finding rather than a flaw. Working with only ~160 patients with matched CT and RNA-seq data posed a significant constraint — early model versions with higher parameter counts overfit badly, producing lower AUCs, which forced us to reduce the fusion head to ~63K trainable parameters and rely heavily on frozen pretrained features. Git LFS was required to push 111MB of model artifacts to GitHub.

Accomplishments that we're proud of

Training a working multimodal deep learning model on real clinical data across three independent cohorts with no data leakage. Achieving an AUC of 0.636 in genomic-only mode using only ~63K trainable parameters — a direct result of carefully balancing model capacity against our limited paired dataset. Building a fully deployed, interactive demo with Grad-CAM, genomic attribution, and Gemini-powered clinical summaries. Getting the project reviewed and approved by Dr. Tshering Lachenpa, MD.

What we learned

Genomic data can carry more staging signal than imaging alone for LUAD — likely because RNA-seq captures molecular subtypes invisible to radiology. Batch correction is non-negotiable when combining multi-institution datasets. Fusion models don't always outperform unimodal baselines, and understanding why is as scientifically valuable as the performance numbers themselves. With small paired datasets, parameter count is as important a hyperparameter as learning rate — overfitting can silently destroy AUC before training loss shows any sign of it.

What's next for LUAD AI

Expanding the paired CT + RNA-seq dataset by incorporating additional public cohorts to overcome the data scarcity that limited this version. Expanding to survival prediction and treatment response modeling. Incorporating mutation data (EGFR, KRAS, ALK) alongside expression profiles. Replacing the linear fusion head with a cross-modal attention mechanism — only feasible with more data. Validating on prospective clinical data with physician feedback in the loop. Exploring federated learning to train across institutions without sharing raw patient data.