Biofusion-AI: Cross-Modal Breast Cancer Detection System

Seeing beyond the scan — fusing imaging and clinical data to catch what single-modality diagnosis misses.

Comment

Inspiration

Breast cancer remains one of the most commonly diagnosed cancers worldwide, and early, accurate detection is often the single biggest factor in patient survival. Mammography and ultrasound are the gold standard for screening, but we kept coming back to one uncomfortable truth: imaging alone often isn't enough. A radiologist looking at a mammogram doesn't just see pixels, they're mentally cross-referencing a patient's age, family history, breast density, and hormone therapy status, often from memory or scattered notes. That context lives outside the image, yet it heavily influences the final diagnosis. We also knew that:

  • Limited context in imaging-only models leads to missed nuance in ambiguous cases
  • Inter-observer variability means two radiologists can read the same scan differently
  • Resource constraints in under-served regions mean specialized diagnostic expertise isn't always available
  • Time pressure from high patient volumes makes faster, more confident triage genuinely life-saving That gap — between what the image shows and what the patient's history means — is what inspired BioFusion-AI. We wanted to build a system that thinks the way an experienced clinician does: by fusing what it sees with what it knows about the patient.

What it does

BioFusion-AI is a cross-modal diagnostic pipeline that fuses three distinct data sources into a single, unified prediction:

  • Mammogram images: analyzed for structural and textural abnormalities
  • Ultrasound images: analyzed for density and mass characteristics
  • Structured clinical history: age, breast density, family history, hormone therapy, and other risk factors Rather than treating these as three separate signals, BioFusion-AI extracts deep features from each modality and merges them into one shared representation before classification. The output is a benign/malignant prediction that reflects the whole patient picture, not just a single scan, reaching 92% accuracy and 91% F1-score in our evaluation.

How we built it

We approached this as a pipeline problem — get each modality into a comparable feature space, then fuse intelligently. 1. Image Feature Extraction We fine-tuned MobileNetV2 CNNs separately for mammogram and ultrasound images, chosen for its lightweight architecture that still transfers well from ImageNet weights to medical imaging. We applied data augmentation (rotation, flipping, zoom, brightness shifts) to combat our limited dataset size and reduce overfitting. 2. Synthetic Clinical History Since paired, real-world clinical records were hard to source (more on this below), we engineered a synthetic structured dataset simulating realistic patient profiles — mixing categorical variables (family history, hormone therapy) and numerical variables (age, density scores) — designed to reflect plausible real-world risk distributions. 3. Feature Fusion Each modality produces a high-dimensional feature vector. We applied PCA and scaling to bring these into a comparable range and reduce dimensionality before concatenating them into a single unified feature vector — the "fused" representation of the patient. 4. Classifier Ensemble We didn't assume one algorithm would win — we benchmarked Logistic Regression, SVM, Random Forest, XGBoost, and Gradient Boosting against the fused feature vector. Gradient Boosting came out on top, hitting 92% accuracy. 5. Validation Beyond a standard train-test split, we ran a sanity check by shuffling labels and re-training — performance collapsed to near-random, confirming the model was learning genuine diagnostic patterns rather than memorizing noise.

Challenges we ran into

The single biggest challenge was data availability. We quickly discovered it was practically impossible to source paired mammogram and ultrasound images from the same patient sample — most public datasets cover one modality or the other, collected from entirely different patient cohorts, with no shared identifiers linking them. This meant we couldn't directly fuse real image pairs at the patient level the way a real clinical workflow would. To work around this, we restructured our approach around synthetic clinical history data and modality-level feature fusion rather than strict patient-level pairing. This let us validate the fusion methodology end-to-end, but it also meant our results represent a proof-of-concept rather than a clinically-deployable system — an important distinction we were careful to keep in mind throughout evaluation. Beyond data, we also wrestled with:

  • Balancing model complexity against a relatively small image dataset (mitigated with augmentation and transfer learning)
  • Getting PCA-reduced image features and structured clinical features onto a comparable scale before fusion
  • Choosing an ensemble strategy that didn't just favor whichever modality had stronger raw signal

Accomplishments that we're proud of

  • Built a fully functional multi-modal pipeline — from raw images and structured data to a single fused diagnostic prediction
  • Achieved 92% accuracy and 91% F1-score, competitive with many single-modality baselines despite the added complexity of fusion
  • Designed a rigorous validation strategy, including the label-shuffling sanity check, to prove the model wasn't just overfitting
  • Successfully combined three fundamentally different data types (two image modalities + structured tabular data) into one coherent feature space

What we learned

  • Cross-modal fusion is powerful, but only as good as your data pairing — the theoretical benefit of combining modalities is only fully realized when the data itself reflects real patient-level relationships
  • Transfer learning is a huge unlock for medical imaging tasks with limited data — fine-tuning MobileNetV2 gave us strong features without needing millions of training images
  • Sanity checks matter as much as accuracy numbers — a high accuracy score means nothing if you can't prove the model is learning real patterns
  • We came away with a much deeper appreciation for why real clinical AI systems take so long to build and validate — the data engineering challenges are often harder than the modeling itself

What's next for BioFusion-AI: Cross-Modal Breast Cancer Detection System

Our top priority is moving from synthetic to real paired patient data — sourcing datasets where mammogram, ultrasound, and clinical history all come from the same patient, enabling true patient-level fusion instead of modality-level approximation. On the modeling side, we plan to refine our fusion technique using cross-attention mechanisms, allowing the model to learn which clinical factors are most relevant to which visual features dynamically, rather than relying on simple concatenation after PCA. This should let the model capture richer interactions between modalities — for example, learning how breast density interacts differently with mammogram texture versus ultrasound echogenicity. Longer-term, we're also exploring:

  • Explainability tools (Grad-CAM, SHAP) so clinicians can see why the model reached a decision
  • Expanding to additional modalities like MRI or digital breast tomosynthesis (DBT)
  • Multi-center validation to test generalizability across different patient populations and imaging equipment
Share this project:

Updates

Submission history