Multimodal AI for Melanoma Decision Support
Inspiration
Skin cancer is one of the most widely diagnosed malignancies worldwide, with melanoma representing one of its most aggressive and life-threatening forms. Early detection is crucial for improving patient survival. When melanoma is identified at a localized stage, the 5-year survival rate can reach approximately 99%.
However, traditional Computer-Aided Diagnosis (CAD) systems primarily analyze dermoscopic images in isolation, overlooking important patient information that dermatologists routinely consider during diagnosis. Furthermore, conventional deep learning models often operate as black boxes, providing predictions without clear explanations of the factors influencing their decisions.
These limitations inspired us to build Multimodal AI for Melanoma Decision Support, a unified AI platform that combines dermoscopic images, structured clinical metadata, Explainable AI (XAI), and Retrieval-Augmented Generation (RAG) to provide transparent, context-aware melanoma screening support.
Our goal is to bridge the gap between powerful AI predictions and interpretable clinical decision support.
What It Does
Our platform integrates visual skin lesion evidence with structured patient information to support melanoma screening through the following capabilities:
1. Multimodal Data Fusion
Combines high-resolution dermoscopic images with structured clinical metadata, including:
- Patient age
- Biological sex
- Anatomical location of the lesion, such as the torso, extremities, or head and neck
Integrating these complementary data sources allows the model to consider both visual lesion characteristics and relevant patient context.
2. Dual-Pipeline Deep Learning Classification
Processes images and clinical metadata through parallel neural network branches.
The visual branch extracts deep features from dermoscopic images using a CNN backbone, while the metadata branch encodes structured clinical variables using a multilayer perceptron (MLP).
The learned representations are concatenated and passed through a classification head to estimate the probability of malignancy.
3. Explainable AI (XAI)
Provides interpretable outputs using three complementary techniques:
- Grad-CAM: Generates visual heatmaps highlighting image regions that influence the model's prediction.
- SHAP: Estimates the contribution of individual clinical features to a prediction.
- LIME: Produces local explanations by analyzing perturbed image regions and their influence on the model output.
These explanations help users inspect whether the model is focusing on meaningful lesion characteristics rather than irrelevant image artifacts.
4. Retrieval-Augmented Generation (RAG)
Uses a FAISS vector index to retrieve semantically similar reference cases from an indexed collection of image-text pairs.
The retrieved examples are supplied as contextual evidence to a Vision-Language Model (VLM), such as LLaVA v1.5, to support in-context classification and structured report generation.
This approach aims to improve contextual reasoning without requiring additional fine-tuning of the VLM for every new case.
5. Interactive Clinical Web Interface
Provides an interactive interface built using FastAPI and/or Streamlit for submitting images and clinical metadata, viewing predictions, and inspecting explainability outputs.
The architecture is designed with potential future integration into Electronic Health Record (EHR) systems and healthcare interoperability standards such as HL7 FHIR.
Important: The platform is intended for research and decision support, not as a replacement for professional medical evaluation or histopathological diagnosis.
How We Built It
We developed an end-to-end multimodal deep learning pipeline using Python and machine learning frameworks, including PyTorch, TensorFlow, and Scikit-learn.
1. Data Preprocessing and Feature Encoding
Dermoscopic Image Processing
Input images are resized to either \(224 \times 224\) or \(128 \times 128\) pixels, depending on the model configuration.
The preprocessing pipeline includes:
- Pixel normalization to improve numerical stability.
- Contrast-Limited Adaptive Histogram Equalization (CLAHE) to improve local contrast.
- Morphological filtering to reduce hair and other visual artifacts.
- Data augmentation using random cropping, vertical flipping, color jitter, and Gaussian blur.
- Region-of-interest segmentation using U-Net, where available, to focus on lesion regions.
These operations help improve robustness to variations in image quality, illumination, and acquisition conditions.
Clinical Metadata Processing
Continuous variables, such as patient age, are standardized using Z-score normalization:
$$ Z = \frac{X - \mu}{\sigma} $$
where:
- \(X\) is the original feature value.
- \(\mu\) is the mean of the feature.
- \(\sigma\) is its standard deviation.
Categorical variables, such as biological sex and anatomical site, are transformed using one-hot encoding.
Missing values are handled using median imputation for continuous variables and mode imputation for categorical variables. Imputation statistics and normalization parameters are fitted on the training set to prevent information leakage.
2. Parallel Neural Network Architecture
The model contains two independent feature extraction branches.
Visual feature encoder: Extracts deep image representations using a CNN backbone, such as ResNet or EfficientNet.
Metadata encoder: Processes structured clinical variables through fully connected layers with Batch Normalization, ReLU activation, and Dropout.
The resulting feature vectors are concatenated and passed through a classification head.
Conceptual PyTorch Fusion Module
import torch
import torch.nn as nn
class MelanomaFusionModel(nn.Module):
def __init__(self, img_backbone, meta_dim, num_classes=1):
super().__init__()
self.img_encoder = img_backbone
self.meta_encoder = nn.Sequential(
nn.Linear(meta_dim, 64),
nn.BatchNorm1d(64),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(64, 32),
nn.BatchNorm1d(32),
nn.ReLU()
)
self.classifier = nn.Sequential(
nn.Linear(512 + 32, 128),
nn.BatchNorm1d(128),
nn.ReLU(),
nn.Dropout(0.4),
nn.Linear(128, num_classes)
)
def forward(self, img, meta):
f_img = self.img_encoder(img)
f_meta = self.meta_encoder(meta)
f_fusion = torch.cat(
(f_img, f_meta), dim=1
)
logits = self.classifier(f_fusion)
return logits
The example assumes that the image encoder returns a 512-dimensional feature vector. The backbone must be configured to produce this output dimension.
The classifier returns logits. During training, these can be passed directly to BCEWithLogitsLoss; sigmoid activation is applied during inference to obtain probabilities.
3. Feature Fusion and Mathematical Decision Layer
Let the image encoder produce a feature vector:
\[ \mathbf{f}_{\text{image}} \in \mathbb{R}^{C_1} \]
and the metadata encoder produce:
\[ \mathbf{f}_{\text{metadata}} \in \mathbb{R}^{d_1} \]
The two representations are concatenated to form a joint multimodal feature vector:
$$ \mathbf{F} = [\mathbf{f}{\text{image}} \, ; \, \mathbf{f}{\text{metadata}}] $$
This representation captures complementary information from the image and clinical data.
The fused features are passed through dense layers to generate a final classification logit:
$$ z = \mathbf{W}_{\text{out}}\mathbf{h}
- b_{\text{out}} $$
The estimated probability of malignancy is obtained using the sigmoid function:
$$ \hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}} $$
where \(\hat{y}\) represents the predicted probability for the positive class.
The model is trained using Binary Cross-Entropy loss, with class weighting to account for imbalanced data. Optimization can be performed using Adam or AdamW with a cosine annealing learning-rate schedule.
4. Retrieval-Augmented Generation and Explainability
The RAG component uses FAISS to index vector representations of reference cases. For a new input, the system retrieves the \(K\) nearest neighbors according to the selected similarity metric.
The retrieved examples are incorporated into a structured prompt for a Vision-Language Model, such as LLaVA v1.5.
The system combines this retrieved context with the current case to support:
- Contextual comparison with similar reference cases.
- In-context classification experiments.
- Structured summaries of model findings.
- More informative explanations of the prediction.
The explainability module complements this process by generating Grad-CAM heatmaps, SHAP feature attributions, and LIME image explanations.
These outputs are designed to make the system's behavior easier to inspect. They are explanations of model behavior, not proof of clinical correctness or causal relationships.
Challenges We Ran Into
1. Severe Class Imbalance
Skin lesion datasets often contain substantially more benign lesions than malignant melanoma cases. This can cause a model to favor the majority class and miss important malignant cases.
Our approach:
- Evaluated balanced dataset configurations where available.
- Used class-weighted loss functions.
- Considered sensitivity, specificity, F1-score, and ROC-AUC in addition to accuracy.
For a class \(i\) containing \(n_i\) training examples, an inverse-frequency weighting scheme can be expressed as:
$$ w_i \propto \frac{1}{n_i} $$
The weights must be normalized or otherwise scaled appropriately for the chosen loss function.
2. Patient-Level Data Leakage
Randomly splitting individual images can place images from the same patient in both training and testing sets. This can lead to overly optimistic performance estimates.
Our approach:
We used a patient-level partitioning strategy with a 70% training set, 15% validation set, and 15% test set, ensuring that patient identities do not overlap between partitions.
3. Cross-Modality Feature Imbalance
CNN feature representations can have much higher dimensionality than structured metadata representations, potentially causing one modality to dominate the fused representation.
Our approach:
- Applied Global Average Pooling to reduce spatial feature maps.
- Encoded clinical metadata using a dedicated MLP.
- Used Batch Normalization and Dropout to improve training stability and regularization.
4. Visual Artifacts and Background Noise
Dermoscopic images can contain hair fibers, ruler markings, uneven illumination, and background artifacts that distract the model.
Our approach:
We incorporated CLAHE, morphological filtering, and lesion-focused preprocessing. U-Net-based region-of-interest segmentation was also considered or integrated where available.
5. Integrating Retrieval with Vision-Language Models
A VLM may produce inconsistent classifications or unsupported statements when it receives insufficient context.
Our approach:
We used FAISS-based nearest-neighbor retrieval to provide similar reference examples in the model prompt. This creates a retrieval-augmented inference pipeline that can be evaluated against zero-shot inference.
Retrieved cases provide contextual evidence, but they do not independently establish the correct diagnosis.
Accomplishments That We're Proud Of
1. Strong Multimodal Classification Results
In our reported experiments, the multimodal fusion model achieved the following performance:
| Evaluation Metric | Reported Result |
|---|---|
| Accuracy | 94.5% |
| Sensitivity | 93.8% |
| Specificity | 95.2% |
| F1-score | 0.94 |
| ROC-AUC | 0.949–0.95 |
The model was evaluated against single-modality baselines, including VGG-16, ResNet-50, MobileNetV2, and EfficientNet.
These results suggest that combining image features with clinical metadata can be a promising approach. The reported values should be interpreted in the context of the dataset, class definitions, test-set size, and evaluation protocol.
2. Focus on Reducing False Negatives
Sensitivity is especially important in melanoma screening because missed malignant cases can delay further medical evaluation.
Our experiments indicated that incorporating clinical metadata improved sensitivity relative to the evaluated image-only baseline. This highlights the potential value of multimodal learning for screening support.
3. Transparent, Multimodal Explanations
We integrated visual and tabular explainability into the prediction pipeline:
- Grad-CAM identifies image regions associated with the model's prediction.
- SHAP estimates the contribution of individual metadata features.
- LIME illustrates local image-region contributions.
Together, these tools allow users to inspect the visual and clinical information influencing a prediction.
4. Retrieval-Augmented VLM Evaluation
In our reported experiments, retrieval-augmented inference improved the Vision-Language Model's F1-score from 0.3729 to 0.6864.
| VLM Evaluation Configuration | F1-score |
|---|---|
| Zero-shot inference | 0.3729 |
| Retrieval-augmented inference | 0.6864 |
These results indicate that retrieving similar reference cases may improve in-context classification performance without additional model fine-tuning.
The comparison is specific to our experimental configuration and should be validated on independent patient-level test data before drawing conclusions about general clinical performance.
What We Learned
1. Clinical Metadata Can Provide Valuable Context
Patient age, biological sex, and anatomical location can provide additional context beyond visual appearance alone.
Our exploratory analysis suggested that certain demographic and anatomical variables were associated with differences in the dataset's lesion labels. These relationships can help the model learn complementary patterns, although associations in a dataset do not necessarily imply causation or universal clinical risk.
2. Multimodal Fusion Can Improve Feature Separation
We used t-SNE to explore the learned representations of image-only and multimodal models.
The multimodal representation appeared to produce more distinct clusters in our exploratory analysis, suggesting that clinical metadata may help separate examples with similar visual characteristics.
However, t-SNE is a visualization technique rather than a quantitative measure of classification performance.
3. Explainability Is Essential for Trustworthy AI
High predictive accuracy alone is not sufficient for a medical decision-support system.
Grad-CAM, SHAP, and LIME provide complementary ways to inspect model behavior. Their explanations must still be assessed for stability, relevance, and consistency with clinical knowledge.
4. Healthcare Interoperability Matters
Practical deployment requires more than a trained model. Healthcare systems need secure data handling, standardized interfaces, appropriate access controls, and well-defined workflows.
Standards such as HL7 FHIR could support future integration with electronic health records, subject to institutional requirements, privacy protections, and clinical validation.
What's Next?
1. Expanded Imaging Modalities
We plan to investigate additional imaging modalities, including thermal imaging and reflectance confocal microscopy, where suitable datasets are available.
Combining different imaging sources may provide complementary information about lesion structure and physiological characteristics.
2. Privacy-Preserving Federated Learning
We aim to explore federated multimodal learning across multiple healthcare institutions.
This approach could allow participating institutions to collaboratively train models while keeping raw patient records within their respective environments. Additional safeguards would still be required to manage privacy risks, security threats, and differences between clinical populations.
3. Automated Clinical Report Generation
We plan to improve the report-generation component using structured templates and carefully validated vision-language models.
A future version could generate summaries aligned with established melanoma assessment frameworks, including the seven-point checklist, while clearly separating model observations from confirmed clinical findings.
4. Robustness and External Validation
Future work will focus on:
- External validation across independent datasets and clinical sites.
- Calibration of predicted probabilities.
- Evaluation of sensitivity at clinically relevant operating thresholds.
- Subgroup performance and fairness analysis.
- Confidence estimation and referral recommendations.
- Prospective evaluation with qualified medical professionals.
5. EHR Integration and Deployment
We plan to develop secure interfaces for structured clinical data exchange using HL7 FHIR and to evaluate integration with clinical information systems.
The long-term objective is to build an interpretable, privacy-conscious, and clinically validated AI decision-support platform.
Conclusion
Multimodal AI for Melanoma Decision Support combines deep learning, structured patient metadata, explainable AI, and retrieval-augmented vision-language intelligence in a single research platform.
By considering both visual lesion characteristics and clinical context, the system aims to provide more informative predictions and transparent explanations than image-only approaches.
Our work demonstrates the potential of multimodal AI as a foundation for melanoma screening research, while highlighting the importance of patient-level evaluation, clinical validation, explainability, privacy, and responsible deployment.
This project is intended for research and educational purposes. It is not a medical device and must not be used as a substitute for diagnosis, treatment, or advice from a qualified healthcare professional.
Log in or sign up for Devpost to join the conversation.