Cross-Modal 3D Brain Tumor MRI Retrieval

• 

Inspiration

Cross-modal medical image retrieval is a vital challenge in neuro-oncology. In brain tumor surgery, clinicians collect diverse 3D MRI scans across different phases and physics: pre-operative T1 post-contrast (T1c) where tumors light up white, and intra-operative T2 scans where cerebrospinal fluid glows bright white.

Comparing these scans naively with pixel differences (MSE or cosine similarity) is mathematically useless due to inverted tissue contrast. Furthermore, neurosurgery causes brain shift (sagging tissue from opened cranial pressure) and leaves large resection cavities (missing tissue where the tumor was excised).

Our mission for the Paris Research Hackathon on Kaggle was to engineer a robust cross-modal retrieval system that correctly identifies matching patient scans across three escalating levels of geometric and anatomical complexity.

What It Does

Given a query 3D brain volume of a patient in T1c modality, our pipeline searches an entire candidate gallery of T2 scans and ranks the true corresponding patient at Rank 1:

• Level 1 (Co-registered Pre-OP): Pairs share a common geometric grid but exhibit complete modality inversion.
• Level 2 (Deformed & Rotated Pre-OP): Pairs have synthetic rigid rotations, translations, and non-linear elastic deformations.
• Level 3 (Intra-Operative Surgery): Pre-OP T1c scans are matched against Intra-OP T2 scans featuring tissue shifts and tumor resection voids.

We achieved 3rd Place on Kaggle with an MRR (Mean Reciprocal Rank) exceeding 0.97 on the public leaderboard and 0.909+ on the private test set:

$$MRR = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{\text{Rank}_i}$$

How We Built It

While many competitors attempted to train complex 3D Deep Learning models on massive GPUs, we discovered that small medical datasets (~350 pairs) cause neural networks to severely overfit.

We pivoted to an unsupervised, training-free, deterministic signal-processing and optimization architecture tailored to the physical distortion of each level:

1. Pre-processing Pipeline:

• Efficient 3D NIfTI loading and foreground-masked percentile normalization ( [1\%, 99\%] ) using NiBabel and NumPy.
• Standardized isotropic grid resampling to 64x64x64 cubes using SciPy.

2. Level 1 — Normalized Mutual Information (NMI):

• Instead of pixel values, we measure joint statistical entropy between modalities:

$$NMI(A, B) = \frac{H(A) + H(B)}{H(A, B)}$$

• Fused with a 3D brain silhouette Dice overlap score to achieve a flawless 1.000 MRR.

3. Level 2 — Rigid-to-Affine Registration & Self-Similarity Descriptors:

• Utilized SimpleITK with a 3-stage cascade: Center-of-Mass alignment, Euler 3D rigid registration (rotation + translation), and 12-DOF Affine registration (scaling + shearing) against a neutral brain template using Mattes Mutual Information.
• Extracted MIND (Modality-Independent Neighbourhood Descriptors) and SSC-12 edge vectors, capturing invariant patch relationships across 12 neighborhood directions.

4. Level 3 — Trimmed Matching ("Resection Cavity Filter"):

• Traditional registration fails on surgery scans because algorithms try to force healthy tissue into resection holes.
• We designed a Trimmed Distance Metric: sorting voxel-wise SSC errors and discarding the worst 25% to 50% outlier voxels (the cavity). The score evaluates only the remaining healthy anatomical cortex!

5. Optimal Bipartite Matching (Hungarian Algorithm):

• Applied scipy.optimize.linear_sum_assignment to globally resolve matching conflicts across the cohort, preventing multiple queries from claiming the same top target.

Challenges We Ran Into

• The Deep Learning Overfitting Trap: Teams leveraging 192 GB AMD MI300X GPUs with 3D InfoNCE contrastive learning saw training loss drop from 4.16 to 1.7, while test MRR crashed to ~0.42. Recognizing this early was our primary strategic turning point.
• SimpleITK Concurrency Bug: Multi-threaded Mattes-MI registration non-deterministically sampled points, flipping top-1 rankings between runs. We enforced thread-pinning (SetGlobalDefaultNumberOfThreads(1)) and deterministic seeding (seed=42) to guarantee 100% byte-identical reproducibility.
• The Planted Coordinate Data Leak: We discovered that Level 3 target headers had been pre-resampled into query coordinates, enabling a 1.0 score by simply matching CSV metadata without opening images. We isolated the leak, proved it dropped to random chance (0.205) when scrambled, and deliberately stripped headers to build an honest, anatomically validated engine.

Accomplishments That We're Proud Of

• 3rd Place Finish in the Paris Research Hackathon.
• Outperforming massive GPU deep-learning baselines with an elegant, zero-learned-parameter mathematical pipeline.
• Achieving 100% deterministic, audit-proof medical retrieval with zero data leakage.

What We Learned

• Data Efficiency > Model Complexity: In low-data regimes, domain-specific physics, information theory, and robust feature engineering beat unregularized deep neural networks every time.
• Scientific Integrity: Always audit your data headers. Catching and rejecting a synthetic leak produces algorithms that doctors can actually trust in an operating room.

Built With

• computer-vision
• data-science
• information-theory
• medical-imaging
• mri
• numpy
• optimization
• pandas
• python
• simpleitk