NeuraLens

Ctrl+Z for brain surgery, one tumor at a time.

Comment

Inspiration

Every year, 300,000 people are diagnosed with brain tumors worldwide. Before a surgeon can operate, they need to understand exactly where the tumor is, how big it is, and where it meets critical brain structures. Today, that process relies on manual annotation using expensive, specialized software — a workflow that takes hours per patient and is simply inaccessible to most hospitals, especially in developing regions.

We asked ourselves: why can't a surgeon just drag four MRI files into a browser and get a 3D game plan in under a minute? That question became NeuraLens.

What it does

NeuraLens is a browser-based tool that automates brain tumor segmentation and visualization from multi-modal MRI scans. A clinician uploads four standard MRI files (T1, T1 contrast-enhanced, T2, and FLAIR), and the system automatically:

  • Segments the tumor into three clinically meaningful sub-regions: the enhancing tumor (actively growing edge), necrotic core (dead tissue), and peritumoral edema (surrounding swelling)
  • Displays interactive 2D slices showing the brain from three angles (top-down, front-on, and side) with the tumor regions highlighted as a colored overlay
  • Renders an interactive 3D model of the tumor surfaces inside a semi-transparent brain, allowing surgeons to rotate, zoom, and toggle individual regions
  • Computes clinical measurements including tumor volumes in cubic centimeters and the estimated anatomical location of the tumor center

The entire pipeline runs automatically — no manual annotation, no specialized software, no barriers.

How we built it

Backend (Python/FastAPI):

  • Built a four-stage ML pipeline: validation, preprocessing, inference, and post-processing
  • Used MONAI transforms for medical image preprocessing — reorienting scans to a standard orientation (RAS+), resampling to 1mm resolution, and Z-score intensity normalization
  • Integrated the Swin UNETR model (a 3D vision transformer paired with a U-Net decoder) for tumor segmentation, using sliding window inference with 128x128x128 patches to handle full brain volumes
  • Extracted 3D tumor surface meshes using the marching cubes algorithm from scikit-image, exported as OBJ files for the browser
  • Generated a semi-transparent brain surface mesh from the MRI data to provide anatomical context in the 3D viewer
  • Computed tumor volumes by counting labeled voxels and estimated tumor location by transforming the centroid to world coordinates

Frontend (Next.js/React):

  • Built a drag-and-drop upload interface that auto-identifies MRI modalities from filenames
  • Integrated Niivue, a WebGL-based medical image viewer, for the 2D multiplanar slice display with crosshair-linked navigation
  • Built an interactive 3D viewer using Three.js with OBJ mesh loading, OrbitControls for rotation/zoom, and per-region visibility and opacity controls
  • Designed a clinical stats dashboard showing tumor volumes, sub-region breakdown, and estimated anatomical location

Infrastructure:

  • Ran GPU inference on Google Colab (T4) using a pre-trained Swin UNETR checkpoint from MONAI Research Contributions, trained on the BraTS 2021 dataset (1,251 annotated cases)
  • Built a demo mode that uses pre-computed results for live presentations without requiring a local GPU

Challenges we ran into

  • Model weights confusion: We initially downloaded the pre-trained Swin ViT encoder weights instead of the fully fine-tuned segmentation checkpoint. The model loaded without errors (thanks to strict=False) but produced garbage segmentation — labeling 90% of the brain as tumor. Tracking down the correct fine-tuned checkpoint from the MONAI Model Zoo and understanding the difference between encoder pre-training and full model fine-tuning was a key learning moment.

  • GPU memory constraints: The T4 GPU on Colab (16GB) couldn't handle sliding window inference with a batch size of 4. We had to reduce the batch size to 1 and lower the overlap ratio to fit within memory.

  • 3D mesh alignment: The tumor meshes and brain surface initially appeared misaligned in the 3D viewer because they loaded asynchronously and each was centered on its own bounding box. We fixed this by using a fixed center offset based on the known BraTS volume dimensions, ensuring all meshes align to the same coordinate system regardless of loading order.

  • File format compatibility: The BraTS 2021 dataset uses different file naming conventions (_t1.nii.gz, _flair.nii.gz) than our pipeline expected (t1n.nii.gz, t2f.nii.gz). We had to build a flexible modality matching system that handles multiple naming patterns. The browser file picker also doesn't recognize .nii.gz as a valid extension, requiring a workaround in the upload component.

  • NumPy/MONAI version conflicts: Colab's pre-installed NumPy 2.0 broke MONAI's spatial transforms, which still used the deprecated arr.ptp() method. Pinning the right combination of package versions to work together on Colab required several iterations.

Accomplishments we're proud of

  • End-to-end in the browser: A clinician can go from raw MRI files to an interactive 3D tumor visualization without installing anything — just a browser.
  • Real clinical data: We moved beyond synthetic mock data to run the full pipeline on real BraTS brain tumor cases, producing anatomically accurate segmentation and realistic 3D renderings.
  • Brain surface context: The semi-transparent brain mesh in the 3D viewer gives surgeons spatial context — they can see exactly where the tumor sits within the brain anatomy, not just floating colored blobs.
  • Crosshair-linked navigation: Clicking in any 2D slice view updates all three views simultaneously, enabling intuitive exploration of the 3D brain volume from any angle.
  • Validated model: Using a pre-trained Swin UNETR checkpoint with a published Dice score of 90.59%, validated through 5-fold cross-validation on 1,251 cases, gives us confidence in the segmentation quality.

What we learned

  • Medical imaging has its own ecosystem: Tools like MONAI, Niivue, and nibabel exist because medical data has unique requirements — specific orientations (RAS+), coordinate systems (MNI space), and file formats (NIfTI, DICOM) that general-purpose ML frameworks don't handle out of the box.
  • Pre-trained doesn't mean plug-and-play: Understanding the difference between encoder pre-training weights and fully fine-tuned model checkpoints is critical. A model can load and run inference without errors while producing completely meaningless outputs.
  • 3D visualization is hard: Aligning multiple meshes in a shared coordinate system, handling asynchronous loading, and rendering transparent surfaces with correct depth ordering all required careful engineering.
  • The gap between research and deployment is real: A model with a 90% Dice score in a Colab notebook is very different from a tool a surgeon can use. File format handling, upload UX, error states, and performance optimization all add significant complexity.

What's next for NeuraLens

  • DICOM support: Real hospitals use DICOM format, not NIfTI. Adding a DICOM-to-NIfTI conversion layer would make NeuraLens deployable in actual clinical workflows.
  • Multi-model ensemble: Combining Swin UNETR with nnU-Net predictions and averaging them could improve segmentation accuracy beyond what either model achieves alone.
  • Longitudinal tracking: Allowing clinicians to upload scans from multiple timepoints and visualize tumor growth or shrinkage over time would add significant clinical value for treatment monitoring.
  • Federated learning: Training on hospital data without it leaving the institution would address privacy concerns and enable the model to improve on diverse, real-world data.
  • FDA 510(k) pathway: With multi-institutional validation and DICOM integration, NeuraLens could pursue clearance as a clinical decision support tool, making it available for real patient care.

Built With

Share this project:

Updates