Inspiration
DoseNet++ grew out of interest in computational medicine and biomechanical modeling, where we’ve been exploring how data-driven systems can simulate complex biological and physical processes. Seeing how neural networks can approximate intricate dynamics inspired us to apply similar deep learning approaches to radiotherapy planning — a domain where precision and speed are both critical.
We wanted to investigate whether a model could learn to predict 3D dose distributions directly from patient anatomy, helping move treatment planning toward more automated and efficient workflows.
What it does
Given a patient’s CT scan and organ masks, DoseNet runs a residual 3D U-Net to predict the full voxel-wise radiation dose distribution across the anatomy. The system then generates interactive slice visualizations and DVH metrics, allowing users to analyze target coverage and organ-at-risk sparing directly within the browser.
How we built it
Data Preparation
We used the OpenKBP head-and-neck radiotherapy dataset, which provides CT scans, organ-at-risk (OAR) masks, tumor target volumes (PTVs), and clinically optimized ground-truth dose distributions.
Each patient’s data is stored as sparse CSV files containing only non-zero voxel indices. We reconstructed these into dense 128 × 128 × 128 3D volumes for CT, dose, and structure masks.
CT values were clipped to a fixed range and normalized to [0,1]. Organ masks were converted into binary 3D tensors, and all inputs were stacked into an 11-channel volume (1 CT + 10 structure masks). The possible dose mask was used to restrict predictions to clinically reachable regions.
Machine Learning Model
Our core model is a residual 3D U-Net implemented in PyTorch. The network takes the 11-channel volumetric input and predicts a single-channel 3D radiation dose distribution in Gy.
The encoder downsamples spatial resolution while increasing feature depth, and the decoder upsamples using skip connections to preserve anatomical detail. Residual blocks with GroupNorm and SiLU activations stabilize training on full 3D volumes.
We trained the model using a composite clinical loss function combining masked L1 dose error, tumor underdose penalties relative to prescription levels (56/63/70 Gy), and organ overdose penalties above threshold. Optimization used AdamW, mixed-precision training (AMP), learning rate scheduling, gradient clipping, and EMA weight averaging for stability.
Validation performance was evaluated using OpenKBP dose score and DVH score metrics.
Visualization & System Design
We built a full-stack system to make predictions interpretable beyond offline training. A FastAPI backend loads the trained checkpoint and accepts uploaded patient folders. After reconstructing sparse inputs, the model performs inference and returns predicted dose as a sparse CSV file.
On the frontend, we implemented a React-based multiplanar reconstruction (MPR) viewer using Canvas 2D rendering. The viewer displays synchronized axial, coronal, and sagittal slices with dose heatmap overlays and smooth isodose contour lines.
We also compute Dose-Volume Histograms (DVH) directly in the browser using a Web Worker, enabling interactive evaluation of tumor coverage and organ sparing. A plan quality scorecard automatically checks key clinical constraints such as D95 for targets and maximum dose limits for OARs.
This results in an end-to-end AI-assisted radiotherapy dose planning pipeline — from raw CT data to interpretable 3D visualization and quantitative plan evaluation.
Challenges we ran into
Training a full 3D U-Net on 128³ volumes pushed GPU memory limits quickly. We couldn’t use large batch sizes, so we switched to batch size = 1, enabled mixed-precision (AMP), added gradient clipping, and used EMA weights to stabilize training.
Reconstructing dense 3D volumes from sparse CSV files was also tricky. A small indexing mistake could completely misalign CT, masks, and dose. We built consistent sparse-to-dense utilities and validated shapes and voxel counts at every step to prevent silent errors.
On the visualization side, raw CT slices looked washed out and noisy, and dose overlays often hid anatomical detail. We implemented proper window/level controls, smoother interpolation, and tuned opacity and color mapping so both anatomy and dose patterns remain readable.
Accomplishments that we're proud of
Building a complete end-to-end system — from raw sparse clinical data to a trained 3D neural network to an interactive visualization tool — in a short time frame.
Successfully implementing a real-time multiplanar (axial, coronal, sagittal) medical viewer with dose overlays, contour lines, and DVH analysis directly in the browser without heavy 3D engines.
Designing a clinically inspired loss function that goes beyond simple regression and captures tumor coverage and organ sparing objectives.
Turning a complex radiotherapy planning problem into a usable AI-assisted workflow that produces interpretable 3D dose predictions and quantitative plan quality evaluation.
What we learned
Deep learning can approximate highly structured clinical optimization problems, but only when the objective reflects real treatment priorities — pure voxel-wise accuracy is not enough.
Radiotherapy dose prediction is fundamentally spatial and constraint-driven. Geometry, prescription levels, and organ limits matter as much as raw image data.
We also learned that model performance alone isn’t sufficient. Interpretability and visualization are critical. Clear CT windowing, readable dose overlays, and DVH analysis are what make the system practically usable, not just technically functional.
What's next for DoseNet
We want to introduce controllable planning modes that allow users to adjust clinical priorities — such as emphasizing tumor coverage or organ sparing — and dynamically observe how dose distributions change. We also want to implement better visualization tools specific to this problem.
Log in or sign up for Devpost to join the conversation.