NIfTI 4D Visualizer with frame insertion

A better result while running for a longer time.

Inspiration

Medical imaging procedures such as CT and MRI are often complex, time-consuming, and burdensome for patients. In particular, cardiac MRI requires multiple scans to capture the entire motion of the heart across time. This not only prolongs the exam but, in the case of CT, also increases radiation exposure. Our inspiration was to design a tool that can enhance existing MRI data without requiring extra scans. By intelligently inserting additional frames, we aim to reduce scan time, minimize patient discomfort and exposure, and provide physicians with a clearer view of cardiac dynamics to accelerate diagnosis.

What it does

Our project, NIfTI 4D Visualizer with Frame Insertion, reads 4D cardiac MRI data (3D volumes evolving over time) and uses a machine learning model to insert intermediate frames. This effectively increases the temporal resolution of the heart sequence, producing smoother and more continuous cardiac motion. Physicians can visualize the result as 2D animated slices or interact with the reconstructed 4D volume. We also developed a Streamlit web app that allows users to upload new 4D NIfTI data and immediately view enhanced frame animations online.

How we built it

We trained our model on the ACDC dataset (Automated Cardiac Diagnosis Challenge), which contains short-axis cardiac MRI volumes with diagnostic labels. Each case provides 4D data (X, Y, Z spatial dimensions plus time), typically covering about 30 frames for a full cardiac cycle.

To enhance this data, we leveraged:

Fourier Feature Embeddings: We mapped spatiotemporal coordinates into a higher-dimensional Fourier basis to capture high-frequency details of heart motion, enabling the model to represent subtle, complex dynamics.
MLP Networks (Multi-Layer Perceptrons): We trained a deep MLP to learn a continuous function of cardiac motion in 4D space-time. Combined with Fourier embeddings, this allowed the MLP to model nonlinear variations in heart dynamics.
NeRF-inspired Sparse Sampling: Borrowing from Neural Radiance Fields, we trained the network using sparse sampled points rather than full voxel grids, drastically reducing computation while reconstructing full 4D sequences from limited data.

We extended 30-frame cardiac sequences to 35 frames by inserting 5 additional frames, generating smoother temporal resolution. Training was accelerated using Automatic Mixed Precision (AMP) to balance speed and memory. Finally, we deployed results in Streamlit, making it possible for users to upload their own scans and instantly view 2D animations of enhanced cardiac motion.

Challenges we ran into

Handling 4D NIfTI data efficiently, including sparse sampling and coordinate encoding.
High computational cost of implicit neural representations (INR) on large volumetric data.
Balancing smooth interpolation with fidelity to original frames.
Optimizing visualization of large medical volumes in a lightweight web interface.

Accomplishments that we're proud of

Successfully implemented Fourier embeddings + MLP + NeRF-style sparse sampling for cardiac MRI frame insertion.
Produced extended sequences with smooth, natural cardiac motion.
Built an interactive Streamlit app for easy upload and visualization of results.
Demonstrated how cutting-edge neural representation learning can help reduce scanning requirements while maintaining clinically useful image quality.

What we learned

Fourier feature embeddings significantly improve the ability to model high-frequency temporal changes.
NeRF-inspired sparse sampling is powerful for medical image reconstruction, reducing compute while preserving detail.
Implicit neural representations with MLPs can successfully model 4D continuous functions of cardiac motion.
Practical experience working with the ACDC dataset and handling 4D medical imaging pipelines.

What's next for NIfTI 4D Visualizer with Frame Insertion

Extend the model to more clinical datasets for better generalization.
Optimize inference to allow real-time frame insertion in clinical workflows.
Integrate cardiac segmentation to focus interpolation on specific regions such as the left ventricle.
Collaborate with clinicians to validate in real medical environments, with the long-term goal of reducing MRI scan time while improving diagnostic speed and accuracy.

Citation

O. Bernard, A. Lalande, C. Zotti, F. Cervenansky, et al. "Deep Learning Techniques for Automatic MRI Cardiac Multi-structures Segmentation and Diagnosis: Is the Problem Solved ?" in IEEE Transactions on Medical Imaging, vol. 37, no. 11, pp. 2514-2525, Nov. 2018 doi: 10.1109/TMI.2018.2837502

Shen, Liyue, John Pauly, and Lei Xing. "NeRP: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction." IEEE Transactions on Neural Networks and Learning Systems 35, no. 1 (2022): 770-782.