RadScan AI

Autonomous Radiology Triage & Multimodal Report Copilot

Comment

Inspiration

Radiologists face extreme burnout worldwide, evaluating 50+ volumetric MRI studies daily. Each study contains 24 to 40 high-resolution DICOM slices, requiring 10 to 15 minutes per case to manually inspect every anatomical plane, verify multi-slice ligament continuity, and dictate repetitive reports.

We built RadScan AI to eliminate this clinical bottleneck. By combining serverless GPU microservices on GCP Cloud Run, a 2.5D Volumetric CNN-BiGRU Neural Network (trained on 819,100 DICOMs / 530 GB data), and Vertex AI Gemini 1.5/3.5 Pro, RadScan AI acts as an autonomous co-pilot that performs 12-target pathology detection, pinpoints lesion coordinates with Grad-CAM visual heatmaps, and drafts structured DICOM reports in under 3 seconds.

What it does

  1. ⚡ 1-Click Clinical Triage: Evaluates pre-loaded 2.5D MRI datasets (ACL Tear, Meniscus Tear, Normal Knee) or custom uploaded .dcm files in sub-second time.
  2. 🧠 2.5D Volumetric Classification Engine: Processes 24 parallel slices across Sagittal, Coronal, and Axial planes using spatial CNN representations paired with depth-wise BiGRU temporal recurrent pooling.
  3. 🔥 Grad-CAM Visual Explainability: Computes gradient-weighted class activation heatmaps overlaying tear coordinates with interactive opacity controls (0% to 100%).
  4. 📊 12-Target Pathology Risk Breakdown: Simultaneously calculates multi-class probabilities across ACL tears, meniscus tears, joint effusion, bone contusions, and collateral ligament injuries.
  5. 🤖 Gemini Clinical Report Synthesis: Vertex AI Gemini 1.5/3.5 Pro autonomously digests tensor feature outputs and generates two distinct clinical views:
    • Radiologist Clinical View: Formal DICOM examination findings, impressions, and orthopaedic recommendations.
    • Patient Portal View: Plain-language, jargon-free medical summaries for patient portals.

How we built it

  • Frontend: Built with Next.js 14 (App Router), React 18, and Tailwind CSS. Features an interactive 24-slice DICOM canvas viewer, Grad-CAM opacity toggles, an interactive 6-step guided tour, and dual PACS UI modes.
  • Backend Microservice: Built with FastAPI and PyTorch, containerized via Docker, and deployed on GCP Cloud Run (with scale-to-zero L4 GPU support).
  • LLM & Agent Framework: Vertex AI Gemini 1.5/3.5 Pro via google-cloud-aiplatform and google-genai SDKs with Google ADK agent orchestration.

Mathematical Formulation

1. Depth-Wise BiGRU Temporal Slice Attention

For a 3D MRI stack (S = {s_1, s_2, \dots, s_{24}}), spatial features (f_t = \text{CNN}(s_t)) pass through a Bidirectional GRU to track inter-slice depth continuity: [ \overrightarrow{h}t = \text{GRU}{fw}(f_t, \overrightarrow{h}{t-1}), \quad \overleftarrow{h}_t = \text{GRU}{bw}(f_t, \overleftarrow{h}_{t+1}) ] [ h_t = [\overrightarrow{h}_t \mathbin{\Vert} \overleftarrow{h}_t] ]

2. Grad-CAM Activation Weighting

Explainability weights (\alpha_k^c) for target class (c) at feature map (A^k) are computed via backpropagated gradients: [ \alpha_k^c = \frac{1}{Z} \sum_{i} \sum_{j} \frac{\partial Y^c}{\partial A_{i,j}^k} ] [ L_{\text{Grad-CAM}}^c = \text{ReLU}\left( \sum_k \alpha_k^c A^k \right) ]

Challenges we ran into

  • Partial Volume 2D Artifacts: Single 2D slices often mimic tears due to volume averaging. We solved this by using a depth-wise BiGRU temporal architecture that tracks ligament signal disruption continuously across consecutive depth slices before flagging a lesion.
  • Cold Starts vs Sub-Second Latency: Keeping PyTorch model weight initializations lightweight on GCP Cloud Run while serving fast inference without racking up 24/7 idle GPU server costs.
  • Strict Gemini JSON Formatting: Ensuring Vertex AI Gemini 1.5/3.5 Pro reliably outputs structured clinical report keys (findings, impression, recommendations) using native JSON schema enforcement (response_mime_type="application/json").

Accomplishments that we're proud of

  • Achieving strong multi-planar validation scores (0.802 LB / 0.809 Ensemble Blend) across all 12 knee joint pathology targets.
  • Sub-3-second end-to-end processing pipeline from raw DICOM tensor extraction to Vertex AI report generation.
  • Zero-cost serverless deployment architecture on GCP Cloud Run that scales down to 0 instances when idle ($0/month idle spend).

What we learned

  • How to package PyTorch deep learning models into lightweight CPU/GPU containers on GCP Cloud Run.
  • How to enforce deterministic JSON schemas and system prompts using Google's new google-genai SDK and Vertex AI platform.
  • The critical importance of visual explainability (Grad-CAM) in building clinician trust for medical AI tools.

What's next for RadScan AI

  • Expanding the 2.5D volumetric pipeline to shoulder, spine, and brain MRI modalities.
  • Integrating FHIR/HL7 EHR data export for seamless hospital PACS integration.
  • Developing multilingual voice-first dictation for hands-free clinical consultation in regional health centers.

Built With

Share this project:

Updates

Submission history