The future of oncology

Onco

Inspiration

Oncologists manage ~1.9M new cancer cases a year in the U.S., and every decision spans radiology, pathology, and genomics. But those signals live in silos: PACS viewers, PDF reports, and variant spreadsheets. Clinicians spend time tab-switching and mentally stitching context that should be unified.

Software engineers have Cursor, an automation layer that understands the whole codebase and helps teams ship faster. We wanted to ask: what if oncology had the same thing, a workflow where patient biology is the codebase, and AI copilots help clinicians triage, explain, and track tumors at the point of care?

What it does

Onco is a Cursor-inspired oncology agentic IDE that unifies patient biological data into a single intelligent workspace where context-triggered AI copilots assist clinical reasoning.

Input: CT DICOM series, pathology reports, genomic variant panels

Output: AI-powered lesion segmentation with RECIST assessment, morphology-to-driver gene mapping, CIViC evidence lookups with therapy recommendations and clinical trial matching

The clinician selects context, Cmd+Click a lesion, highlight pathology text, click a variant row, and the appropriate copilot activates instantly. Outputs persist to a shared patient state, enabling cross-modality handoffs similar to a tumor board discussion.

ML and Backend (FastAPI + Python)

MedSAM2 Integration: Deployed promptable medical image segmentation (MedSAM2) on Modal for fast 2D lesion segmentation from CT slices. Users Cmd+Click on a lesion, and the model returns contour points and measurements.

RECIST-Style Assessment Engine: Implemented RECIST 1.1-compliant response evaluation logic, computing longest diameter, percent change from baseline, and categorizing response (CR/PR/SD/PD) with clinical thresholds.

Radiomics Feature Extraction: Built intensity statistics (mean/std HU, skewness, kurtosis), texture analysis (entropy, contrast, homogeneity), and shape metrics (sphericity, elongation) for tumor characterization.

Tumor Classification Pipeline: Combined radiomics features with a heuristic classifier to assess malignancy probability, distinguish tumor types (adenocarcinoma vs benign nodules), and generate clinical narratives.

CIViC Evidence Integration: Live-first GraphQL queries to CIViC (Clinical Interpretation of Variants in Cancer) with intelligent caching fallback, returning actionability assessment, evidence levels, and therapy recommendations.

Clinical Trial Matching: Gene/variant-aware trial matching from curated ClinicalTrials.gov data, returning recruiting studies with phase, status, and intervention details.

Pathology-to-Driver Mapping: Rule-based extraction of diagnoses from pathology text with morphology-to-likely-driver gene mapping (e.g., lung adenocarcinoma -> EGFR, ALK, KRAS G12C).

Live-First Reliability Architecture: Custom reliability manager enforcing time budgets with automatic fallback to cached artifacts, ensuring demo-safe execution while attempting inference first.

SSE Terminal Streaming: Server-Sent Events (SSE) streaming copilot logs to the frontend terminal, providing step-by-step visibility into AI reasoning with explicit fallback notifications.

Frontend (Next.js + Tailwind)

IDE-Style Three-Pane Layout: Dark-mode, terminal-aesthetic workspace with Left (Patient Repository), Center (Active Workspace), and Right (Agent Terminal) panes, mirroring developer tool ergonomics.

DICOM Viewer with Cornerstone.js: DICOM CT viewing with slice navigation, window/level controls, and multi-planar reconstruction (MPR) views showing sagittal and coronal cross-sections.

Interactive Segmentation Overlay: Visualization of AI-generated contour masks overlaid on CT slices, with click-position markers and processing state indicators.

Pathology Dual-View Interface: Split-pane WSI (Whole Slide Image) viewer with OpenSeadragon-style zoom/pan and synchronized pathology report with LLM-powered text explanation capability.

Genomics Variant Table with 3D Structures: Interactive NGS panel display with actionability badges, VAF percentages, classification status, and embedded NGL-powered 3D protein structure visualization showing mutation sites and bound drugs.

Context-Selection Triggers: Cmd+Click lesion selection (radiology), text highlighting (pathology), and row click selection (genomics) each dispatch to their specialized copilot with terminal log streaming.

Session State Management: React hooks managing session lifecycle, patient state synchronization, SSE connections, and copilot orchestration, enabling seamless cross-modality handoffs.

End-to-End Pipeline: Unified workflow from DICOM upload -> AI segmentation -> RECIST assessment -> pathology analysis -> genomics evidence lookup, all accessible in a single integrated interface.

Challenges we ran into

Finding the right balance between technical depth and demo reliability was crucial. Live inference can fail, so we designed a "live-first with cached fallback" architecture that attempts inference but degrades gracefully. Integrating Cornerstone.js for proper DICOM viewing required careful handling of image IDs, viewport management, and coordinate transformations between display and pixel space. We also navigated the complexity of making cross-modality handoffs feel natural, ensuring outputs from one copilot inform the next.

Accomplishments that we're proud of

Medical imaging AI: Integrated MedSAM2 for promptable lesion segmentation that works on CT data with coordinate scaling and DICOM pixel spacing for accurate measurements.

Clinical response assessment: Implemented RECIST 1.1 logic including the 5 mm absolute increase threshold for progressive disease, aligned with clinical trial criteria.

Evidence integration: Live CIViC API queries returning variant-specific actionability, evidence levels, and therapy recommendations backed by curated evidence.

3D protein structure visualization: NGL-powered molecular viewer showing mutation sites, drug binding pockets, and ligand interactions in the genomics workflow.

Radiomics-based tumor characterization: Extracted quantitative imaging features (intensity, texture, shape) for malignancy assessment with interpretable narratives.

Seamless pipeline integration: Built an end-to-end workflow where radiology findings inform pathology review, which suggests genomic testing, with context persisted in shared patient state.

What we learned

We deepened our understanding of medical imaging standards (DICOM, pixel spacing, window/level), clinical response criteria (RECIST), and variant interpretation workflows in precision oncology. Building for reliability taught us that graceful degradation is a feature. The live-first, cached fallback pattern ensures the system always delivers value while still attempting inference.

What's next for Onco

Expand modality support: Full 3D volumetric segmentation, longitudinal lesion tracking with deformable registration, and WSI-based computational pathology (CLAM, attention-based MIL)
Broader knowledge integration: OncoKB, ClinVar, and VEP annotation pipelines beyond CIViC
LLM-powered reasoning: Deeper integration of Gemini/Claude for synthesizing multimodality findings into tumor board summaries
Production deployment: HIPAA-compliant infrastructure with EMR integration via HL7 FHIR
Collaborative features: Multi-user tumor board mode with real-time annotation sharing