Inspiration
My dad recently had to undergo an urgent heart procedure because of serious coronary blockages. It was a complicated case, and my family struggled to find a specialist who could perform it. That experience got me interested in the practical challenges of surgery and how engineering could improve the tools physicians use.
While researching surgical workflows, I came across an interesting problem: once surgeons scrub in, they cannot interact with a normal keyboard or mouse without breaking sterility. Instead, they often have to ask someone else to navigate their medical images for them. Existing touchless viewers address part of this problem, but most only replace mouse movements with hand gestures. Surgeons still have to manually locate structures, take measurements, and interpret what they are seeing.
We wanted to go beyond making medical imaging touchless. What if a surgeon could interact with a CT scan naturally, ask questions about the anatomy, and explore different surgical plans without ever touching a computer?
That became Scrub+, a gesture-controlled, AI-powered surgical planning interface that runs in a browser.
What it does
Scrub+ lets users manipulate real 3D CT scans using a regular laptop webcam, voice commands, or natural language.
Users can rotate scans with a pinch gesture, move a virtual cutting plane with an open palm, measure distances with two fingers, and lock a proposed cut with a fist. As the cutting plane moves, Scrub+ recalculates measurements including remaining liver volume, tumor clearance, and intersected blood vessels.
We also built an AI agent that can understand requests such as "optimize the cut with a 10 mm margin" or "size a stent for this aorta." Rather than generating medical measurements itself, the agent calls tools that perform calculations directly on the segmented CT data.
Users can upload their own CT scans, automatically segment anatomical structures using NVIDIA VISTA-3D, and even click a location in the scan to ask what structure they are looking at.
The goal is to make complex medical imaging more interactive and accessible while keeping quantitative calculations grounded in the actual anatomy.
How we built it
We started by building the 3D imaging interface using TypeScript, Vite, three.js, and WebGL/GLSL. We processed CT scans from the Medical Segmentation Decathlon using Python and anatomical segmentation tools, then built a browser-based volume renderer that displays the original scan alongside segmented anatomical structures.
For gesture recognition, we used MediaPipe to track 21 landmarks on each hand. We implemented gesture classification, smoothing, and state transitions to map different hand movements to controls for rotation, zooming, measurement, and surgical planning.
The next challenge was making the interface do something more useful than simply navigate images. We built a deterministic geometry engine that calculates distances, organ volumes, tumor margins, and vessel intersections directly from CT voxels. We also implemented a search algorithm that evaluates alternative cutting planes to identify configurations satisfying geometric constraints.
We then expanded Scrub+ into an agentic system built around NVIDIA's open models and Nebius AI Cloud.
Our agent uses Nemotron to translate natural-language requests into structured tool calls. We created 13 tools covering imaging navigation, measurements, optimization, stent sizing, and planning. The model decides which tools to call, but the geometry engine performs the actual calculations.
To make the agent faster and less expensive, we built our own model distillation pipeline. We generated 6,000 surgeon-style requests, used Nemotron 3 Super as a teacher to produce tool-calling trajectories, and verified the trajectories against the geometry engine. We retained 4,133 verified traces and fine-tuned Nemotron Nano 9B v2 using LoRA on Nebius H100 GPUs.
On 256 held-out commands, fine-tuning increased task success from 46.1% to 75.8%, with a median latency of 0.8 seconds for the fine-tuned model.
For anatomical segmentation, we deployed NVIDIA VISTA-3D through Nebius Serverless Jobs and Endpoints. This lets users upload CT scans for automatic segmentation or interactively identify anatomical structures using point prompts. Across eight liver scans, the model achieved a median liver Dice score of 0.967, with median segmentation inference time of 5.2 seconds.
We also integrated Tavily to retrieve device instructions and specifications for aortic stent planning. The system checks relevant geometric measurements against extracted device constraints and links back to the original sources.
Throughout the system, we made one architectural decision especially important: the language model is responsible for choosing actions, not inventing measurements. Every numerical value in an agent response must be grounded in actual tool outputs.
Challenges we faced
One of our earliest challenges was making hand gestures feel natural. Our first implementation used palm orientation to switch between different controls, but it was difficult to maintain consistent gestures. We redesigned the interaction system around more distinct movements, such as pinching to rotate and opening the palm to manipulate the cutting plane. We also had to account for overlapping fingers, noisy landmark tracking, and the limited depth information available from an ordinary webcam.
Rendering large CT volumes while continuously updating segmentation overlays and planning measurements was another challenge. We had to carefully preprocess the imaging data and optimize the rendering pipeline to keep the interface responsive.
The hardest technical challenge in the expanded system was making the AI agent both reliable and fast. Generating tool calls was relatively straightforward, but ensuring that those calls actually satisfied a user's request was much harder. We built a separate Python implementation of our geometry tools, added parity tests against the TypeScript engine, and used machine-checkable goal conditions to evaluate agent behavior.
We also discovered that a smaller model with the right training data could outperform its untuned baseline by a substantial margin. Building that training pipeline required generating realistic requests, executing thousands of tool-calling traces, filtering incorrect outputs, and getting our fine-tuned model running on our own inference endpoint.
Integrating cloud infrastructure introduced additional challenges, especially GPU provisioning delays, model deployment compatibility, and keeping the interactive portions of the application responsive.
Accomplishments we are proud of
We originally built Scrub+ at HackNC 2026, where it won first place overall, and continued developing it into a more technically complete system.
One of our proudest accomplishments is that Scrub+ does not depend on an LLM to perform its core measurements. The geometry engine calculates actual values from segmented CT data, and the AI agent provides a natural-language interface to those calculations.
We are also proud that we built and evaluated our own fine-tuned Nemotron agent rather than relying entirely on a general-purpose model. Improving held-out task success from 46.1% to 75.8% demonstrated how much difference task-specific, verified training data could make.
Finally, we integrated 3D rendering, real-time computer vision, medical image segmentation, deterministic geometric computation, and agentic AI into a working browser application.
What we learned
The biggest thing we learned is that building an impressive AI demo and building a reliable system are two very different problems.
At first, we focused on making CT scans respond naturally to hand movements. But as we developed the project, we realized that reducing the number of interactions required to find clinically relevant information was much more valuable than gesture recognition alone.
We also learned that AI works best in this application when it acts as an interface to trustworthy computational tools rather than trying to reason out medical measurements on its own. Distilling a smaller model from verified tool-use examples reinforced the importance of designing good evaluation systems, not just choosing powerful models.
Working on Scrub+ gave us experience across computer vision, 3D graphics, medical imaging, model fine-tuning, inference infrastructure, and full-stack engineering. More importantly, it taught us how difficult it is to connect all these components into a coherent system that actually solves a problem.
What's next
We want to make Scrub+ more capable of supporting realistic surgical workflows, including more sophisticated resection geometry, additional anatomical structures, and better interactions for exploring complex procedures.
We also want to evaluate the interface with actual medical professionals to understand which features are useful, which interactions are unintuitive, and what would need to change before a system like this could be considered for clinical environments.
Scrub+ is currently a research prototype, not a clinically validated medical device. But building it has convinced us that combining natural interfaces with grounded computational agents could fundamentally change how people interact with complex 3D medical data.
Log in or sign up for Devpost to join the conversation.