3D cross section of CT, prepped for optimization
website

Scrub+

Rethinking how surgeons interact with medical imaging, without touching a computer.

Inspiration

My dad recently had to undergo an urgent heart procedure because of serious coronary blockages. It was a complicated case, and my family had a hard time finding a specialist who could perform it.

Through that experience, I became interested in the practical challenges of surgical workflows and how engineering could help address them.

One problem that caught my attention was how surgeons interact with medical imaging during operations. Once surgeons are scrubbed in, they cannot use a regular mouse or keyboard without compromising sterility. Instead, they may need to ask someone outside the sterile field to navigate CT scans for them.

Finding a specific anatomical structure, changing the viewing angle, or taking a measurement can require several rounds of verbal instructions. In a 2016 study, completing an image-navigation task by directing an assistant took an average of 246 seconds, compared with 215 seconds using gestures.

Touchless medical imaging systems already exist, but many focus primarily on basic navigation, such as scrolling, zooming, and rotating. Surgeons still have to manually search through scans to find specific information, and some systems require specialized hardware.

I wanted to build something that went beyond replacing the mouse with hand gestures.

What it does

Scrub+ is a touchless, spatial computing interface that lets users explore real 3D CT scans through hand gestures and voice commands using a regular laptop webcam.

Users can rotate medical images, move cutting planes through anatomical structures, measure distances, locate specific anatomy, and compare hypothetical surgical plans.

The main feature is an anatomy-aware voice copilot. Instead of manually navigating through a scan, users can ask Scrub+ to find the information they need.

For example:

"Scrub, show me the tumor relative to the portal vein." Highlights both structures, calculates their closest distance, and moves the camera to show their relationship.
"Scrub, show me where the tumor is closest to a major vessel." Searches the available vessel segmentations, identifies the nearest vessel, and displays the measurement alongside a CT cross-section.
"Scrub, focus on the left kidney." Automatically locates and frames the kidney.
"Scrub, save this plan." Saves the current cutting plane and camera view.
"Scrub, compare this cut to the original plan." Compares tumor margin, remaining liver volume, and vessel intersections between two hypothetical resection plans.
"Scrub, show me the original plan." Restores the previously saved configuration.

Scrub+ also supports gesture-controlled surgical planning. Users can move a cutting plane through a liver CT scan and see measurements update as the plane changes.

A geometric optimization algorithm searches 3,751 nearby cutting-plane candidates to identify alternatives based on selected planning constraints.

The application also includes a virtual aortic stent-graft demonstration with simplified sizing, positioning, and flow calculations.

All anatomical measurements are calculated from the CT data and segmentation masks rather than generated by a language model.

How I built it

Medical imaging and preprocessing

I used a real liver CT scan from the Medical Segmentation Decathlon and processed the imaging data using Python. I also used TotalSegmentator to generate additional anatomical labels.

The processed CT volume and segmentation masks are included with the application, allowing users to explore the demo without running the preprocessing pipeline themselves.

Users can also import their own NIfTI CT scans directly into the browser. Imported scans can be viewed and measured, while compatible segmentation masks enable the anatomy-aware planning features.

3D volume rendering

I built a custom volume renderer using three.js, WebGL, and GLSL shaders.

Instead of displaying a static anatomical model, Scrub+ renders the underlying CT volume together with its segmentation labels. This allows individual structures, such as tumors and blood vessels, to be highlighted within the surrounding anatomy.

A movable cutting plane exposes cross-sections through the CT volume and supports interactive exploration of hypothetical resections.

Hand tracking and gesture recognition

I integrated MediaPipe to track 21 landmarks on each hand using a regular webcam.

Scrub+ supports several gestures:

Open palm to move and tilt a cutting plane.
Pinch and drag to rotate the 3D scan.
Two fingers to place distance measurements.
Fist to lock the cutting plane.
Two-handed pinch gestures to zoom.

To make the controls more stable, I implemented landmark smoothing and a gesture state machine to reduce jitter and unintended transitions between gestures.

I also experimented with a small neural network trained on synthetic hand-landmark data to improve gesture recognition when fingers overlap.

Because a regular webcam cannot directly measure depth, the application estimates relative hand depth from apparent hand size.

Geometry and surgical planning

I implemented the measurement and planning calculations using the underlying voxel data and segmentation masks.

These include:

Liver remnant volume, calculated from segmented liver voxels on either side of the proposed cut.
Tumor margins, calculated using distance transforms.
Vessel intersections, determined from the segmented vascular structures.
Anatomical distances, calculated directly from the 3D segmentation geometry.
A geometric optimizer that evaluates 3,751 nearby cutting-plane candidates.

The planning measurements update as the user manipulates the cutting plane.

Voice copilot

The voice interface uses the browser's Web Speech API and the wake word "Scrub."

Common requests are handled by a predefined command parser. An optional language model interprets less structured requests and maps them to supported application actions.

The language model does not calculate distances, volumes, or surgical margins. Those calculations are performed by the geometry engine.

The core viewer, gesture controls, measurements, and optimizer work independently of an LLM.

Challenges I ran into

Making gestures intuitive

My first implementation used palm orientation to switch between rotating the model and manipulating the cutting plane.

Although the gestures were technically recognizable, they were awkward to use consistently. I eventually changed the controls to use an open palm for cutting-plane manipulation and a pinch for rotation.

This made the interactions more natural and reduced accidental mode changes.

Handling unreliable hand landmarks

MediaPipe can produce unstable landmark predictions when fingers overlap or become partially obscured.

I added filtering and gesture-state logic to improve stability. I also experimented with a lightweight neural network for gesture recognition under occlusion.

The neural network showed improvements on synthetic hand data, although real-world validation is still needed.

Estimating depth without specialized hardware

I wanted Scrub+ to work with an ordinary laptop webcam rather than requiring a dedicated depth camera.

To approximate hand movement toward and away from the camera, I used changes in the hand's apparent size.

This provides useful relative movement for interaction, but it is not equivalent to precise 3D hand tracking.

Rendering and processing large CT volumes

Medical imaging data can be computationally expensive to process and render interactively.

Combining volume rendering, segmentation overlays, hand tracking, and continuously updating geometric measurements required preprocessing and performance optimization to keep the interface responsive.

Accomplishments that I'm proud of

Built an interactive 3D medical imaging viewer that renders real CT data directly in the browser.
Implemented touchless gesture controls using a regular laptop webcam.
Created an anatomy-aware voice copilot that can locate structures and calculate measurements.
Built planning measurements that respond dynamically to changes in the cutting plane.
Implemented a geometric optimizer that evaluates thousands of candidate cuts.
Verified measurement calculations against independent NumPy implementations.
Built the main functionality without requiring an LLM.

In one test configuration, the geometric optimizer increased the measured tumor margin from 6.3 mm to 10.2 mm and reduced intersected segmented vessel components from two to zero.

This demonstrates an improvement in the selected geometric metrics, not proof that the alternative cut would produce a better clinical outcome.

What I learned

One of my biggest takeaways was that making an interaction technically possible doesn't necessarily make it useful.

My initial goal was to control medical images with hand gestures. But I realized that simply replacing mouse movements with gestures doesn't eliminate the work involved in finding a particular anatomical structure or measurement.

That led me to focus more on anatomy-aware navigation and direct information retrieval.

I also learned that the most useful role for an LLM in this application isn't making medical decisions. It's interpreting what the user wants and invoking the appropriate deterministic tools.

Finally, working with real CT data made me appreciate how much complexity exists between a visually convincing medical imaging prototype and a clinically usable system.

What's next for Scrub+

Scrub+ is currently a research and educational prototype, not a clinically validated surgical planning system.

My next steps are to test the interface with surgeons and clinical trainees, evaluate gesture reliability under realistic operating-room conditions, and validate the anatomical measurements and planning algorithms more extensively.

I also want to expand the planning capabilities with more detailed vascular anatomy, liver segment mapping, and more realistic resection geometries.

Ultimately, my goal is to make medical imaging more accessible during complex procedures and reduce unnecessary friction in surgical workflows.

With further development and clinical validation, systems like Scrub+ could help clinicians retrieve information more efficiently and spend less time navigating software.

Built with

TypeScript and Vite
three.js, WebGL, and GLSL
MediaPipe
Web Speech API
Python, NumPy, SciPy, scikit-image, and nibabel
TotalSegmentator
Medical Segmentation Decathlon
Playwright
Gemini / Claude for optional command interpretation

References and Data Sources

Medical imaging datasets and segmentation

Medical Segmentation Decathlon (MSD) — Task03 Liver. Source of the real liver CT volume and liver/tumor segmentation data used in Scrub+. Antonelli, M., Reinke, A., Bakas, S., et al. (2022). The Medical Segmentation Decathlon. Nature Communications, 13, 4128. https://doi.org/10.1038/s41467-022-30695-9

Dataset: https://registry.opendata.aws/msd/

TotalSegmentator. Used to generate additional anatomical segmentation labels from the CT data. Wasserthal, J., Breit, H.-C., Meyer, M. T., et al. (2023). TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images. Radiology: Artificial Intelligence, 5(5), e230024. https://doi.org/10.1148/ryai.230024

GitHub: https://github.com/wasserth/TotalSegmentator

Research supporting the problem

Touchless medical image navigation in operating rooms. Wipfli, R., Dubois-Ferrière, V., Budry, S., Hoffmeyer, P., & Lovis, C. (2016). Gesture-Controlled Image Management for Operating Room: A Randomized Crossover Study to Compare Interaction Using Gestures, Mouse, and Third Person Relaying. PLOS ONE, 11(4), e0153596. https://doi.org/10.1371/journal.pone.0153596