surGestic

Gesture & voice control for sterile spaces. Real-time computer vision + speech recognition let surgeons navigate and analyze ct-scans touchlessly, keeping hands sterile and workflows fast.

Comment

Inspiration

In high-stakes environments like surgical operating rooms, maintaining a sterile field is critical. Yet, surgeons frequently need to review digital CT scans during procedures. Traditional inputs like mice, touchscreens, or keyboards require physical contact, breaking the sterile field, creating cross-contamination risks, or forcing nurses to act as human intermediaries.

I asked myself: What if the human body and voice could act as a completely touchless, sterile interface?

That was the spark for surGestic (Sterile OR Assistant)—a dual-modal gesture and offline voice recognition platform designed to make medical imaging navigation hands-free, intuitive, and sterile.

What it does

surGestic combines real-time computer vision and offline speech recognition to allow hands-free interaction with CT scan slices.

  • Touchless Gesture Control: Leverages MediaPipe for hand landmark tracking. A single pinch combined with a vertical swipe down or up advances or rewinds the CT slice index.
  • Voice-Activated Grid Selection: Holding a double pinch activates an offline VOSK voice recognition engine. The user can call out grid coordinates (e.g., "C3", "A4") to center on a specific region of interest (ROI) or say "reset" to restore the default view.
  • Dynamic Distance Zooming: After selecting a grid cell, moving the hand further away from the pinch origin zooms in dynamically; moving closer zooms back out.
  • Low-Latency Architecture: Syncs real-time vision and audio processing with a React frontend via a fast REST API state pipeline.

How I built it

I built surGestic as a modular, low-latency system designed to maintain a smooth 30 FPS experience, leveraging GPT-5.6 and OpenAI Codex during the OpenAI Build Week Challenge:

  • Vision & Voice Engine (Python): Built using OpenCV for camera feed capture and Google's MediaPipe HandLandmarker for pinch and swipe tracking. Speech recognition is handled offline via VOSK running in a dedicated audio buffer thread to prevent blocking the vision loop.
  • Backend API (FastAPI): Built with FastAPI and Uvicorn to act as a central state hub (holding slice index, active grid cell, zoom factor, and pan offsets) with sub-100ms response times.
  • Interactive Frontend (React + Vite): Built a high-performance React UI that continuously polls the backend API to render CT scans, overlay coordinate grids, execute smooth CSS zoom/pan transforms, and display real-time audio visualizers.

Challenges I ran into

  1. Non-Blocking Multi-Modal Input Sync: Running a 30 FPS camera feed alongside real-time microphone processing can easily cause lag. I solved this by isolating the VOSK speech recognition pipeline into a concurrent thread using sounddevice and queue, allowing audio buffering without stuttering the OpenCV video pipeline.
  2. Coordinate Transformation & Spatial Mapping: Mapping camera-space 3D hand landmark coordinates to 2D image grid anchors and CSS transform offsets for smooth dynamic zooming was complex. I utilized OpenAI Codex to help generate and refine the spatial mapping formulas.

Accomplishments that I'm proud of

  • Smooth 30 FPS Tracking: Achieving reliable, low-latency hand gesture navigation over standard webcams without needing expensive depth-sensing hardware.
  • Privacy-First Offline Voice: Integrating offline speech recognition (VOSK) so voice commands run locally in the OR without external cloud calls or network delays.
  • Clean Full-Stack Integration: Successfully connecting low-level Python vision/audio processing with a modern React + Vite frontend via a FastAPI state hub.

What I learned

  • Human-Computer Interaction (HCI) in Medical Domains: Designing touchless controls for surgeons requires minimal cognitive load—gestures and vocal commands must be simple, precise, and physically easy to hold.
  • Concurrent Pipeline Architecture: I learned how to manage multi-threading in Python effectively so that audio stream loops and frame processing operate in parallel without state sync bottlenecks.

What's next for surGestic

  • AI Assistant Integration: Building on the existing UI setup, fully integrate an AI assistant triggered by voice commands to perform tasks such as comparing older and current scans, generating automated analyses, or detecting and interpreting critical regions directly within CT scans.
  • Native DICOM & PACS Support: Move beyond generated CT image assets to full, native integration with hospital PACS archives and standard DICOM viewers.
  • Expanded Voice & Gesture Controls: Introduce advanced vocal and physical gestures for window/level (brightness/contrast) adjustments and dynamic measurement tools.

Built With

Share this project:

Updates