-
-
A view of our UI with Kaggle Dataset images of X-Rays
-
Hand X-Ray
-
Teeth X-Ray
-
Chest X-Ray
Inspiration
Coming from families who live in underdeveloped areas, we have seen firsthand how hard it can be to get necessary medical care. Relatives who work in healthcare and deal with the daily nurse shortage inspired us to create a locally hosted software prototype to help doctors and surgeons during surgery when there aren’t enough nurses. In under-resourced clinics, a surgeon often has no circulating nurse to pull up records, scroll through imaging, or keep a surgical log. Breaking scrub to do it themselves is costly and raises the risk of contamination.
What it does
Hands-Off OR lets surgeons zoom in and out, switch between X-rays and files, and keep a log of the surgery while staying scrubbed in. It works through hand gestures captured by a webcam and voice commands through a microphone, so it’s easy to use in any operating room situation. A floating cursor follows the surgeon’s hand, and pinching lets them pan and zoom a DICOM-style imaging viewer. A live Surgical Log sidebar records instruments as they’re called out, and the system can flag a mismatch in the instrument tray count with a full-screen alert.
How we built it
We built the basic system design to ensure a decoupled architecture that allows smooth, local hand tracking that doesn't rely on the internet and avoids high latency. We used agentic workflows to minimize bugs, edge cases, and integration issues across the project, from OpenCV to voice activation.
Challenges we ran into
Some challenges we ran into were with the voice commands, as it was hard to get accurate speech detection from the microphone. Beyond that, keeping hand tracking smooth at 30 FPS while also streaming video frames to the AI took careful separation of the two data paths. Coordinating several AI coding tools without them writing conflicting code also required us to lock down the shared schema early.
Accomplishments that we're proud of
We are very proud of how much we got done in such a short time. For the time we were given, we produced a well-working, efficient, and useful tool. We’re especially proud that gesture control, voice input, and AI function calling work together in one touchless interface that runs with nothing more than a webcam and a microphone.
What we learned
We learned how to combine real-time edge computer vision with a multimodal AI model in a single pipeline, and why splitting fast, local processing (hand tracking) from slower cloud reasoning (Gemini) keeps the interface responsive. We learned how to design a shared WebSocket schema so separate components can be built independently, and how to use function calling to turn AI output into concrete actions in a UI. We also learned a lot about the practical limits of voice input in noisy, real-world environments, and about designing for a sterile setting where every interaction has to be touchless.
What's next for Hands-Off OR
Next, we want to make Hands-Off OR as ready for a real operating room as we can. That starts with better voice recognition through noise filtering, since surgical environments are loud and masks muffle speech. We also plan to support real DICOM files and 3D slice scrolling, so surgeons can pinch and drag through an MRI sequence, and to add AI-annotated imaging that highlights anomalies like those on chest X-rays. Finally, we want to expand our gesture set with per-surgeon calibration for different hand sizes and glove types, and pilot the tool with clinicians in under-resourced clinics, with stronger privacy and offline capability so patient data stays local.
Log in or sign up for Devpost to join the conversation.