MedBridge

Every minute, a patient's life hangs in the balance because the expert who can save them is a thousand miles away. We close that distance with AR, teleporting a specialist's hands into any OR.

Comment

Inspiration

Every year, millions of preventable deaths occur simply because the right medical expert isn't in the right place at the right time. A specialist in New York could save a life in rural Nebraska, but geography stands in the way. This disparity in access to specialized surgical and medical knowledge is a critical global challenge.

Our inspiration was to shatter this barrier. We envisioned a world where a seasoned surgeon could guide a complex procedure from thousands of miles away, not just by talking, but by being virtually present. We were inspired by the idea of using Augmented Reality to "teleport" a specialist's hands and expertise into any clinical setting, democratizing life-saving skills.

What it does

MedBridge is a real-time, interactive AR platform that "teleports" a specialist's hands into a local clinician's field of view. The remote expert uses their webcam to control a virtual 3D hand. The local clinician, using a tablet or smart glasses, sees this virtual hand overlaid onto the real world, pointing and guiding them through complex procedures as if the expert were right there in the room. The system provides live video and audio for seamless communication, creating a truly collaborative remote surgical environment.

How we built it

Our architecture combines three core systems:

  1. Computer Vision Backend: A Python server using Flask, OpenCV, and MediaPipe tracks the expert's hand movements from their webcam feed in real-time, converting them into 3D coordinates.
  2. WebRTC Signaling Server: A lightweight Node.js server establishes a direct, low-latency peer-to-peer connection between the two users for streaming video, audio, and the crucial hand-tracking data.
  3. AR Web Clients: Vanilla JavaScript applications run in the browser. The clinician's app uses Three.js to render the 3D virtual hand based on the incoming coordinates, effectively creating an AR overlay on their device's camera feed.

Challenges we ran into

  1. Latency is Life: In a surgical context, every millisecond counts. Our biggest challenge was minimizing the "glass-to-glass" latency, the time from the expert moving their hand to the clinician seeing the virtual hand move. We had to optimize our hand-tracking algorithm and rely on WebRTC's direct data channels instead of slower server-based communication to achieve the necessary speed.

  2. 3D in the Browser: Rendering and smoothly manipulating a 3D object in real-time based on a constant stream of data was complex. We had to carefully manage the rendering loop in Three.js and interpolate coordinate data to ensure the virtual hand's movement was fluid and not jittery, which could be dangerously distracting for the clinician.

  3. System Integration: Juggling three different environments (Python, Node.js, and the browser) was a significant integration challenge. Ensuring the data formats were consistent, the servers were robust, and the WebRTC handshakes were flawless required meticulous coordination and debugging across the entire stack.

Accomplishments that we're proud of

We are incredibly proud of creating a functional, end-to-end prototype that proves our core concept is viable. Achieving near-real-time latency for the hand tracking over a standard internet connection was a major success. Successfully integrating Python computer vision, a Node.js signaling server, and browser-based 3D rendering into one cohesive system felt like a huge accomplishment. Most of all, we're proud to have built something that has the potential to make a real-world impact on people's lives.

What we learned

This project was a deep dive into the technologies shaping the future of communication and medicine. We gained invaluable experience in:

  • Real-time Computer Vision with Python's OpenCV and MediaPipe libraries.
  • Full-stack WebRTC implementation, from signaling to establishing peer-to-peer data and video streams.
  • Applied 3D Graphics using Three.js to create a practical and interactive AR overlay in a web browser.
  • System Architecture, learning how to design and integrate multiple disparate services into a single, cohesive, and performant application.

What's next for MedBridge

The future for MedBridge is focused on moving from a proof-of-concept to a clinical-grade tool. Our next steps include:

  • True AR Integration: Porting the clinician application from a tablet to true AR glasses (like the HoloLens or Magic Leap) for a hands-free experience.
  • Enhanced Accuracy and Tools: Improving the precision of the hand tracking and adding more virtual tools, such as pointers, scalpels, and markers, that the expert can switch between.
  • AI-Powered Assistance: Integrating AI to provide real-time feedback, such as measuring incision length or warning if the clinician's hand strays from the expert's guided path.
  • Clinical Trials: Partnering with medical institutions to begin testing the platform in simulated and, eventually, real surgical scenarios to validate its efficacy and safety.
Share this project:

Updates