Remote Proctoring for Ultrasound Needle Guidance

• 

  A user provides needle placement suggestions to another user. Both users see the scene from their own viewpoint.

This HoloLens application is designed to help doctors collaboratively place needles under ultrasound guidance. Needle placement involves complex 3D spatial relationships that must normally be imagined in the doctor’s mind. It takes a great deal of hands-on experience to achieve proficiency. Even experts require sticking the patient multiple times. Our application tracks the real needle and ultrasound probe with an external tracking system, to show the physicians where in the body the needle is headed before actually sticking into the patient.

The remote doctor sees the needle trajectory and ultrasound image in 3D, from her own viewpoint, giving her a better understanding than if she were watching 2D video from a webcam. The remote doctor offers suggestions to the local doctor on how to better position the needle to approach the target. She does this by moving a holographic red cube to the spot from where she wants the local doctor to enter the skin. Both doctors see the needle, needle trajectory, ultrasound probe, and ultrasound scan (which are controlled by the local doctor), and the moving red cube (controlled by the remote doctor).

We developed a C++ application to interface a PC with the external tracking system, and created the HoloLens application using Unity. One of the challenges we faced was maintaining a smooth frame rate with low latency when sending the ultrasound image and tracking data wirelessly from the PC to each HoloLens. We created a custom protocol to send tracking and ultrasound data over UDP, and connected the PC and HoloLens devices to a wireless router to minimize issues with wireless congestion on the public WiFi network.

Next, we plan to get and incorporate feedback from doctors, refactor our code to FDA-appropriate standards, and find a way to get HoloLens technology into a commercial FDA-approved device so that we can sell our app.

Built With

• c#
• c++
• unity