HoloRay Dynamic Annotation Module

Inspiration

After listening to the HoloRay presentation during the opening ceremony, our team wanted to learn more about the technology that they were building. We attended the HoloRay workshop and were immediately fascinated by the potential of real time anatomical annotation visualization.

What it does

Our project enhances the HoloRay platform by allowing surgeons to annotate directly onto the video feed while providing AI-assisted real-time anatomical identification. Unlike typical annotation tools, our annotations remain anchored to the intended location even as the video moves, ensuring accuracy and context are preserved. Additionally, the system can analyze any image input, providing immediate AI feedback about the anatomical structures or objects within the frame.

How we built it

We used OpenCV to handle video input, extract frames, and render annotations dynamically. Gemini AI was used to analyze images and providing information about the identified medical objects.

Challenges we ran into

When first presented with the HoloRay challenge, we were confused about where to begin as the problem seemed very daunting to solve. Our team did not have a lot of experience dealing with Opencv and image/video processing. Once we managed to get started, we also ran into the problem of not knowing how to split up work in a way that would not leave people not having parts to work on. Once we had some sort of a working video tracking system, it was difficult to make the tracking more robust and reliable as there were still huge inconsistencies between renders.

Accomplishments that we're proud of

As mentioned before, our team did not have much experience dealing with the technology required to complete this challenge, so it was a huge learning experience for everyone on the team. Despite this, we were able to bridge many different technologies together to build a working prototype for our project.

What we learned

We learned how to handle continuous video streams, extract frames, and overlay dynamic annotations that track moving content. By using Gemini AI, we gained practical experience in sending images to AI models and interpreting responses for real-world applications.

What's next for HoloRay Dynamic Annotation Module

Increasing the reliability of the tracking is an ongoing aspiration as the current version of our project still needs some tweaking to achieve a reliable and finished product. Our real time anatomical identification tool would ideally be paired with a LLM specifically trained to recognize medical objects rather than one like Gemini AI which is best suited for everyday use for the general public.