Noma

Reducing Mohs Surgery errors by leveraging intelligent telemedicine, Noma revolutionizes care with real-time 3D visualization of patient faces and automated medical transcriptions of procedures.

Comment 1

Noma: A Game-Changer in Telemedicine for Facial Surgery

Inspiration

During our research into healthcare disparities, we uncovered a critical issue: a significant gap in the availability of specialized medical professionals, especially in rural regions of the U.S. This shortage is particularly concerning when it comes to dermatologists performing Mohs Micrographic Surgery (MMS), a precise procedure used to treat skin cancer like Melanoma. Due to this lack of access, rural patients not only experience treatment delays but also face a higher risk of procedural and surgical errors during their treatment.

Our inspiration for Noma emerged from the potential to reduce these errors through telemedicine. By providing surgeons with real-time 3D visualization and automated medical transcriptions, Noma enhances surgical accuracy and ensures that patients, regardless of their location, receive expert, error-free care.

What it Does

Noma addresses these challenges by enhancing communication between surgeons during Moh's surgery.

  • Real-time 3D Facial Visualization:

    • We create an interactive 3D visualization of the patient’s face during surgery through a live recording of the surgery
    • Consulting surgeons can interact with the model and make precise incisions based on real-time updates.

  • Automated Medical Transcriptions:

    • Noma captures live video and audio from the operating room.
    • It generates accurate, time-stamped medical transcriptions, helping to document procedures and identify potential errors.

Together, these features enable remote collaboration between surgeons in rural areas and specialists, ensuring high-quality care.

How We Built It

To build Noma, we used several advanced technologies:

  • 3D Visualization:

    • We tested multiple 3D reconstruction techniques, including Neural Radiance Fields (NeRF) and Gaussian Splatting.
    • Ultimately, we chose Instant Splat for its speed and high-quality rendering, ensuring real-time updates during surgery.
    • This model takes in an initial video of the patient's face and outputs a high-quality 3D reconstruction of the face

  • Automated Transcriptions:

    • We used AWS Transcribe for speaker diarization to distinguish between different speakers during surgery.
    • LITA (Language-Image Transformer Agent) helped us transcribe actions in real-time from the video feed.
    • We built a knowledge graph using Neo4j, which maps relationships between surgical tools, procedures, and surgeons, and updates it in real time with GraphRAG.
    • This knowledge graph is then passed into Meditron 7B, an LLM finetuned on medical transcriptions and guidelines, to produce time-stamped medical transcriptions.

By combining these tools, we created a system that captures real-time events and generates highly accurate medical documentation.

Challenges we ran into

  • Generating 3D images that update real time: Each of the three 3D reconstruction models we tested has inference times of at least 45 minutes. We initially wanted to have the 3D images update in real time as the surgery continues; instead, we load an initial scan of the patient for the consultant to see.
  • Real-time 3D Rendering: Achieving high-quality, low-latency 3D facial visualizations was difficult due to the complexities of rendering and the need for fast processing in live surgery environments.
  • Automated Transcriptions Accuracy: Integrating multiple transcription tools while maintaining high accuracy in a live setting was challenging, especially in distinguishing between various medical terms and procedures.

What's next for Noma

  • Extended Medical Use Cases: Expanding beyond Mohs surgery to support other critical surgical procedures, allowing more widespread adoption of the technology.
  • Improved User Experience: Continuing to optimize the interface for both surgeons and consultants, making interactions smoother and more intuitive.
  • Further Telemedicine Integrations: Enhancing the telemedicine aspect by incorporating more robust, real-time communication features that allow for seamless collaboration between remote medical experts and on-site teams.
Share this project:

Updates