MediAnnotateAI

MediAnnotateAI — See, Annotate, and Collaborate on Patient Care in Mixed Reality

Comment

Inspiration

During my work on NGH NextGenerationHIS at GPI Group I developed multiple structured patient history forms ensuring clinicians could capture update and retrieve patient information efficiently While NGH solved the problem of digital health record management I noticed a critical gap much of patient care is visual and spatial

  • Explaining conditions to patients
  • Planning procedures
  • Coordinating care between remote specialists MediAnnotateAI was inspired by this need to make patient data interactive visual and collaborative in real time

What it Does

MediAnnotateAI is a Mixed Reality application that empowers healthcare professionals and patients to interact with clinical data and anatomical visualizations in a spatial intuitive way Using the Logitech MX Ink stylus on Meta Quest headsets clinicians can

  • Annotate 3D patient models precisely
  • Collaborate remotely with specialists
  • Provide interactive visual explanations to patients
  • Operate securely in offline-first workflows MediAnnotateAI extends the structured workflows of NGH into immersive MR turning static patient histories into actionable spatial experiences

How We Built It

  • Expo Go React Native Apps on Meta Quest Built the first version of the app to capture and annotate 3D patient screens save drawings to the system and share them for review
  • YOLO11 Pose Detection Integration Added the ability to track patients in real time during examinations allowing annotations to follow the patients movement and save them directly onto the detected pose model
  • Unity 3D Anatomy Overlay Integrated fullbody anatomical models in spatial 3D over the YOLO11 pose detection providing enhanced visualization of muscles veins and other anatomical details for richer clinical context
  • Stylus Integration MX Ink stylus enables precise annotations workflow marking and patient education overlays
  • AI Layer Summarizes annotations flags abnormalities and generates structured clinical notes from MR interactions
  • Offline Architecture Secure encrypted local storage with synconconnect inspired by NGHs reliability
  • Remote Collaboration Multiuser mode allows remote specialists to annotate patient models in real time maintaining structured clinical workflows

Challenges We Ran Into

  • Translating structured NGH patient forms into interactive 3D MR annotations
  • Maintaining offlinefirst security and compliance for sensitive patient data
  • Ensuring stylus input was precise enough for clinicallevel annotations
  • Balancing MR complexity with usability for clinicians unfamiliar with immersive technology

Accomplishments That Were Proud Of

  • Built a fully functional MR workflow for patient care using Expo Go React Native enabling doctors to capture annotate and share 3D patient screens in the first version
  • Integrated YOLO11 Pose Detection allowing annotations to follow patient movement during examinations and saving them directly on the pose model
  • Added Unity 3D anatomical overlays for muscles veins and other structures providing richer context and actionable insights for clinicians
  • Developed an AI Assistant that supports fast searching summarizes annotations flags abnormalities and generates structured clinical notes
  • Enabled remote collaboration between clinicians in real time maintaining structured clinical workflows and enhancing patient care
  • Designed a secure offline first architecture ensuring reliable operation in lowconnectivity or sensitive healthcare environments

What We Learned

  • How to translate structured clinical workflows into 3D immersive interactions
  • The value of offline first design in sensitive healthcare environments
  • How to integrate AI for annotation summarization and workflow guidance
  • How enterprise level healthcare systems like NGH inform MR design for hospitals and clinics

Whats Next for MediAnnotateAI

  • Rehabilitation tracking with 3D motion overlays and AI guidance
  • Remote telemedicine consultations for clinics in lowconnectivity regions
  • Medical education and residency training using MR simulation
  • AI assisted clinical decision support for routine and complex procedures
  • Direct body annotations using YOLO11 Pose Estimation enabling precise MR overlay of 3D anatomical annotations directly on the patients body

Ultimately MediAnnotateAI aims to bring patient data to life creating a natural visual and collaborative extension of modern hospital information systems

Built With

  • built-with:-javascript
  • c#-(unity)
  • chatgpt
  • cloud-sync
  • collaboration
  • copilot
  • encrypted-local-storage-(sqlite/realm)
  • gemini
  • llama
  • logitech-mx-ink-sdk
  • meta-quest-sdk
  • on-device-ai
  • react-native-(expo-go)
  • remote
  • typescript
  • unity-3d-anatomical-models
  • yolo11-pose-detection
Share this project:

Updates