seeCOLe: MARY (Medical Assistant and Research sYstem)

Empowering clinicians with voice-activated AI insights and real-time predictive risk assessments through EHR integration and augmented reality.

Comment

Inspiration

The mission of seeCOLe (see Clinical data On Lens) inspired MARY (Medical Assistant and Research sYstem), to enable hands-free clinical data interaction. Clinicians often face EHR fatigue, inefficient data retrieval, and documentation challenges, which take time away from patient care. MARY was created to enhance seeCOLe by integrating AI-powered clinical decision support, allowing providers to retrieve patient data, receive recommendations, and streamline workflows using natural language and hands-free commands in an augmented reality environment.

What it does

MARY acts as an AI-driven clinical assistant within seeCOLe, allowing clinicians to:

✅ Retrieve patient records and complete real-time actions hands-free using SMART on FHIR APIs.
✅ Analyze real-time lab results, medications, and vitals for decision-making.
✅ Receive AI-powered treatment recommendations based on evidence-based guidelines.
✅ Streamline documentation by summarizing clinical notes and encounters.
✅ Enhance patient safety by flagging potential medication interactions or abnormal lab results.
✅ Enable hands-free interaction through augmented reality and voice commands.

How we built it

MARY was designed as an AI-powered extension of seeCOLe, integrating:

🔹 FHIR API (SMART on FHIR, MeldRx) to securely pull patient data from EHRs.
🔹 Machine Learning (ML) & NLP to process voice-activated clinical queries and structured requests.
🔹 Augmented Reality (AR) Visualization to present real-time clinical insights.
🔹 Cloud-based architecture for scalability and seamless integration into hospital systems.

Challenges we ran into

1. Context Window Overload in LLM Integration

  • Problem: When integrating LLMs with FHIR APIs, the system generated vast amounts of data per query. Each interaction accumulated previous API responses, quickly overwhelming the model’s context window. After 4-5 interactions, the available context space became too limited, making the chat unusable.
  • Solution: Implemented a dynamic, per-query data retrieval system that processes FHIR resources in real time without persistent storage. This approach keeps the working context and saves context separately, ensuring efficient model interactions.

2. Generating Accurate FHIR Resource Requests

  • Problem: When adding patient resources via FHIR, the system frequently generated incorrect or incomplete request bodies, leading to validation failures.
  • Solution: Developed a validation mechanism to dynamically detect and correct placeholder values. Additionally, implemented a context-specific guidance system for different resource types and an error feedback loop that iteratively refines request bodies until a successful (200) response is received.

3. Limited Documentation & Error Handling from MeldRx

  • Problem:
    • Lack of detailed documentation required extensive trial and error during implementation.
    • MeldRx error responses provided only an error_id without explaining the issue, forcing the team to frequently contact MeldRx support for resolution.

Accomplishments that we're proud of

  • ✅ Successfully maintained a continuous chat flow while integrating LLMs with FHIR APIs.
  • ✅ Developed an automated debugging system for FHIR responses using LLMs, reducing manual intervention.
  • ✅ Constrained and bound the LLM within FHIR API functions, ensuring it prioritizes structured API responses over its own knowledge base.
  • ✅ Gained expertise in FHIR (Fast Healthcare Interoperability Resources) and its various components.
  • ✅ Optimized database queries, improving performance and ensuring faster response times.

What we learned

  • 📌 Mastered the design of a robust chat flow, leveraging system instructions, custom function usage in LLMs, and dynamic, user-friendly error message handling.
  • 📌 Gained a deep understanding of FHIR and its resources for managing healthcare data effectively.
  • 📌 Improved AI prompt engineering to enhance interaction with LLMs.

What's next for MARY (Medical Assistant and Research sYstem)

🚀 Future enhancements include:

  • Expanding FHIR API integrations to work with more EHR vendors.
  • Enhancing predictive analytics to provide proactive patient risk assessments.
  • Improving NLP and voice interaction to make MARY even more intuitive for clinicians.
  • Integrating with wearable XR devices like Microsoft HoloLens and Apple Vision Pro for a fully immersive experience.
  • Piloting MARY in live hospital settings to gather real-world feedback and refine usability.

Built With

Share this project:

Updates