LiveCallAnalytics

An AI-powered healthcare support assistant that helps agents resolve medical queries via email, chat, audio, or video - securely, in real-time, with LLM and patient-aware assistance.

Comment

💡 What Inspired Us

In today’s fast-paced healthcare environment, patients expect seamless, real-time support across multiple communication channels. Our goal was to create a unified agent experience that empowers support teams to resolve critical medical, clinical, and insurance-related issues with speed, intelligence, and empathy.

Key Inspirations Behind the Experience:

  • Omnichannel Support: Patients may reach out through email, live chat, or real-time calls (audio/video). We wanted to ensure that no matter the medium, agents can handle and resolve queries from a single, intuitive platform.
  • Agent Empowerment via AI: Live agent interactions are enhanced with the support of a powerful LLM agent. Agents can pause conversations, ask for assistance, and quickly provide accurate, context-aware answers to patients.
  • Real-Time Intelligence: Speech-to-text captures live audio conversations and displays them as chat input, ensuring the agent never misses a detail. This reduces friction and improves turnaround times for resolutions.
  • Secure, Compliant Handling of Sensitive Data: With Guardrail agents performing PII checks and integrating with a Retrieval-Augmented Generation (RAG) system for patient documents, responses remain safe, compliant, and informed.
  • Human-Centric Resolution Flow: From email queries to live calls and chat sessions, the journey is designed with clarity and empathy. Every resolved issue ends with a thoughtful closure email, reinforcing trust. This solution is inspired by the vision of blending human expertise with AI-powered assistance, ensuring that healthcare support becomes faster, smarter, and more compassionate than ever before.

⚙️ What It Does

This intelligent support system is designed to handle and resolve critical medical, clinical, and insurance-related user queries across multiple channels — all from a single agent interface.

Core Capabilities:

  • Multichannel Query Handling
    Agents can receive and respond to user queries via:
    • Email
    • Live Chat
    • Live Audio Call
    • Live Video Call
  • Dynamic Query Resolution
    Agents select from priority medical issues and begin resolving them. For example:
    • An email query gets auto-populated into a text area.
    • Live chat requests can be accepted and answered with the help of LLM-generated responses.
    • Incoming voice calls transcribe user speech in real time, allowing agents to view and interact with the conversation seamlessly.
  • AI-Powered Agent Assistance
    During a live interaction, agents can:
    • Pause the conversation
    • Consult the LLM Agent for support
    • Retrieve relevant medical or insurance information stored in a RAG (Retrieval-Augmented Generation) system
    • Resume the conversation and respond with confidence and accuracy
  • Built-in Guardrails for Data Privacy
    Every user query is checked for sensitive information (PII) before a response is generated, ensuring compliance and patient safety.
  • End-to-End Resolution Workflow
    Once a query is resolved:
    • Calls and chats are closed cleanly
    • A closure email is automatically sent
    • The issue is marked as resolved in the system This solution is crafted to streamline healthcare support, reduce resolution times, and enhance both agent productivity and patient satisfaction.

🛠️ How We Built It

We built this intelligent healthcare support assistant by integrating modern AI tools, secure data handling practices, and seamless communication APIs — all tied together in a unified agent experience. Architecture Diagram

Key Components:

  • Frontend Interface
    • A responsive dashboard where agents can:
    • View priority medical/clinical/insurance queries
    • Accept and handle email, live chat, audio, and video calls
    • View incoming requests and respond in real-time
    • Pause/resume interactions while retrieving AI assistance
  • Multichannel Communication Stack
    • Integrated with APIs to support:
    • Email ingestion and response
    • Live chat (bi-directional messaging with typing indicators)
    • Voice calls with real-time transcription
    • Video call support for high-touch queries
  • LLM-Powered Agent Assistance
    • Used OpenAI/Gemini for natural language understanding and response generation
    • Agents can invoke the LLM at any time for query resolution
    • Patient context and query history passed for richer responses
  • Guardrails and PII Detection
    • Built-in PII scanners (via Guardrails AI agent)
    • All queries and documents undergo filtering before LLM interaction
    • Ensures compliance with healthcare data policies
  • RAG (Retrieval-Augmented Generation) System
    • Patient documents and historical data indexed in a vector store
    • LLM retrieves and references relevant content dynamically
    • Provides accurate, personalized responses based on prior records
  • Backend Orchestration
    • Secure microservices manage:
    • Session tracking
    • Channel routing
    • Call/chat/email lifecycle
    • Audit and closure notifications
  • Hosted on AWS Cloud
    • The entire solution is deployed on AWS, leveraging services like:
    • EC2 for compute
    • S3 for storage
    • Lambda for serverless tasks
    • CloudWatch for monitoring
    • Ensures scalability, security, and high availability
  • Closure Workflow
    • After resolution, the system triggers:
    • Automatic closure email
    • Feedback loop for continuous improvement

⚠️ Challenges We Ran Into

Building a real-time, AI-assisted, multichannel healthcare support system came with several challenges — both technical and operational. Here's what we encountered:

Integration Complexities

  • Managing seamless communication across email, chat, audio, and video channels required integrating multiple APIs and ensuring synchronized session states across them.
  • Ensuring consistent UX across these modes was tricky, especially during transitions (e.g., switching from email to live call).

PII Detection & Compliance

  • Healthcare data is sensitive. Detecting and redacting Personally Identifiable Information (PII) in real-time before passing queries to the LLM was crucial and complex.
  • We had to implement Guardrail agents and validate that every response adhered to HIPAA-like compliance and audit standards.

Contextual LLM Querying

  • Getting relevant, accurate answers from the LLM agent wasn’t always straightforward — especially when queries involved context from past interactions or uploaded patient documents.
  • Fine-tuning our RAG system to return just the right context for each case required multiple iterations.

Real-time Speech-to-Text Accuracy

  • During live calls, converting user speech into accurate, readable chat input in real-time was challenging, especially with accent variability and medical terminology.
  • Handling overlapping speech or background noise required refining our transcription pipeline.

Latency & Performance

  • Ensuring low latency for LLM responses while handling multiple concurrent sessions was challenging, especially during peak traffic.
  • Optimizing LLM inference + RAG retrieval + frontend updates to feel instantaneous took performance tuning across layers.

Cloud Infrastructure & Scalability

  • Balancing cost-efficiency and scalability on AWS was non-trivial, especially while supporting voice/video processing and session persistence.
  • Monitoring distributed services (Lambda, EC2, S3, API Gateway) required robust logging and alerts through CloudWatch.

These challenges helped shape a more resilient, privacy-first, and responsive healthcare support solution — ready for real-world deployment.

🏆 Accomplishments We Are Proud Of

Building this AI-powered, multichannel healthcare support assistant has been a challenging but rewarding journey. Here are some of the accomplishments we’re most proud of:

Seamless Multichannel Support

  • Successfully enabled real-time query handling across Email, Chat, Audio, and Video Calls — all within a single agent dashboard.
  • Ensured smooth transitions between channels while maintaining session context.

Smart Agent + LLM Collaboration

  • Integrated a Large Language Model (LLM) to assist agents with accurate, contextual responses in real time.
  • Enabled agents to pause live conversations, ask the LLM for guidance, and resume with confidence — improving resolution speed and quality.

Robust Privacy & Compliance

  • Implemented a Guardrail Agent to automatically detect and filter PII/sensitive data before processing queries.
  • Ensured all interactions adhere to privacy standards — critical in healthcare and insurance use cases.

Effective Use of Patient Data via RAG

  • Indexed patient records and support documentation in a RAG (Retrieval-Augmented Generation) system.
  • Delivered personalized and document-grounded responses using LLM with minimal hallucination.

Live Call Transcription & Resolution

  • Enabled real-time transcription of voice calls into chat input, improving accessibility and agent comprehension.
  • Allowed agents to respond with precision and track user intent instantly.

Automated Closure Workflow

  • Built a closure pipeline that sends follow-up emails and logs resolution data automatically, saving agent time and maintaining professional communication.

Fully Cloud-Native Deployment

  • Successfully hosted the entire platform on AWS Cloud, ensuring scalability, high availability, and secure storage.

These milestones reflect our commitment to building a smart, responsive, and secure healthcare support system — one that truly empowers agents and puts patients first.

📚 What We Learned

This project pushed us to innovate at the intersection of healthcare, AI, and real-time communication. Along the way, we learned valuable lessons — both technical and human-centered.

AI Is Powerful, But Needs Guardrails

  • While LLMs are great at understanding and generating responses, they must be guided by context, filtered for safety, and supplemented with facts.
  • Implementing PII detection and document-grounded RAG systems was crucial to maintain both accuracy and trust. ### Human + AI Collaboration Enhances Support
  • Empowering agents with AI-backed assistance—without removing the human in the loop—results in faster and more empathetic resolutions.
  • Features like pause-resume for live calls and on-demand LLM help proved essential for real-world usability. ### Healthcare Domain Needs Extra Care
  • Building for healthcare means extra responsibility: privacy, compliance, terminology, and emotional sensitivity all matter deeply.
  • We learned how to align technical decisions with domain-specific regulations and patient trust. ### Multichannel Coordination Is Complex (but Worth It)
  • Synchronizing logic across email, chat, audio, and video streams was technically challenging — but it enabled a truly unified agent experience.
  • Each channel has its own quirks, but shared context across them improves continuity and reduces friction. ### Real-Time Systems Need Deep Optimization
  • Building low-latency interactions (especially for voice and LLM responses) taught us the value of performance tuning, preloading context, and streamlining API calls.
  • Tools like AWS Lambda, EC2, and CloudWatch helped monitor and scale the system effectively. ### Small Details Make Big Differences
  • From closure emails to UI feedback loops, we learned that micro-interactions matter — they build user trust and polish the experience. This journey taught us how to blend cutting-edge AI with compassionate support — creating something both technically exciting and deeply impactful.
Share this project:

Updates

Submission history