Medipal

AI healthcare companion transforming the ER experience for patients and staff. Offering personalized support, streamlined workflows, and better communication.

Comment

🧠 MediPal – AI-Powered Healthcare Companion

🏥 Project Overview

MediPal is a cutting-edge AI-powered healthcare companion designed to revolutionize the Emergency Department (ED) experience for both patients and medical staff. This full-stack prototype demonstrates how artificial intelligence can provide personalized support, emotional comfort, and workflow enhancements throughout the entire ER journey.

🎯 Core Mission

  • For Patients: Deliver an intelligent, empathetic companion that helps them navigate their ER visit, understand medical updates, and receive emotional support—especially during long wait times.
  • For Nurses: Equip clinical staff with a smart dashboard that surfaces real-time insights, triage recommendations, and communication summaries to enhance patient care and efficiency.

🏗️ Technical Architecture

🧩 Frontend Stack

  • Framework: Next.js 15 (App Router)
  • Language: TypeScript
  • Styling: Tailwind CSS + Custom Components
  • UI Components: Radix UI + shadcn/ui
  • State Management: Jotai
  • Icons: Lucide React
  • 3D Graphics: Three.js + React Three Fiber + Drei

🧠 AI & Voice Communication

  • Conversational AI: SambaNova Cloud (LLM)
  • Voice Communication: OpenAI Realtime API + WebRTC
  • Audio Processing: Web Audio API (gain control, stream cleanup)
  • Function Calling / RAG: Context-aware Q&A over medical data (insurance, history, reports)

🧍‍♂️ 3D Avatar System

  • Real-time lip sync via phoneme-to-viseme mapping
  • Morph target facial animations and gestures
  • High-quality lighting and rendering for realism
  • Rendered with GLTF + Three.js

🔄 Data Flow Overview

↓ WebRTC + Audio API OpenAI Realtime API (Streaming STT & TTS) ↓ LLM → Function Call → Custom APIs (Insurance, History, Reports) ↓ Response → Text & Audio → 3D Avatar (Real-Time Expression)

🧪 Key Features by Role

👤 Patient Interface (/patient)

  • Voice-first Chat: Realtime text/audio interaction with streaming feedback
  • 3D AI Avatar: Lifelike doctor with real-time lip sync and gestures
  • Function Calling AI:
    • ai_insurance_rag: Understand insurance coverage & claims
    • medical_history_rag: Retrieve treatment records and medications
    • medical_report_rag: Break down complex test results or imaging
  • ER Session Tracker:
    • QR check-in
    • Location updates
    • Wait time estimation
  • Smart Health Records:
    • Chronological history, symptoms, allergies, medications
    • AI insights: pattern detection, risk alerts, treatment efficacy
  • Insurance Navigator:
    • Policy overview, deductible tracking, appeal guidance, EOB analysis
  • Family & Support Network:
    • Emergency alerts, health proxy access, secure chat, shared care records

👩‍⚕️ Nurse Dashboard (/nurse)

  • AI Triage System:
    • Smart dynamic forms
    • Symptom risk scoring & vitals input
  • Voice Simulation:
    • Simulated patient voice for assessment training
  • Decision Support:
    • Protocol recommendations
    • Drug interaction alerts
    • Bed/resource allocation
  • Live Analytics Dashboard:
    • Queue visualization
    • Time-to-assessment tracking
    • Staff and room utilization metrics

☁️ AWS Infrastructure

MediPal uses a serverless-first, modular architecture on AWS for scalability, compliance, and real-time operation.

🧬 Services Used

Service Purpose
AWS Lambda Core logic and event-driven compute (containerized)
Amazon API Gateway Secure API endpoint triggers
Amazon S3 file storage (transcripts, session data)
Amazon EventBridge (Optional) Scheduled tasks and alerts
Amazon ECR Docker container storage for Lambda

🔗 Infra repo: AWS Lambda & Infra Files
🖼️ Diagram: ./public/architecture_diagram.png

✅ Accomplishments

  • Developed full AI agent with 3D avatar + voice chat
  • Integrated insurance & medical report RAG querying
  • Prototyped both patient-facing and nurse-facing interfaces
  • Collaborated with SKH and Raffles Medical for feedback

🧠 What We Learned

  • Effective AI in healthcare requires more than answers—it needs empathy
  • Real-time voice streaming is complex but highly engaging
  • Serverless AI pipelines scale beautifully when built around modularity
  • Clinician involvement ensures relevance and safety from the start

🔮 What's Next for MediPal

  • ✨ Clinical Pilots with SKH and Raffles Medical
  • 🧒 Launch PsychAI: child mental health support
  • 👵 Build Elderly Companion with family dashboards
  • 🧬 Add predictive triage models and risk scoring
  • 🔗 Integrate with hospital EHR systems
  • 🌐 Localize with multilingual support
  • 📱 Publish mobile React Native version

MediPal is a bold step toward collaborative, human-centered AI in healthcare. We don’t replace human care—we amplify it.

Built With

Share this project:

Updates