AAA Healthcare Voice Assistant Pinky AI

A healthcare co-pilot that supports patients and clinicians. Dial the demo number now and speak to our live agent.

Comment

Inspiration and Problem Statement

Healthcare systems around the world struggle with a deceptively simple problem: missed appointments and overlooked tasks. These gaps don’t just cost clinics revenue they disrupt continuity of care, delay treatments, delay payments, add strain on doctors and nurses, ultimately reducing patient outcomes.

Our Solution:

"What if an AI agent could proactively close that loop reaching patients, nurses and doctors in the most natural way possible: a reminder phone call before a task is due."

I already have an EHR/ CRM, how is this any different?

Your CRM is a database. Pinky is a workforce. Most healthcare software acts like a digital filing cabinet—it’s great at storing names and dates, but it can't pick up the phone. A filing cabinet doesn't help you when your receptionist calls in sick or your front desk is buried under a mountain of paperwork. Pinky is the "digital staffer" who bridges the gap between your data and your patients.

Your CRM is passive. Pinky is proactive. Traditional CRM systems typically wait for people to take action. They might send an automated text, but they can't have a 5 minute conversation with a patient who is confused about their pre visit instructions.

  • Pinky can compliantly call and text 100+ patients per day, volume that would take a human staffer an entire week of dedicated "phone time."
  • Pinky works when patients are actually home. She can handle Sunday evening reminders for Monday morning appointments, ensuring your high value slots are always filled.

Pinky doesn’t replace your team; she protects them. She isn't another "tool" for your staff to manage. She is a highly trained assistant who plugs directly into your existing workflow. By taking over the repetitive, high-volume tasks like identity verification and scheduling follow-ups, Pinky eliminates the #1 cause of front desk burnout. She handles the busy work so your human team can focus on the patient standing right in front of them.

How it Works

Pinky AI is an interoperable healthcare voice assistant that:

  • 📞 Initiates reminders and missed appointment recovery via time-based triggers
  • 🩺 Supports nurses and doctors with proactive task reminders
  • 🗣️ Uses natural voice AI for empathetic, real‑time interaction and scheduling
  • 🧠 Converts conversations into structured RAG powered summaries
  • 🧾 Logs outcomes into a FHIR aligned record
  • 💡 The best part is Pinky can operate off Google or Microsoft Sheets where clinicians work

Example flow:

  1. Patient or clinician has an upcoming appointment or misses a task
  2. Agent places a call using a time-based trigger, or receives an incoming call
  3. Voice interaction occurs
  4. Conversation is transcribed, analyzed pushed to a sheet of record
  5. Longterm data storage in AWS achieving the highest level of compliance.

-> Record is updated in an FHIR style dataset:

|Caller_Phone | |Call Summary | |Call Duration (s) | |Call Date | |Call Time | |Language | |Agent |

|Status | = "Call Back"

How we built it

We focused on a lean, low-code architecture to maximize speed while still demonstrating interoperability:

Architecture Overview

-> MCP (Model Context Protocol) - Prompt Opinion marketplace defines agent behavior - System prompt + guardrails + knowledge base

-> A2A (Agent-to-Agent Communication) - Voice agent interacts with scheduling logic - Modular flow simulates coordination between agents

-> FHIR Layer (Mock Implementation) - For simplicity Google Sheets is used as a FHIR datastore with structured fields

Tools Used

  • Twilio → voice call infrastructure
  • ElevenLabs → natural voice conversations using real world training transcripts
  • Prompt Opinion → agent orchestration
  • Google Sheets → mock FHIR records
  • Zapier / Make/ N8N → no-code automation
  • AWS → Final secure and compliant data store.

Flow; Human/MCP→A2A→FHIR

This pipeline demonstrates how modern AI agents can integrate with healthcare standards without heavy backend engineering and reduced cost.

Challenges we ran into

1. Simulating FHIR without a backend FHIR servers are complex. Instead of building one, we:

 - Modeled FHIR resources in spreadsheets
 - Focused on structure over infrastructure.

2. Voice reliability in a live demo

Voice agents can be unpredictable:

 - Latency issues
 - Speech recognition variability

We solved this by:

  • Designing a controlled demo scenario
  • Close to 200 live calls across the home services vertical: a high volume high stakes environment where compliance is mandatory
  • Using clear conversational guardrails and a lean scenario with limited variables. On this assignment the voice agent has 1 task: verify identity, then remind or reschedule the client about an appointment and take note of the response in a compliant manner.

3. Balancing compliance and usability

Healthcare AI must be careful: - No diagnosis - No sensitive overreach - No unauthorized access to patient data

We implemented strict prompt guardrails:

- Only scheduling-related and reminder actions no medical advice
- Clear escalation boundaries
- Information and timelines only discussed once identity has been verified.

Accomplishments that we're proud of

  • 🎯 Built a fully working voice agent demo that can be called at anytime of the day
  • 🔗 Demonstrated true interoperability (MCP → A2A → FHIR)
  • 📞 Created a callable phone experience
  • 🧾 Generated structured healthcare compatible outputs
  • ⚡ Delivered a low-code solution with real world relevance => Trained the model on real world calls in the home services vertical.

What we learned

- Interoperability is not just technical—it’s experiential and requires regular fine tuning
- Voice interfaces dramatically increase accessibility in healthcare, especially with the elderly
- You don’t need full infrastructure to prove a concept—clarity beats complexity
- Standards like FHIR become powerful when combined with AI agents that operate within set guardrails and system prompts.

We also learned how to design systems where:

       Unstructured conversation→Structured healthcare data

The Roadmap: The Future of Pinky AI

Pinky isn't just an assistant; she is the orchestrator of a high-compliance healthcare ecosystem. Our next phase focuses on moving from "interoperable" to "embedded."

1. Deep Clinical Integration

  • Production FHIR Connectivity: Moving beyond mock data to live synchronization with Epic, Cerner, and MyChart for real-time scheduling.
  • Insurance Gatekeeper: Automating real-time eligibility checks during the initial voice interaction.

2. The Multi-Agent Orchestrator

  • Specialized Agent Cells: Deploying a "Federated AI" model where dedicated agents handle Scheduling, Billing, and Compliant Reporting in parallel.
  • Predictive Recovery: Leveraging machine learning to identify "high-risk no-shows" and triggering preemptive, personalized multimodal outreach via text, whatsapp and phone calls,

3. Global Accessibility

  • Multilingual Fluency: Expanding native voice support to serve diverse patient populations, ensuring healthcare equity through language.
  • The Wellness Hub: Integrating our Mobile Dashboard (Mockup shown) to provide patients with a unified interface for reminders, goal tracking, and direct voice-line access to their care team. In high stakes oncology and renal care departments this application could literally be life saving.

Battle-Tested Deployment & Safety

Our demo isn't just a prototype; it’s a hardened, live environment. We’ve implemented enterprise-grade controls to ensure "Pinky" remains professional and efficient:

  • Click-to-Call Integration: A seamless, one-page lander that initiates a live voice session with our v3 agent.

  • Token Optimization: Intelligent session limits (3 mins max) and daily call quotas to ensure sustainable operations.

  • Harassment & Time-Waster Defense: We have developed a proprietary Time-Waster Detection Dataset. Pinky is engineered to identify and redirect non-productive or inappropriate interactions, ensuring clinical resources are never compromised, a sample dataset may be viewed here: https://www.kaggle.com/datasets/lifebricksglobal/llm-rag-chatbot-training-dataset

  • Real-World Resilience: Ported from 200+ calls in the home services sector, our NLU is fine-tuned to handle interruptions, background noise, and complex verification flows.

AAA Healthcare Voice Assistant Pinky AI is a glimpse into a future where AI agents don't just talk. They coordinate, comply, and care.

Built With

  • a2a
  • elevenlabs
  • google-sheets-(fhir-mocked-datastore)
  • n8n
  • prompt-opinion-(mcp)
  • qwen
  • rag
  • twilio
Share this project:

Updates