Booking Doctor's Appointment using Vitalis
Onboarding

Inspiration

Vitalis began with a scare close to home. One of our teammate’s grandfathers, who takes Warfarin for atrial fibrillation, nearly suffered a stroke after eating a spinach salad—unaware that leafy greens could dangerously interfere with his blood thinner. That moment revealed how even small, everyday choices can carry serious medical consequences.

We realized his story wasn’t unique. Many elderly patients struggle to manage complex medication schedules, forget critical doses, or accidentally combine drugs that shouldn’t be taken together. For them, healthcare apps can be too technically confusing, and they often lack someone who understands how their everyday choices affect their health based on their personal medical conditions.

Every year, 125,000 Americans die from medication errors, and non-adherence costs the U.S. healthcare system $290 billion. We wanted to build more than a reminder app—we wanted to create an AI health companion that truly understands your medical history, watches for hidden risks, and acts as a 24/7 guardian for your well-being.

Our Solution

Vitalis is an intelligent healthcare agent that combines Retrieval-Augmented Generation (RAG) with agentic decision-making and tool use to provide personalized, proactive patient support.

Unlike static chatbots, our system continuously retrieves context from a patient’s medical history and decides when to take meaningful action—such as scheduling appointments, sending reminders, or asking follow-up questions about symptoms.

Our architecture integrates a large language model with a dual-memory system and a tool orchestration layer:

The vector-based RAG component (ChromaDB) recalls semantically relevant health information.
Gemini serves as the decision agent, interpreting patient context, reasoning over retrieved data, and autonomously. The Gemini API drives the project through three key components. In agent.py, gemini-2.0-flash powers the main medical chatbot, handling user queries, scheduling, and email interactions while managing tool orchestration. The gemini-2.5-flash model in tools/history.py extracts structured medical data from free-text inputs and stores it in ChromaDB, while gemini-pro in tools/email_notification.py generates professional doctor alerts summarizing patient details and conditions with fallback templates for reliability. Together, these models form an integrated system that automates medical dialogue, data extraction, and communication seamlessly.

These layers are coordinated through an autonomous reasoning loop, enabling the agent to interpret inputs, retrieve data, and invoke external APIs (e.g., Gmail or Calendly) to complete real-world actions.

Implementation

Voice Interface

Many elderly users struggle with typing or navigating complex interfaces. We integrated ElevenLabs Speech-to-Text to enable hands-free, natural speech input with accurate medical term recognition, making the agent intuitive and accessible for all users.

Medical Memory

Traditional healthcare chatbots struggle with long-term recall and context retention — especially when dealing with complex, evolving patient histories. RAG systems, however, enable dynamic retrieval of past information, but are rarely optimized for persistent, semantically rich medical data.

To address this, we use ChromaDB as a vector-based medical memory that encodes and retrieves a patient’s health history, symptoms, and conversations based on semantic similarity rather than exact matches. This allows Vitalis to understand patterns across time, such as recurring symptoms, past medication adjustments, or lifestyle changes. In this way, Vitalis stands out compared to pure LLM use; it recognizes when old conditions re-emerge, retrieve relevant historical context, and make informed, proactive decisions. In healthcare, where symptoms can resurface years later or medications interact with past treatments, this persistent and context-aware memory is crucial for delivering safe, personalized care.

Drug Interaction Prevention

Polypharmacy can lead to dangerous drug or food interactions that often go unnoticed. Our system performs real-time cross-checks between medications, allergies, and conditions whenever users introduce new drugs or lifestyle changes. Every recommendation is safe, contextual, and personalized to the patient’s health profile.

Intelligent Conversational AI

Healthcare chatbots often lack context and empathy. Powered by the Letta Agent Framework and Gemini 2.5 Flash, Vitalis recalls prior sessions through agentic tool usage and adapts explanations to each user’s level of understanding. The result is clear, friendly, and medically responsible guidance.

Challenges

Structured Output and Prompt Optimization

While Gemini 2.5 Flash provided exceptional reasoning and speed, we encountered a key challenge: its output was often too verbose or unstructured for direct front-end display. For instance, when integrating with Calendly, Gemini initially returned long-list-form text responses (e.g., detailed time ranges) instead of concise, structured availability blocks suitable for UI rendering.

Solution: We optimized our system through iterative prompt engineering and output schema control. By defining clear role instructions, response templates, and JSON-like output structures, we ensured Gemini consistently generated machine-readable and UI-friendly responses. This allowed the agent to dynamically populate front-end components—such as appointment slots, reminders, or medication summaries—in a clean and intuitive format.

Ethical AI in Healthcare

A key challenge was ensuring our AI never oversteps into providing medical diagnoses or unsafe advice.

Solution:

Context-aware guardrails escalate cases with sensitive symptoms (e.g., chest pain) to a doctor immediately.
When uncertain, the agent responds with safe fallbacks such as: “I’m not a medical professional, but it would be best to contact your doctor to discuss this further.”
Human-in-the-loop design ensures Vitalis enhances communication between patients and providers rather than replacing them. For critical cases, the system can proactively email or schedule appointments through integrated APIs.

Learnings

We learned that RAG isn’t just about retrieval quality; it’s about instruction precision. Even with a strong memory system, unoptimized prompts can lead to irrelevant or overly verbose outputs that confuse users or break the front-end flow. Effective RAG integration isn’t just a technical challenge—it’s a communication design problem, where the quality of instructions determines how intelligently and clearly the agent behaves, and is uniquely shaped by the language model and agent architecture.

We also recognized that autonomy without boundaries is dangerous in healthcare. Even the most advanced language models can produce unsafe advice. Doctor referral fallbacks ensure a safety-first approach.

Most importantly, we discovered that the true value of a healthcare agent is not to replace doctors, but to enhance communication and extend accessibility through human-in-the-loop AI.

Next Steps

Expand the drug database through integration with the FDA Drug Interaction API, covering over 10,000 medications, supplements, and over-the-counter drugs.
Partner with senior living facilities for pilot programs to validate impact and improve system design.
Integrate wearable data (e.g., Apple Watch, WHOOP) to correlate medication adherence with heart rate, sleep, and activity, enabling proactive detection of side effects.