AVA: An AI Companion for Seniors

AVA is an AI companion for older adults that holds natural voice conversations, then provides families with meaningful insights and patterns about their loved one's physical and emotional health.

Comment

🌸 AVA - AI Companion for Seniors

AVA is an AI companion for older adults that holds natural voice conversations, then provides families with meaningful insights and patterns about their loved one's physical and emotional health.

Links

Inspiration

Every year, millions of seniors live alone, often separated from their families by distance or busy schedules. While families want to stay connected and ensure their loved ones are healthy, they struggle to maintain regular contact and detect subtle signs of cognitive decline, emotional distress, or physical health issues. By the time red flags become obvious, conditions have often progressed significantly.

We were inspired by stories of seniors who felt lonely and isolated, longing for regular conversation and connection. At the same time, their adult children worried from afar, wishing they could be more present but unable to visit as often as they'd like. Traditional monitoring solutions are either intrusive (cameras, sensors) or miss the human element entirely. We asked ourselves: What if there was a warm, conversational companion that seniors could talk to anytime - and as a natural byproduct of those conversations, families could stay informed about wellbeing without being invasive?

AVA was born from this dual vision:

  • For seniors: A caring voice companion available whenever they want to chat, providing comfort and reducing loneliness
  • For families: Peace of mind knowing that if any health concerns arise during these natural conversations, they'll be alerted early

What it does

AVA is a voice-first AI health monitoring system designed specifically for seniors and their families. It consists of two seamlessly integrated interfaces:

For Seniors:

  • A simple, voice-activated companion accessible with a single tap
  • Natural, empathetic conversations about daily life, health, and wellbeing
  • Real-time audio responses that feel like talking to a caring friend
  • No typing, no complex interfaces - just conversation

For Families:

  • A comprehensive dashboard showing conversation history and health trends
  • AI-powered assessments across 5 critical health dimensions:
    • Physical Wellbeing (mobility, pain, energy)
    • Cognitive Function (memory, clarity, confusion)
    • Emotional Wellbeing (mood, anxiety, depression)
    • Social Connection (isolation, relationships)
    • Daily Independence (ADLs, self-care)
  • Severity-based alerts (Routine → Minor Concern → Moderate → High Concern → Urgent)
  • Actionable recommendations for intervention
  • Privacy-focused design (no intrusive transcripts shown to families)

AVA acts as an early warning system, detecting subtle changes in health status through natural conversation patterns, word choice, speech clarity, and emotional tone - all while maintaining the dignity and privacy seniors deserve.

How we built it

Architecture Overview

Architecture Diagram (Reduced)

Architecture Diagram - Reduced

Architecture Diagram (Complex)

Architecture Diagram - Complex

Frontend (React + Vite + Tailwind CSS):

  • Three-interface design: Landing page, Senior chat interface, Family dashboard
  • Real-time WebSocket connection for bidirectional audio streaming
  • Minimal, accessible UI with color-coded severity indicators

Backend (Python + FastAPI):

  • WebSocket server managing real-time audio streaming
  • Integration with OpenAI's Realtime API for natural voice conversations
  • Custom prompt engineering for empathetic, health-focused dialogue
  • Asynchronous conversation analysis pipeline

AI & Prompt Engineering (The Heart of AVA):

We invested significant effort in prompt engineering to make AVA's conversations feel natural while systematically gathering health information. This multi-layered prompting system is the core innovation of our project:

  1. Conversational Prompt Design:

    • Engineered sophisticated prompts that guide AVA to ask open-ended questions naturally embedded in conversation
    • Carefully balanced clinical empathy with strategic health observation
    • Designed conversation flows that organically elicit information about pain, mobility, memory, mood, and social engagement without feeling like an interrogation
    • Our prompts include context-aware follow-ups that adapt based on previous responses

  2. Intelligent Severity Scoring System:

    • Developed a weighted scoring algorithm that transforms conversational cues into quantifiable health metrics
    • Each health dimension (physical, cognitive, emotional, social, functional) uses:
      • Multi-factor concern detection with severity weighting (0.5-3.0 scale)
      • Contextual scoring that considers both explicit statements and implicit signals
      • Dynamic threshold mapping to clinical severity levels
    • The prompt engineering ensures consistent, reliable scoring across diverse conversation styles and senior communication patterns

  3. Multi-Stage Analysis Architecture:

    • Real-time conversation → Structured transcription → AI-powered health analysis → Clinical severity mapping → Family-friendly visualization
    • Each stage uses carefully crafted prompts optimized for specific outputs
    • Red flag detection system trained to identify urgent concerns requiring immediate attention

Database (Supabase/PostgreSQL):

  • Conversation storage with metadata
  • Analysis results stored as structured JSON
  • Efficient querying for dashboard aggregations

Deployment:

  • Frontend: Vercel (auto-deployment from GitHub)
  • Backend: Railway (containerized Python app)
  • Database: Supabase managed PostgreSQL

Challenges we ran into

  1. Prompt Engineering for Natural Conversations:

    • Balancing clinical information gathering with natural conversation flow
    • Preventing the AI from sounding like a medical questionnaire
    • Ensuring consistent severity scoring across different conversation styles
    • Solution: Iterative prompt refinement with extensive testing, weighted scoring algorithms

  2. Privacy vs. Insight Trade-off:

    • Families need health insights but seniors deserve privacy
    • Showing full transcripts felt invasive
    • Solution: Display only structured assessments and severity scores, not raw conversations

  3. Deployment Synchronization:

    • Coordinating WebSocket server (Railway) with static frontend (Vercel)
    • CORS configuration for production domains
    • Managing environment variables across platforms
    • Solution: Centralized configuration, Railway + Vercel CLI automation

  4. Severity Calibration:

    • Initial scoring was either too sensitive (false alarms) or too lenient (missed concerns)
    • Solution: Developed a weighted concern system with clear severity thresholds

Accomplishments that we're proud of

  1. Production-Ready Voice AI: Built a fully functional, real-time voice conversation system that feels natural and responsive
  2. Sophisticated Prompt Engineering: Developed a multi-layered prompting system that balances empathy with clinical observation
  3. Privacy-Conscious Design: Created a monitoring system that respects senior dignity while providing family peace of mind
  4. Beautiful, Accessible UI: Designed interfaces that work for both tech-savvy families and seniors who've never used apps
  5. End-to-End Deployment: Successfully deployed a complex full-stack application with real-time components to production
  6. Severity Scoring Algorithm: Implemented a clinically-inspired weighted scoring system that provides actionable insights

What we learned

Technical Skills:

  • Real-time WebSocket audio streaming in browser environments
  • OpenAI Realtime API integration and optimization
  • Advanced prompt engineering for clinical-grade structured outputs
  • Designing conversation flows that balance empathy with data collection
  • Full-stack deployment orchestration (Vercel + Railway + Supabase)
  • Audio processing with Web Audio API and PCM16 encoding

Product Design:

  • The importance of simplicity in interfaces for seniors
  • How to balance monitoring with privacy and dignity - seniors are not patients to be surveilled
  • The power of voice as the most natural, accessible interface for older adults

Prompt Engineering Mastery:

  • How to craft prompts that extract structured clinical data from unstructured conversation
  • Weighted scoring methodologies that translate subjective health signals into objective metrics
  • Context management and conversation memory in multi-turn AI interactions

What's next for AVA

Immediate Priority - Telephony Integration:

Our most exciting next feature is transforming AVA into a phone-accessible service:

  1. Inbound Call System: Seniors can call a dedicated phone number anytime they want to talk to AVA - no app, no screen, just pick up the phone
  2. Scheduled Check-ins: Families can schedule regular wellness calls where AVA proactively calls their loved ones at preferred times (e.g., daily morning check-ins)
  3. Emergency Callback: If AVA detects urgent health concerns, families receive immediate notifications with the option to call their loved one directly
  4. Multi-Number Support: Configure multiple family member phone numbers to receive alerts and updates
  5. Voicemail Integration: If a senior doesn't answer a scheduled call, AVA can leave a gentle reminder and try again later

This telephony feature solves a critical barrier: many seniors don't use smartphones or apps, but everyone knows how to use a phone. By making AVA accessible via traditional phone calls, we expand reach to the populations who need it most - isolated seniors with limited tech literacy.

Enhanced Conversational Intelligence:

  1. Persistent Conversational Memory:

    • Build long-term memory systems where AVA remembers personal details across conversations
    • Reference past discussions naturally: "How's your grandson doing? Last time you mentioned he started college."
    • Track recurring topics, hobbies, and relationships to make conversations feel genuinely personal
    • Memory-aware prompts that adapt based on conversation history and senior preferences

  2. Adaptive Prompt Engineering Based on Senior Feedback:

    • Implement feedback loops where AVA adjusts communication style based on senior responses
    • Detect when questions feel intrusive or uncomfortable and pivot gracefully
    • Learn preferred conversation topics and communication patterns per individual
    • Adjust pacing, question depth, and follow-up strategies based on engagement signals
    • Continuously refine prompts using real-world conversation data to improve empathy and effectiveness

  3. Advanced Health Metrics & Trend Analysis:

    • Longitudinal tracking: Visualize health scores over weeks and months to detect gradual decline patterns
    • Sentiment analysis: Track emotional tone trends to detect depression or anxiety early
    • ADL timeline tracking: Monitor mentions of daily activities (cooking, bathing, dressing) for functional independence patterns
    • Sleep and energy patterns: Correlate mentions of sleep quality with daytime energy and mood
    • Medication adherence signals: Detect indirect mentions of missed medications or confusion about prescriptions

  4. Multi-Language Support:

    • Expand to Spanish, Mandarin, Hindi, and other languages serving diverse senior populations

Clinical Integration & Validation:

  1. Healthcare Provider Dashboard:
    • HIPAA-compliant portals allowing doctors and nurses to access AVA assessments with patient consent
    • Integration with Electronic Health Records (EHR) systems

Built With

Share this project:

Updates