CareProxy – A Voice-First Health Companion for Elderly Care

Inspiration

Many elderly people live alone, managing chronic conditions, minor symptoms, and sudden health concerns without immediate support. When something feels “off,” they often face three problems at once:

• They don’t know how urgent the situation is
  
• They struggle to explain symptoms clearly, especially under stress
  
• Their caregivers and physicians receive fragmented or delayed information
  

From personal experience watching family members navigate these moments, we realized that the gap isn’t medical knowledge, it’s calm, structured coordination at the moment of concern.

CareProxy was inspired by a simple question:

What if an elderly person could speak naturally, and that conversation itself became structured, actionable medical context for caregivers and physicians?

---

What It Does

CareProxy is a voice-first healthcare companion designed for elderly individuals and people living alone.

It listens, asks the right questions, remembers past interactions, and knows when to escalate.

---

Core Flow (MVP)

1. Voice Conversation

• A user speaks naturally about a health concern
  
• CareProxy asks clarifying questions (severity, duration, symptoms, history)
  
• No diagnosis, only careful information gathering
  

2. Triage & Decision

• The agent classifies urgency:
  • Emergency
  • Urgent
  • Monitor
    
• Red flags trigger escalation logic
  

3. Automatic Handoff

Generates:

• A caregiver-friendly summary
  
• A physician-ready clinical report
  

These can be shared immediately if escalation is needed.

4. Health Memory

• Non-urgent conversations are stored as structured memory
  
• Over time, this builds a longitudinal medical history for caregivers and clinicians
  

CareProxy turns unstructured voice into clarity, continuity, and confidence, in seconds.

---

How We Built It

• LiveKit Agents Framework
  Low-latency, real-time voice interaction (STT, TTS, turn-taking)

• OpenAI (Whisper + GPT-4o)

  • Whisper for speech-to-text
    
  • GPT-4o for structured triage reasoning and report generation
    

• Deterministic Prompting
  Designed to:

  • Ask questions before recommending action
    
  • Avoid diagnoses
    
  • Produce consistent, staff-readable outputs
    

• Arize Phoenix (Observability)
  Used to trace conversations, debug decision paths, and improve prompt reliability

• FastAPI Backend
  Stores conversation memory and exposes report APIs

• Simple Web Dashboard
  Displays urgency, summaries, and reports for caregivers

The system is intentionally focused:

voice → understanding → action → memory

---

Challenges We Ran Into

• Making voice interactions feel calm and human
  
• Managing long conversations without losing clinical relevance
  
• Coordinating async voice, LLM reasoning, and persistence
  
• Staying focused on MVP scope without overengineering
  

---

Accomplishments We’re Proud Of

• Built a fully working voice-first healthcare agent
  
• Implemented real triage logic, not just chat
  
• Generated distinct caregiver and physician reports
  
• Created a longitudinal memory system
  

---

What We Learned

• Voice works best when paired with structure and intent
  
• Asking the right questions matters more than fast answers
  
• Healthcare tools must reduce cognitive load
  
• Observability is critical for trustworthy agentic systems
  

---

What’s Next for CareProxy

CareProxy is designed as a foundation for long-term elder care coordination:

• Browser-based LiveKit voice integration
  
• Vector-based memory for richer longitudinal reasoning
  
• Secure sharing with caregivers and clinicians
  
• Chronic condition tracking
  
• Proactive check-ins and pattern detection
  

Long-term goal:
To make CareProxy a quiet, trusted presence, always available, never overwhelming.

Built With

• arize
• livekit