Memoria

Inspiration

The summer before my senior year of high school, I took care of my grandmother before she passed away. She had late stage dementia, so she didn't remember who I was, or much of anything for the fact of the matter. Seeing how that frustrated and scared her led me to this idea. There's so tens of millions of seniors developing alzheimer's and dementia around the world, and as all the money has gone to medication to slow the progress of the illness, not enough thought has been put into how to help them live everyday life.

What it does

Memoria is an agentic AI companion for Alzheimer's patients featuring:

• Realistic Avatar Interaction - A lifelike avatar that is always on and that patients can talk to naturally, powered by HeyGen and WebRTC streaming

• Perfect Semantic Memory - Every conversation is analyzed by multiple AI agents. These agents can extract meaningful facts, store them, retrieve similar facts, contact emergency contacts, search for medical advice, etc.

  • Has functionally infinite memory, filters and stores facts worth storing in a database and uses elastic search to access relevant context during live conversations
  • has in built timers that allow Memoria to proactively reach out with reminders (if you tell Memoria you usually take meds at 7pm in normal conversation, it will start reminding you so you don't forget)

• Proactive Safety System - Automatically warns about medication interactions and allergy risks

• Voice-First Design - No typing or complex apps to navigate—just talk

How we built it

Lots of AI help lol :) Multi-Agent Architecture:

• Classifier Agent - Determines if information is worth remembering
  
• Extraction Agent - Pulls out key facts (names, relationships, medications)
  
• Medical Safety Agent - Cross-references allergies and drug interactions
  
• Response Agent - Generates warm, patient-centered replies
  

Tech Stack:

• Frontend: Next.js 15, React 19, TypeScript, TailwindCSS
  
• Backend: FastAPI, Python, LangGraph for agentic orchestration
  
• AI: GPT-4o-mini, OpenAI TTS, semantic embeddings
  
• Memory: Elasticsearch with vector search
  
• Avatar: HeyGen LiveAvatar SDK, LiveKit WebRTC
  

Challenges we ran into

• Integrating real-time avatar lip-syncing with custom TTS audio
  
• Building a semantic memory system that retrieves relevant context without keyword matching
  
• Designing agent coordination so safety checks happen automatically without slowing responses
  

Accomplishments that we're proud of

• The avatar feels genuinely present—patients respond to faces better than text
  
  • Our safety agent caught a simulated allergy interaction in testing
    
  • Memory retrieval works semantically ("Where does my daughter live?" matches "Sarah moved to Seattle")
    

What we learned

• Agentic AI isn't just about calling tools—it's about orchestrating multiple specialized systems
  
• Voice interfaces need careful latency optimization
  
• Designing for vulnerable users requires extra care around tone and safety
  

What's next for Memoria

• Proactive check-ins via SMS when patients haven't interacted
  
• Caregiver dashboard with conversation summaries and alerts
  
• Integration with smart home devices for ambient reminders
  
• HIPAA compliance for healthcare deployment
  

Built With

• elasticsearch
• fastapi
• gpt
• heygen
• langgraph
• liveavatar
• python
• react