Memora

checkcheck

Comment

Inspiration

Watching a loved one struggle with memory loss is heartbreaking. For those affected, it turns everyday life into a series of small, exhausting challenges: forgetting familiar faces, losing track of conversations mid-sentence, misplacing essential items, and needing constant reassurance about where they are and what’s happening. We were inspired to create a companion that could gently whisper helpful reminders, like having a caring family member always by their side, helping them navigate conversations, recognize the people they love, and guide them through their everyday lives.

What it does

Memora is an AI-powered memory companion designed for people with memory difficulties and their caregivers. It:

  • Recognizes family members in real-time using facial recognition
  • Whispers helpful context when someone approaches, such as their name, relationship, and what they last talked about
  • Captures and extracts memories from conversations automatically, building a searchable timeline of meaningful moments
  • Monitors for safety hazards by detecting allergens and tracking medication to prevent accidental overdoses
  • Provides a caregiver dashboard to manage family circles, add notes, and review the memory stream

How we built it

  • Backend: FastAPI with SQLite, handling WebSocket streams for real-time video processing
    • Face Recognition: DeepFace with Facenet embeddings, supporting multiple detector backends (RetinaFace, MTCNN, OpenCV)
    • AI/LLM: Perplexity AI for generating natural whispers, extracting facts from conversations, and classifying memories by topic
    • Voice: ElevenLabs for natural text-to-speech with pyttsx3 as a fallback
    • Vision API: Overshoot for advanced hazard detection in Sentinel mode
    • Frontend: Vanilla JavaScript SPA with WebSocket-powered HUD displaying real-time recognition cards and transcription

Challenges we ran into

  • Reliable person detection: Face recognition can fail in poor lighting, so we built a dual-mode system with color-based detection as a fallback
    • Context management: Knowing when a conversation ends and a new one begins required careful state tracking with timeout-based resets
    • Real-time performance: Streaming video frames over WebSocket while running face embeddings demanded optimization and graceful degradation
    • Balancing accuracy and simplicity: Making the system work for non-technical caregivers while maintaining robust AI features

Accomplishments that we're proud of

  • Graceful fallback architecture: Every critical feature has multiple fallback layers (face to color detection, ElevenLabs to pyttsx3, Perplexity to template generation)
    • The whisper system: Generating warm, natural reminders that feel like a caring family member rather than a robotic assistant
    • Sentinel Mode: Real-time hazard detection with medication tracking that resets daily, which is a genuine safety feature for vulnerable populations
    • Caregiver-first UX: A dashboard that makes it simple to enroll faces, add family members, and review memories

What we learned

  • Face recognition in the browser is challenging, andserver-side processing with WebSocket streaming proved more reliable
    • LLM prompts for fact extraction need careful engineering to avoid hallucination while capturing meaningful details
    • Building for vulnerable populations requires thoughtful defaults, clear fallbacks, and designs that account for varying technical ability
    • Real-time applications benefit enormously from WebSocket architecture over polling

What's next for Memora

  • Purpose-built wearable kit: a lightweight chest-mounted (or glasses-mounted) camera paired with a discreet earpiece for all-day, hands-free “whispers” without pulling out a phone
    • Mobile companion app for caregivers to receive alerts and add notes on-the-go
  • Voice-activated interactions so users can ask "Who is this?" without needing to look at a screen
  • Multi-language support for families who speak different languages
Share this project:

Updates