Masccot
Architecture
Semantic Recall
Semantic Evidence
Reminder Confirmation
Morning Recall

Project Memoria: The Dementia Assistant

Inspiration

When someone with dementia forgets where they left their keys, what they did an hour ago, or what they talked about with their son, that gap turns into stress for them and for the caregiver. We wanted an assistant that does not just answer when asked, but remembers for the patient and speaks up at the right moment.

Our earlier version could recall memories and detect falls, but it was reactive and its memory only ever grew. Trivial events piled up in the index and slowed every answer. This rebuild fixes that: memory now curates itself, and the proactive side is fast enough to actually help.

What it does

Memoria turns quiet home monitoring into grounded memory help. It runs entirely on Qwen Cloud: qwen3.7-plus for reasoning and routing, text-embedding-v4 for semantic memory, qwen3-vl for vision and spatial grounding, and qwen3-asr-flash for transcription.

Self-organizing memory. A cleanup agent scores each memory's importance, folds old trivia into day summaries, and keeps safety-critical memories forever.
Fall detection. A custom YOLO model watches the room feeds and emails caregivers the moment a fall is confirmed.
Grounded chat. Questions answered from stored evidence, never guessed: "What was I doing in my bedroom last night?"
Finding objects. qwen3-vl-plus locates and highlights items in a snapshot: "Where is my water bottle?"
Semantic recall. Fuzzy questions grounded in real transcripts: "What did I talk to my son about?"
Proactive guidance. The agent starts the conversation: safety warnings, morning reports, and reminders that fire on the right moment ("don't forget your water bottle" as the patient heads out).

How we built it

The main lesson from the old version was that a memory agent's quality lives in its memory system, not its model. So we built three tiers.

The Memory Cleanup Agent. Every event is scored 0 to 1 for importance at ingest, and safety events are pinned so they never fade. A background agent then finds old, low-importance, unpinned events, groups them by day and room, and folds each group into one factual day summary. The summary goes into ChromaDB and the raw events leave the vector index, but nothing is deleted from MongoDB; the originals are marked consolidated and kept. The index stays small and fast instead of drowning in trivia. The sweep is idempotent and safe to re-run.

Scored recall. Because every memory carries importance, retrieval ranks by more than similarity:

final_score = similarity * exp(-age_days / half_life) * (1 + importance)

Fresh, important, and pinned memories win; stale trivia decays out. The ranked evidence is packed into a fixed token budget, so the model reasons over the best evidence, not the most. A judge step decides whether the hits are strong enough to answer, need timeline grounding, or should admit uncertainty.

Working memory. A tiny block of active profile facts and live reminders is injected into every prompt through plain indexed reads in milliseconds, never embedded, never searched. This is why a reminder fires the instant a matching camera event arrives, and why "remind me" confirms with no second model call.

Proactive channel. Messages are created the instant something happens: a stored safety alert, the first sighting of the day, a due reminder, or a camera event that matches a reminder's room, time, and behavior. Web Push wakes the installable PWA even when it is closed, and a single atomic poll delivers each message exactly once.

A qwen3.7-plus router classifies each query (object, time, semantic, reminder, general) and routes it to the right agent. Recorded videos reach Qwen video understanding through a presigned Alibaba OSS URL, with a Gemini fallback so ingestion never blocks. The backend is FastAPI; the client is a React PWA.

Challenges we ran into

The real tension was intelligence versus speed and honesty. A proactive assistant that is too eager becomes noise, too slow is useless, and one that hallucinates is dangerous. Making reminders feel instant meant cutting an LLM call out of the hot path. Making delivery reliable meant an atomic pending-to-delivered step so a warning is never double-sent or dropped. Making recall trustworthy meant a judge that can say the evidence is too weak, which is harder than writing a fluent guess.

The cleanup agent had to be non-destructive, idempotent, and aware of what to protect, which is why safety events are pinned and skipped by every sweep.

Accomplishments that we are proud of

Memory that cleans up after itself: lean index, instant reminders, nothing important lost.
A system that admits uncertainty instead of inventing details.
Proactive delivery that reaches the patient even with the app closed, exactly once.
MongoDB as the single source of truth, with ChromaDB always rebuildable from it.

What we learned

A memory agent's intelligence is in its architecture, not its size: what to trust, what to distill, what to protect, and what to keep instantly at hand. The jump from a search tool to a proactive companion came from organizing memory, not from a bigger model.

What is next

Voice-first interaction with qwen3-asr-flash and qwen3-tts-flash.
A caregiver dashboard of derived context, never raw surveillance.
Geofence check-ins when the patient leaves a safe zone.
Smarter event-reminder matching learned from the patient's own routines.
One-command Alibaba Cloud deployment for home setups.

Built With

Qwen Cloud (qwen3.7-plus, text-embedding-v4, qwen3-vl-plus, qwen3-vl-flash, qwen3-asr-flash), Python, FastAPI, MongoDB, ChromaDB, React, Vite, Web Push, Alibaba Cloud OSS, Ultralytics YOLO, OpenCV, Gemini (fallback).