Memory Vault, Bee-Your Assistant

Dementia means asking the same questions all day. Bee, your Assistant remembers the day via the Bee wearable, answers every time with patience, and calls in the caregiver when needed.

Comment

Memory Vault

A companion for the moments you need help remembering, even when no one else is there. Always present, always listening, always available. Built on the Bee wearable, using a Strands agent on Bedrock, gated by a local Strands decider model for memory recall and governed at runtime with Hooks and Steering, so it answers the same question warmly every time and never invents.

Track: Bee (Wearable AI) Mini challenge: AWS Builder (Amazon Bedrock + Strands Agents SDK + strands-decider) Repo: https://github.com/Gopsee/Memory-Vault (MIT)

Inspiration

Walt asked what day it was three times before breakfast. Margaret answered every time. She always does.

Walt and Margaret are fictional, but their morning is real for millions. More than 55 million people live with dementia, and most are cared for at home by family. Recent memories fade first: who called, what day it is, whether the pills were taken. The people who love them answer the same questions all day, every day, but the hardest part is answering the fifth time as warmly as the first.

"What did we decide?" "When was that appointment?" "Did I promise to call?" Our days are full of conversations and details we expect ourselves to remember. We leave a conversation meaning to follow through, then lose some details we need, and finding it again means searching messages, retracing the day, or asking someone else. A busy professional wants to revisit a commitment. Someone running a household wants to recall a plan.

And memory loss is not one condition, and it is not only Walt's. Amnesia after a head injury or surgery. Chemo brain. Long COVID. Depression and grief. Plain ageing. The same things go first: recent events, names, what comes next. Some people are cared for by family, some by friends, and many have no one present to help. A person who lives alone and forgets the way home on a walk has nobody to ask.

Bee already hears the day. We asked: what if the thing that hears the day could hold it for you, with the patience of a machine and the manners of a good carer, and be there whether or not anyone else is? Not a productivity assistant that is "concise and actionable", but a companion that never says "you already asked", never quizzes, never argues, never makes something up, and knows who to call when you say you fell.

That is a hard brief for a language model, and especially for a small, fast, cheap one. So the project became as much about governing the model at runtime as about prompting it.

What it does

Memory Vault is a web and mobile app that sits on top of a person's Bee account.

  • Chat. A warm companion that answers from today's Bee conversations and from long-term memory. "Did Daniel call?" is answered from the transcript Bee actually recorded. "What day is it?" is answered the same gentle way every time it is asked. "I think I'm lost" gets the emergency contact's name and number, right away.
  • Highlights. "You like to know" cards: the questions the person asks again and again, answered ahead of time from today's date, memory and today's Bee summary, before they are asked. Below them, Bee's own insights, takeaways and action items.
  • Conversations. Every Bee conversation with its full transcript, plus "Learning about you" cards (facts heard once, waiting to be heard again) and "Questions you ask" (repeats being counted toward a card).
  • Profile. The long-term memory a caregiver, a friend, or the person themselves can read and edit: profile, people, routines, preferences, facts. Protected facts (the emergency contact, the diagnosis, a death) can never be overwritten from chat. Plus Delete my data.

The everyday needs behind the app are simple:

  • "What did I agree to do?" Revisit recorded conversations, recover commitments, manage the related to-dos.
  • "When did we say we would meet?" Find it in the transcript instead of reconstructing the conversation.
  • "What day is it?" Orientation, with patience, every time.
  • "What medicine am I supposed to take?" Surface recorded routines and configured reminders, without inventing instructions or treating a routine as proof that a dose was taken.
  • "I'm out walking and I can't find my way home." Recognise distress and point the person to their saved contact.

The responsive web app brings five sections together: Highlights, Chat, Conversations, To-dos and Profile, with large type, clear navigation and prominent answer cards. The app answers through chat and syncs Bee recordings periodically. It does not false detect emergencies, give directions home or place calls, and the distress flow assumes an emergency contact has been configured.

In the demo, Walt asks what day it is three times, asks whether Daniel called (yes, about Leo's Saturday Little League game, straight from the Bee recording), reports that "someone from the bank wants my card number" (the companion stays calm and tells him to hang up and call Margaret at her number), and asks whether his brother Frank, who died three years ago, is coming on Sunday. It does not repeat the news. It says: "Tell me about Frank. I'd love to hear a happy memory."

How we built it

Four engines share the work.

Engine Role Cost
Regex checks in code Memory mention, safety concern, small talk, states-a-fact Instant
strands-decider 2B (local, CPU) Is this message on topic? Which stored memories does it need? No network, no cloud cost
Amazon Bedrock, Nova Micro via Strands Agents The companion itself, plus fact extraction, topic grouping and daily answers One cloud call each
Hooks and Steering in code (Strands harness) Intercept every tool call and every reply and check them against the care rules Instant, deterministic

The data path. The bee CLI syncs every Bee conversation with its full transcript into local Markdown, at most once a minute on demand and every five minutes in the background. Today's conversations, each with its summary, are placed in the agent's pinned opening message on every turn. The Bee MCP server (bee mcp serve-http, Streamable HTTP) gives the agent facts, to-dos, daily summaries and search. The mobile page reads and writes the Bee account through the same CLI, so the Bee token never reaches the browser.

The agent. A Strands Agent on apac.amazon.nova-micro-v1:0 with the Bee MCP tools, a conversations tool over the local transcripts, and remember / recall tools over a three-tier memory. A custom ConversationManager offloads old exchanges by topic once the chat passes six exchanges. Hooks coerce tool-argument types, ask Allow / Decline before any write to the Bee account, log every turn and report token usage. A Steering plugin judges every tool call and every reply.

The front end. A FastAPI server with a single agent worker thread; each chat message becomes a job the page polls, and a write tool pauses the job until the person answers Allow or Decline. A web chat and a mobile view (Highlights, Chat, Conversations, To-dos, Profile) with big type and plain language.

The technical ideas we are proudest of

1. Runtime governance with Hooks and Steering: care rules in code, not in hope

The Strands harness gives two layers of control over a running agent, and Memory Vault uses both.

Hooks attach to the agent's lifecycle events. BeforeToolCallEvent hooks coerce string arguments to the types the tool schema expects (small models send "False" and "100") and pause the turn for Allow / Decline before any tool that creates, updates or deletes in the Bee account. AfterToolCallEvent and AfterModelCallEvent hooks log every call and record token usage per chat job.

Steering judges intent. A SteeringHandler returns Proceed or Guide with a reason the model reads.

Before a tool runs. A remember note must be grounded: at least half of its content words must appear in one message the person actually sent. A question is not a fact. replaces must name an existing memory item exactly. Protected facts, the emergency contact, the condition, a death, anything the caregiver set, can never be overwritten from chat. recall is only for a topic offloaded this session. No Bee lookup to answer a greeting. A broken rule cancels the call and tells the model what to do instead.

After the reply. Code checks style (plain text, at most one question, short sentences, under 120 words), kindness (no "you already asked", no quizzes, no blunt corrections), and four safety rules: never contradict the person about what happened today; never deliver death news or say whether a loved one who has died will visit, under deceased_policy: gentle_redirect; always point distress, being lost, or a scam to the emergency contact by name and number; never mention a medicine or dose that is not in memory or the chat. Problems trigger a rewrite with every fix in one message. After three rewrites a style problem is let through, but a safety problem replaces the reply with a fixed, known-safe sentence, personalised from the profile: "I'm here with you. Please call Margaret now. The number is (650) 555-0142."

An unsafe reply therefore has no path to the screen. The same checks run on the daily "You like to know" answers.

2. A 2B local model decides what the cloud model gets to see

strands-decider answers yes/no questions with calibrated probabilities and runs on a laptop CPU. We ask it one question per message: is this something a personal assistant for their Bee wearable should handle? The criteria are written for this user, so a question about a person or interest in memory counts as on topic even if it looks like trivia, while homework and general knowledge do not. Below P = 0.5 the message gets a fixed reply and zero Bedrock calls. Regex short-circuits run first, so "Hi" is answered in about two seconds and a fall or a scam is never filtered out.

3. A memory retriever built on strands-decider, not on embeddings or tool calls

Long chats are offloaded by topic into session memory. The usual design hands the model a recall tool and hopes it remembers to call it. Small models often don't.

Instead, we build one yes/no question per stored topic, each carrying that topic's summary, and evaluate all of them in a single strands-decider call. Topics with P ≥ 0.5 (at most two) are rendered as notes and placed in the opening message before the agent runs. The agent gets exactly the memories the message needs, with a probability attached, with no tool call, no vector index and no cloud round trip. Steering then blocks any recall for a topic that was already preloaded, so the model cannot waste a turn re-fetching it.

4. Memory that needs evidence

A memory companion must not turn one confused remark into a permanent fact. So memory has three tiers: session, intermediate, persistent. At the end of a session, Bedrock extracts facts from the person's own words only (never from what the wearable overheard, never from the assistant's replies), each fact must be grounded in a sentence they typed, and each counts once per session. Heard in two sessions, it is promoted. Otherwise it stays as a "Learning about you" card that can be settled with Remember now or Forget. Questions work the same way: asked three times in five days, a question earns a "You like to know" card and is answered before it is asked.

5. Live Bee data, no tool call needed

Because today's recorded conversations and summaries are already in the opening message, "Did Daniel call?" is answered correctly on the first model call, from Bee's real transcript, with no tool round trip. A small model that rarely calls tools still gets the day right.

What makes it original

The central idea is continuity of memory: carrying context from everyday life into the moments anyone needs help remembering, including when no other person is beside them. Three implementation choices work together to make that real:

  • Retrieval happens before generation, with a small local decision model choosing what the cloud model sees.
  • Memory earns persistence, with an intermediate review stage and protection for the facts a caregiver set.
  • Behaviour is governed at runtime, with Hooks and Steering enforcing bounded correction and a known-safe fallback instead of relying on a compassionate prompt.

"You like to know" closes the loop: the conversation teaches the interface what to make easier to find tomorrow. Repetition stops being a symptom and becomes a signal.

Why it is reliable

  • Deterministic safety floor. Safety rules end in a reply that is known to be safe.
  • Grounding, not trust. Every saved fact is traceable to something the person said. Protected facts are guarded twice, in steering and in the memory merge.
  • Bounded loops. Three rewrites and six tool blocks per turn, then a decision, so a stubborn model cannot spin.
  • Graceful degradation. A failed Bee sync leaves the local copy usable; a failed topic grouping still offloads; a failed extraction saves the notes only.
  • Crash safety. Atomic writes, .prev.json undo copies, and consolidation of a session left unsaved by a crash.
  • Tested with a simulated customer. simulate.py plays seven scripted personas against the real assistant in scratch memory and checks earlier replies, tool calls and saved memory, including a correction with no shared words and a long day that forces offloading and recall. The conversations and chat responses are real-time.
  • Inspectable. Session logs record the topic-screen probability, recall picks and scores, every tool call and status, every steering intervention, and token usage.
  • Local first. Memory, logs and transcripts stay on the computer; off localhost the server refuses to start without a password.

Research, compliance and privacy

The needs behind Memory Vault follow established dementia-care guidance. The National Institute on Aging describes wandering and getting lost and recommends practical aids such as timers and written reminders for managing medicines. These sources support the problem; they do not establish that Memory Vault improves clinical outcomes.

Memory Vault also handles health-adjacent data about vulnerable adults, so we researched the US rules that apply (HHS guidance on HIPAA covered entities among them) and designed for them. Not legal advice; engineering diligence.

  • COPPA. The app is for adults and the people who help them, and says so. It is not child-directed. Bee may record grandchildren near the wearer, but the app reads text summaries only, never stores audio, and counts chat questions only, never recordings.
  • HIPAA and the FTC Health Breach Notification Rule. A consumer app a family runs on its own is generally not HIPAA-covered, but a memory-care facility or home-care agency deploying it would turn the stored data into protected health information. So the design starts from the safeguards HIPAA expects: demo data only, the gate and retriever run locally so nothing leaves the machine unless a model call needs it; secrets in environment variables; a password the moment the server leaves localhost; a per-session log of every turn and tool call; tracked questions that expire after five days; and one-action Delete my data.
  • BAAs. Amazon Bedrock is HIPAA-eligible once the AWS BAA is accepted and a HIPAA-eligible model and region are pinned (the apac. profile routes across regions). strands-decider runs locally, so no third party is involved. Bee is a consumer product with no known BAA for Bee data; that is the honest blocker for a clinical deployment, and we say so rather than paper over it.

Challenges we ran into, and how we solved them

  • Small models skip instructions. Nova Micro is fast and cheap, but it under-weights a long system prompt once tools are attached, writes its reasoning inline as <thinking> tags, and sometimes prints "Tool call: remember {...}" as text instead of calling the tool. To solve this, every one of those became a code check: the care rules moved into Steering, the memory moved into a pinned opening exchange the model does respect, replies are cleaned, and offloaded exchanges are rendered as notes rather than transcripts so the model does not copy the format.
  • Not calling tools. The model rarely calls recall or looks up a conversation on its own. The decider-based preload and the per-turn conversation index solve this by putting what is needed in front of the model instead of asking it to fetch.
  • Bee summaries speak in the third person. Bee's summaries call the wearer "the man", "the father" or "the husband", and a conversation title alone ("Family Care During Memory Decline") is too vague to tell that Daniel called. We therefore include each conversation's summary, not just its title, and tell the model it took part in all of them.
  • Conversations still being recorded. Bee may not have saved a conversation yet. The companion must say "I can't see it yet", never "that didn't happen". The contradict rule in Steering enforces this.
  • Grief and policy. Whether to repeat the news of a death is a judgement for the family, not the model. It became a caregiver-set deceased_policy in memory, enforced by code.

Accomplishments we are proud of

  • A companion whose safety floor is deterministic, built from a general-purpose agent harness's Hooks and Steering rather than a bespoke rules engine.
  • Using a 2B yes/no model as a calibrated memory retriever, in one call, with no embeddings.
  • A memory system where facts and standing questions earn their place with evidence, and the person or their caregiver stays in control.
  • A demo on live Bee data that handles a scam call, a repeated question and a question about a loved one who has died, all gracefully.
  • A design that took COPPA, HIPAA and BAA requirements into account from day one, and is honest about what a clinical deployment would still need.

What we learned

Prompts describe behaviour; they do not guarantee it. Everything a carer would never forgive had to become a check in code with a known-safe fallback, and the Strands harness turned out to have exactly the right seams for that: Hooks for mechanics, Steering for intent. The surprise was how far a small local decider can go when it is asked precise yes/no questions: it can gate, it can retrieve, and it costs very little per call. Relevant, fast and cheap is a design principle, not just a cost saving.

What's next

  • Evaluate with real users: general users, people living with memory loss, caregivers and domain experts, and measure retrieval quality and the failure cases of the care checks systematically.
  • A support path for people with no caregiver or emergency contact configured.
  • Personalisation of tone and assistance, so a busy professional and a person with dementia each get the right companion rather than the same one.
  • Speak the reply and accept voice, so no one has to type, and so a person lost on a walk can just talk.
  • Consent-based location from the phone, so "I'm lost" can include where you are when it tells you who to call.
  • Answer cards on Echo and Fire TV, so "You like to know" is on the kitchen screen, not only the phone.
  • A helper mode: a concise, factual view of the same memory for Margaret, or for a friend who checks in, with a daily digest of what was asked and how the day went.
  • Source tags on every fact (person vs helper), and a confirmation flow for things learned from Bee's own facts.
  • Extra gentleness after sunset, when confusion often worsens, using the time of day already in the prompt.
  • The remaining steps toward a deployment a care provider could use: encryption at rest, HTTPS by default, per-user logins and an access log, a retention schedule, a pinned HIPAA-eligible region, and the open question of a Bee BAA.

Built with

Bee CLI, Bee MCP server (Streamable HTTP), Strands Agents SDK (agent, Hooks, Steering plugin, ConversationManager, MCP client, agent skills), strands-decider 2B (local CPU), Amazon Bedrock (Amazon Nova Micro), Python, FastAPI, Pydantic, HTML/JS.

Built With

Share this project:

Updates

Submission history