Encore: a taste-memory agent for dementia caregivers

When someone with dementia can no longer tell you what they love, Encore uses Qloo's taste graph to rebuild the songs, films and places of their youth, then learns from every session.

Comment

Try it in 60 seconds: open encore-qloo.fly.dev → Try Dorothy's session → Open today's session → Compare with a generic AI (same model, no Qloo). Then Start session and tap Upset on any item to see it stop and recover. Back on Dorothy's page, Rebuild live with Qloo shows the agent working call by call, and Engagement report shows what it has learned. Group sessions for care homes are at /facility.

Inspiration

I'm a medical doctor, and I've seen people with dementia and their families in my clinical work. Dementia takes the words before it takes the person. Caregivers are told that familiar music and films can bring someone back for a moment, but "play some oldies" is advice for a decade, not a person. A woman who grew up in Brooklyn in the 1950s singing along to Ella Fitzgerald, and a man from San Antonio raised on Pedro Infante and Trío Los Panchos, need completely different afternoons. Their caregivers usually know three or four things they loved. I wanted to turn those three or four things into a whole world, and to build it the way I'd want any clinical tool built: safe by default, honest about the evidence, and able to show why it suggests what it suggests.

Who it's for

  • Family caregivers. More than 55 million people live with dementia (WHO, 2021), a number projected to reach 78 million by 2030. Each of them has someone deciding what to do with the afternoon.
  • Memory-care activities staff. US nursing homes must offer an activities program built around each resident's interests (42 CFR §483.24(c)). Encore builds the session and a printable activity log.
  • What exists today. Personalised-music programs such as Music & Memory start from the songs a family can already name. Encore starts where that list runs out, and reaches past music into film, TV, books, food and places.

What it does

Encore is a taste-memory agent for family caregivers and memory-care staff.

  • Two-minute intake. First name, birth year, hometown, two to five things they loved, and anything to avoid. Every "they loved…" is resolved against Qloo, and when it's ambiguous the caregiver picks the right one (the 1960 Magnificent Seven, not the 2016 remake). Encore pre-selects the match from the person's era but never guesses silently.
  • A Taste Profile, built live. Encore works out the years the person was 10 to 30 (the "reminiscence bump", when memories are most vivid) and runs an agent over Qloo's taste graph: singers, films, TV and books from those years, plus food spots and places in their hometown. The caregiver watches every step, and every item says why it was picked: "Because Dorothy loved Frank Sinatra and Roman Holiday."
  • Daily sessions. The agent composes a 15-20 minute arc of 6-8 items (a warm opening song, variety in the middle, a calm closer). Prompts follow a clinical rule: recognition, not recall. They invite and name things ("This is Dean Martin. Shall we listen?") and never test memory ("Do you remember…?").
  • Session Mode. A calm, tablet-first screen: one big item, big type, one play button, no timers and no auto-advance. A separate caregiver bar records Lit up, Sang or talked along, Neutral or Upset. Upset stops the item at once and offers something they already enjoyed.
  • For memory-care facilities. An activities director picks a few residents and Encore builds one group session from their pooled seeds, ranking picks by how many residents they connect, with every pick saying whose favourites led to it.
  • Common Ground. A caregiver adds three of their own favourites; Qloo Analysis Compare finds the taste they share with their loved one and suggests things to enjoy together.
  • Their years, as a picture. A timeline of the reminiscence window with the films, shows and books Encore found placed at their years.
  • In their own language. Many people with dementia hold on to their first language longest, so when it isn't English, every caregiver prompt is also written in it ("Este es Pedro Infante. ¿Lo escuchamos juntos?") and checked by the same clinical rule.
  • Engagement report. What works for this person by kind and by Qloo genre, best moments, what's kept out, and a printable activity log for families and care teams.
  • It learns. Reactions update the profile deterministically. Songs they sang along to are promoted into the Qloo signal, upsetting items and their genre are excluded for good, and the agent expands the profile to v2 while you watch. A "What I learned" note explains the changes in plain words.

Why Qloo

Encore only works because of Qloo's cross-domain taste graph. A language model on its own gives the famous hits of the decade, repeats what you told it, and can't show where anything came from. The Compare with a generic AI tab runs the same intake through the same model with no tools and scores the two sessions side by side. Encore's picks are new finds by design (the seeds are excluded from every query), each one traceable to the person's favourites through Qloo explainability and linked to a real player, poster or map. The generic session mostly hands the seeds back, cites nothing, and in my runs asked the person to remember things in most of its prompts.

How I use the API:

  • /search resolves seeds across artists, films, TV shows, books, places and destinations, and resolves the hometown to a Qloo locality ID.
  • /v2/insights produces every recommendation, with signal.interests.entities (seeds, plus promoted favourites after each session), signal.demographics.age (55 and older), filter.release_year.min/max and filter.release_country for film and TV, filter.publication_year.min/max for books, filter.location (locality ID) plus filter.tags (cuisine tags) for food and places, filter.exclude.tags (war, horror, disaster, funerals, violence, the avoid list, and genres learned from upsetting reactions), filter.exclude.entities (excluded items and the seeds), and feature.explainability on every call.
  • Explainability powers every "Because they loved…" chip and the aggregate seed influence in the trace.
  • /v2/tags turns free text ("war footage", "fish and chips") into real tag IDs.
  • /entities fetches seed details: images, Spotify and IMDb IDs, and notable songs.
  • /v2/insights/compare (Analysis Compare) powers Common Ground: shared taste tags between a caregiver's favourites and their loved one's.
  • Explainability as attribution. In Facility mode, each pooled seed belongs to a resident, so the top explainability drivers say which residents a song connects.

How I built it

  • Next.js 16 (App Router, TypeScript, Tailwind 4), deployed on Fly.io, with Supabase Postgres for profiles, items, sessions, reactions, agent traces and a private Qloo response cache.
  • Vercel AI SDK 7 with Claude. The profile build is a tool-calling loop (qloo_insights, qloo_tags_search, save_profile) streamed to the browser as NDJSON. Session composition uses structured output validated with Zod.
  • Guardrails around the model. The server fills era, country, location, age and exclusions for every Qloo call (the model can only override them). Any entity ID that didn't come back from Qloo in that run is rejected. Prompts that test memory are replaced. Every LLM step has a deterministic fallback.
  • Learning is plain code. sang_along +2, lit_up +1, upset means excluded forever; the LLM only explains the result.
  • No login. An anonymous workspace cookie. The three example people are read-only templates, built by a script against the live API; opening one copies it into your workspace, so judges never see each other's reactions.
  • Illustrations were generated for the project in one mid-century print style. No photos of real patients or caregivers appear anywhere; artist and film images come from catalog data.

Challenges I ran into

  • Silent failures. Qloo ignores parameters a type doesn't support and still returns plausible results. A release-year filter on artists or books changes nothing. I wrote a per-type parameter whitelist that throws in development, and found the right params by probing (books use publication_year; release_country needs "United States", not "US"; location radius is in meters).
  • Music has no era filter, so I filter artists on their career start year after fetching.
  • Places have no era filter either. Restaurants and landmarks come from the hometown as it is now, not as it was in 1955. Finding places that were open in the person's youth is next on the list.
  • Search is not era-aware. That made the candidate picker essential rather than optional.
  • Coverage varies by region. My Indonesian persona (Koes Plus, Titiek Puspa) had good music and food but thin film and TV data in the reminiscence window, so I replaced her with a Liverpool persona and list Southeast Asian coverage as future work.
  • Clinical tone. Writing prompts that invite instead of quiz took explicit rules in the prompt and a code check.

Accomplishments that I'm proud of

  • The agent visibly adapts. It broadened a food search that came back empty, and when Roberto's food results weren't Mexican it retried with a cuisine tag on its own.
  • Every recommendation is traceable to the favourites it came from, through Qloo explainability.
  • The before/after: Profile v1 → v2 shows which favourites now steer the taste graph, what is out for good, and the new finds they led to.
  • Bilingual prompts that stay clinically safe: the never-test-memory rule is enforced in code for English, Spanish, French, Italian, Portuguese, German and Indonesian.
  • Session Mode is built for the person, not the demo: large type, large targets, no timers, and a calm path out when something upsets them.

What I learned

The taste graph is most powerful as a conversation partner for an agent rather than a lookup: seeds in, explainability out, reactions back in as new seeds. And as a doctor, I think the right product for dementia care is honest about evidence. Music-based activities probably lift mood slightly and may help social connection, but probably don't reduce agitation (Cochrane, 2025). So Encore is a connection and mood tool, not a treatment, and it says so.

What's next

  • A facility pilot: Facility mode already builds group sessions; next is rosters, scheduling and per-bed pricing for memory-care operators.
  • Era-aware places: favour long-standing spots that were open when the person was young.
  • Southeast Asian catalog coverage, starting with Indonesia, where I practise medicine, and caregiver prompts in more languages.
  • Exact songs and trailers through YouTube and TMDB, and sharing a session between family members.

Built With

Share this project:

Updates

Submission history