Madeleine

• 

• 

• 

• 

Inspiration

1. The Human & Clinical Reality of Dementia

Over 57 million people worldwide currently live with dementia (World Health Organization), with nearly 10 million new cases every year. For families and care-home staff, one of the most heartbreaking moments is visiting someone you love and running out of ways to connect. Small talk ("What did you have for lunch?" or "Do you remember who came yesterday?") relies on short-term episodic memory — the very first cognitive system Alzheimer's disease destroys. Asking those questions inadvertently turns a loving visit into a stressful memory test, triggering anxiety, withdrawal, or agitation.

2. The Neuroscience: Why Music & Youth Endure

Two foundational discoveries in cognitive neuroscience and gerontology anchor Madeleine:

• Preservation of Musical & Cultural Memory: Landmark neuroimaging research (Jacobsen et al., "Why musical memory can be preserved in advanced Alzheimer's disease", Brain, 2015) demonstrated that the brain regions responsible for encoding long-known music — specifically the caudal anterior cingulate and ventral pre-supplementary motor area — show some of the lowest rates of cortical atrophy and amyloid disruption in Alzheimer's disease. A person who can no longer recognize a relative may still sing every word of a song from 1958.
• The "Reminiscence Bump" (Ages 10 to 30): Autopilot autobiographical memory is not distributed evenly across life. Psychologists have long established the reminiscence bump: human beings form their deepest, most durable identity memories during adolescence and young adulthood (roughly ages 10 to 30). The records we spun at seventeen, the movie stars we admired at twenty-two, and the neighbourhood soda fountain we visited on Saturdays remain accessible long after recent decades fade.

3. The Proustian Spark — and Why Plain Chatbots Fail

Madeleine takes its name from Marcel Proust's In Search of Lost Time, where the taste of a single scallop-shaped madeleine cake dipped in tea brought his entire childhood in Combray rushing back involuntarily.

Reminiscence therapy works — but only when it is personal. Families usually remember only three or four fragments ("Grandma loved Ella Fitzgerald, Roman Holiday, and Brooklyn egg creams"), and busy care-home staff have even less. When you ask a standard LLM or chatbot about someone born in 1942, every single person gets the exact same stereotypic list: Elvis Presley, Casablanca, I Love Lucy, Marilyn Monroe. That is a decade, not a person. A seamstress in Brooklyn who loved vocal jazz and French couture lived in a completely different cultural universe than a railway clerk in Bombay who loved Kishore Kumar and Sunday matinees, or a teenager in Liverpool who queued outside the Cavern Club.

An LLM knows history, but Qloo knows taste. We built Madeleine to bridge those three or four family-remembered fragments into a rich, 8-domain, era-accurate cultural world using Qloo's graph of 250+ million cultural entities.

---

What It Does

Madeleine transforms a 60-second family intake into a living, adaptive reminiscence ecosystem designed for two distinct tempos: a rich Studio for families and care coordinators, and a calm, distraction-free Session Player for the person living with dementia.

1. Low-Friction, Trauma-Aware Family Intake (/new)

• Voice or Text: Family members can type or use built-in Web Speech API dictation to speak naturally about their loved one — often much easier when memories come in a rush.
• Core Signals: Name, birth year, hometown (where they grew up), current city (for localized outings), primary language, a few things they loved, brief life story notes, and Things to Avoid (trauma, hospitals, bereavement, or distressing historical events).
• One-Click Cross-Cultural Personas: Four pre-configured, culturally diverse fictional people let judges test Madeleine immediately:
  • Rose (born 1942, Brooklyn → Boca Raton): Seamstress who copied Audrey Hepburn's dresses; loved Ella Fitzgerald, Frank Sinatra, Roman Holiday, and egg creams.
  • Raj (born 1950, Bombay → Leicester): Railway clerk at Victoria Terminus; loved Kishore Kumar, Mohammed Rafi, Sholay, Irani cafés, and Marine Drive.
  • Margaret (born 1947, Liverpool → Manchester): Typist at the Liver Building who danced at the Cavern Club; loved The Beatles, Cilla Black, Doctor Who, and Blackpool summers.
  • Carmen (born 1955, Mexico City → Los Angeles): Primary school teacher and choir singer; loved Pedro Infante, José José, Rocío Dúrcal, El Chavo del Ocho, and Xochimilco (generates full Spanish-language sessions).

2. Live "Agent Theater" Streaming UI

• Instead of hiding behind a loading spinner, Madeleine streams every thought, tool invocation, Qloo query, latency metric, and entity resolution live over NDJSON so families and judges can watch the Curator and Designer agents reason through Qloo's taste graph in real time.

3. The Cultural World & Studio (/profile/[id])

• Interactive 3D Memory Carousel: Built with React Three Fiber (@react-three/fiber & @react-three/drei), presenting the person's cultural world as tangible objects floating in warm projector light.
• Vinyl Groove Taste Fingerprint: Visualizes Qloo's weighted urn:tag taste analysis as concentric vinyl grooves showing the aesthetic threads (e.g., Traditional Pop, Lush Orchestration, Classic Hollywood, Romantic Comedy) that tie their memories together.
• 8 Cross-Domain Skeuomorphic Memory Cards: Color-coded with Kodachrome palettes so the physical form tells you what kind of memory it is:
  • Music (urn:entity:artist): Vinyl record sleeves with real 30-second audio previews hydrated via the iTunes Search API.
  • Film (urn:entity:movie): Cardboard 35mm Kodachrome slide mounts with release years and personalized "why this matters" notes.
  • Television (urn:entity:tv_show): Vintage cathode-ray tube frames.
  • Stars & Faces (urn:entity:person): Classic Polaroid prints.
  • Books (urn:entity:book): Hardcover cloth editions.
  • Everyday Brands (urn:entity:brand): Nostalgic household labels and scents.
  • Destinations (urn:entity:destination) & Local Outings (urn:entity:place): Airmail postcards and perforated admission tickets for real-world venues near their current city that match their lifelong taste.
• Autobiographical Life Timeline & Comfort Kit: Maps cultural milestones alongside their personal life story (birth, formative years at ages 10–30, marriage, career) and curates a tangible sensory kit (scents, foods, textures) plus a strict clinical Avoid List.

4. Guided 20-Minute Session Player (/profile/[id]/session)

• Dementia-First Ergonomics: Full-screen, zero clutter, warm contrast, oversized touch targets, full keyboard navigation (←, →, Space), and Atkinson Hyperlegible typography (engineered by the Braille Institute for low-vision readers).
• Validated Reminiscence Arc: Every session follows clinical dementia-care structure:
  1. Opening Song (familiar music with spinning vinyl and audio preview to orient and soothe)
  2. Beloved Face or Film (visual trigger with slow Ken Burns pan)
  3. Personal Life Story Moment (bridging cultural era to their own lived history)
  4. Sensory / Tactile Trigger (a familiar taste, scent, or object to hold)
  5. Gentle Closing Song (calm wind-down)
• "Invite, Never Quiz" Prompt Design: Prompts are strictly engineered to never test short-term memory (never "Do you remember who sang this?"). Instead, they invite open emotional expression ("Does this voice remind you of Saturday nights?") accompanied by a practical Caregiver Tip on every slide.
• Multilingual Read-Aloud: Integrated Web Speech API (speechSynthesis) reads prompts aloud in a calm cadence in the person's native language (English, Spanish, etc.).

5. The Reflector Agent & Continuous Learning Loop

• During a session, the carer logs effortless one-tap reactions on each moment: Lit up, Engaged, Neutral, or Upset.
• Clicking "Plan Next Session" triggers the Reflector Agent:
  • Entities that made the person Light up are promoted into primary Qloo seed signals (signal.interests.entities) to discover deeper adjacent memories.
  • Entities that caused Upset / Distress are permanently added to Qloo's filter.exclude.entities blocklist so they are never surfaced again.
  • Generates transparent "What Madeleine Learned" clinical notes explaining how the profile evolved.

6. Printable Bedside Care Card (/profile/[id]/card)

• A print-ready (@media print) one-page summary designed to be taped inside a care-home resident's room or handed to a relief nurse — listing who they are, the songs that calm them, conversation starters that work, sensory comforts, and topics to avoid.

7. Live "Compare with Plain AI" Benchmark (/profile/[id]/compare)

• A built-in scientific control group on every profile. Runs the exact same LLM with the exact same person's intake without Qloo tools, resolves both lists in Qloo, and scores both sets of recommendations against the person's true taste seeds using Qloo's affinity engine on an identical 0–100% scale.

---

How It's Qloo-Powered

Qloo is not a post-processing filter or decorative lookup — Qloo is the cognitive graph engine that drives every stage of Madeleine's multi-agent pipeline.

1. Scoped Agent Tool Loop (src/lib/qloo.ts & src/lib/agent/tools.ts)

Our Groq-hosted Curator Agent interacts with Qloo (https://hackathon.api.qloo.com) autonomously through five specialized tools over a multi-step tool-calling loop:

1. search_qloo → Entity Resolution (GET /search)
   • Batch-resolves raw family text (singers, films, actors, shows, brands, cities) into canonical Qloo Entity IDs across typed URNs (urn:entity:artist, urn:entity:movie, urn:entity:tv_show, urn:entity:person, urn:entity:book, urn:entity:brand, urn:entity:destination, urn:entity:place).
2. taste_fingerprint → Aesthetic Tag Extraction (GET /v2/insights?filter.type=urn:tag)
   • Passes resolved cultural seed IDs via signal.interests.entities to extract the top 30 weighted taste tags (urn:tag).
   • Domain hygiene: We filter seed inputs to purely cultural domains (music, film, tv, stars, books) before querying tags so food or location seeds don't pollute aesthetic taste tags with pricing or geographic metadata. Weights are normalized against the maximum tag weight to render the Taste Fingerprint grooves.
3. discover → 8-Domain Era-Filtered Insights (GET /v2/insights)
   • Queries Qloo across all 8 domains in parallel using signal.interests.entities (resolved seeds), optional signal.interests.tags, filter.exclude.entities (any distressing items), and automatic server-side demographic & reminiscence-bump filters:
     • Audience Demographics: signal.demographics.age mapped dynamically from birthYear (55_and_older, 36_to_55, 35_and_younger).
     • Film & TV Era Window: filter.release_year.min = (birthYear + 10) - 6 and filter.release_year.max = (birthYear + 30) + 2.
     • Books Era Window: filter.publication_year.min = (birthYear + 10) - 80 and filter.publication_year.max = (birthYear + 30) + 2 (accounting for classic literature read in youth).
     • Stars & Faces Birth Window: filter.date_of_birth.min = (birthYear - 40)-01-01 and filter.date_of_birth.max = (birthYear + 8)-12-31 so discovered icons were contemporaries or idols of their youth.
     • Geolocated Outings: For urn:entity:place, passes filter.location.query set to the person's currentCity (or hometown) so families get actionable local places to visit today that match the person's lifelong aesthetic taste.
4. rank_by_taste → "LLM Knows History, Qloo Knows Taste" Hybrid Ranking (GET /v2/insights)
   • Because Qloo's urn:entity:artist endpoint does not expose a native career-year filter, pure graph traversal can occasionally return modern artists appreciated by the same demographic.
   • When the Curator sees modern skew or sparse results, it invokes rank_by_taste: the LLM proposes up to 12 historically era-correct candidates, resolves their entity IDs via /search, and calls /v2/insights with filter.results.entities=<candidate_ids> and signal.interests.entities=<seed_ids>. Qloo scores and orders those historical candidates by the person's exact taste affinity!
5. find_tags → Genre & Movement Steering (GET /v2/tags)
   • Looks up specific cultural movement tags (e.g., bebop, doo-wop, film noir, golden age bollywood, bolero) to steer discovery via signal.interests.tags.

2. Zero-Hallucination Entity Registry (ctx.registry)

Every entity returned by any Qloo tool call is stored in a server-side RunContext registry (Map<string, Entity>). When the Curator submits its final picks (finish tool) and the Designer writes the session, every entityId is strictly validated against ctx.registry.has(id). The LLM chooses and writes context, but it cannot invent or hallucinate an entity.

3. Closed-Loop Adaptation (POST /api/adapt)

When a carer completes a session, the Reflector pipeline partition reactions into lovedIds (lit_up + engaged), flatIds (neutral), and upsetIds (upset). It re-queries Qloo with signal.interests.entities prioritizing lovedIds and passes upsetIds into filter.exclude.entities, dynamically evolving the taste graph after every visit.

---

How We Built It

1. Three-Stage Agentic Architecture (src/lib/agent/pipeline.ts)

• Stage 1 — The Curator Agent (Tool-Calling Loop): Powered by Vercel AI SDK (generateText) running on Groq (openai/gpt-oss-120b with automatic failover to openai/gpt-oss-20b / kimi-k2-instruct), executing parallel tool calls (parallelToolCalls: true) with a strict latency budget (AbortSignal.timeout) and a deterministic Qloo top-up fallback that guarantees every domain hits its target count (PICK_TARGETS) even if an LLM step times out.
• Stage 2 — The Designer Agent (Validated Structured Output): Receives the verified Qloo entities, affinity scores, and family story, and synthesizes the cultural portrait, per-item <18 word explanations, 20-minute session script, sensory kit, and caregiver tips. Uses Zod schemas (DesignSchema, SessionSchema, MomentSchema) with resilient field-level .catch() defaults and multi-tier JSON extraction repair so a partial model response never breaks a family's session.
• Stage 3 — Real-Time Audio Hydration (src/lib/itunes.ts): Concurrently queries the iTunes Search API (pMap concurrency-limited worker pool) matching artist + era song title to attach real 30-second AAC audio previews and album art to every music card and session moment.

2. Frontend, Visual Craft & Accessibility

• Framework: Next.js 16.4 (App Router, Turbopack) + React 19 + TypeScript 5 + Tailwind CSS v4.
• Real-Time Streaming Protocol: Server routes (/api/build, /api/adapt, /api/compare) stream Newline-Delimited JSON (NDJSON) events (phase, thought, tool_start, tool_end, profile, error) consumed by a custom ReadableStream client (src/lib/stream-client.ts).
• 3D WebGL Hero Scene: Built with Three.js, React Three Fiber, and Drei (src/components/hero/HeroScene.tsx), rendering floating memory artifacts and spinning vinyl in warm volumetric projector light.
• Editorial & Accessible Typography:
  • Fraunces (variable optical size & swash italics) + Newsreader (editorial serif) for warmth, dignity, and nostalgic bookishness.
  • Atkinson Hyperlegible Next for all body copy, prompts, and caregiver controls to ensure maximum legibility for older eyes.
• Kodachrome Design System (src/app/globals.css): Custom CSS film grain overlay, projector vignette, developing-photo keyframe animations (@keyframes develop), Ken Burns still motion, and physical object skeuomorphism (.slide-mount, .polaroid, .postcard, .ticket).
• Privacy-First Local Architecture: Zero database, zero cookies, zero tracking. All profiles and session logs live exclusively in the family's browser localStorage (src/lib/store.ts), while the server holds only an ephemeral, in-memory TTL cache for Qloo responses.

---

Challenges We Ran Into

1. Solving Era Drift in Music & Brand Recommendations:
   • Challenge: While Qloo's /v2/insights supports filter.release_year for movies and TV shows and filter.publication_year for books, urn:entity:artist and urn:entity:brand represent timeless entities without a single release year. Early tests for someone born in 1942 occasionally surfaced modern artists who happen to share jazz/pop crossover audiences today.
   • Solution: We engineered the rank_by_taste hybrid tool. The LLM leverages its historical pre-training to propose candidates strictly from the person's 10–30 age window, resolves them in Qloo, and passes their IDs into /v2/insights via filter.results.entities. This combines the LLM's chronological precision with Qloo's real-world taste affinity scoring.
2. Groq Strict Tool-Schema Validation vs. Live Streaming Speed:
   • Challenge: Large tool-calling models on Groq occasionally emit slightly varied domain names (e.g., "movie" instead of "film", or "singer" instead of "music"), which causes strict enum schemas to reject the entire tool call with a 400 Bad Request.
   • Solution: We made tool input schemas permissive (z.string()) and built a server-side synonym normalizer (toDomain in src/lib/agent/tools.ts) plus a multi-stage JSON repair fallback for structured outputs.
3. KeepingTaste Fingerprints Pure Across Mixed Seeds:
   • Challenge: When a family includes a food or city in their initial seeds (e.g., "Coney Island" or "egg creams"), querying urn:tag across all seeds pulled in restaurant-pricing or regional tags alongside musical and cinematic moods.
   • Solution: In taste_fingerprint, we partition the seed registry so aesthetic tag analysis runs on cultural entities (music, film, tv, stars, books) while still using all seeds for cross-domain entity discovery.
4. Writing Prompts That Never Feel Like a Clinical Test:
   • Challenge: Default LLM behavior when generating conversation prompts is to ask factual trivia questions ("Who directed Roman Holiday?" or "What year did you get married?"), which are distressing for someone with dementia.
   • Solution: We baked strict clinical reminiscence communication rules into DESIGNER_SYSTEM and REFLECT_SYSTEM — enforcing open sensory invitations, first-person emotional curiosity, and explicit caregiver coaching tips on every single moment.

---

Accomplishments That We're Proud Of

• A Demo That Makes People Go Quiet: Watching a few rough phrases about a grandparent transform in 25 seconds into spinning vinyl with real audio from their youth, vintage slide mounts, and gentle, non-quizzing conversation prompts consistently creates an emotional reaction.
• True Cross-Cultural & Multilingual Reach: Built and verified end-to-end across four distinct cultural worlds — 1950s Brooklyn, 1960s Bombay, 1960s Merseybeat Liverpool, and 1970s Mexico City (including full Spanish-language session generation and read-aloud synthesis).
• 100% Grounded in Qloo Entities: Not a single cultural recommendation is hallucinated; every card carries a verified Qloo Entity ID, affinity score, and domain provenance.
• Quantitative Proof via the Live Compare Page: Rather than merely claiming Qloo is better than a plain LLM, we built a live benchmark right into the app (/profile/[id]/compare) that proves Madeleine's superior taste affinity on the exact same Qloo measurement scale.
• Dignified, Accessible Craft: A UI that rejects sterile "medical software" aesthetics in favor of warm projector light, tactile analog objects, and low-vision accessibility.

---

What We Learned

• LLMs and Taste Graphs Are Complementary Opposites: LLMs are extraordinary at narrative synthesis, empathy, and historical chronology, but poor at authentic, non-stereotypical taste correlation. Qloo's taste graph is extraordinary at cross-domain cultural affinity, and pairing them via an agentic tool loop unlocks capabilities neither has alone.
• Designing for Two Cognitive Realities: Care software has two users in the same room — a stressed caregiver who needs rich controls, explanations, and printouts (the Studio), and a person with cognitive decline who needs calm, singular focus and sensory warmth (the Session Player). Separating those two modes transformed the usability of the product.

---

What's Next for Madeleine

1. Care-Home & Memory-Care Pilots: In the United States and UK, residential care facilities are legally mandated to provide person-centred activities tailored to each resident's life history (CMS F-Tag 679). Madeleine can onboard a new resident in 2 minutes and print their bedside Care Card immediately.
2. Collaborative Family Links: Optional encrypted link-sharing so siblings, grandchildren, and nieces in different cities can each contribute a few memories and family photos to a shared profile before a visit.
3. Full-Length Spotify & Apple Music Playback: Upgrading from 30-second iTunes previews to OAuth-connected full-track playback for extended music therapy playlists.
4. Longitudinal Care Coordinator Analytics: Tracking mood and engagement trends across weeks of sessions to discover which eras, genres, or times of day (e.g., mitigating "sundowning" agitation) consistently bring comfort.

Built With

• groq
• motion
• nextjs
• qloo
• react-three-fiber
• tailwindcss
• three.js
• typescript
• vercel-ai-sdk