Narrative Memory Agent

You don't build the character. You meet them.

Comment

Inspiration

There are characters you finish a book and can't put down — who feel less like something an author made up and more like someone who was already there, glimpsed through a page you can't step past. I've felt it at 2 a.m. as a reader, and as someone who writes: the sense that the character knows something I don't, and if I could just ask them—

I serialize the Caelvorn Series on Dreame (Book 1 complete), so I live on both sides of that page. I've wanted to talk to my own characters the way a reader wants to talk to theirs — not to a persona I'd write out and puppet, but to the person the manuscript already made. Every companion app I tried answered in a mask I'd designed. None of them surfaced the character who was already there. NMA is the door I built to get as close as I could.

What it does

NMA reads novel text and builds a real cognitive profile for each character — traits, behavior patterns, motivation, arc stage, and a relationship network — then lets you talk to them in their own voice.

You ask a character a question, and NMA generates 2–5 distinct in-voice responses, each scored for how in-character it is. You pick the one that fits, and the system learns your preference. Characters remember previous conversations across sessions, and they change: an offline Sleep cycle consolidates what happened into evolving traits, so the same question can yield a new, self-reflective answer it couldn't have given before.

The difference from a companion chatbot is where the character comes from. Those apps ask you to write a persona and act it out with you; what answers is a mask you designed. NMA surfaces the person the book already wrote — extracted from the text itself. You don't build the character. You meet them.

The demo runs on two novels at once: Elizabeth Bennet and Darcy from Pride and Prejudice (pre-loaded on a fresh install), and Lena, Mira, and Caelan Ashmark from my own Caelvorn Series — ingested live through the pipeline, straight from my manuscript. NMA is dogfooded against the exact problem it was built to solve.

How we built it

Dual Circuit Engine. Circuit A (Generation) prompts Qwen 3.6-flash with the character's full profile — backstory, traits, motivation, arc stage — to produce diverse responses. Circuit B (Validation) scores each option on five OOC factors: trait consistency, behavior patterns, semantic distance (from real ChromaDB embeddings, not LLM-estimated), self-consistency, and surprise value. Each option is classified as fitting, surprising, or out-of-character.

Two memory layers + a schema store. Working Memory buffers current context and persists to SQLite for cross-session recall. Episodic Memory holds a Zwaan-indexed event timeline that decays over time. The Character Schema Store (SQLite) holds the profile — written by ingestion and by Sleep, read by Circuit A for generation. A separate ChromaDB vector store powers the semantic-distance score, but is deliberately not treated as a memory layer — it's a search tool.

GenBias learning loop. When you mark a response as fitting, NMA updates a 5-dimensional preference vector via EMA and injects that bias into future generations — the system learns what each user prefers for each character.

Sleep consolidation. Inspired by human memory: an NREM phase extracts facts, a REM phase abstracts behavior patterns into traits, and a Self-Audit reports contradictions and confidence.

Stack: FastAPI (Python 3.11) + Next.js 16 (TypeScript), SQLite, ChromaDB, Qwen 3.6-flash + text-embedding-v3 via the OpenAI-compatible dashscope-intl endpoint — the whole LLM layer is the standard OpenAI SDK pointed at a different base URL, so switching models is a one-line env change. 124 unit tests. Containerized with Docker / docker-compose and deployed on Alibaba Cloud with a persistent data volume, so no memory is lost between restarts.

Challenges we ran into

The hard part wasn't the model — it was earning the project's own name.

The bug the docs found before the code did. Mid-project, I audited the repository against its own documentation. The docs claimed cross-session memory: close the browser, come back tomorrow, and the character remembers. The code told a different story — the ask and feedback endpoints weren't actually writing events into episodic memory. Session resumption restored the conversation, but the character had learned nothing from it. The feature the entire project is named after was, at that moment, half-real. It's fixed now — every question and every user choice becomes an episodic event — and the lesson became the project's testing philosophy: documentation isn't a description of your system; it's a list of claims that each needs a test. I ended with 124 unit tests. The number kept me honest.

Three bugs stacked like a bad joke. Days before the deadline, the Sleep cycle — the centerpiece — died mid-dream in production. It turned out to be three bugs nested inside each other: a ChromaDB version drift (get() stopped returning embeddings unless asked explicitly), which uncovered my own defensive or [] guard throwing on a NumPy array, which uncovered the same NumPy-in-a-boolean mistake wearing a different disguise a few lines down. The data was never in danger — every embedding was intact the whole time — but the code kept insisting it was. The permanent lesson is now in my ops notes: a NumPy array must never enter a Python boolean context.

Cutting my favorite layer. The original design had three memory layers — working, episodic, semantic — mirroring the cognitive psychology literature. Halfway through, I cut semantic-layer persistence entirely. Not because it didn't work, but because a hackathon rewards a system that is demonstrably solid over one that is theoretically complete, and the semantic layer was where bugs went to hide. Two layers plus a schema store, each doing one job well, beat three layers doing overlapping jobs badly. The docs say "two active memory layers," and I stand by the honesty.

Deploying lean. The production ECS instance turned out to be too small to build the Docker images at all — so images are built and packaged elsewhere and shipped over (docker save | gzip, SCP, docker load). Production is a lean box that only ever runs, never builds. Accidental best practice via insufficient RAM.

Accomplishments that we're proud of

  • A memory system that actually persists and evolves — characters remember across sessions and visibly grow after Sleep, not just replay a static persona.
  • OOC validation grounded in real embeddings, not an LLM grading itself — the semantic-distance score comes from ChromaDB, so "in-character" is measured, not asserted.
  • Dogfooded on real, unpublished text. The demo ingests my own manuscript live, so it's tested against the exact problem it was built to solve — not a toy dataset.
  • A clean judge path: docker compose up with pre-loaded demo data, MIT-licensed, 124 tests green.

What we learned

The interesting problem in a MemoryAgent isn't storage — it's what to forget and when. Decay, importance scoring, and Sleep consolidation matter more than raw capacity; a character that remembers everything equally is as unconvincing as one that remembers nothing. Grounding "in-character" in real embedding distances rather than the model's own judgment made consistency something I could see and tune instead of hope for.

The harder lesson was knowing when to stop. When the organizers extended the deadline by eleven days, every instinct said add a feature. I did the opposite: froze the code — every bug gets fixed, nothing new gets built — and spent the time on whether anyone would understand what this was. The README stopped opening with a product category and started opening with the actual feeling that made me build it. A judge doesn't need a product; a judge needs an MVP that makes its own point. The full story of that decision — and the bugs behind it — is in my build journal on Libellus — Build Notes I (building it) and Build Notes II (knowing when to stop).

What's next for Narrative Memory Agent

I built NMA starting with the one person I was certain wanted it — me. Next is finding out who else does: opening it up to more of my own cast and to other novels, for both sides of the book. Readers who finish a series and keep replaying the fork a character didn't take. Authors who need to check what a character actually knows before writing the next line. On the technical roadmap: restoring the semantic memory layer — cut deliberately during the sprint, it belongs in the version that isn't racing a deadline. The want is real on both sides — NMA is how I find out how many people share it strongly enough to build a product on.

Built With

Share this project:

Updates