Forget Me Not

A four-store memory layer for LLM agents that resolves contradictions, decays gracefully, and retrieves under a token budget, proven against 3 baselines.

Comment

What inspired it

Every "agent with memory" demo I saw seemed to fall into one of two traps. Either the agent remembered nothing between sessions and repeatedly asked for information it had already learned, or it simply stuffed the entire conversation history back into every prompt and called that "memory."

The second approach works until a user changes their mind. If someone corrects a preference or updates a fact, both the old and new versions remain in the context with no principled way for the model to know which one is current. The system remembers everything, but understands nothing about change.

I wanted to build the infrastructure that actually handles evolving knowledge. Instead of treating memory as a transcript, I wanted a system with explicit policies for what should be stored, when existing memories should be replaced, how different memories should age, and what deserves to survive when context is limited. Just as importantly, I wanted to measure whether those policies genuinely improved behaviour instead of simply assuming they would.

What I learned

The biggest lesson was that memory is fundamentally a policy problem rather than a storage problem.

Persisting data is straightforward. The difficult questions are deciding when two memories represent conflicting information, how quickly different categories of memory should decay, and which memories deserve to occupy a limited context window. A temporary note from ten minutes ago should not live as long as a long-term user preference, and a corrected fact should not compete equally with the outdated version it replaces.

Building MemoryBench alongside the memory engine rather than after it turned out to be invaluable. During evaluation, I found what initially looked like a major flaw in the memory system. It consistently returned the most recent information, yet the benchmark reported lower accuracy than simpler baselines. The real issue was the benchmark itself. Some synthetic evaluation probes expected answers that had already been superseded by later corrections in the conversation. The memory system was correct, but the expected answers were stale.

Fixing the probe generator rather than the memory engine resolved the discrepancy and reinforced an important lesson. When a benchmark and the implementation disagree, the benchmark deserves just as much scrutiny as the system it is evaluating.

How I built it

The memory engine is built around four independent memory stores: episodic, semantic, preference, and working memory. Every record shares a common schema and is managed by a single MemoryManager, while each store applies its own decay policy and behaviours. Working memory, for example, is explicitly cleared when a session ends instead of naturally decaying over time.

The first behaviour is selective writing. Every conversation turn is passed through a Qwen qwen-plus extraction model, which decides whether anything in that interaction is worth remembering. Most conversations produce no new memories at all, keeping storage focused on information that is likely to matter later.

The second behaviour is contradiction resolution. Rather than continuously appending new facts, every memory belongs to a logical key such as preference: editor or fact:paper_x:bleu_score. When new information arrives for an existing key with different content, the previous record is marked as superseded, linked to the replacement, and retained for auditability instead of being deleted.

Memory importance is updated continuously through a weighted scoring function:

$$ \text{score}(r,t)=w_r\cdot\text{recency}(r,t)+w_a\cdot\text{access}(r)+w_i\cdot\text{importance}(r) $$

where recency follows an exponential decay:

$$ \text{recency}(r,t)=0.5^{\frac{\text{age_days}(r,t)}{\text{half_life}(\text{store}(r))}} $$

The default weights are (w_r=0.5), (w_a=0.2), and (w_i=0.3). Access frequency uses a log-scaled count capped at one, while each memory store has its own half-life ranging from roughly six hours for working memory to ninety days for semantic knowledge. Records whose scores fall below a configurable threshold are archived instead of deleted, preserving a complete history while keeping active retrieval focused on relevant information.

Retrieval itself is treated as an optimisation problem rather than a simple top-k ranking. Given a token budget (B), the system solves a classic 0/1 knapsack problem:

$$ \max_{x\in{0,1}^n}\sum_{i=1}^{n}\text{relevance}i x_i \quad\text{subject to}\quad \sum{i=1}^{n}\text{token_cost}_i x_i\le B $$

using dynamic programming. This allows several moderately relevant but inexpensive memories to be selected instead of a single expensive memory that exhausts the available context. This behaviour is verified directly through a dedicated unit test, test_knapsack_beats_naive_topk.

To evaluate the complete system, I built MemoryBench, which exercises the production memory implementation rather than a simplified simulation. The benchmark contains thirty synthetic multi-session conversations with evolving user preferences and facts that are intentionally corrected over time. It compares the proposed memory system against three baselines: no memory, full history prompting, and a naive top-k retrieval approach that ranks memories solely by embedding similarity without contradiction handling. Evaluation measures recall under a fixed token budget, stale memory retrieval rate, context size per turn, and an illustrative cost and latency model.

Challenges faced

The biggest challenge was discovering that the benchmark itself could produce misleading results. Early versions of the synthetic probe generator occasionally evaluated memories after a correction had already occurred while still expecting the outdated answer. This unfairly penalised systems that correctly prioritised the latest information. Restricting every evaluation probe to the valid time window before the next correction resolved the issue.

Another challenge was building a deterministic evaluation pipeline without relying on paid embedding APIs. A simple hashed bag-of-words representation initially yielded poor relevance rankings because common phrases dominated the similarity calculation. Introducing stop word filtering significantly improved retrieval quality while keeping the benchmark completely free and reproducible.

Finally, I wanted to remain transparent about what was real and what was simulated. The benchmark reports cost and latency estimates using documented pricing and simple linear models based on token counts rather than live billing or API measurements. This keeps experiments reproducible while clarifying which metrics are empirical and which are illustrative.

Built With

  • alibaba-cloud
  • dashscope
  • ecs
  • faiss
  • faiss-cpu
  • fastapi
  • matplotlib
  • nextjs
  • nginx
  • numpy
  • oss
  • oss2
  • pydantic
  • pypdf2
  • pytest
  • python
  • qwen
  • qwen-max
  • qwen-plus
  • react
  • tailwindcss
  • text-embedding-v3
  • typescript
  • uvicorn
Share this project:

Updates