Qwen Memory Agent

Inspiration

Long-running AI assistants face a difficult tradeoff. Sending the entire conversation history is slow and expensive, but aggressively compressing it can remove the exact detail needed later. Existing memory demos often retrieve text successfully without explaining where a memory came from, whether it is still current, or why it influenced an answer.

We built Qwen Memory Agent to explore a more practical approach: create durable, source-grounded memories in the background, then recall only the most relevant evidence when the user needs it.

What it does

Qwen Memory Agent is a persistent memory layer for multi-session AI assistants.

The foreground agent responds immediately using committed memories and recent conversation turns. After the answer is delivered, a separate background pipeline:

Preserves the original conversation turns as source notes.
Uses Qwen to extract compact, atomic observations.
Links every observation to its supporting source messages.
Detects supported relationships and knowledge updates.
Persists the resulting memory and retrieval indexes.
Makes the new memory available to future conversations.

When answering a later question, the agent combines lexical and semantic retrieval, expands relevant source links, and packs the evidence into a bounded context window. Outdated observations can be marked as superseded without deleting their history.

A React Flow workbench visualizes this process in real time. Users can inspect source messages, observations, retrieval scores, provenance links, supersession relationships, background jobs, and the evidence used for each answer.

How we built it

The memory core is written in Python and keeps storage, retrieval, observation extraction, consolidation, and answer generation as separate components.

We use:

Qwen through Alibaba Cloud Model Studio for observation extraction, memory consolidation, embeddings, and answers.
SQLite and FTS5 for durable local storage and restart-safe lexical search.
BM25, Chroma, and Qwen embeddings for lexical and semantic retrieval.
Reciprocal Rank Fusion to combine independent retrieval signals.
AgentScope for the foreground task agent and tool execution.
FastAPI and Server-Sent Events for live conversations and pipeline events.
React, React Flow, and Dagre for the interactive memory graph.
LongMemEval for reproducible multi-session memory evaluation.

The live system uses two independent per-conversation queues. The answer queue preserves conversational order, while the memory queue performs observation extraction and embedding work after the response. The assistant answer and deferred memory job are committed in one transaction so interrupted jobs can be recovered after a restart.

Challenges we faced

The hardest problem was not simply retrieving similar text. It was deciding what should become a durable memory while retaining enough evidence to audit that decision.

Structured model output also required careful validation. We validate observation provenance, reject unknown source references, bound model inputs, verify embedding dimensions, and handle incomplete provider responses explicitly.

Another challenge was avoiding benchmark overfitting. We removed fixed keyword intent rules, regex-based recall logic, question-type prompts, and handcrafted benchmark answers. The benchmark adapter records predictions, recalled notes, context budgets, source evidence, and persisted store paths, but it does not change the core memory behavior for individual test cases.

Finally, background memory processing had to remain invisible to the user. Separating foreground answers from durable memory writes improved perceived latency while preserving cross-session consistency.

What we learned

High-quality memory starts at write time. A complex answer prompt cannot reliably repair memories that were stored without provenance, temporal context, or update relationships.

We also learned that observability is part of memory quality. A graph showing exactly which source produced an observation and which memories in