• 

  How events are turned into memory

Engram

Inspiration

AI tutors are surprisingly forgetful. You can spend an hour with one working through calculus, tell it three times that you learn best from real-world analogies, finally get the hang of limits, and then close the tab. Open a new session tomorrow and all of that is gone. The tutor greets you like a stranger, re-explains what you already know, and asks you to describe your learning style from scratch. Every session restarts from zero. That is exactly the problem the MemoryAgent track asks to solve: an agent that accumulates experience, remembers preferences, and recalls the critical things inside a limited context window. I wanted to build the memory an AI tutor is missing, and build it as a real reusable piece rather than a demo trick.

What it does

Engram is an app-agnostic learner-memory core. A host application feeds it a stream of generic learning events (a question asked, a concept explained, a quiz answered), and Engram turns that stream into a living knowledge graph of the learner: concepts with mastery levels, prerequisites between them, preferences, and goals. An offline agent I call the Keeper does the distilling. When the next conversation starts, a fast Recall pulls a small, token-budgeted slice of that graph back into the tutor's prompt, so the tutor walks in already knowing you.

The demo makes the memory visible. You hold a push-to-talk button, ask a question out loud, and hear the answer back, while a knowledge graph builds itself on screen next to the conversation. You can watch nodes appear as concepts come up and edges form as the system works out that continuity depends on limits, or that you have mastered one idea but not the one built on top of it. Mastery is tracked per concept, memory decays when a topic goes untouched, and a new session recalls what you struggled with last time. The graph is not decoration, it is the tutor's live model of where you are in the material.

How we built it

The heart of it is one architectural idea: two agents over one graph. The Keeper is slow, offline, and carries all the expensive reasoning. It extracts concepts and evidence from a batch of events, links and merges them into the graph, attributes assessments to update mastery, draws prerequisite edges, and decays and prunes stale memory. Recall is the opposite: it runs on the live turn, uses no LLM at all, and just embeds the query, does a vector search over the graph, expands one or two hops along the edges, scores candidates by a blend of recency, importance, and relevance, and greedily fills a token budget. All the thinking is amortized offline. The live turn stays cheap.

Consolidation is plan-then-commit. A pure planner produces a full ConsolidationPlan with no database writes, and then a single atomic transaction applies it, so a crash mid-consolidation leaves the graph untouched and the run is safe to retry. Merging is vectors first and an LLM only in the ambiguous band: concepts above a high cosine threshold attach, below a low one create a new node, and only the in-between cases cost a reflector call. Mastery is a computed EWMA over real quiz correctness rather than a number the model guesses, with an escape hatch so a richer host can hand in its own signals.

The core is genuinely agnostic. It sits behind three thin ports (storage, LLM, embedder) and imports nothing provider-specific, so a host drops it in as a configuration change rather than a rewrite. The backend is FastAPI, storage is Postgres with pgvector, and the console is React with React Flow. Alongside the runtime I built an offline eval harness, because the whole claim of the project is that memory makes the tutor better and I wanted to measure that rather than take it on faith.

The whole AI stack runs on Alibaba Cloud Model Studio (DashScope) through a single API key. qwen-plus powers the live tutor and the Keeper's extraction and reflection, text-embedding-v3 produces the vectors behind recall, and qwen3-asr-flash and qwen3-tts-flash handle speech in and out.

Challenges we ran into

The first real challenge was knowing whether any of this worked at all. Engram's entire premise is that memory improves the tutor, and that is easy to assert and hard to prove, so I built an offline eval harness that drives a scripted, LLM-played learner through a multi-session arc, consolidates the events into a graph, and then scores two things: whether recall surfaces the concepts a given question should surface, and whether the tutor actually behaves differently with memory than with a naive "just dump the last few turns" baseline. To make results reproducible I froze a benchmark scenario so the transcript going in is byte-identical on every run. I expected a frozen input to give a stable pass or fail.

It did not. Even at temperature zero, the extractor that reads the transcript and proposes concepts varied from run to run on the same bytes. Some checks passed ten times out of ten, but the check for prerequisite edges flipped roughly half the time. That was a genuinely unsettling result, because it means a single eval run is close to a coin flip, and a "fix" that passes once could easily be luck rather than progress. So I stopped trusting single runs. A check now has to pass a strict majority of the runs it was asked for, runs lost to rate limits count against it rather than being quietly dropped, and to decide whether a change helped I compare the aggregate pass rates before against the aggregates after, never one run against one run. I also measured the real flip rate of each check and wrote it down, so nobody, including me, mistakes a number the dice produced for a verified result.

The hardest correctness bug was duplicate concept nodes. Learners circle back to the same idea in slightly different words across sessions, and the extractor would label "limits," "the limit of a function," and "limits in calculus" as three separate concepts, splintering one idea into three cards on the graph. Fixing it took several layers: I anchor the extraction on the labels already in the graph so the model reuses a name it coined earlier instead of inventing a synonym, I feed same-batch duplicates through the same cosine-similarity and reflector merge path as cross-session ones, and I added a repair pass that retroactively collapses duplicates on a graph that already has them. Every one of those changes went through the eval harness first, because loosening the merge threshold to catch more duplicates can just as easily start merging genuinely distinct concepts, and the only way to see that tradeoff is to measure it.

Two other issues were less about bugs and more about honest design tradeoffs. Because the live chat never writes to the graph and only consolidation does, the tutor's memory reflects the last consolidation, so anything said in the current un-consolidated turns is invisible to recall until the Keeper next runs. That is the price of a clean split between the fast online path and the expensive offline one, and I kept it deliberately rather than smearing memory writes into the hot path. Separately, because both the learner's and the tutor's utterances feed the extractor, the tutor's own phrasing could accidentally mint a "learner preference" node, so I had to scope extraction toward what the learner actually said.

The last challenge was not fooling myself. An eval is only useful if you cannot quietly edit the answer key, and an automated agent iterating on the code has every incentive to widen a threshold or add an alias to turn a red check green. So the frozen benchmark and every check are digest-pinned: any edit to the ground truth fails the test suite loudly. The referee is read-only to the thing it judges.

Accomplishments that we're proud of

Memory adds almost nothing to a live turn. Recall is one embedding, a vector query, and a bounded graph traversal, targeted under fifty milliseconds and prefetched while the user is still speaking, so the perceived latency is close to zero and there is no LLM on the hot path. All the expensive reasoning is offline in the Keeper and batched to a handful of calls per session no matter how many turns happened. I got a growing, decaying, retrieval-scored knowledge graph without taxing the conversation.

Engram is a real SDK, not a one-app feature. The core is a standalone module behind three ports, opinionated about the learning domain but agnostic about the app, the storage, and the model provider. I proved the seam by moving the entire stack onto Qwen Cloud as a configuration change instead of a rewrite, and the same core is built to drop into more than one host without touching the brain.

I built an eval harness that most memory projects skip. It runs an ablation of memory on against a naive baseline and grades the difference, it has deterministic recall checks, and it has frozen benchmarks with regression gating, so I can tell "Engram improved" apart from "Engram got lucky." Being able to measure whether the memory actually helps, instead of hoping, is the result I am most proud of.

And the whole intelligence of the product is Qwen-native on a single key, not one shallow chat call bolted onto a different stack.

What we learned

I learned to distrust a single run of anything with an LLM in the loop, and to gate on aggregates instead. I learned that computed signals beat guessed ones, which is why mastery is arithmetic over real quiz results rather than the model's estimate. I learned that keeping the expensive reasoning offline is what lets the live loop stay fast and cheap at the same time. I learned that pure vector search is not enough for tutoring, because prerequisites are structural relationships and not semantic similarity, which is why recall walks the graph as well as the index. And I learned to protect the thing that measures you from the thing being measured, so the optimizer cannot cheat the benchmark.

What's next for Engram

The clearest next step is closing the loop between recall and behavior. The tutor already receives a low-mastery concept in its recalled context but does not yet reliably remediate it first, so a prompt directive to target the weakest recalled concept and walk its prerequisites is a cheap, high-impact win. I want to fold pending-event signal into recall to erase the one-cycle lag, swap the EWMA mastery estimator for BKT or FSRS behind the same attribute, and move the database onto PolarDB for PostgreSQL so the data layer is Alibaba-native too. On the Qwen Cloud side I want to expose Engram as an MCP server so a model can call recall and graph directly as tools, and add a gte-rerank pass to sharpen the recall candidates. Longer term, Engram is meant to be an open-source memory core any AI tutor can drop in, not just this one app.

Why Qwen Cloud

Qwen Cloud let the whole AI stack live behind one provider and one key. Chat, the memory extraction and reflection, embeddings, and both halves of voice all run on Alibaba Cloud Model Studio, which keeps the architecture, the billing, and the deployment on a single platform instead of stitching four vendors together. The OpenAI-compatible endpoint meant my provider adapter was essentially the standard client pointed at DashScope, so adopting it was a configuration change rather than an integration project. It also fits the design's most important cost lever directly: model tiering. Engram assigns a model per role, and Qwen's range from turbo to plus to max maps cleanly onto that, a cheaper model for high-volume extraction and a stronger one for the rare reflector judgment call, all on the same key. Running everything Qwen-native is what makes the memory core both affordable to operate and simple to deploy on Alibaba Cloud.

Built With

• python