Mnemosyne

Mnemosyne is the first sparse-memory substrate that lets language models learn continuously without retraining — replacing dense attention with a biologically-plausible consolidation loop.

Comment

Inspiration

Transformer attention is one of the most important inventions in modern AI — and it has a structural flaw. Attention is being asked to do two jobs at once: be a local-context operator (syntactic and immediate-semantic processing within the working window) and a global-memory operator (long-range retrieval across everything the model has ever seen). It is structurally bad at the second job.

What it does

Transformer attention is one of the most important inventions in modern AI — and it has a structural flaw. Attention is being asked to do two jobs at once: be a local-context operator (syntactic and immediate-semantic processing within the working window) and a global-memory operator (long-range retrieval across everything the model has ever seen). It is structurally bad at the second job.

Three limitations follow directly from this conflation. First, attention scales quadratically with context length — O(n 2 ) — which is why context windows are measured in thousands of tokens rather than millions of pages. Second, attention is stateless; every inference starts from an empty slate, which is the root cause of hallucination. Third, attention cannot distinguish what was just perceived from what was consolidated long ago — there is no learning.

The industry's response — longer context, sparse attention variants, retrieval-augmented generation — treats the symptoms. None diagnoses the disease. The disease is that attention is the wrong primitive for memory.

The inspiration came from biology. Biological brains solved this problem half a billion years ago: the hippocampus uses sparse, content-addressable episodic memory for recent experience; the neocortex stores slowly-consolidated semantic memory; a sleep consolidation pass transfers information between them. The two systems have different computational properties, different scaling laws, different failure modes. Transformers conflate them into a single dense matrix. Mnemosyne imports the biological split into language models.

How we built it

The prototype runs entirely in the browser — no external ML model dependency, no GPU, no API keys. The rules say "perfection is not expected, originality is," so I prioritized a live interactive demo over a heavyweight ML pipeline.

Core engine (src/lib/mnemosyne.ts): 256-dim sparse encoder with lemmatization + substring fuzzy matching for morphological variants ("consolidates" matches "consolidating"). Append-only memory store. Top-k MIPS retrieval with a learned gate (sigmoid over retrieval scores). Async k-means consolidation pass (k=3). Benchmark suite and single-exposure learning demo. API: 8 Next.js API routes — write, query, consolidate, reset, seed, stats, benchmark, single-exposure. UI: Next.js 16 + TypeScript + Tailwind 4 + shadcn/ui. Each memory entry's 16 active dims are rendered as a 16×16 grid, making the sparsity tangible. Framer Motion handles the retrieval animations. Moonshot Paper: A separate ~4,200-word PDF with 10 APA references (Kanerva 1988; Rolls & Treves 1998; Vaswani 2017; Lewis 2020; Rae 2016; Graves 2014; Hopfield 1982; Kumaran 2016; Beltagy 2020; Zaheer 2020), covering the problem, the first-principles insight, scientific foundations, architecture, preliminary results, and long-term implications. A crucial design choice: the memory is non-differentiable. Writes are append-only — there is no gradient through the buffer. This is the opposite of differentiable neural computers. The non-differentiability is deliberate: it means the memory is stable (old entries are not silently rewritten), the system can learn from a single exposure, and the consolidation pass can be approximate without propagating errors.

Challenges we ran into

The benchmark honesty problem. My first benchmark design had Mnemosyne beating the baseline 10–0, which I didn't believe. On investigation, I found that my queries shared exact tokens with the target memories, so the baseline was winning for the wrong reason. I redesigned the queries to use morphological variants where the lemmatizing encoder has a genuine structural advantage. The honest result on a small buffer is that Mnemosyne and the exact-token baseline achieve comparable accuracy — but Mnemosyne's structural advantages (sparse representation, O(n⋅k) retrieval, single-exposure learning) appear at scale.

The "is this just RAG?" framing. The single hardest design decision was figuring out how to make the demo visibly different from RAG. The answer was the single-exposure learning demo: write a brand-new fact, immediately query it, watch Mnemosyne retrieve it. RAG retrieves from a fixed external corpus; Mnemosyne retrieves from a per-model buffer that is continuously updated by the model's own experience. RAG cannot learn "Brewlab on 3rd Avenue" without updating the document store. Mnemosyne learns it from one append to the buffer.

Scope discipline in a 17-hour window. The hardest challenge was resisting scope creep. I cut: a real sentence-transformer encoder (replaced with a hashing-based proxy with lemmatization), a real generative decoder (the prototype retrieves but doesn't generate), and a true async consolidation pass (it's synchronous in the demo). Each cut was painful but each one kept the project shippable.

Accomplishments that we're proud of

The single-exposure demo works on the first query. I expected the hashing-based sparse encoder to be too weak for single-exposure learning to actually retrieve a fact written milliseconds earlier. In practice, the lemma + substring fingerprint captures enough semantic signal that 3/3 facts are retrieved on the first query. The principle holds. The architecture is retrofit-ready. Because the decoder is frozen and only the sparse projection + retrieval gate are trained, Mnemosyne can be added to any existing pretrained transformer without retraining the base weights. This makes the path from prototype to production unusually clean. The honest benchmark. I chose not to game the benchmark. The prototype admits that on a 12-entry buffer, sparse retrieval and exact-token retrieval are comparable on accuracy — and then makes the structural argument (sparsity, complexity scaling, lifelong learning) for why Mnemosyne wins at scale. Honesty is a feature.

What we learned

Sparse encoders are surprisingly effective. I expected the hashing-based sparse encoder to be a weak proxy for a real sentence-transformer. In practice, the lemma + substring fingerprint captures enough semantic signal that single-exposure learning works on the first query. This validated the architecture's core claim: the primitive matters more than the encoder quality. The "just RAG" objection has a real answer. The hardest intellectual work was figuring out why Mnemosyne is not RAG with extra steps. The answer: RAG retrieves from a fixed external corpus; Mnemosyne retrieves from a per-model buffer that is continuously updated by the model's own experience. This is a different computational category, not an incremental improvement. Non-differentiability is a feature, not a bug. I initially planned to make the memory end-to-end differentiable (differentiable neural computer style). I'm glad I didn't. The append-only design is what enables single-exposure learning — there is no gradient to silently rewrite old entries. Stability comes from refusing to learn the wrong thing.

What's next for Mnemosyne

Built With

Share this project:

Updates