Context Memory System

• 

  "Git-style version control for AI agent memory

Inspiration

Every long-running AI agent eventually hits the same wall: it either runs out of context space or silently forgets things that mattered. We noticed that most memory solutions treat this as a simple storage problem — summarize old messages, drop what seems irrelevant, move on. But this "black box" approach means users have no idea what got kept, what got compressed, or why.

We asked ourselves: what if AI memory worked more like version control? Developers trust Git with their most important work because every change is tracked, reversible, and explainable. We wanted to bring that same trust and transparency to AI agent memory.

What it does

Our system treats every memory update as a versioned commit — recording what was kept word-for-word, what was summarized, what was discarded, and the reasoning behind each decision. On top of this foundation, we built:

• Checkout — jump back to any earlier memory state and resume from there
• Branching — explore an alternate direction without losing the original path
• Merging — combine branches back together, with the system flagging genuine conflicts instead of silently overwriting facts
• Memory Lineage — every fact carries its history: when it changed, why, and how confident the system is
• Self-Healing — a background check continuously scans for contradicting facts (e.g., a budget figure that changed without explanation) and asks the user to confirm which version is correct
• Relevance Scoring — a lightweight heuristic estimates how likely a piece of information will be needed again, guiding what stays detailed and what gets compressed

The result is a memory system where nothing is hidden: users can inspect, correct, branch, or roll back the agent's understanding at any point, instead of restarting from scratch.

How we built it

We started by mapping the problem to a familiar mental model — Git — and worked backward from there:

1. Storage layer: a lightweight database (SQLite) stores each memory update as a commit object with fields like parent_id, timestamp, topic_tag, action, reason, and confidence, forming a tree structure similar to a commit graph.
2. Classification layer: an LLM call scores each incoming piece of conversation for importance and relevance, deciding whether it should be kept in full, summarized, or discarded.
3. Branch/Merge logic: branching simply forks the commit tree from a chosen point; merging compares commits across branches and uses an LLM to detect and resolve conflicting information.
4. Self-healing check: a periodic scan compares facts sharing the same topic tag, flagging contradictions for the user to resolve rather than silently picking one.
5. Interface: a dashboard visualizes the commit history as a timeline and the branches as a simple tree, so users can see and interact with the agent's memory the same way they would browse commit history in a code repository.

We built the backend with Python/FastAPI, used an LLM API for the reasoning-heavy steps (classification, conflict resolution, relevance scoring), and kept the frontend lightweight so the focus stayed on making memory operations genuinely visible and interactive.

Challenges we ran into

• Deciding what counts as a "conflict" vs. a natural update. Not every change in a fact is a contradiction — sometimes information is simply updated over time. We had to design a way for the system to distinguish an intentional update from a genuine unresolved conflict, and ask the user rather than guessing.
• Balancing compression with the ability to explain "why." It's easy to summarize aggressively to save tokens, but that risks losing the reasoning trail. We had to make sure every compression decision still left behind enough of a trace to answer "why did you drop this?" later.
• Keeping relevance scoring honest. We were careful not to oversell this as a "predictive AI model" — it's an LLM-based heuristic, not a trained prediction system, and we made sure our design and messaging reflected that accurately rather than overclaiming.
• Making version control intuitive for non-developers. Git's mental model is powerful but unfamiliar to most end users, so a large part of our effort went into the UI — making branches, commits, and merges understandable at a glance, not just technically correct.

What we learned

Treating memory as a first-class, inspectable object — rather than an invisible cache — changes the trust relationship between a user and an AI agent entirely. Small design choices, like requiring a "reason" field on every memory decision, forced us to think much more carefully about explainability from the ground up rather than bolting it on afterward.

Built With

• anthropic
• claude
• css3
• fastapi
• git
• gpt-4
• html5
• javascript
• json
• llm
• node.js
• openai
• prompt-engineering
• python
• react
• rest-api
• sqlite
• tailwindcss
• vector-embeddings