Palimpsest

Memory that shows its edits. A DataHub agent that writes down only what changed, and supersedes a belief instead of overwriting it.

Comment

Open the console → — real beliefs out of the running system: what it believes about eight datasets, what it used to believe, and the reasoning that replaced it. No install.

Inspiration

An agent pointed at a data catalog re-reads the whole thing on every run, because it has nowhere to put what it learned last time. It pays for the same conclusions twice. Worse, it cannot answer the question people actually ask about a catalog: what changed?

Bolting a vector store onto it does not fix that. It makes it worse in the one place a catalog needs it most. A dataset whose owner changed three times ends up with three equally-retrievable memories, all stated as fact, and the agent has no way to know which one is current.

What it does

Palimpsest gives that agent a memory that revises itself.

It reads DataHub through the official MCP server, compares what it sees against what it already knows, and writes down only the difference. On an unchanged catalog it writes nothing at all. When the catalog contradicts a memory - the owner changed, columns were dropped, the description was rewritten - the old belief is not overwritten. It is superseded, kept, and linked to the evidence that replaced it.

A palimpsest is a manuscript where new text is written over old and the old text still shows through. That is the whole design.

Measured on the sample catalog, from the recorded demo:

  • first pass: 27 observations written
  • second pass, nothing changed: 0 written, 27 skipped, 100%
  • after one ownership change: 1 written out of 27, flagged as contradicting a held belief

And it gives the conclusion back to the catalog. palimpsest publish attaches an institutionalMemory link to each dataset, described by what is actually in memory - "Agent memory - 5 beliefs, 5 revisions (latest 2026-08-05)" - so the next person who opens that dataset in DataHub sees that something has been reasoning about it, and can click through to what it believes and what it used to. A link rather than a description: a description is somebody's writing, and an agent that overwrites one gets switched off within the week. The aspect is replaced wholesale on write, so existing links are read back and rewritten alongside ours - nobody's runbook disappears.

And it is reachable from the agent doing the work. Palimpsest ships a DataHub Skill in the same SKILL.md format as the DataHub skills registry, so it installs beside datahub-search and datahub-lineage rather than competing with them:

npx skills add GDLevSargsyan/palimpsest

The skill writes down the discipline the schema already enforces, where an agent will read it: get the brief before touching a dataset, never present a superseded belief as current, quote the consolidator’s reasoning rather than paraphrasing it, and write episodes — never facts. Its routing table sends catalog questions back to the DataHub skills. This one only answers the questions DataHub cannot, which are the ones about what changed.

How I built it

Three memory tiers live in one CockroachDB, with the vectors beside the rows:

Tier What lives there Written by
working the task scratchpad, deliberately not embedded the agent, per task
episodic what was observed, embedded, decaying unless recalled the scanner
semantic what is believed, versioned, with evidence attached the consolidator only

The live agent never writes to semantic memory. A separate consolidation pass - sleep-time, not request-time - clusters episodes, distills them into checkable claims, and reconciles each against what is already believed: a restatement reinforces, a contradiction supersedes and records why, anything else is new.

Episodes decay and are reinforced on recall, so a memory earns its keep by being useful. An episode that decayed, was never recalled, and has already become a fact is dropped - unless it is the last evidence under a live belief, which is never discarded.

DataHub tools used: the Cloud Managed MCP Server as the primary catalog source (search, get_entities, get_lineage, list_schema_fields), with a GraphQL source kept for backfills.

Stack: CockroachDB Cloud with distributed vector indexing, Gemini embeddings, and Groq / Gemini / Claude / Ollama behind one interface for the consolidation judgements.

Challenges I ran into

Six defects, and every one of them was silent. None threw an error; each reported success while doing the wrong thing. That pattern is the real story of this build.

  1. The scanner deduplicated by vector distance and quietly dropped lineage observations, because every sentence contains the dataset's name so unrelated aspects sit close together in vector space. It reported a high skip rate and looked efficient. Fixed by deduplicating per aspect, not per distance.

  2. Switching catalog source read as a catalog change. MCP returns the description someone edited in the UI, GraphQL the one the source system published. An observation now belongs to its observer.

  3. The model invented its own predicates - owner on one run, upstream_dependency on the next. Since contradictions are only matched within a predicate, a renamed predicate means an ownership change never supersedes anything. The predicate now comes from the observation, not the model.

  4. Episodes were marked consolidated even when the model never answered. A rate-limited free tier was retiring evidence permanently. A cluster the model could not answer for is now left for the next pass.

  5. Re-consolidation invented changes. Distillation is not deterministic, so re-running it over one episode produced a differently worded claim and the reconciler read two wordings as a disagreement. A belief is now identified by the evidence it rests on, not by its phrasing.

  6. Forgetting cascaded away the evidence under live beliefs - 39 of 43 facts were left unfalsifiable. The system still answered questions; it just could no longer show why it believed anything.

Accomplishments that I'm proud of

Finding the six above rather than shipping over them. Each one had a green dashboard on top of it.

Also: every number in the demo, the README and the video is real output. Clone the repo and you get these exact strings back, odd counts and all.

What I learned

That a memory layer fails quietly by default. Every failure mode here degraded into a system that looked like it was working - high skip rates, clean logs, confident answers - and only became visible by checking the invariant behind the number rather than the number itself.

Contributed back

Two fixes to mcp-server-datahub, both found by building on it, and both about the same thing: the caller is a model, and a message it cannot act on costs a round trip it has no way to ask a follow-up question about.

  • #184 — get_lineage’s docstring documents a COUNT PARAMETER and tells the caller to raise count. The signature has no count; the knob is max_results. An agent that believes the docstring passes an unknown keyword and gets a validation error naming neither the right parameter nor the wrong one.
  • #185 — platform IN () raised Expected value at position 13, got ), which names the token the parser wanted rather than the mistake the caller made. That empty list is usually one the model computed a moment earlier, and the right move is to drop the condition. The new message says so. The parser’s own docstring listed this under Known limitations; the patch removes the bullet by fixing it.

What's next

Confidence that means something (it is near-constant for catalog metadata today), decay constants tuned against a real deployment rather than reasoned about, and column-level memory so a schema change points at the field that moved.

Built With

  • agent-memory
  • claude
  • cockroachdb
  • datahub
  • datahub-skills
  • fastapi
  • gemini
  • groq
  • mcp
  • psycopg
  • python
  • vector-search
Share this project:

Updates