Chronical Graph

Chronicle Graph gives AI agents durable memory for data incidents: real DataHub context, calibrated confidence, human-confirmed patterns, and approval-gated write-back to DataHub

Comment

Inspiration

AI agents are good at producing a plausible answer once. They're much worse at remembering what happened across weeks of incidents, telling a confirmed root cause apart from a tempting guess, and explaining why they trust one piece of context over another. We wanted to build the missing memory layer for that — one where an agent's confidence is earned from real evidence, not asserted.

What it does

Chronicle Graph investigates data incidents using live DataHub metadata, lineage, and cached evidence from real public postmortems. It ships with three investigable cases: one demo incident wired to a live local DataHub instance, and two real public postmortems (GitHub's 2020 database outage, Firetiger's 2026 ingest outage) investigated by the exact same strict-JSON pipeline — no scripted content anywhere. The Archivist keeps competing hypotheses visible, cites evidence IDs for every claim, and returns a calibrated confidence score with a stated reason. When a resolved incident is human-confirmed, it's compiled into a reusable failure pattern with a bounded, capped confidence boost — never enough to override current evidence. Any DataHub change starts as a proposal and only applies after an explicit human approval, verified end-to-end against a real running DataHub instance. Every node in the investigation graph is required to trace back to something real — a DataHub entity, fetched postmortem text, or a confirmed pattern in Postgres — never an AI-generated artifact.

How we built it

Four boundaries: (1) a DataHub adapter using the Agent Context Kit with a GraphQL fallback, caching live context locally; (2) the Chronicle graph itself, storing entities, evidence, confirmed patterns, and proposals in PostgreSQL; (3) an investigation agent that builds a selective context package per case and returns strict JSON-schema-validated output with retries; (4) a validation and approval layer that gates every DataHub write behind an explicit decision. On top of that, we built a full MCP server (@modelcontextprotocol/sdk) exposing the same context-search, pattern-matching, and proposal tools over both Streamable HTTP and stdio, so any MCP-compatible agent host can drive Chronicle directly. The runtime is Node.js, a local DataHub Docker stack, the DataHub Python SDK for seeding and for applying approved writes, and OpenAI's Responses API with local content-addressed caching.

Challenges we ran into

Getting DataHub running locally on Windows was the first hurdle — a Python 3.11 Conda environment and a secrets-encoding issue in the quickstart. The harder problems were about trust, not plumbing. Our evidence scraper was silently truncating real postmortems before their actual content even started, feeding the model pure page navigation and producing a useless investigation — we caught it by checking where recognizable incident language actually appeared in the raw text. Separately, we caught the model fabricating "confirmed pattern matches" with invented IDs for incidents that had zero real historical overlap — it was treating a schema field as an invitation to always produce something, rather than reporting nothing when nothing was there. Both needed explicit, adversarial fixes (extraction anchored to real text markers; instructions that require an empty result when there's no real match), not just better prompting. We also had to resist the urge to let a nice-looking graph win over an honest one — a UI iteration that added "alternative explanation" and "unresolved question" nodes looked great but blurred the line between AI reasoning and sourced fact, so we pulled it and replaced it with real named entities read literally off the postmortem text instead.

Accomplishments that we're proud of

  • Two independent real public postmortems are investigated by the identical strict-JSON pipeline as the DataHub-connected demo case — and correctly return zero fabricated pattern matches.
  • Every node in the investigation graph is traceable to a real source; we treat "looks impressive" as disqualifying if it isn't backed by something real.
  • The full governed write-back loop is verified live: a proposal, an approval, and a tag that we independently confirmed landed in a running DataHub instance via its own API.
  • A real MCP server, not a placeholder — Streamable HTTP and stdio both tested end-to-end.
  • We found and fixed two separate trust-undermining bugs (truncated evidence, fabricated patterns) before they could ship, by treating our own model's output with the same skepticism we're asking it to apply to evidence.

What we learned

Persistent memory is more useful as linked, inspectable knowledge than as a giant transcript, and confidence needs a shared rubric and a stated reason, not just a number. The bigger lesson: an LLM doesn't self-correct just because the rest of your prompt is careful — it will fabricate a "match" or misread a truncated document unless you give it an explicit, adversarial instruction and actually check its output against ground truth. Trustworthy memory has to be verified, not assumed.

What's next for Chronicle Graph

A stateful, multi-turn MCP session for longer-running agent conversations; a labeled evaluation set to calibrate confidence instead of relying on rubric self-report; extending the real-named-entity approach so more of the graph (teams, deployments) draws from literal declared facts instead of staying dataset-only; and a genuine upstream contribution back to the DataHub project itself — a documentation fix or small Agent Context Kit example — which we haven't done yet and want to be upfront about.

Built with

DataHub, DataHub Python SDK, GraphQL, Agent Context Kit, Model Context Protocol, OpenLineage concepts, Node.js, Python, Docker Compose, PostgreSQL, OpenAI Responses API, strict JSON Schema, local content-addressed caching.

Built With

Share this project:

Updates