Hindsight - The Context Operations Platform for DataHub

Ten agents over a time machine for DataHub's context graph. Reconstruct what the catalog said at any past instant, replay any agent decision against it, and prove if agent or the metadata was wrong.

Comment

Inspiration

DataHub's own strategic thesis says the capability regulated buyers should weigh most heavily is a "timestamped, versioned context graph" that can "reconstruct exactly what context an agent acted on."

DataHub cannot currently do this. We verified it in the source at commit b011808d, not in the docs:

Claim Evidence
No getAspectAsOf(timestamp) API exists grepped the codebase, there is none
GET /aspects/{urn}?aspect=X&version=N returns no timestamp VersionedAspect.pdl is {aspect, version} only
The lineage UI cannot show historical lineage docs, verbatim: "Selecting time windows will not show you historical lineages. It will only filter the view of the latest version."
DataHub discards its own history by default retention.yaml: maxVersions: 20, applied on every ingest, not retroactively fixable

When an agent reads and writes production metadata at scale and gets it wrong, nobody can answer the only question that matters:

Was the agent wrong, or was the catalog wrong?

Those have opposite remedies. One says retrain or reprompt the agent. The other says fix the metadata and go find everyone who consumed it while it was wrong. Today you cannot tell them apart, so blame gets assigned by vibes and teams either distrust every agent or trust all of them.

What it does

Ten specialist agents share one time-aware context graph, one orchestrator, one findings ledger, and one policy gate. They hand findings to each other, so running them together produces strictly better answers than running them apart.

Agent What it does
chronicle Reconstructs the catalog at any past instant, replays agent decisions against the context they actually had, finds context defects
chokepoint Solves monitor placement as a weighted set-cover over source-to-consumer lineage paths, proves it with Monte Carlo failure injection
contrail Derives consumer-oriented data contracts from column lineage intersected with real SQL and profile history
stormbreak Collapses an alert storm to one root cause via lineage dominators, resolves one owner, suppresses the rest
quarantine Column-scoped blast radius — halts only the affected subgraph and keeps the rest running
datasheet Renders training-data provenance documents from ML lineage with Annex IV gap analysis
escheat Infers owners for unowned assets from four independent graph signals, scored by agreement
ledger Cost flows down lineage, value flows up — finds assets with cost above zero and consumers at zero
parallax Detects semantic drift: who consumed a metric whose definition silently changed
actuary Failure-probability table from six graph and history features, with fitted coefficients and reported calibration

Three handoffs are wired:

stormbreak ──root cause──▶ quarantine   contain what actually broke, not a guess
chronicle  ──defect urns─▶ parallax     was the correction semantic?
escheat    ──owners──────▶ stormbreak   page someone real

Demo scenario: On 2026-06-16 an agent read prod.raw.users and found email_hash described as "an irreversible one-way hash containing no recoverable personal data," the asset tagged safe-to-export, and no PII tag anywhere. It included the column in a partner marketing export — correct call, given the catalog.

On 2026-06-29 privacy review corrected the record: the hash is salted and re-identifiable by anyone holding the salt table.

HINDSIGHT replays the decision against the context it actually had and returns CONTEXT_DEFECT: the agent reasoned correctly, the catalog was wrong. It bounds the damage — 13 days and 4 downstream entities — and writes an incident, a context defect report, and structured properties back into DataHub so the next person or agent inherits the knowledge.

Three surfaces: MCP server (mounts alongside DataHub's own), Slack bot with human-approval gates on every proposed write, and an offline web console (hindsight serve --offline, no build step, no external services).

How we built it

There is exactly one bridge across DataHub's historical data, and it is undocumented. Every Timeline API change transaction carries a versionStamp:

schemaMetadata:4;globalTags:2;ownership:1;glossaryTerms:3

That string is the complete aspect-to-version map at that instant. Join it to the versioned aspect store and the graph reconstructs exactly, with a receipt for every field:

1. GET /openapi/v2/timeline/v1/{urn}?startTime=0&endTime=<T>
2. take the last transaction at or before T, parse its versionStamp
3. for each (aspect, version):  GET /aspects/{urn}?aspect=..&version=..
4. hydrate timeseries aspects over the window

DataHub ships git log for metadata. HINDSIGHT adds git checkout, git blame and git bisect, then puts ten agents on top of it.

Every agent produces its findings with --no-llm. The LLM is permitted exactly three jobs: classify a delta as CORRECTION / EVOLUTION / NOISE, filter deltas to those causally relevant to a decision, and narrate structured findings into prose. It is never allowed to invent a URN, a timestamp, a lineage edge, or a number. Every quantity comes from the graph.

That split means the simulations are seeded and reproducible, and a judge can verify the engine independently of model behaviour.

Challenges we ran into

versionStamp is entirely undocumented. We reverse-engineered it from TimelineServiceImpl.java — it is the only key to point-in-time reconstruction and is easy to dismiss as an internal implementation detail. Without it, the whole platform is impossible. With it, it is four HTTP calls.

retention.yaml sets maxVersions: 20 by default, applied on every ingest. By the time anyone asks "what did we know in March", March is already gone — and unlike most configuration mistakes, it cannot be fixed after the fact.

Two bugs we shipped and fixed both produced plausible wrong answers rather than crashes: a monitor-placement simulation that scored leaf-heavy placements at 100% because it never checked whether detection happened before a consumer saw the data, and a replay path that folded the asset name into the delta key, silently downgrading real context defects to INDETERMINATE. Both are now regression-tested. A plausible wrong answer is worse than a crash.

Accomplishments that we're proud of

  • Discovered and documented versionStamp as the key to point-in-time reconstruction — entirely absent from DataHub's public docs
  • Built a deterministic replay engine: every quantity comes from the graph, LLMs only narrate
  • record_decision MCP tool that turns any MCP-speaking agent into an auditable one without changing its code
  • aiAgent entities with upstreamLineage — registering an agent puts it into the lineage graph so impact analysis on a table shows which agents depend on it (landed OSS v1.7.0, 2026-08-04)
  • Eight upstream contributions to DataHub: docs fixes, feature proposals, and an RFC for lineage edge validity intervals
  • An actuary agent that reports its own AUC and falls back to a documented prior — because an uncalibrated risk score is astrology

What we learned

Reading DataHub's Java and PDL source rather than its documentation changed the project twice. It is how we found versionStamp — the only reason point-in-time reconstruction is possible at all — and how we found the 20-version retention default, which quietly makes it impossible if you do not set it before you need it.

The second lesson was about honesty in output. Two bugs we shipped and fixed both produced plausible wrong answers rather than crashes. Both are now regression-tested. A plausible wrong answer is worse than a crash, and an uncalibrated risk score is astrology, which is why actuary reports its own AUC and falls back to a documented prior when there is not enough history to fit.

What's next for Hindsight - The Context Operations Platform for DataHub

  • Wire validTo on lineage edges upstream so historical lineage can prove edge removal — the one genuine gap that remains
  • Column-level fineGrainedLineage reconstruction at a point in time, so quarantine can be column-precise historically and not just currently
  • Ship record_decision as a DataHub Skill hook so every agent in an organisation becomes auditable by default
  • Point ledger's price flags at real warehouse billing exports

Built With

  • anthropic
  • datahub
  • datahub-mcp
  • fastmcp
  • httpx
  • networkx
  • openai
  • pydantic
  • python
  • rich
  • slack-bolt
  • sqlglot
  • sqlite
  • svg
  • typer
  • vanillajs
Share this project:

Updates