Hindsight

The on-call agent that gets cheaper every time it runs: it files each postmortem inside DataHub, so the next incident starts from the last one. Measured: 20 tool calls cold, 15 warm.

Comment

Plenty of agents in this gallery walk DataHub lineage to a root cause. This one is the one that gets cheaper the second time.

Hindsight files its postmortem inside DataHub, and the next investigation retrieves it before it walks a single edge: 20 tool calls cold, 15 warm — both runs on record, replayable offline, and the example states the cost as well as the gain.

Then it re-reads every change it wrote through the GMS GraphQL API — not the MCP tools that wrote it. That is how it caught a real defect: verified 5/6, a second run silently overwriting the first one's incident banner. Both runs had reported success. The failing file ships unaltered.

And none of it is locked in this codebase: npx skills add gmassello/hindsight runs the same procedure in your own CLI with no Python in the loop, and handed the same incident it reaches the same fourteen owners, set for set — proposed upstream to the official DataHub skills repo.

The memory loop, measured

The learn phase saves the postmortem with save_document, inside DataHub. The next investigation's recall finds it before it starts walking lineage, and turns it into hints that steer root_cause. That is the 20-vs-15 above: the warm run did not re-derive what the cold run had already concluded.

The gain is not free, and the example says so out loud: the warm run swept 6 consumers against the cold run's 29. Memory bought speed and cost coverage. Both runs ship with their raw event streams, so hindsight replay examples/02-cold-vs-warm/warm reproduces either one with no DataHub and no API key.

The memory also outlives the agent. It lives in the catalog, so the next on-call inherits it whether or not they ever run this thing.

An agent that reports its own success proves nothing

So hindsight verify re-reads every mutation in a run's audit log straight from DataHub through the GMS GraphQL API — a different channel from the MCP tools that wrote it — and exits non-zero if anything is missing.

It immediately found something the agent had not: two runs against the same table silently overwrote each other's incident banner, because update_description replaces rather than appends. Both runs had reported success. The failing verify.txt ships as it came out, the failure included.

None of it is locked in this codebase

The same procedure also ships as a portable Agent Skill. npx skills add gmassello/hindsight installs it into whatever CLI you already use — Claude Code, Cursor, Codex, Gemini CLI and seven more — with no Hindsight code in the loop and nothing cloned.

Handed the same incident as the reference implementation, it reached the same 14 owner URNs, set for set, in 9 investigation tool calls, applied three approved mutations, and saved a postmortem that a re-search retrieved. It is proposed upstream to datahub-project/datahub-skills.

What it does

Give it a free-text alert — "orders in order_entry_db is showing NULL values in customer_id since 03:00 UTC, downstream order reports look wrong" — and it runs eight phases against DataHub:

  1. intake — parses the report into symptom + mentioned assets
  2. resolve — turns names into URNs, and states the ambiguity it resolved (three tables are called orders; it says which one it picked and why)
  3. recall — searches past postmortems stored as DataHub documents, before investigating
  4. impact — walks downstream lineage and computes the blast radius with a deterministic formula: 29 consumers ranked, 14 deduplicated owners to page
  5. root_cause — walks upstream and ranks competing hypotheses, each citing the URNs it rests on
  6. propose — an action plan as a dry run, with a reason per mutation
  7. → human approval gate ←
  8. commit — applies add_tags, update_description, add_owners over MCP
  9. learn — saves the postmortem with save_document, inside DataHub

Inspiration

Data on-call is a job of re-derivation. A dashboard is wrong since 03:00, and whoever is paged opens a lineage graph and starts clicking: which table actually broke, who consumes it, who do I wake up, what changed upstream. Two weeks later the same thing happens and none of that work exists anymore — the conclusion died in a Slack thread while the catalog, the one place everybody looks, still says the table is fine.

That is the gap Hindsight is built for. Not "an LLM that answers questions about your metadata", but an agent that finishes the job the way a human on-call has to finish it: with the catalog updated and a postmortem someone else can find.

How we built it

  • A deterministic phase state machine with an LLM inside each phase, not a free ReAct loop. Each phase gets a whitelisted subset of MCP tools and a terminal submit_* tool whose schema is a Pydantic model. Phases are marked critical or not: a non-critical failure emits an error event and the run continues with partial state.
  • The math is not in the model. impact has the LLM report raw consumers; the score is computed in Python. The same graph always produces the same ranking.
  • 13 DataHub MCP tools, 8 read and 5 mutation, with tool schemas discovered at runtime — the mutation prompts are built from the live schemas rather than hardcoded shapes. Mutations that fail over MCP retry through GMS GraphQL.
  • Python 3.13 + FastAPI with SSE, a React frontend with zero runtime dependencies beyond React (native EventSource, plain CSS), Gemini as the default provider, and a hosted demo on GitHub Pages that replays captured runs with no backend at all.

Challenges we ran into

  • datahub datapack load returns before the lineage graph exists. The ingestion reports success in a second while the index fills for minutes. A run started too early produces a real-looking investigation with a blast radius that is quietly wrong — no error, just fewer consumers. It cost a full re-capture of every recorded run.
  • The negative control we could not honestly build. We wanted a run where the agent exonerates the pipeline. Four attempts, none exonerated — and the diagnosis was not the prompt: the demo catalog has no freshness, no row counts, no job history, so "the pipeline is healthy" is not a claim that metadata can support. We cut the scenario and kept what stands on its own: a three-state verdict where exonerated forces an empty plan in code, and a grounded-hypothesis rule — every hypothesis must cite a URN a tool actually returned.
  • Gemini rejects $ref/$defs in function declarations, and a property legitimately named title has to survive the same pass that drops title as schema metadata.

Accomplishments that we're proud of

  • Five captured runs across four directories, each replayable offline with hindsight replay <dir> — no DataHub, no API key.
  • The clean-clone test. The repo cloned from GitHub into a fresh directory, following only the README: uv sync, ruff, pytest (30 passed), hindsight serve answering on :8000, npm ci + npm run build. It surfaced three real defects, all fixed.
  • Verifying the Skill surfaced three defects in the Skill, all fixed before it was proposed upstream. Proving portability was not a formality; it was a bug hunt.

What we learned

Verification through a second channel is worth more than any amount of self-reported success — and the corollary is that honest limits are cheaper to ship than to hide. Every caveat we ended up writing down (what memory costs, the negative control we could not build, the mutation that never fired in a captured run) turned out to be a stronger claim than the version that pretended it did not exist.

What's next for Hindsight

Append-only incident banners, freshness and volume signals so the agent can genuinely exonerate a healthy pipeline, and raising a formal DataHub incident as part of the commit phase.

Built With

Share this project:

Updates