Witness Graph

Tells you whether a metric dropped because customers left or because the instrument broke. Every claim cites a DataHub URN, and an external scorer reads only the run ledger.

Comment

Inspiration

Every team that has run a dashboard has had the same argument. Signups drop, and half the room says churn while the other half says the pipeline. The two answers lead to opposite work: one is a retention campaign, the other is a day reading ingestion logs. Picking wrong costs a week either way.

The distinction is not visible in the metric. It is visible one layer up, in the assets that feed it, which is exactly what a catalog holds and a dashboard does not.

What it does

Witness Graph takes one symptom, walks the DataHub lineage above the affected metric, forms competing hypotheses, gathers evidence for each, and refuses to commit to a verdict until every hypothesis has been argued against. The write-back path that returns the finding to the catalog is connected and gated behind approval: with approval, a run tags the datasets its verdict cited and records the finding as their editable description. Without approval nothing downstream of the approval node runs at all, which is the default.

It answers with one of four verdicts: instrument failure, upstream data defect, customer behaviour, or definition change.

How we built it

Eleven nodes. Only five are agents. The other six are code, and that is the design: the checks that keep an investigation honest are not prompting.

  • witness_gate fails a run unless every hypothesis has a witness, every source resolves against the catalog, at least one hypothesis is refuted, and no claim is unsourced. A confirmation-bias run cannot pass it.
  • citation_gate fails a run when a cited witness is absent from the ledger, or when a rejected alternative has no refuting evidence.
  • human_approval is the only node that can authorise an external change, and it has no auto-approve path.

Everything is written to an append-only ledger. The scorer reads the ledger and never the agent's own summary, so a run cannot flatter itself.

The benchmark

Eleven cases with sealed answer keys. Each is generated by seeding a specific fault into the estate, so ground truth is determined by construction rather than by our opinion.

Three cases are controls where nothing is broken and customers really did change, and a fourth is a definition change rather than a defect. They exist because a benchmark made only of broken pipelines measures bias instead of judgement: an agent that always answers "instrument failure" scores 6 of 11.

harness/audit_citability.py asserts that every asset an answer key requires exists in the warehouse, or in the catalog under a name the scorer can match. It found four cases whose keys pointed at assets nothing published, and it fails the build if that recurs. It does not yet check that a tool in the node's allowlist can discover that asset, which is a gap two assertions passed through.

Challenges

The instructive failures were not model failures.

An early run answered correctly and scored zero on citation, because the scorer compared logical names against urns and the two vocabularies had never met. A second run refuted the correct hypothesis on sound evidence, because the defect lived in a view definition and nothing in the tool descriptions said view text was readable. A third case asked the agent to cite a job that the estate builder constructed and never published. A fourth pointed at an assertion that was in the catalog and discoverable by no tool at all, because search does not return that entity type. A fifth described a symptom in data the generator never seeded.

Each looked like the agent reasoning badly. Each was the harness failing to say what was available. We fixed the harness and the same model started answering.

What we learned

Detection accuracy is not a property of the model. It is a property of what the harness makes reachable and what it says is reachable, and those are different things. Every gain in this project came from closing the gap between them.

What's next

Extending the corpus beyond metric investigation, and turning the graph engineering rubric in docs/RUBRIC.md on graphs that have nothing to do with metrics. The rubric scores eight dimensions and defines its top level the same way throughout: the claim can be falsified from artifacts alone, without asking the model what it did.

Built with

Python, DataHub, DataHub MCP server, SQLite, JSON Schema, Anthropic Claude.

Try it out

Project URL: https://hdpark95.github.io/witness-graph/

Every figure on that page is the output of harness/score.py, run when the page was built, so the site cannot disagree with the repository. Each case links to the full investigation: which of the eleven nodes fired, every tool call, every witness gathered, and the verdict set against the answer key. The scorer's raw output is published beside it as summary.json.

Repository, Apache 2.0. README.md explains the method without requiring a checkout. Scoring needs no API key and no DataHub instance:

python3 harness/warehouse.py --all --out warehouse/
python3 harness/score.py --cases cases/ --runs runs/
python3 harness/check_submission.py --warehouses warehouse/

The last command is the one worth running. It fails the build when the repository is not in a state a judge could evaluate: a node declaring a tool that is not registered, a committed ledger that answers its own case wrongly, a case corpus that no longer regenerates identically, a local path leaking into a ledger. Each of those is a defect it caught here, not a hypothetical.

Built With

  • anthropic-claude
  • datahub
  • datahub-mcp-server
  • json-schema
  • python
  • sqlite
Share this project:

Updates