dhdr (decision records for DataHub agents)

• 

  dhdr refusal

• 

  dhdr certificate

 ## Inspiration

 I'd been chasing a specific failure in policy engines: two decisions, made
 against what looks like
 identical input, with identical logged reasoning, coming out opposite ways
 — and nothing in either
 record revealing which regime was actually in force. The log is honest.
 It's just answering a
 different question than the one you're asking three weeks later.

 When I looked at DataHub's agent surface, the same shape was there. An
 agent asks what depends on
 a table, gets an answer, decides. The answer is correct. But the answer is
 *about a moment*, and
 the record doesn't say which moment.

 What made it worth building rather than complaining about is that DataHub
 already stores the
 missing piece. Versioned aspects, `systemMetadata.version`, millisecond
 `lastObserved`, and an
 endpoint that will hand you any prior revision. The coordinate exists, one
 API call away — it just
 isn't reachable from where the agent stands. That's a much better problem
 than a missing feature:
 the fix is a binding, not an architecture.

 ## What it does

 An engineer wants to drop a column. An agent checks DataHub, finds nothing
 downstream reads it,
 and allows the drop. Three weeks later a dashboard is broken and someone
 asks why. The log says
 "no consumers found" — but lineage moves, and nothing says whether that was
 true *then* or merely
 true of a stale read.

 **dhdr binds every context read to the metadata revision in force when the
 read happened.** The
 record stops saying "zero consumers" and starts saying zero consumers
 *according to a named
 aspect revision, observed at a recorded instant*. The incident review can
 finally tell the two
 worlds apart.

 Three things follow from that, and they matter more than the binding
 itself:

 - **It refuses.** When a read can't be tied to a revision, the capability
 class collapses to
   *none* — never "C2 with a warning", which is exactly the phrasing a
 hurried reader takes as
   certification. It reports a class, never a percentage, because a score
 averaged over
   incommensurable kinds of missing evidence manufactures the confidence
 this exists to prevent.
 - **It does work, not just audit.** Each decision carries the concrete
 change: `ALTER TABLE …
   DROP COLUMN promo_code;` when the drop is safe, and a deprecation comment
 naming the consumer
   that still reads it when it isn't. A refusal that proposes the safe
 alternative is the useful
   part. Proposed, never applied — there's deliberately no code here that
 runs a migration.
 - **It writes back.** The certificate lands in the dataset's
 `institutionalMemory` as a resolvable
   link, so the next agent or engineer inherits it, and as a SARIF
 annotation on the pull request,
   where it can still change the outcome instead of sitting in a terminal
 somebody ran once.

 The demonstration is one agent making the identical `get_lineage` call
 twice, seconds apart,
 through DataHub's real MCP server, while a pipeline change lands in
 between. Identical call.
 Opposite decisions. The agent's ordinary reasoning does not identify which
 metadata world it saw;
 dhdr does — each decision names the revision it was bound to.

 ## How we built it

 **A falsification probe before any product code.** The premise was that the
 MCP surface drops a
 coordinate the platform stores. If that were wrong, the entry was wrong,
 and I wanted to know on
 day one rather than day six. The probe settled it from shipped source:
 `grep -rn systemMetadata gql/`
 returns nothing, and `schemaMetadata(version: 0)` is hardcoded at
 `gql/entity_details.gql:1652` —
 version 0 being DataHub's latest-sentinel. The one place the API takes a
 version is pinned shut on
 the way up.

 **Test-first, throughout.** Every unit began as a failing test I watched
 fail for the right
 reason. That discipline caught things I'd otherwise have shipped: an
 unbound read that quietly
 claimed `bundled` provenance, a certifier that would happily pair one
 decision's record with
 another decision's reads.

 **Real transport, real data.** The agent goes through FastMCP's client
 against the actual
 `mcp-server-datahub`, not an imported tool function — if you're going to
 claim you built *with*
 the Agent Context Kit, the claim should survive someone reading the code.
 And every scenario
 decides over real `showcase-ecommerce` entities, because a hand-invented
 `analytics_orders` reads
 as a toy and, more practically, MCP resolves upstreams that exist *as
 entities*, so a fabricated
 URN produces a demo that silently binds to nothing.

 **A second scenario to test the architecture rather than assert it.**
 Access governance reads
 `glossaryTerms` instead of `upstreamLineage` — different aspect, different
 predicate, different
 question. It needed no change to the capture core, which is the claim. It
 also immediately found a
 bug the first scenario couldn't: the coordinate layer was reading through
 the OpenAPI v2 endpoint,
 which returns 400 for `glossaryTerms` while working fine for
 `upstreamLineage`. A lineage-only
 test suite reported a healthy read layer that couldn't read three aspects
 at all.

 **An ablation that could have gone against us, and did.** Remove each
 captured field, see what the
 certifier can still claim. The result: deleting the revision from the
 record costs *nothing*,
 because the upstream verifier doesn't model DataHub versions. I published
 that rather than
 deleting the row, because the honest conclusion is stronger — the
 load-bearing part is the
 *refusal*, not the annotation.

 ## Challenges we ran into

 **The lineage cache, which cost the single largest block of time.** A
 lineage change reached the
 graph index in 2–4 seconds and `get_lineage` kept returning the old answer.
 GMS logged
 `Lineage Graph = size:0` while the identical GraphQL query by hand returned
 the edge. So I went
 hunting for a broken indexing pipeline that didn't exist — OpenSearch
 health, the graph index
 (the edge was there), Kafka consumer lag (zero), the MCL consumer. The
 actual cause is that
 `get_lineage` reads through a cache whose default TTL is a day.

 **Two platform gaps that shaped the design.** `institutionalMemory` has no
 registered patch
 template, so a server-side append fails with a null-template NPE — and
 `If-Version-Match` isn't
 enforced on the write endpoint, so a stale-version write returns 200 and
 overwrites. Both closed
 off atomic write-back. Rather than claim a guarantee I couldn't make, the
 README states the
 limitation in those words and a test pins it, failing the day either
 mechanism starts working.
 Then I wrote the missing template and opened a PR.

 **Aspect retention silently breaking history.** Retention keeps ~20
 versions, so on a busy aspect
 version 1 is gone. My history walk scanned upward from 1, stopped at the
 first gap, and reported
 "no history at all" — no error, every read binding to nothing, on exactly
 the instances that have
 been alive long enough to matter.

 **A cold read that found what I'd missed.** Late on, I had a separate AI
 agent — deliberately
 given no project context, and barred from my notes and git history — follow
 the README exactly as
 a judge would. It got **7 failed tests** where the README promised a clean
 run. Two
 causes: `.[dev]` never installed `acryl-datahub` or `mcp-server-datahub` —
 declared nowhere, and
 invisible to me because my own environment had them from the probe work —
 and the test suite
 decided "is DataHub up?" by trusting a bare 200 on port 8080, the most
 contended port in software.
 My own verification had missed both because I'd pointed the check at a dead
 port, which skipped
 the very tests that would have exposed them.

 **And a self-inflicted one worth admitting:** for a while, the demo didn't
 exercise the path the
 project claims. `dhdr demo` read through the aspect API and never
 published. The MCP path and the
 write-back existed and were tested — just not in the thing a judge would
 watch. Caught in review
 and fixed; the demo now runs the real MCP reads and writes each certificate
 back.

 ## Accomplishments that we're proud of

 **The flip works live, through the real MCP server.** Not a recording, not
 a simulation — the
 world genuinely moves between two protocol reads, because MCP has no
 time-travel parameter and I
 refused to fake one. The two decisions land on consecutive aspect
 revisions, one either side of
 the pipeline change that flipped the answer, each bound to the revision
 that justified it.

 **The certificate arrives where the decision is made.** It's on a real pull
 request, as a real
 code-scanning annotation, naming the decision, the dataset, the revision
 and the proposed change.

 **We shipped a fix upstream, not just a complaint.**
 `InstitutionalMemoryTemplate` plus
 registration and four tests, and those tests pass on DataHub's own CI.
 Filing an issue is table
 stakes; fixing the thing that blocked you is a different category.

 **The central claim is verified against DataHub's own storage, not against
 our own tests.** A
 differential test asks MySQL which aspect was in force at an instant, and
 requires our answer to
 match. It found a real defect — `lastObserved` records when metadata was
 last *observed*, not when
 a revision took effect, so the proposal step can be wrong. And it showed
 the architecture already
 contained that: fact-matching refused to bind, turning a wrong proposal
 into an honest "unbound"
 rather than a false certificate. The safety net was built for a different
 reason and caught this
 too.

 **The unflattering results are in the README.** The write-back isn't atomic
 and says so. The
 ablation reports that our own revision field is inert to the verifier. The
 third scenario is
 missing and the reason is stated. On a project whose entire claim is "don't
 overclaim", the
 credibility comes from the parts that cost something to admit.

 ## What we learned

 **Verify from outside the thing you're verifying.** I confirmed a firewall
 rule was working by
 curling the box's own public IP — from the box. Traffic to your own address
 never traverses the
 forwarding path, so the test was meaningless in both directions. Only an
 external probe told the
 truth: five services, including an unauthenticated OpenSearch, were
 reachable from the internet.

 **A workaround can hide the gap it works around.** The fact-matching in the
 coordinate layer
 exists only because the MCP response can't date itself. It works well
 enough that it would be easy
 to stop noticing the parameter should exist upstream. Writing the "what
 changes if this ships"
 section forced me to separate what's mine from what's a symptom.

 **Self-testing has a blind spot shaped exactly like your own setup.**
 Everything worked on my
 machine because my machine had dependencies the package never declared. A
 cold reader with no
 context found in minutes what my own verification had walked past for a
 day. The reason it worked
 is that they were not allowed to know anything I knew.

 **The same lesson, in code.** An independent oracle — DataHub's own storage
 — found in an hour
 what dozens of my own tests had missed for days. Those tests were not bad.
 They were derived from
 the same model as the code, so they could only ever agree with it. If you
 want evidence rather
 than confirmation, the check has to come from somewhere that does not share
 your assumptions.

 **Two scenarios are worth more than twice one scenario.** The second didn't
 add a feature; it
 proved the core was domain-ignorant *and* found a bug the first
 structurally couldn't.

 ## What's next for dhdr (decision records for DataHub agents)

 **A third scenario: ownership-driven retirement.** Designed and
 deliberately not built before
 submission — a third domain would have been engineering in front of a
 deadline. Worth noting the
 first draft had the policy backwards ("no owner → retirement allowed"),
 which reads as reckless to
 anyone who's watched orphaned data get deleted. It's now owner present →
 propose and notify; no
 owner → refuse and escalate.

 **Atomic write-back, the moment the upstream PR lands.** The test that pins
 the current limitation
 fails on that day, which is the signal to simplify `publish.py` to a
 server-side patch.

 **Wider use of the context graph.** Today the agent reads one aspect per
 decision. Reading
 ownership, tags and domain into the same record would make the certificate
 carry more of what
 DataHub actually knows.

 **And the gate as something teams install, not something we demo.** A
 certificate printed by a CLI
 is read once, by whoever ran it. The second design law is the one that
 decides whether any of this
 matters: *a record nobody keeps certifies nothing.*

Built With

• ai-agents
• data-governance
• datahub
• fastmcp
• mcp
• model-context-protocol
• opensearch
• provenance
• python
• sarif