Attest

Verify what an AI agent claims about your data — zero verdicts decided by a model — and publish each verdict back into DataHub, so the next agent inherits it from the catalog, not from Attest.

Comment

Attest: the auditor that checks what your AI says about your data — and files the verdict back

Attest verifies an agent's claims against DataHub, decides every verdict in code, and — once a human publishes — writes each one back into the catalog as its own durable artifact. A sentence goes in. A cited verdict comes out, and the next agent inherits it from DataHub rather than from Attest. Zero verdicts are decided by a model, and that is an invariant the pipeline asserts on every run, not a promise in a README.

Named for what it does and refuses to do: it attests, so it never guesses. When the catalog is silent, Attest says so — rather than calling silence a disagreement.

Category: Open / Wildcard · Repo: open source, Apache-2.0 · Built solo · Full detail in the README.

The one-minute version

  • Zero verdicts decided by a model. Freshness, ownership, classification and schema verdicts are date math, set membership and string comparison. A run that violates it is FLAGGED and cannot be published (HTTP 409).
  • Falsifiable, not asserted. Break one checker on purpose and accuracy over the 40-case benchmark falls from 1.000 to 0.675. That sabotage runs on every just check, not on demand.
  • Writes back to DataHub, not just reads it. Each published verdict becomes its own content-addressed Assertion with append-only verdict events in DataHub — readable with no Attest database at all (ClaimReader(client, store=None)).
  • Absence is not disagreement. A third verdict exists because "we reviewed it and it's clean" and "nobody has ever looked" are different facts. The catalog grants closed-world reasoning per entity; Attest never assumes it.
  • Nothing reaches the catalog without a human. POST /audit writes nothing, ever. Publishing a verdict and accepting the agent's correction are separate decisions, either of which can be withheld.
  • The MCP Server was measured, not skipped. On the current 17-dataset seed, parity fails on 17/17 datasets: 136 mismatches across 491 comparisons, and four of five true claims change verdict. Issues #168 and #169 were filed upstream with reproductions, and PR #182 proposes a fix for #168.

The problem

An agent summarises a warehouse for a governance review and writes: "customer_profile holds no PII." Nobody checks it — there is nothing to check it against — and the sentence goes into the report.

The table is tagged PII. Its email column is tagged PII. The catalog said so the whole time.

The failure underneath that one is worse. The next table has no tags at all, and "we reviewed it and it's clean" is indistinguishable from "nobody has ever looked." A tool that reports those as the same thing cries wolf on every under-documented table in the warehouse — which is most of them. That is why the third verdict exists and is load-bearing.

What it does

Attest verifies an agent's claims against DataHub Core v1.5.0.6 as ground truth and returns one of three verdicts — Supported, Contradicted, or Insufficient-Coverage. Once a human publishes one, it writes the verdict back into DataHub as its own content-addressed claim artifact with append-only verdict events, so the next agent inherits it from the catalog rather than from Attest's database.

A real run, nothing canned. Three claims about a seeded catalog go in as prose. Out comes: customer_profile contains no PII → Contradicted (four PII signals converge — an explicit table tag, two PII-node glossary terms, hasPII=true, and two tagged columns, each cited back to the catalog field it came from); customer_profile is owned by alice.chen → Supported; raw_events is owned by bob.martinez → Insufficient-Coverage, because the ownership aspect is absent and the catalog is genuinely silent. 15.1s, 6,109 tokens, $0.0013. Attest's configured model proposes a revision to the contradicted claim, and that revision is re-checked by the same deterministic checker against the same frozen dataset snapshot.

Zero verdicts decided by a model — and how that is falsifiable

Freshness, ownership, classification and schema verdicts come from date math, set membership and string comparison. checkers/ imports no model client and never sees agent text.

But "the source doesn't import it" is only as good as the next commit, so that is not what the guarantee rests on: every pipeline step records what kind it is (deterministic / llm / io) alongside its token spend. The kind is what a step claims; the token count is the evidence. A checker that quietly started calling a model fails the run even if its answers are right — and a violating run is FLAGGED and cannot be approved (HTTP 409), so a report the pipeline could not vouch for never reaches the catalog.

The model is used for three non-verdict tasks — decomposing prose into typed claims, phrasing explanations, and proposing revisions to contradicted claims. Every proposed revision is re-checked by the same deterministic checker; explanation prose passes three gates (crosscheck, lexical faithfulness, polarity) before it can ship. Any gate failing ships a deterministic template instead.

Validation

I did not want to claim "deterministic verdicts" on the strength of a demo, so I measured it. Every number below is backed by a committed JSON receipt in the repo.

  • The deterministic core, 40 hand-labeled claims: accuracy 1.000, macro-F1 1.000, pass@5 = 1.0. All 12 (claim type × verdict) cells populated — a dark cell would be a silent misclassifier.
  • 100% is a conformance gate, not a capability score. The checkers are code implementing exactly the rules the labels encode, so anything below 100% would be a bug, not a difficulty signal. Which is why the next row matters more than the first.
  • The vacuity check: break one checker on purpose and accuracy falls to 0.675, Supported precision to 0.536, with 9 correctness and 4 coverage failures named. A benchmark that cannot fail measures nothing. This runs on every just check.
  • Cross-family calibration: an independent model family (Nemotron) re-labels every case from the same declared policy at temperature 0 — 39/40 agreement. It disputed a different case on each of two runs, and flipped a verdict it had previously agreed with. A judge that answers its own question differently cannot be ground truth; it can only calibrate it.
  • Crash-recoverable settlement: proven by a real SIGKILL at four write points and once during recovery. A durable write-ahead intent is persisted before the first catalog write and replayed on the next start.
  • What the benchmark does not prove, stated because omitting it is what would make the 100% hollow: it runs against a seeded catalog, not a messy real one; the labels apply the policy rather than validate it; and the catalog is both oracle and input, so it cannot tell you whether the catalog is right about the data.

The benchmark earned its numbers by failing first. Its cross-family labeler surfaced a governance rule that lived as a comment inside a checker rather than in the declared policy — a semantic buried in an if, which is the exact thing the policy file exists to prevent.

What the next agent inherits

One claim becomes one custom DataHub Assertion, at a URN derived by sha256 over the canonical claim, plus one appended run event per verdict. Two consequences, both load-bearing: two claims about one dataset coexist (they hash differently, so they land at different URNs), and re-running the write is safe (no run id, no clock in the identity). A re-audit appends; it never overwrites.

The URN this repo's README documents — urn:li:assertion:attest-a6016c69300d32bf5a0a — is computed offline from the claim, before any write runs. It is the URN DataHub then holds. A judge can verify that in ten seconds.

The proof that this is inherited rather than merely recorded is a reader constructed with no Attest database at all — ClaimReader(client, store=None). Every claim artifact, current verdict and reviewer comes back out of DataHub alone; per claim, Attest retrieves the newest 50 verdict events and exposes the catalog's total plus whether older history exists.

Nothing reaches the catalog without a human. POST /audit writes nothing, ever. Every audited claim parks for a per-claim decision, and publish is separate from accept_correction — because "your claim was wrong, and the fix you proposed is also wrong" is a thing a reviewer needs to be able to say. There is no ?auto_approve=true and no "approve all."

Engaging with the DataHub MCP Server

DataHub's agent tooling includes the MCP Server as a way to obtain catalog context, so rather than skip it I built to the one-method seam an adapter would implement and measured whether its responses carry the facts a deterministic verdict is made of.

The server runs — compatibility is the failure everyone expects here, and it is not what happened: correct OSS detection, correct version-gating, every call answered for every seeded dataset without error. The finding is about what those successful responses contain.

Measured on mcp-server-datahub 0.6.0 against the pinned Core: parity fails on 17/17 seeded datasets — 136 mismatches across 491 comparisons — and four of five true claims change verdict, including customer_profile.email is PII reading back Contradicted. The tag arrives as the display name "PII", the column reads unlabelled, the table is Verified, and my own completeness rule turns the loss into a confident denial. A transport that is lossy for a language model is not merely lossy for a checker — it is inverting.

That is a finding about structured consumers, not a defect for the server's intended use, where the compaction is a feature. Two issues were filed upstream with reproductions — #168 and #169 — and PR #182 proposes a targeted fix for #168 with regression coverage. just spike-mcp exits non-zero by design: if it ever goes green, the finding has expired and the decision is worth reopening.

How I built it

Python 3.12 and FastAPI. LangGraph for the pipeline — the retry cap is a graph edge, not a counter, and the human checkpoint is a real interrupt_before pause rather than a flag saying someone should look. Direct GraphQL over httpx to DataHub. SQLite for durable state. React + Vite + TypeScript for the UI.

Governance semantics are declared as reviewable data in checkers/policy.py, not buried in an if. Which signals count as PII, how they resolve when they disagree, and when absence may be read as denial are all readable in one file.

Challenges I ran into

A transport that was lossy for a model turned out to be inverting for a checker. The MCP finding was not the compatibility problem everyone expects — the server worked fine. It was that each optimisation for a language model removes something a deterministic checker needs, and the loss is silent: nothing in the response indicates a URN existed.

Settlement had to survive a real crash. The three catalog writes are sequential, not atomic. Closing that meant a durable write-ahead intent persisted before the first catalog write, and proving it with a real process kill at four points plus one during recovery — not a mock raising an exception, which proves a caught failure and nothing about memory loss after a remote commit.

My fakes could not fail the way real things fail. Three separate bugs shipped green because the test doubles had no fault injection on the path that broke: a provider 500 that walked past every degradation path, a dead socket filed as a malformed question (producing a complete run with zero verdicts — silence rendered as an answer, in the tool built to catch exactly that), and a stale request overwriting a newer one. Each was found by making the fake capable of failing, and each is now pinned by a test that goes red without the fix.

What I learned

An invariant nobody can falsify is decoration. "Zero verdicts decided by a model" is worth nothing asserted. The sabotage that collapses the benchmark to 0.675 is the only reason to believe it.

Absence is the hard part, not disagreement. Most of the design difficulty was in never letting "the catalog is silent" become "the agent is wrong." It is why there are three verdicts, why null must round-trip as null and never collapse to empty, and why a provider outage returns a 503 rather than a clean report with no claims in it.

Honest limits read as credibility, not weakness. The README lists its own gaps, the benchmark states what it does not prove, and the MCP write-up says plainly that the finding is version-bounded. None of that made the work look smaller.

What it does not do

The benchmark runs against a seeded catalog, not a messy real one, and its labels apply the policy rather than validate it. The guards are lexical detectors; they do not prove arbitrary natural-language entailment. The full DataHub-backed application runs locally and unauthenticated. The public GitHub Pages experience is an interactive replay of a real recorded Attest transaction, not a hosted live DataHub deployment. There is no bulk publication; a human checkpoint on every claim is what this build ships, and a real deployment at scale would need a policy layer.

Built With

Share this project:

Updates

Submission history