EPISTEME — Autonomous Trust Layer for DataHub Metadata

Inspiration

Enterprises just handed AI agents the keys to their data catalogs. Through DataHub's MCP Server and Agent Context Kit, agents can now read the catalog, take action, and write metadata back automatically. Descriptions, tags, glossary terms, lineage notes, all written at a scale no human team could ever match.

But there is a dangerous gap nobody had filled. Agents can write, but nothing verifies what they write.

An AI agent writes "This table contains finalized customer orders." It sounds reasonable, so it is accepted. Another agent grounds on it as fact. A governance bot certifies it. A dashboard is built on it. A decision is made from it. And it was wrong. The table actually contained five order states, most of them not finalized. Nothing caught it.

We call this failure mode Metadata Collapse. As AI agents write more and more, catalog coverage rises while catalog truth silently degrades. Every wrong claim becomes grounding for the next one. No catalog platform on the market measures or defends against it.

So we built EPISTEME, the first fully autonomous adversarial trust layer for AI-written metadata. It runs on its own, with no human in the loop, and it verifies every write within seconds of detecting it.

What it does

EPISTEME sits transparently between AI agents and DataHub as an MCP proxy. It is a one-line configuration change with zero code changes to any existing agent. Every metadata write is intercepted and put on trial by an autonomous adversarial agent pair.

The Blue Agent proposes the metadata claim. The Red Agent then tries to falsify it, using only real DataHub evidence such as schema, profiling statistics, query logs, and source code. Red is structurally forbidden from using its own world knowledge. Its only job is to break the claim.

The verdict routes automatically. A supported claim commits and writes an attestation back to DataHub. An unsupported claim goes to a human through a DataHub Change Proposal. A contradicted claim is blocked and the agent's reliability is penalized. An unverifiable claim commits but is flagged with infinite grounding depth. There is no human in the loop at any point.

The core science

EPISTEME is built on four novel mechanisms, each mathematically grounded.

Grounding-depth trust decay

Every claim receives a grounding depth that measures its distance from a primary, verifiable source:

$$ g(c) = \begin{cases} 0 & \text{if the actor is a primary observer, such as a profiler, ingestion job, or human owner} \ 1 + \min_{e \in E(c)} g(e) & \text{if evidence exists and the verdict is SUPPORTED} \ \infty & \text{otherwise, a bare model guess with no evidence chain} \end{cases} $$

Trust decays multiplicatively with grounding depth:

$$ \tau(c) = \rho(a) \cdot \gamma^{\,g(c)} \cdot v(c) $$

Here $\rho(a)$ is the actor's reliability, $\gamma = 0.60$ is the per-hop decay factor, and $v(c)$ is verifier confidence. Every citation hop away from real evidence costs forty percent of trust.

Metadata Collapse Index

We aggregate trust across the whole catalog into a single health metric, weighted by real downstream blast radius:

$$ \text{MCI}(S) = \frac{\sum_{c \in S} w(c) \cdot \mathbf{1}[\,g(c) \geq 2\,]}{\sum_{c \in S} w(c)} $$

Here $w(c)$ is the downstream blast radius pulled from real DataHub lineage. A rising Metadata Collapse Index means the catalog is collapsing.

Citation-DAG immune response

Every claim records the evidence it cited, forming a citation directed acyclic graph in PostgreSQL. When a claim is later falsified, EPISTEME computes the set of all transitively contaminated claims using a recursive query:

$$ R(c_0) = {\, c : c \rightsquigarrow c_0 \,} $$

It then reverts them in reverse topological order, newest first, using DataHub PATCH writes guarded by If-Version-Match conditional headers. This guarantees the one property that makes autonomous rollback safe to deploy. It can never overwrite a human's edit. If a human changed a value since the AI wrote it, the version will not match, the revert is skipped, and the claim is flagged for human review instead. It is git revert, for metadata contamination.

Bayesian actor reliability

Every agent carries a Beta reliability posterior, updated on every verdict:

$$ \rho(a) \sim \text{Beta}(\alpha_a, \beta_a), \qquad \alpha_a \mathrel{+}= 1 \text{ on SUPPORTED}, \qquad \beta_a \mathrel{+}= 1 \text{ on CONTRADICTED} $$

with posterior mean reliability:

$$ \mathbb{E}[\rho(a)] = \frac{\alpha_a}{\alpha_a + \beta_a} $$

An agent whose reliability falls below the write-revocation threshold of 0.25 automatically loses write access. Reliability also feeds back into the trust score, so a historically unreliable agent's future claims start with lower trust before they are even verified. EPISTEME does not just catch bad claims. It progressively demotes the agents that make them, and rehabilitates the ones that improve.

How we built it

EPISTEME Core is built in Java 21 and Spring Boot 3. It consumes DataHub's MetadataChangeLog Kafka topic with a plain Kafka consumer, skipping its own writes to avoid infinite loops. It runs the grounding-depth engine, the Beta reliability tracker, and the citation-DAG recursive queries.

The Red and Blue verifier is built in Python 3.12 with LangGraph. The adversarial flow moves from proposing a claim, to gathering evidence, to falsifying, to a verdict. It is powered by NVIDIA NIM running Llama 3.1, with Red pinned to temperature zero for deterministic, auditable verdicts.

The Trust Firewall is a FastMCP proxy that transparently wraps DataHub's official MCP server. It intercepts every mutation and forwards reads with in-band trust annotations.

The immune response performs transitive reverts through PATCH writes with If-Version-Match, and opens real DataHub Incidents on contaminated downstream assets.

The frontend is built in React 18, TypeScript, Vite, Tailwind, Framer Motion, and D3, with nineteen pages all wired to real ledger and DataHub data. In total, EPISTEME touches fourteen distinct DataHub surfaces, including the MetadataChangeLog Kafka topics, systemMetadata, If-Version-Match conditional writes, the PATCH change type, structured properties for attestations, Change Proposals, Incidents, lineage, dataset profiling, query history, Timeline versioning, and the MCP Server itself.

Challenges we ran into

The first challenge was feedback loops. EPISTEME's own attestation writes generate new change-log events, so we had to tag its writes by actor type to make sure the consumer never re-verifies its own output into an infinite loop.

The second was falsification without hallucination. Early versions of Red would agree with fluent claims. We solved it by structurally binding Red to exactly four evidence tools and rejecting any verdict that did not cite a specific evidence field.

The third was safe autonomous rollback. Naive reverts risk destroying human corrections, and If-Version-Match conditional writes were the key to making autonomy safe to deploy rather than dangerous.

The fourth was running deterministic demonstrations on a non-deterministic model. We pinned temperature to zero and cached verdicts by evidence-snapshot hash so the pipeline produces identical results on every run.

What we learned

The hard part of trustworthy AI is not making models smarter. It is making them accountable. An agent that must cite real evidence, that earns or loses standing over time, and whose mistakes can be traced and reversed, is fundamentally more trustworthy than one that simply sounds confident. Grounding beats fluency.

What's next

Real-time drift detection that re-verifies supported claims automatically when the underlying data shifts. A self-generating Grounding Agent that creates the missing evidence to make unverifiable claims verifiable. And upstreaming our metadata attestation aspect as an RFC to the DataHub project itself.

EPISTEME. Trust, earned, not assumed.