Gatehouse

DataHub gives agents context. Gatehouse makes context decide what they may do: an MCP gateway that checks every agent tool call against live graph facts before it runs.

Comment

Inspiration

Every "agents + data catalog" demo this year ends the same way: the agent reads the catalog, says something intelligent about the data, and then does whatever it wants. The catalog is a lookup service. Nothing in the loop can say no.

That gap is not theoretical. The moment you let an agent hold a warehouse credential, you have handed a non-deterministic process your PII tables, your deprecated exports, and your write path — and your only controls are the coarse database grants you wrote for humans. Meanwhile DataHub already knows exactly which table is PII, which is deprecated and why, who owns it, what domain it belongs to, and whether it is currently broken. That knowledge is sitting one GraphQL call away from the decision that needs it.

So we asked a narrower question than "what can an agent do with a catalog?" — we asked what if the catalog decided?

What it does

Gatehouse is an enforcement gateway that sits between AI agents and the data stack.

It wraps the official DataHub MCP server unchanged (mounted in-process, so agents see the standard DataHub tool surface), adds a governed run_sql warehouse tool, and puts a policy middleware in front of every tool call. Before a call runs, the engine fetches the relevant facts from the DataHub graph — PII tags, glossary terms, ownership, domains, deprecation status, active incidents — and returns a verdict:

  • ALLOW — the call proceeds
  • DENY — the call never executes; the agent gets back the rule id, the graph facts, and the audit id, so an LLM agent can self-correct instead of retrying blind
  • REQUIRE_APPROVAL — the call blocks on a human at the console

Every decision streams to a live console with a fleet-wide emergency stop, seals into a hash-chained audit log, and is written back into DataHub as first-class metadata.

No LLM sits in the decision path. Policy is deterministic, testable, and sub-millisecond: p50 0.054 ms per decision with cached facts, +59 ms measured end-to-end overhead over raw SQL, 28/28 tests passing.

The moment that explains it

  1. A quality agent scans mart_billing and finds 1,215 negative billing amounts and 277 impossible stay lengths — the corruption DataHub planted in its own healthcare starter dataset.
  2. It calls the governed report_data_issue tool. Gatehouse checks its scope, then raises a real Incident on the dataset in DataHub.
  3. Seconds later a different agent — the analytics agent — asks a routine revenue question against the same table. Gatehouse reads the graph, sees the active incident, and refuses, citing the incident the first agent just wrote.

The two agents never exchanged a message. There is no message bus and no shared memory. One agent's discovery changed what another agent was allowed to do, through the catalog alone. That is the context graph working as a control plane instead of a lookup service.

What gets written back to DataHub

Gatehouse event Becomes in DataHub
Registered agent DataJob under the gatehouse/agent-fleet DataFlow, with owner, scopes, risk tier
Agent session DataProcessInstance run with real inlets/outlets, visible in lineage and Runs
Policy violation Incident on the offending dataset, citing the rule and the graph facts
Access to a governed dataset Run results on a custom agent access policy assertion (Validations tab)
Confirmed data issue found by an agent Incident via the governed report_data_issue tool

The graph everyone already maintains becomes the policy that governs the fleet — and every decision the fleet makes flows back into the graph as an auditable trail no agent can quietly rewrite.

How we built it

Python 3.12 and FastMCP for the gateway. The official mcp-server-datahub is mounted in-process, so agents get the standard DataHub tools plus run_sql, and every one of them crosses the policy middleware first. Agents identify with an X-Gatehouse-Agent header; everything else is ordinary MCP, so any MCP client works with no SDK and no code change — we drove it from Claude Code as well as from scripted agents.

The policy engine is a deterministic rules evaluator (policies.yaml) over facts pulled from DataHub via GraphQL behind a 15-second TTL cache. SQL is parsed with sqlglot to resolve the actual tables a statement touches, which is what gets checked — not the string. Write-back uses the acryl-datahub SDK low-level emitters: MetadataChangeProposalWrapper, DataProcessInstance, raiseIncident, upsertCustomAssertion. The console is a single self-contained HTML page over a WebSocket. The demo warehouse is DataHub's own healthcare and NYC-taxi starter datasets, planted quality issues and all, running on DataHub OSS v1.7.0 from datahub docker quickstart.

The audit log is a hash chain: each record commits to the previous one, so the console's "verify audit" is a real integrity check, not a log tail.

Challenges we ran into

Live and cheap at the same time. Reading the graph on every call adds hundreds of milliseconds. TTL-caching the facts fixes the cost but changes the semantics, so we made that explicit: the TTL is the fleet's maximum staleness. When an incident lands, every agent in the fleet inherits it within 15 seconds. That is a stated property, not an accident.

Making DENY useful instead of a wall. A bare refusal makes an LLM agent retry blind. Our deny message names the rule, the exact graph facts with their URNs, and the audit id, so the agent can adjust its approach — and a human can reconstruct the decision later.

A demo that could not be trusted to be up. The DataHub stack was unstable on our dev box (Docker daemon flapping, OOM cascades under 16 GB). Rather than gamble on live infra, we built a full offline fixture mode: the gateway, policy engine, console, audit chain, approvals, e-stop and run_sql all run for real against a snapshot of real graph facts, with no Docker required. It started as a hedge and turned into the best thing we shipped — a judge can see the whole product in about two minutes. Live write-back is proven separately, with the verbatim captured GraphQL response in examples/.

Accomplishments we're proud of

Enforcement that is genuinely free at runtime (0.054 ms p50), a two-agent coordination moment that needs no coordination protocol at all, an audit log you can actually verify, and two ways to run the whole thing — with or without Docker.

What we learned

DataHub's graph is already a near-complete policy source. PII classification, ownership, deprecation, domains and incidents map onto agent access rules with almost no translation layer — the hard part was not modelling policy, it was deciding to let the metadata decide. The interesting product is not more metadata. It is putting the metadata in the path.

What's next

OAuth per-agent identity, push invalidation from the DataHub Actions event stream instead of a TTL, Snowflake/BigQuery/Postgres executors, column-level masking, and Slack approvals routed through DataHub subscriptions.

Built With

Share this project:

Updates