Inspiration
What it does## Inspiration
Production ML models fail silently. A data engineer drops a column, relaxes a nullability constraint, or a feature's distribution drifts — and the model just keeps serving predictions, quietly worse, until someone notices accuracy dropping in production. By then the engineer who caused it has moved on to something else, and often doesn't even know the model existed.
DataHub already has the lineage graph to answer "what's downstream of this table" — that part isn't the gap. The gap is that nobody's using that graph to actually stop a bad deploy before it ships. We wanted a gate, not a dashboard you check after you've already been paged.
What it does
Undertow sits in CI, between a model and a deploy. It walks DataHub's graph backward from an mlModel through its features and staging layers to the raw tables underneath, diffs what it finds against the last approved state — not just the current graph — and blocks the deploy when something upstream broke. It names the engineer whose change caused it, not the person running the deploy.
A schema change (a dropped column) is graded CERTAIN and can block. Statistical drift (a shifted mean, a null-rate jump) is graded PROBABLE and can warn, but can't block unless a team opts in — a distribution moving is evidence something might be wrong, not proof.
An investigation loop can optionally add context — what SQL actually reads the affected column, whether the change was documented, which other models sit on the same path — by calling DataHub's own MCP tools. It cannot change the verdict. That's not a prompt instruction; it's structural. The loop consumes and produces a Finding, and Finding has no severity field. We have a test asserting the verdict is byte-identical with and without the agent running.
Everything Undertow learns gets written back into DataHub as native metadata — tags, structured properties, a real assertion with run history — so the next run, on any machine, starts from what the last one learned. Undertow keeps no database of its own.
How we built it
Six decoupled layers: a resolver that walks the graph via DataHub's MCP server (with an SDK fallback), a differ that compares schema, governance, and statistical profiles against a stored baseline, an attributor that traces findings back to root cause and resolves the owning engineer, a policy engine that evaluates deterministic rules from undertow.yaml, an optional investigator agent loop, and a reporter that renders console output, GitHub PR comments, and writes verdicts back to DataHub.
The demo fixture isn't hand-built — staging.transactions_clean's schema and column-level lineage come from running DataHub's own sqlglot_lineage parser over real SQL, the same parser its Snowflake and BigQuery connectors use. The graph itself was recorded from a live DataHub OSS v1.7.0 instance, so undertow demo replays real captured state through the exact same differ, attribution, and policy code a live run uses — no mocked logic, just a swapped-out connection.
We also found a real gap in DataHub itself: datahub/specific/ ships PATCH-based builders for chart, dashboard, dataJob, dataProduct, dataset, form, and structuredProperty — but not for mlModel, which meant its aspects were UPSERT-only. UPSERT is lossy: one full-aspect write silently destroys any property a different writer didn't know about. We built MLModelPatchBuilder to close that gap and opened it upstream as datahub-project/datahub#18979.
Challenges we ran into
Getting the agent boundary right was the hardest design problem, not the hardest engineering one. It would have been easy to let the investigation loop influence severity "just a little" — richer context, maybe a confidence adjustment. We decided a gate you can argue with isn't a gate, so we made it structurally impossible: the type the loop works with doesn't have a field to write a verdict into, even if it wanted to.
We also had to decide what "no drift found" should mean when a column was never profiled in the first place. Silently treating "not profiled" the same as "checked, clean" is a much stronger claim than the evidence supports, so every report says exactly how much of the footprint it could actually inspect.
DataHub's OSS MCP server ships with mutation tools disabled by design, so reads and writes take genuinely different paths — reads go through MCP, writes go through the REST emitter. Getting both to resolve the same graph and produce the same verdict took real testing.
Accomplishments that we're proud of
The byte-identical verdict test. It's one assertion, but it's the whole thesis of the project compressed into a single test case: an agent can add context to a production gate without becoming the thing deciding whether a deploy ships.
What we learned
That the most convincing part of an "agent + guardrail" story isn't the guardrail described in prose — it's the guardrail made structurally impossible to violate, then demonstrated live rather than just claimed.
What's next
- Real ingestion-time coupling — reacting to change events as they happen in DataHub, rather than only at explicit
checktime - Team-level policy config beyond
undertow.yaml— org-wide defaults with per-model overrides - Extending the investigation loop's tool surface as DataHub's MCP server surface grows
Log in or sign up for Devpost to join the conversation.