UnderWrite

• 

  churn_model_v2 BLOCKED — target leakage on live DataHub FineGrainedLineage

• 

  Evidence path: discount_history → … → raw_billing.retention_discount (post_outcome)

• 

  Same gate, remediated model — APPROVED (CLEAN), no forbidden provenance

Inspiration

A model can pass every unit test and still be unsafe to deploy.

Take a churn model trained on a feature called discount_history. Nothing in the model repository looks suspicious. But DataHub's lineage shows that discount_history ultimately descends from retention_discount — a value written after the customer churned, the exact outcome the model is supposed to predict. That's target leakage. Code review alone may not catch it, because the leak isn't in the code. It's in the data's history.

DataHub already has the evidence. CI just doesn't act on it.

That's why I built UnderWrite: it turns DataHub's context graph into a deployment authorization boundary for ML systems, and demonstrates it against exactly the discount_history → retention_discount scenario above.

What it does

Before a model ships, UnderWrite asks DataHub one question: does the evidence say this deployment should be allowed?

1. Starts from an ML model registered in DataHub
2. Reads its feature and dataset lineage
3. Traverses fine-grained provenance upstream (bounded, cycle-safe)
4. Evaluates that evidence against a deterministic policy
5. Blocks or approves the deployment in CI
6. Writes the governance outcome back to DataHub
7. Uses the Agent Context Kit to explain how a blocked deployment can be fixed

The repo ships two versions of the churn model from the Inspiration example:

• churn_model_v2 traces back to retention_discount → BLOCKED, CI fails
• churn_model_v2_fixed has no forbidden lineage path → APPROVED, CI passes

Sample outputs for both runs are in examples/, so a judge can see the exact BLOCKED/APPROVED payloads without running the project.

How we built it

Deployment request → UnderWrite → DataHub context graph → deterministic policy → BLOCK/APPROVE → DataHub write-back

UnderWrite pulls fine-grained lineage, schema fields, tags, glossary terms, and ML entities from DataHub, normalizes them into a graph, and runs bounded traversal to reconstruct where a model's inputs actually came from. The target-leakage policy checks whether any feature's lineage reaches a field tagged as post-outcome evidence.

A deployment can only pass when the verdict is APPROVED and the evidence came from a live DataHub instance — if DataHub is unreachable or evidence is incomplete, UnderWrite fails closed rather than defaulting to approval. The verdict is exposed through a CI-friendly API and GitHub Action, and after every decision UnderWrite writes the governance event back into DataHub, so the graph accumulates not just what the org knows about its data, but what happened because of that knowledge. Remediation guidance for a blocked deployment is generated by an LLM grounded in the Agent Context Kit — it explains the blocked path, it doesn't rule on it.

Challenges we ran into

Making metadata enforceable. Reading lineage was easy. Turning a metadata graph into a decision safe enough to gate CI was not — traversal had to be bounded and cycle-safe, and missing evidence could never silently read as "approved."

Separating reasoning from generation. It was tempting to just hand the metadata to an LLM and ask "does this look safe?" That's a flashier demo and a much weaker control. UnderWrite keeps the verdict deterministic and uses the LLM only for grounded explanation and remediation after the verdict.

Making DataHub bidirectional. I didn't want DataHub to be a database UnderWrite queries once and forgets. The write-back loop was the harder half to build, but it's what turns a one-off check into institutional memory.

Accomplishments that I'm proud of

Removing DataHub breaks UnderWrite outright — the deployment decision depends on context the model repo itself doesn't contain. Specifically, it:

• traces fine-grained lineage rather than dataset-level relationships only
• distinguishes live DataHub evaluations from offline fixtures
• fails closed when evidence is unavailable
• handles cyclic lineage safely
• gates CI directly on the verdict
• writes governance outcomes back into DataHub
• keeps generative AI outside the authorization boundary

And the demo proves both directions on the same model family: unsafe provenance → BLOCKED, clean provenance → APPROVED.

What we learned

Lineage is usually something a human inspects after a problem shows up. Building UnderWrite reframed it as machine-readable evidence at an authorization boundary, evaluated before anything ships.

I also learned an LLM at every step doesn't make a system more agentic — for a deployment gate, predictability matters more than flexibility, so the consequential decision stays in deterministic code and AI is scoped to explanation and remediation.

Write-back changed how I think about a metadata platform's role: once decisions flow back into DataHub, the graph preserves not just what the org knows, but what it did about it.

What's next for UnderWrite

This build intentionally focuses on one policy: target leakage. The architecture is policy-shaped rather than leakage-specific. The next step is testing whether the same DataHub-backed authorization pattern generalizes to other metadata-driven deployment decisions without weakening its deterministic, fail-closed guarantees.

DataHub provides the evidence. UnderWrite decides. CI enforces.

Built With

• agent-context-kit
• ci-cd
• datahub
• fastapi
• fine-grained-lineage
• github-actions
• langchain
• mlops
• pydantic
• python
• react
• vite