Aegis Governance Engine

• 

  Final

Inspiration

Modern data teams move fast, and that speed creates a blind spot. Every time an analyst duplicates a column like email into a downstream table — and renames it along the way, say to em_c — traditional keyword-based security scanners lose the thread entirely. Under regulations like GDPR or Quebec's Law 25, that silent gap is a real liability, and it usually only gets caught during a manual, hours-long schema audit by a senior engineer.

We wanted to see if an agent could close that gap automatically, using the one thing a renamed column can't hide from: its lineage.

What it does

Aegis Governance Engine is an autonomous agent connected live to a DataHub instance. Instead of scanning column names, it walks the fine-grained lineage graph to trace where every field actually came from. If a column descends from a source tagged as personally identifiable information — even after being renamed, even across platforms — Aegis flags it.

Once it finds an ungoverned propagation, it does three things, in order:

1. Drafts a remediation patch — a dbt model applying SHA-256 masking to the exposed field, preserving joins and distinct counts while removing readability.
2. Writes governance metadata back into DataHub — a restriction tag on the exact schema field URN, plus an audit note with a timestamp and the reasoning behind the flag.
3. Prepares a pull request — never merges it. A human always makes the final call.

How we built it

• Detection runs on DataHub's UpstreamLineageClass.fineGrainedLineages, reading real field-to-field lineage rather than table-level or name-based matching. Each violation records its detection method and a confidence score, and transformations already applying masking or hashing (transformOperation) are excluded to avoid false positives.
• Remediation uses the DataHub Python SDK's emitter (MetadataChangeProposalWrapper) to write tags and audit notes, following a read-modify-write pattern so existing governance metadata is never overwritten.
• Action uses the GitHub REST API to open a pull request containing the generated dbt patch — never touching production directly.
• Every write path defaults to dry-run. Nothing lands in DataHub or GitHub until that's explicitly turned off.
• We also built an honest fallback: when fine-grained lineage isn't available for an asset, the agent falls back to name-based correlation and labels that violation as degraded precision rather than presenting it with the same confidence as a lineage-verified one.

Challenges we ran into

Getting a full local DataHub stack running — Docker, WSL2, the SDK — took longer than the actual agent logic, which is a fair trade for anyone who's set up a metadata platform from scratch. The more interesting challenge was matching our detection logic to DataHub's real aspect model: several early drafts referenced a table-level PII tag pattern, which is the wrong granularity — sensitivity in DataHub lives on the schema field URN, not the dataset URN. Rebuilding detection around per-column tags was the right call, but it meant redoing the lineage traversal to be field-aware rather than table-aware.

Accomplishments that we're proud of

The anti-evasion property is real, not aspirational: we tested it against a scenario where a PII column is deliberately renamed during transformation (email → em_c), and the agent traced it back to its source with 95% confidence through fine-grained lineage — with zero name matching involved. We also kept the whole write path honest by default: dry-run everywhere, explicit logging of what would happen, and a fallback path that tells you when it's less sure of itself instead of hiding it.

What we learned

Column-level lineage is a genuinely different detection primitive than table-level lineage — it's the difference between "this table might be risky" and "this exact field is risky, and here's the two-hop chain that proves it." We also learned, the practical way, that DataHub's aspect model rewards reading the SDK source directly rather than assuming a table-level mental model transfers cleanly to field-level governance.

What's next for Aegis Governance Engine

Multi-hop lineage traversal (tracing risk through three or more transformation steps instead of one), support for additional sensitivity classifications beyond PII, and packaging the agent as a redeployable configuration so it can be pointed at a new DataHub instance without code changes — the shape it would need to take to run as a standing service for a data team rather than a one-off scan.

Built With

• cryptography
• datahub-sdk
• dbt
• docker
• github-api
• python