Inspiration
Every data platform team knows this pattern: a new column lands in production, nobody tags it for PII, and three weeks later compliance finds out the hard way — usually during an audit, not before one. Manual PII tagging doesn't scale with the rate columns get added, and full automation is too risky to trust blindly on sensitive data. We wanted an agent that closes that gap without asking a human to babysit it: auto-decide the obvious cases, and put the ambiguous ones in front of a person with one click instead of a ticket queue. DataHub's MCP Server and Actions Framework made it possible to build this as an actual event-driven loop instead of a cron job that scans the whole catalog and hopes.
What it does
AutoSteward watches DataHub for schema changes in real time. When a new or modified column shows up, it pulls schema, lineage, and docs context for that column through the DataHub MCP server, then runs it through a two-tier classifier: a fast regex pass on the field name first, backed by an LLM self-report (Groq, llama-3.3-70b-versatile) for anything the regex can't call. High-confidence results get tagged automatically. Medium-confidence ones go to the dataset owner as an interactive Slack card — High Risk PII, Low Risk / Non-PII, or Custom, one click. Whatever the answer, AutoSteward writes the tag, glossary term, and an audit description (who confirmed it, when, how) straight back into DataHub's metadata graph, so the next person or agent that touches that column inherits the answer instead of re-deriving it. Classification never sees row-level or sample data — only schema, lineage, and docs metadata ever reach the LLM.
How we built it
A custom DataHub Action consumes schema-change events off DataHub's Kafka topic and forwards the changed URN and field name over HTTP to a FastAPI agent. The agent is the whole decision loop: MCP client wrapper for context fetch, the two-tier classifier, a rule-based confidence router, a Slack Block Kit card builder/parser, and a write-back module built on the DataHub Python SDK. Slack interactions come back through a signature-verified webhook, and confirming a card updates the message in place rather than leaving a stale card behind. We built it commit-by-commit as separate, independently testable pieces — regex classifier, LLM classifier, router, MCP client, Slack card, write-back, then wired them together in the FastAPI app — with a pytest suite covering each module against mocked boundaries (Groq, Slack, DataHub SDK, MCP session), so the whole thing runs and is verifiable without any live infrastructure. Late in the build we added a mock demo mode that replays the same decision-and-write-back path against a local page instead of Slack, so judges can see it work without standing up a Slack app or a Groq key.
Challenges we ran into
DataHub's Quickstart ships an internal schema registry rather than a standalone one on port 8081 — pointing the Kafka consumer at the wrong SCHEMA_REGISTRY_URL silently failed to deserialize events, and it took real debugging to trace that back to the registry endpoint rather than the Action itself. The MCP aspect payload also comes back as JSON-encoded bytes, not a parsed object, which broke field-name extraction the first time around. On the write-back side, DataHub's glossary term API requires an explicit auditStamp that isn't optional the way it looks in some examples, and getting Slack's signature verification to fail closed (not just fail loud) on a missing signing secret needed its own dedicated test. Each of these turned into a targeted fix rather than a rewrite, but they were exactly the kind of "works in the demo, breaks on real infra" gaps that only show up when you wire the real DataHub stack end-to-end instead of mocking it from the start.
Accomplishments that we're proud of
The full loop is real, not simulated: a genuine schema-change event on a genuine local DataHub instance triggers a genuine Kafka-consumed Action, a genuine MCP context fetch, a genuine Slack card, and a genuine write-back through the DataHub Python SDK — the same code path runs whether the trigger is a live Slack click or the mock demo page. We're also proud of keeping the classification boundary honest: the agent is architecturally incapable of seeing row-level or sample data, only metadata, which matters a lot for a tool whose whole job is deciding what counts as sensitive. And the test suite mocks every external boundary individually, so the whole system is verifiable offline in seconds, not just "trust me, it worked when I ran it."
What we learned
Event-driven beats polling for this problem — reacting to a schema-change event the moment it happens is a fundamentally different (and cheaper) shape than periodically re-scanning an entire catalog for anything that might be PII. We also learned to respect the confidence boundary: routing only the genuinely ambiguous cases to a human, instead of everything or nothing, is what makes a human-in-the-loop system actually get used rather than ignored. And building against DataHub's real Actions Framework and MCP server — instead of just reading its docs — surfaced infrastructure details (schema registry routing, aspect payload encoding, required audit stamps) that no amount of API reading would have caught.
What's next for AutoSteward
Extend the Tier 1/Tier 2 classifier to lineage-aware inference — if a column is fed directly by a known-PII source column, that's a strong signal before any LLM call is needed. Add a feedback loop so human corrections (the "Custom" button path) retrain or adjust the router's confidence thresholds over time instead of staying static. Support batch backfill mode for existing, already-cataloged columns that predate AutoSteward, not just newly changed ones. And extend write-back beyond tags and glossary terms into DataHub's data contracts, so a confirmed PII classification can also gate downstream access policy, not just describe it.
Log in or sign up for Devpost to join the conversation.