Change Guardian

Inspiration

Every data engineer has lived this nightmare: a Friday rename of one upstream column passes all unit tests, merges cleanly, and by Saturday morning the Looker executive dashboard is broken and the ML pipeline throws KeyError in production. The on-call engineer spends four hours manually tracing lineage and hand-writing compatibility patches — work that should never have been human work. Meanwhile, storage bills quietly accumulate from thousands of stale, unused datasets that no team trusts their lineage visibility enough to safely delete. DataHub holds the ground truth — lineage, schemas, ownership, domain policies. The missing piece was an agent that acts on that context, not just reads it.

What it does

Change Guardian is an autonomous, closed-loop governance engine with two modes. In schema drift mode, it detects an upstream column rename via DataHub, traverses the multi-hop lineage graph to find every broken downstream consumer, generates technology-aware compatibility patches, opens a validated pull request, creates a Jira incident ticket, and writes a pending-migration tag back to DataHub. In FinOps mode, it identifies stale, un-queried datasets costing real money, confirms zero active downstream consumers as a hard safety gate, and deprecates the asset in DataHub only after human approval. Every session is sealed with a SHA-256 cryptographic hash chain for tamper-proof auditability.

How we built it

We designed a 6-stage Tarot state machine — each phase named after an archetype matching its function:

The CodeGate Validation Engine routes patches by file typ. Then compiles every patch through Python ast.parse and auto-repairs illegal SQL constructs before any PR is opened. The reconcile daemon polls merged PR diffs against original patches to train a local learning cache, then executes the final DataHub writeback atomically.

Challenges we ran into

The hardest problem was technology-aware code repair. A naive LLM applies col AS alias to every file — valid SQL, but a runtime crash in Python. We built a full AST validation layer that detects file type, applies the correct strategy, detects illegal patterns like ORDER BY col AS alias, and auto-repairs them before surfacing results. The second challenge was closed-loop reconciliation: a governance run doesn't end when the PR opens — it ends when the PR is merged and DataHub is updated, requiring a careful state machine to prevent duplicate writebacks across concurrent sessions.

Accomplishments that we're proud of

We built a genuinely working, end-to-end governance loop. DataHub lineage is traversed live, real Gitea PRs are opened, real Jira tickets are created, and DataHub catalog state is mutated and reconciled.

What we learned

DataHub is a governance action platform, not just a passive catalog. The ability to write incidents, attach tags, mark deprecations, and resolve them on PR merge transforms it into the central state machine of an autonomous agent loop. We also learned that false confidence is worse than no automation: if a "stale" dataset has any active lineage consumer, the agent must stop completely and explain why — not silently proceed and create an outage.

What's next for Change Guardian

Phase I replaces the local ChromaDB style cache with a pgvector Postgres store — embedding Abstract Syntax Trees of merged patches so the agent retrieves cross-domain structural patterns org-wide.

Phase II adds a real-time Kafka DataHub MCL consumer for sub-second drift triggers and expands lineage traversal to Snowflake, BigQuery, Databricks, and Looker in a unified transitive downtime cost graph.

Phase III moves human approval from email/Jira polling to interactive Slack and Teams action blocks — where a data engineer can approve, edit, or reject a patch inline and the decision writes back to the learning cache instantly.

Phase IV packages Change Guardian as a reusable change-guardian/action@v1 GitHub Actions step, shifting governance left into the PR pipeline before schema changes ever reach production.

Built With

• agents
• datahub
• gitea
• governance
• graphql
• jira
• llm
• python