Metadata Guardian

An AI data steward that detects stale metadata from engineering changes, reasons using DataHub, and automatically proposes or applies accurate metadata updates.

Comment

Inspiration

Enterprise AI agents are only as reliable as the context they are given. Platforms like DataHub hold that context, but it decays: a column gets renamed on a Tuesday and the catalog keeps describing the old one for the next eight months. Nobody is at fault — documentation simply moves slower than code.

We did not want to build another chatbot on top of a catalog. We wanted to fix the catalog itself, continuously, and answer the harder question that comes with it: how do you let an agent write to a production metadata store without ever making it worse?

What it does

Metadata Guardian watches engineering changes through GitHub webhooks and keeps DataHub synchronized without anyone maintaining it by hand.

When code, schemas or data models change, the agent:

  • Receives the change as an HMAC-signed GitHub webhook, verifies the signature, and persists the event before doing any analysis — so a slow model call can never lose a schema change.
  • Parses the commit into file-level change sets, recognising SQL migrations and dbt YAML models.
  • Retrieves the current state from DataHub, then classifies each change by entity type and impact.
  • Detects which catalog metadata is now stale, missing, contradictory or orphaned by comparing the code against what DataHub currently claims.
  • Generates concrete correction proposals, each with written reasoning.
  • Scores every proposal from 0.0 to 1.0 across four named factors: clarity, source quality, risk and reversibility.
  • Routes by that score, then writes back the approved ones to DataHub, recording every decision for audit.

The routing rule — the heart of the design

  • 0.85 and above → written to DataHub automatically, then recorded for audit.
  • Below 0.85 → held for a human and pushed to Slack as an interactive Block Kit card with the agent's reasoning attached.
  • Below the 0.70 threshold → never applied unsupervised. A person decides, or the proposal expires.

That rule is what makes an autonomous writer safe to point at a production catalog. In the demo run the agent produces five proposals, applies two, and holds three.

The ones it holds are the interesting part. A new email_hash column almost certainly holds personal data, and the agent proposes tagging it as PII — correctly. But tags drive access policy, so a wrong write here changes who can read the data. The risk factor drops, the score lands below the bar, and the change waits for a person. The agent knows the difference between being right and being safe.

How we built it

GitHub webhook → FastAPI → LangGraph agent → Slack review → DataHub (REST / MCP)
                              ↕
     PostgreSQL (events, proposals, approvals, change_sets, agent_runs)

The eight-node LangGraph pipeline. Four nodes call a model; four are deterministic.

  • parse — extracts file-level change sets from the webhook payload.
  • classify — LLM labels each change by entity type and impact.
  • detect — LLM finds stale, missing, contradictory or orphaned metadata.
  • generate — LLM drafts concrete correction proposals with reasoning.
  • score — LLM scores confidence across the four factors.
  • route — splits into auto-apply and human review, and notifies Slack.
  • approve — marks the high-confidence proposals approved.
  • writeback — pushes the aspects to DataHub and persists the run.

Core components

  • FastAPI for the REST API, webhook reception and orchestration.
  • LangGraph for the agent state graph, with versioned Jinja2 prompt templates configurable per stage.
  • DataHub integration through a REST client for writes, with an MCP transport for environments running the DataHub MCP Server.
  • OpenAI GPT-4o / Anthropic Claude for reasoning, with automatic provider fallback via an LLMFactory.
  • GitHub webhooks with HMAC signature verification.
  • Slack SDK for human-in-the-loop approval through Block Kit messages with interactive approve/reject buttons.
  • PostgreSQL with async SQLAlchemy for events, proposals, approvals, change sets and agent runs.
  • Docker + Azure Pipelines for a three-stage CI/CD path, with k6 and Locust for load testing.
  • Playwright for the demo recording — a single continuous take with the narration locked to the measured length of every scene.

Challenges we ran into

Designing for restraint, not coverage. The first version applied everything it proposed. The real engineering was in the scoring factors and the routing bands — deciding what an agent should refuse to do on its own.

A routing bug that silently dropped human work. The scorer emits auto_approve and requires_human alongside its factor breakdown, but the router read them from inside confidence_factors. They never arrived, so the router fell back to raw score comparisons and proposals in the review band were quietly rejected instead of reaching a human. The tests passed because they mocked the nested shape. We only found it by watching a real run.

Timezone-naive columns against a timezone-aware domain. Every model column was DateTime(timezone=False) while the domain layer produced timezone-aware UTC datetimes, so every Postgres insert failed. Invisible under SQLite in tests.

Endpoints that returned 200 and did nothing. Approve, reject and batch-approve issued their UPDATE but never committed the session. The entire human-review path was a no-op that looked perfectly healthy from the outside.

An incomplete write path. The DataHub REST client was missing four of the eight update methods, and its signatures took tag= and owner= while the writeback node calls value= — so the documented REST fallback could not actually write anything.

Async SQLAlchemy across agent boundaries. Session lifetime through LangGraph nodes needed careful expire and refresh handling, and mocking an ordered sequence of model calls needed a call-counting side effect rather than fragile keyword matching.

Four of those defects were exposed only by building the demo and driving the real system end to end. A green test suite had been hiding every one of them.

Accomplishments that we're proud of

  • A genuine end-to-end loop: signed webhook → eight-node agent → confidence routing → DataHub write, with every decision persisted and auditable.
  • An agent that writes less than it could. In the demo it applies two of five proposals and can articulate, per proposal, exactly why it held the other three.
  • Every number in the console is live. Nothing in the UI is a fixture, and the health panel refuses to call a dependency "verified" until it has seen real traffic on the wire.
  • A demo that proves rather than asserts. The model-call trace with real latency and token counts, the per-node evidence, and the DataHub aspects shown on the receiving end are all captured from the actual run.
  • 43 passing tests, zero lint violations, and a three-stage Azure DevOps pipeline.
  • Four real defects found and fixed, because we refused to demo against mocks.

What's next

  • Trigger writeback on human approval, so an approved proposal reaches DataHub without waiting for the next agent run.
  • Complete the MCP stdio transport so MCP becomes the primary writer with REST as the documented fallback.
  • A proactive metadata auditor that scans DataHub for drift instead of waiting for a commit.
  • dbt integration for direct model-to-metadata lineage.
  • Organization-wide, multi-repository webhook management.
  • GitHub PR comments showing metadata impact next to the code change.
  • Learning from reviewer decisions to tune the confidence factors per team.

Built With

Share this project:

Updates