Project MetricGaurd

MetricGuard finds conflicting SQL definitions for the same business metric, proves how much they disagree, helps approve one canonical truth, writes it to DataHub, and guards against future drift.

Comment

MetricGuard — Semantic Conflict Intelligence for DataHub

Inspiration

A company can have one business metric and several trusted answers.

Finance, Sales, and executives may all calculate “weekly revenue” using production SQL that looks reasonable, while quietly disagreeing about filters, timezones, deduplication, aggregation, or eligible customers. Traditional data-quality checks will not catch this because every query can run successfully and every dashboard can remain green.

DataHub is becoming the governed home for semantic models and metrics. But organizations cannot govern conflicts they have not discovered. I built MetricGuard to find those hidden disagreements, prove their real numerical impact, help a human establish canonical truth in DataHub, and prevent the conflict from returning.

What it does

MetricGuard follows a five-stage loop:

  1. Discover competing metric definitions from observed SQL in DataHub.
  2. Compare them as structured semantic signatures.
  3. Prove the disagreement by executing both definitions against Postgres.
  4. Govern the resolution through human approval and DataHub write-back.
  5. Guard the approved definition against future semantic drift.

Instead of comparing SQL as text, MetricGuard extracts the dimensions that determine business meaning:

  • aggregation;
  • entity and grain;
  • timezone;
  • filters;
  • deduplication;
  • null handling;
  • source population.

This allows aliases, casing, whitespace, and predicate ordering to change without producing false alarms.

For each time period, numerical divergence is calculated deterministically:

In the verified weekly-revenue demonstration, two production definitions disagree by 13.07% on average, with a $28.9 million cumulative gap across 71 periods.

How I built it

MetricGuard is a Python application built around a strict boundary:

The agent decides what to investigate. Deterministic code proves the facts. A human governs the truth.

The deterministic layer uses:

  • SQLGlot for SQL parsing and normalization;
  • Pydantic models for semantic signatures and evidence;
  • Postgres for executing competing definitions;
  • pure Python comparison, divergence, clustering, and drift engines.

The agent layer uses LangChain with provider-agnostic model selection. The LLM can decide which definitions deserve investigation, gather additional context, explain trade-offs, and recommend a canonical definition. It never calculates semantic equivalence or numerical divergence.

MetricGuard uses DataHub’s official MCP server for:

  • observed-query discovery;
  • ownership, domains, tags, and lineage;
  • entity and governance context;
  • approval-gated write-back.

Approved resolutions are recorded using existing DataHub capabilities, including tags, structured properties, descriptions, documents, and consumer warnings. Every mutation passes through one approval choke point; the agent can stage proposals but cannot approve them.

MetricGuard also includes Sentinel, which monitors DataHub query definitions. Cosmetic changes are dismissed deterministically without an LLM call. New or semantically changed definitions start an investigation that must end as:

  • staged_resolution;
  • needs_human_decision; or
  • dismissed_with_evidence.

I also submitted the reusable workflow as DataHub Skills PR #65.

Challenges I faced

Distinguishing textual changes from semantic changes

SQL is extremely easy to rewrite without changing its meaning. Building cosmetic invariance while still catching differences such as COUNT(*) versus COUNT(DISTINCT user_id) required a structured representation rather than text similarity.

Keeping the LLM away from verifiable facts

It was tempting to ask the model whether two queries were equivalent. That would make the most important result probabilistic. I instead built deterministic tools for parsing, comparison, execution, and divergence, exposing them to the agent as callable evidence sources.

Making autonomous behavior safe

A useful agent should act without waiting for every instruction, but it should not silently redefine organizational truth. Sentinel can initiate investigations autonomously, while all DataHub mutations still require explicit human approval.

Working with real integration boundaries

The DataHub MCP server exposes different capabilities depending on configuration and version. MetricGuard discovers capabilities at runtime, supports both stdio and HTTP transports, and degrades cleanly when DataHub, the warehouse, or an LLM is unavailable.

Making the result independently verifiable

A judge may not want to launch DataHub, Postgres, and an LLM just to understand the project. I created a frozen Mission Control replay and committed machine-readable warehouse proofs so every public number can be inspected and reproduced.

What I learned

The central lesson was that agentic systems become more trustworthy when judgment and proof are separated.

LLMs are valuable for deciding what evidence matters, connecting context, explaining trade-offs, and recognizing when a human decision is required. They should not replace deterministic computation when the answer can be parsed, executed, measured, or tested.

I also learned that governance is not only about publishing a canonical definition. The harder problem is discovering the conflicting definitions already operating across an organization—and ensuring the disagreement does not quietly return.

Try it

MetricGuard: Discover conflicts. Prove impact. Govern one trusted definition.

Built With

Share this project:

Updates