Metriccourt

The problem

Every data team has the same fight. Three dashboards, three "revenue" numbers, and nobody can say which one is correct without spending half a day reading SQL.

The definitions diverge for boring, mechanical reasons: a different filter, a different join grain, a different time window. But that cause is invisible in the catalog. DataHub knows the assets exist. It doesn't know which definition is canonical, or why the others disagree.

What it does

MetricCourt reconciles metric definitions in a dbt + DataHub stack.

It finds every asset computing the same business metric. It parses each one's compiled SQL with SQLGlot and diffs it against the canonical definition across five deterministic patterns: filter, join grain, time window, aggregation, and source table. Rules decide the verdict. The language model only writes the prose, and --no-llm turns it off without changing a single result.

Verdicts are CONFORMING, DIVERGENT, or UNKNOWN. UNKNOWN is a real outcome, not a fallback. It means the AST couldn't decide, and we'd rather say that than guess.

Then it writes one canonical definition back into DataHub: a glossary term with a markdown definition, metric_canonical and metric_nonconforming tags, a documentation block, and a canonical_sql structured property. Every write is verified by reading it back.

Finally, DataHub's own Analytics Agent answers the revenue question from that definition instead of guessing from a tag.

The output is a self-contained HTML report: which assets claim the metric, the mechanical evidence fragment by fragment, the verdict, and a receipt of what got written back.

How we built it

The write path is proven end to end against the OSS quickstart. Glossary term creation, merge-safe tag attachment (a pre-seeded tag survives our write), idempotent documentation blocks, and structured properties on both dataset and glossary term all round-trip on read-back. The strategy is a hybrid because it had to be: SDK v2 where merge semantics matter, the classic emitter for entities the SDK has no class for.

The agent loop is closed and measured across 8 runs on two DataHub versions. Each run resets the catalog, asks in a fresh conversation, runs MetricCourt with write-back, then asks again in another fresh conversation. All 8 after-runs selected the canonical fct_revenue and cited the glossary definition in their answer text. We don't headline the quality score, because measured across identical resets it ranged from 2 to 5. The behaviour is stable. The number isn't.

The diff engine runs over 12 dbt mart models on a live Postgres warehouse with hand-verified seed numbers. 168 unit tests, 24 live integration tests, layer boundaries enforced by import-linter, and a one-shot script that goes from empty Docker to an open report.

We also pointed it at 6 public dbt repositories. GitLab's data team: 2709 models, 236 metric candidates, 20 clusters, 11 auto-triaged, 2 confirmed after hand-reading. Plus 13 more confirmed divergences across Tuva Health, dbt-ga4, snowplow-unified, dbt_stripe and dbt_netsuite.

Challenges we ran into

No single DataHub write surface was enough. SDK v2 ships entity classes for datasets and tags but not for glossary terms or structured properties. The classic emitter can create those, but it overwrites whole aspects.

The Analytics Agent silently loads zero catalog tools when the GMS token is empty, which is the quickstart's no-auth default. It answers fluently and grounds in nothing. That cost us hours.

Our own measurement nearly lied to us. A dead conversation still returns a plausible score of 3 with the reason "No context lookups yet". And a whole-transcript check for "did it mention the term?" always passes, because the tool-call JSON contains the term URN. Both produce a clean green run that proves nothing. We rewrote the harness to score only the final answer text.

What we learned

Determinism is what made this defensible. Because the rules decide and the model only narrates, every verdict on screen can be re-derived by anyone who clones the repo.

The measurement was harder than the build. Our first green numbers came from conversations that had already died. And when we re-ran the identical loop after a DataHub version bump, the scores moved while the behaviour we actually care about held on every run. So we report the behaviour.

What's next

Close the detection gap the real-repo scans exposed. On GitLab our clusterer surfaced the divergences for triage, but across the five round-2 repos it auto-surfaced 0 of the 13 findings that hand-reading confirmed. Clustering by name token and shared lineage is too weak for real warehouses. Column-level lineage and semantic grain inference are the next lever.

Adopt DataHub's showcase-ecommerce datapack as a second corpus. Its four Tableau custom-SQL revenue sheets are a genuine divergence set with a published answer key.

Upstream: a metric-reconcile skill for datahub-skills, and an RFC for a first-class MetricDefinition aspect.

Built With

• acryl-datahub
• apache
• claude-sonnet-5
• datahub
• datahub-analytics-agent
• dbt
• docker
• jinja
• langchain
• openrouter
• postgresql
• pydantic
• python
• sqlglot
• typer