GroundTruth

An agent that reads your real DataHub schema before writing SQL and then proves it did.

Comment

Inspiration

Every AI coding assistant that has ever written a dbt model for me has written it against a schema it had never seen. It doesn't say "I don't know your warehouse" — it guesses, fluently. stg_orders instead of orders. refunded_at instead of return_date. A join key that sounds exactly right and doesn't exist. The code looks reviewable, so it gets reviewed, and the error surfaces later as a broken pipeline or, worse, a number in a dashboard that nobody questions.

The frustrating part is that the answer is usually sitting right there. Teams running DataHub already have every table, column, description, tag, and lineage edge catalogued. The model just never gets asked. So we built the thing that asks — and then, because "the agent says it checked" is not evidence, we built the thing that grades it.

What it does

Groundtruth takes one prompt — "build a dbt model reporting on orders that were returned or refunded" — and runs it two ways against a live DataHub instance.

Ungrounded: the agent gets no metadata access. It writes confident, plausible SQL.

Grounded: the agent queries the real DataHub MCP server first — search, entity lookup, schema fields, even the catalog's own documentation — and only then writes SQL.

Then every table and column reference in both models is checked against the live metadata graph and scored:

                            UNGROUNDED                GROUNDED
  ────────────────────────────────────────────────────────────────
  grounding score           0%                        100%
  invented tables           3                         0
  unresolvable columns      8                         0
  DataHub queries made      0                         9

The prompt is a deliberate trap. The showcase warehouse tracks returns only — there is no refund table, no refund column, no refund concept anywhere in 67 datasets and 816 columns. Return data lives as a return_date field inside ORDER_ITEMS, joined back to ORDERS. You can prove the trap yourself before the demo even starts:

$ groundtruth catalog --search refund
  No table or column matching "refund" exists — searched all 67 datasets / 816 columns.
  closest columns: promotion_end_date, return_date, return_status, region_id

The ungrounded run invents stg_refunds, exactly as predicted. The grounded run discovers the real structure and — this is the part we care about most — writes the gap into the model itself:

"Verified against DataHub, this warehouse tracks RETURNS ONLY. There is no refund concept in the catalog... The refunded half of the request cannot be answered from the warehouse as it stands."

It didn't invent the missing half. It told us it's missing, and named the upstream feed that would be needed to answer it. That's the difference between output you review and output you merge.

Everything ships as a self-contained HTML report (both models side by side, each reference underlined verified-green or invented-red, with the full DataHub call timeline), and the run ends by opening a real pull request — gated on the score, so a model with a hallucinated reference cannot become a PR.

How we built it

DataHub surfaces used. Groundtruth touches DataHub through two deliberately independent paths. The agent uses the DataHub MCP Server (acryldata/mcp-server-datahub), exercising 8 distinct tools across a grounded run — search, get_entities, list_schema_fields, get_lineage, get_lineage_paths_between, get_dataset_queries, grep_documents, and search_documents. The verifier never uses MCP; it queries the DataHub GMS GraphQL API directly, so the grade isn't produced by the same channel that gathered the facts. Underneath both: a local datahub docker quickstart stack, the DataHub CLI's ingestion (datahub datapack load), the showcase-ecommerce datapack (~1,050 entities across Snowflake, Looker, PowerBI, Tableau, dbt, Spark, Postgres, S3), and the DataHub UI for confirming the metadata is real.

  • Agent runtime: Claude Code in headless mode (claude -p --output-format stream-json), with the stream parsed live so you watch each DataHub tool call and the bytes of real metadata it returns.
  • Metadata source: the official DataHub MCP Server against a local datahub docker quickstart, loaded with the showcase-ecommerce datapack (~1,050 entities across Snowflake, Looker, PowerBI, Tableau, dbt, Spark, Postgres, S3).
  • Verifier: a scope-aware SQL reference extractor plus a DataHub GMS GraphQL client.
  • Output: dbt models, an HTML report, a gh-opened PR, and a non-zero exit code.
  • Runtime dependencies: none. The CLI is plain Node 18+ with an empty dependency tree.

Two design decisions carry most of the weight, and both exist because the obvious implementation would have proven nothing:

The verifier does not use MCP. It talks to the DataHub GMS GraphQL API directly. If it graded the agent through the same channel the agent used to gather facts, a clean scorecard would just be the agent marking its own homework. Two independent paths to the same graph is the whole basis for trusting the number.

The grounded run happens in an empty sandbox. Our repo contains a catalog snapshot, a previously generated model, and a README that spells out the real schema in prose. An agent with filesystem access could read any of those and produce a perfect-looking model without ever touching DataHub — and it would score 100%. So the grounded run gets a throwaway directory containing nothing but an empty dbt scaffold. There is nothing on disk to find; the only way to learn the schema is to ask. And provenance is scored separately from correctness: if a run makes zero DataHub queries, its output is discarded, not reported as a success.

Scoring is deliberately un-gameable in the small ways too. Every reference lands in exactly one bucket — verified, hallucinated, orphaned (a column read off a table that doesn't exist), or local (a CTE name or column alias, which is not a claim about the warehouse at all). Local references are excluded from the denominator on purpose: counting them would let a model inflate its grade by adding CTEs.

Challenges we ran into

Proving a negative. "It didn't hallucinate" is a much harder claim than "it produced output." Most of the build went into making that claim falsifiable rather than asserted — the independent verification path, the empty sandbox, provenance as its own metric. The scorecard had to be something we could try to cheat and fail.

Parsing SQL well enough to grade it. A naive regex-based extractor gets this wrong in ways that quietly flatter the model. Comments can smuggle in fake references. CTE names look exactly like table names. A column read off an invented table would get counted twice — once as a bad table, once as a bad column — turning one mistake into two. Each of those is now a test; there are 22, and they run with no DataHub instance required.

MCP cold-start timeouts. The server kept reporting status: "failed", which sent us hunting for an auth or config bug that wasn't there. It was a timeout: uvx mcp-server-datahub unpacks ~120 packages into a fresh venv and imports the DataHub SDK before it speaks stdio — 13–30 seconds cold, past the agent's default MCP startup budget. The fix is a raised timeout plus a persistent uv tool install, but the real lesson was that a failure this ambiguous shouldn't be diagnosed by guessing. So groundtruth check now completes a real MCP handshake and reports the actual time in seconds, rather than checking that uvx is on PATH and hoping.

A DataHub CLI bug on Windows. datahub datapack load died with KeyError: 'Did not find a registered class for c' — the local-file ingestion source parses the C: drive letter as a URL scheme. Patched fs_base.get_path_schema to treat a single-letter scheme as file and filed it away in the README troubleshooting section.

Non-determinism in the thing being demoed. The ungrounded run has to hallucinate for the comparison to land, and LLM output varies. Rather than fake it, we chose a prompt where the trap is structural — the warehouse genuinely has no refund concept, so any model that answers the prompt literally must invent something. The failure is reproducible because the gap in the schema is real.

Accomplishments that we're proud of

  • The scorecard is evidence, not a claim. 0% → 100% on the same prompt, same grader, with the grader deliberately built on a path the agent can't reach.
  • The header comment in the generated model. The agent hitting a request the warehouse can't satisfy and documenting the gap — rather than papering over it with a plausible column name — is the behaviour we actually wanted from all of this, and it emerged from grounding rather than from prompt instructions telling it to say that.
  • It's useful outside the demo. groundtruth verify exits 2 on a hallucinated reference, so it drops into CI on any dbt repo as a pre-merge check — and it doesn't care whether the model was written by an agent or a person.
  • groundtruth check. Every dependency probed, including a real handshake, before you waste a run. It is the least glamorous part of the project and the one we'd miss most.
  • Zero runtime dependencies, 22 passing tests, and a demo that runs end to end in one command.

What we learned

The interesting question turned out not to be "can an agent use metadata" — it obviously can, and MCP makes it easy. It's "how would you know if it did?" An agent that has metadata access and quietly doesn't use it produces output that is indistinguishable from a grounded run right up until it's wrong. That's why provenance ended up as a first-class metric next to correctness, and why a zero-query run gets discarded instead of scored.

We also learned how much of a grounding evaluation is really about controlling what's within reach. Our first grounded run scored beautifully and proved nothing, because the agent could read our own README. Most of the rigour here is subtractive — taking things away until the metadata graph is the only remaining source of truth.

And the failure mode we found most interesting wasn't a wrong column. It was confident completion of an impossible request. Asked for something the data can't answer, the ungrounded model didn't hesitate or hedge — it invented the missing table and moved on. The grounded model stopped and said so. Knowing what isn't in the warehouse is as valuable as knowing what is.

What's next for GroundTruth

  • Beyond dbt. The verifier only needs a reference extractor per language — Airflow DAGs, Spark jobs, plain warehouse SQL, and Looker/LookML all have the same hallucination surface.
  • Lineage-aware verification. Today we verify that references exist. The next step is verifying they're appropriate: joining on a key with no lineage relationship, or reading from a deprecated table when DataHub knows the replacement, are both real bugs that pass today's check.
  • Governance signals as guardrails. DataHub already carries PII tags, deprecation status, ownership, and glossary terms. A model that selects a PII-tagged column into an unrestricted mart should fail the same gate that catches a hallucinated one.
  • Ship the CI action. groundtruth verify is already a working pre-merge gate; packaging it as a GitHub Action with an annotated diff comment is a small step with an outsized payoff for teams already on DataHub.
  • A regression harness. Run the grounded/ungrounded pair across many prompts and models, and track grounding score over time — turning a one-shot demo into a benchmark for how well a given model actually uses the metadata it's handed.

Built With

  • anthropic
  • claude
  • claude-code
  • dbt
  • docker
  • for-devpost's-built-with-field-(comma-separated-tags):-datahub
  • gh-cli
  • gh-cli-if-the-field-is-capped-or-you-want-it-tighter
  • github-actions
  • graphql
  • javascript
  • mcp
  • model-context-protocol
  • node.js
  • python
  • snowflake
  • sql
Share this project:

Updates