Inspiration
I've seen this happen firsthand: someone renames a field or redefines a metric, three dashboards quietly go wrong, and six months later nobody can explain why the change was ever approved.
DataHub already tracks what's connected to what — lineage, owners, glossary terms, all of it. What it doesn't do is remember why a decision was made, or notice when the reasoning behind it stops being true. That's the gap I wanted to build against — not another audit log, not another approvals tool. I wanted DataHub's own graph to be the place where that reasoning actually lives.
What it does
Given a proposed change — in the demo, renaming order_total to recognized_revenue on a real DataHub dataset — Evidence Gate pulls real context from DataHub (lineage, glossary terms, ownership, quality state), scores risk using explicit, deterministic rules, and runs a real read-only validation query. It detects a 13.16% aggregate revenue shift, well above the 1% tolerance, so the change is blocked, and a real dbt compatibility patch and migration test get generated so the block isn't a dead end.
It then writes the complete decision back onto the DataHub asset itself — rationale, evidence, risk score, required approvers, structured properties, a documentation link, a native incident. A completely separate process can later ask "why was this blocked?" and retrieve the real answer directly from DataHub. If something the decision relied on changes later — I remove a glossary term link in the demo — the stored decision automatically becomes stale. When a similar change comes in afterward, Evidence Gate finds the earlier decision as precedent and explains exactly what still applies and what's different.
One thing I didn't plan for: because the pipeline stopped assuming every asset has an owner, it surfaced that several downstream BI assets in the demo dataset have no owner assigned in DataHub at all — a genuine metadata gap the tool discovered on its own.
How I built it
I built this project solo. I leaned heavily on Claude for architecture and code review, and an agentic CLI (Antigravity, running inside GitHub Codespaces) for execution — my laptop doesn't have enough RAM to comfortably run DataHub's Docker stack, so the whole project was developed in the cloud from day one.
The core stack is intentionally straightforward: Python and FastAPI for orchestration, direct GraphQL/REST calls to DataHub GMS for discovery, DuckDB for deterministic validation, and Gemini for turning already-computed facts into readable rationale. Gemini never decides whether a change is approved or blocked, and that boundary isn't enforced by prompting alone — there's an actual hallucination check, and if the model references a risk signal that never fired, the output is rejected and replaced with a deterministic template.
I also evaluated DataHub's own Agent Context Kit and Analytics Agent before deciding not to use them. Agent Context Kit reads lineage from the search index, and during testing it returned zero downstream consumers for assets that direct GraphQL traversal could clearly see, so I kept direct graph queries instead. Analytics Agent generates SQL through an LLM, which conflicts with one design rule I wasn't willing to compromise on: validation has to stay deterministic and allow-listed, full stop. So I kept DuckDB.
Challenges I ran into
Honestly, the infrastructure fought back more than the reasoning logic did. Running DataHub's full stack — GMS, OpenSearch, MySQL, Kafka, the frontend — inside a GitHub Codespace meant repeatedly hitting disk limits and watching GMS get killed under memory pressure. At one point, right after an OpenSearch reindex, the UI briefly showed a completely different sample dataset instead of my own, and for a minute I genuinely thought I'd lost everything. Tracking that down turned into one of the most useful lessons of the whole project: my Decision Provenance write-back had survived every restart because it was safely stored in MySQL the entire time. The search index was stale. The underlying data never was.
The reasoning layer had its own version of the same problem. Early on, Gemini generated rationale that referenced risk signals which had never actually fired — a real hallucination, and I only caught it because I compared the output against the actual evidence bundle instead of trusting it because it sounded convincing. Fixing that meant carefully limiting which facts the model receives and adding a phrase-level check so it can't reference signals that don't exist. A quieter version of the same issue showed up in how required approvers got resolved: across otherwise identical runs, the list changed from six names, to three, then two. The cause was a lineage query returning partial results while OpenSearch was still catching up after a restart. Neither bug would have surfaced if I'd trusted one clean run and moved on.
Accomplishments that I'm proud of
Everything in this submission is backed by something I actually reran and verified, not something that merely should work. Decision Provenance genuinely survives container restarts. Staleness detection updates a real record on a real DataHub asset when the glossary term is removed. Precedent retrieval finds a real previous decision and explains what's actually different about the new one. Every screenshot, demo, and validation result comes from a real execution.
I'm also proud that the open-source contribution is genuine — not a repository link dressed up as one, but a real pull request against datahub-project/datahub-skills, submitted from my own account, in my own voice.
A community member even opened a real PR catching a decision-ID collision bug after I'd already submitted — I reviewed it properly rather than merging blindly, confirmed the fix against my live DataHub instance, and merged it with credit. Having the project attract and survive real outside scrutiny felt like a good sign in itself.
What I learned
The biggest lesson was that the gap between "the demo works" and "the demo is actually true" is where most of the real engineering lives. It's easy to make a model produce text that sounds authoritative. It's much harder to build a system that can't quietly mislead you, because you've built the checks that catch it when it tries. I also came away with a much deeper understanding of DataHub's metadata model — GraphQL, aspects, write-back mechanics, and how metadata actually flows through the platform — mostly by debugging real failures and reading real stack traces rather than relying on documentation alone.
What's next
The upstream datahub-decision-provenance skill was intentionally written to extend beyond this single revenue-rename scenario. The next step is validating the same Decision Provenance pattern against additional change types, like a schema type change or a data quality override. The goal isn't to assume the approach generalizes — it's to verify, with the same level of evidence, whether these deterministic risk rules and graph-linked write-back patterns actually hold up outside the exact scenario I built and validated here.
Log in or sign up for Devpost to join the conversation.