LineageGate

Inspiration

Data teams make high-impact schema changes through pull requests, but the most important context often lives somewhere else: lineage graphs, ownership records, domains, quality signals, and usage evidence in the data catalog. A code diff can look small while affecting dashboards and downstream models that the reviewer cannot see. We built LineageGate to bring that missing context into the merge decision and make the result reproducible.

What it does

LineageGate is evidence-backed change control for data pull requests. When a signed GitHub webhook arrives, it compares the base and head dbt manifests, identifies changed models and columns, resolves the affected assets through DataHub MCP, freezes the evidence, and evaluates deterministic governance policy.

The result is published as a GitHub Check with explicit reason codes and obligations. Missing, stale, truncated, or ambiguous catalog evidence fails closed. If approval is required, LineageGate binds it to the exact repository, pull request, reviewer, role, and head commit. A new commit automatically makes an older approval stale.

After remediation and current-commit approval, LineageGate rechecks the obligations, updates the same Check, writes a durable decision document to DataHub, verifies it through readback, and preserves the evidence hash and audit trail in PostgreSQL.

How we built it

LineageGate is a modular TypeScript service on Node.js 24 with a Hono webhook entrypoint and a ports-and-adapters architecture.

The ingress layer preserves the raw request body, enforces a size limit, verifies the GitHub signature, authorizes the installation, and deduplicates deliveries.
The dbt change detector converts validated base and head manifests into stable change facts.
The DataHub adapter uses MCP operations for asset resolution, schema evidence, downstream lineage, and decision-document writeback.
The policy engine is deterministic: it has no model calls, network calls, database access, or hidden clock.
A durable workflow coordinates retries, approval waits, current-SHA revalidation, and idempotent external effects.
PostgreSQL stores deliveries, immutable analysis runs, evidence snapshots, policy decisions, approvals, published effects, and audit events.
An optional bounded model adapter can produce grounded explanations and remediation drafts, but it cannot override policy or mutate GitHub or DataHub.

The system was developed locally with Docker, PostgreSQL, DataHub Core and DataHub MCP. Vitest, fast-check, contract fixtures, integration tests, and real signed-webhook runs provide layered verification.

How we used DataHub

DataHub is the evidence system at the center of the workflow, not a decorative integration. LineageGate resolves changed dbt assets to DataHub URNs, gathers entity and schema context, follows downstream lineage, records evidence coverage and freshness, and treats catalog gaps as first-class policy inputs.

When a change is finally approved, LineageGate saves the decision to DataHub with the commit SHA, policy version, obligations, approval reference, and evidence hash. It then reads the document back and compares the immutable fields before allowing the GitHub Check to become successful.

Verified demo

Our 2:15 demo follows one correlated proof sequence:

A pull request removes a schema field.
The LineageGate Check fails closed.
DataHub evidence explains the affected context and evidence gap.
A remediation commit restores the field and creates a new immutable run.
Approval for the previous SHA is rejected as stale.
Approval for the current SHA is accepted.
The same Check becomes successful.
The DataHub decision and PostgreSQL audit trail provide durable readback.

The evidence package also verifies duplicate-delivery handling, restart recovery, authorization checks, post-success approval revocation, and a race-safe final approval check. All published screenshots and identifiers are synthetic and sanitized.

Challenges we faced

The hardest part was preserving trust across asynchronous systems. GitHub deliveries can be retried, approvals can change while DataHub I/O is in flight, and a branch can move after a reviewer approves it. We addressed this with immutable SHA-bound runs, stable idempotency keys, approval rechecks after external I/O, and durable effect records.

A real webhook run also exposed a timestamp-normalization bug: GitHub omitted fractional seconds while PostgreSQL normalized the stored instant. The first review delivery correctly failed instead of granting approval. We changed the store to compare timestamp instants, added a PostgreSQL regression test using GitHub's exact timestamp form, and proved the recovery with a fresh signed delivery.

What we learned

Governance automation is strongest when uncertainty is represented explicitly. Missing catalog context should not quietly become a safe answer. Separating deterministic policy from generated explanation made the system easier to test, audit, and trust. We also learned that commit-bound approvals and idempotent effects are essential for any merge gate that spans multiple external systems.

What's next

The verified implementation is local-first. Next steps are a hardened hosted runtime, multi-runner lease renewal, broader dbt artifact acquisition, configurable policy packs, and additional DataHub evidence signals. The pull request will remain the primary interface so maintainers can make governed decisions without adopting another dashboard.