LineageGuard AI

An AI agent that uses DataHub’s metadata graph to analyze schema changes, identify affected pipelines and ML models, generate safe fixes, and write results back to DataHub.

Comment

Inspiration

A schema change that looks harmless in one warehouse can quietly break a dbt model, a dashboard, or an executive metric several hops downstream.

Data teams already store much of the required context in DataHub: schemas, lineage, ownership, domains, tags, and criticality. The missing piece is a trustworthy agent that can turn this context into an actionable decision without becoming an uncontrolled write path.

We built LineageGuard AI around one principle:

An agent should not merely sound confident. It should identify the exact asset, show the evidence behind every important claim, expose uncertainty, survive independent review, and require explicit human approval before writing anything back.

What it does

LineageGuard AI is an evidence-first control plane for data-change decisions, built for the Agents That Do Real Work track.

It combines five connected experiences:

  1. Live 3D DataHub cartography
    A server-cached, non-blocking view of the DataHub catalog and lineage graph. Users can explore assets, platforms, relationships, and exact URNs while background enrichment continues.

  2. Verified Agentic RAG assistant
    A LangGraph workflow uses Qdrant to retrieve candidate metadata, confirms it through allow-listed DataHub MCP reads, locks the exact target asset, reasons over live evidence, and verifies factual claims before answering.

  3. Read-only change-impact analysis
    Supports ADD_COLUMN, DROP_COLUMN, RENAME_COLUMN, and CHANGE_TYPE. It performs deterministic preflight checks, multi-hop lineage traversal, risk scoring, affected-asset discovery, remediation planning, rollback guidance, and evidence recording.

  4. Independent model review
    An optional NVIDIA advisory critic and two independent judges, OpenAI and Groq, review grounding, technical correctness, completeness, safety, and actionability. An unavailable or failing judge never becomes an approval.

  5. Governed DataHub write-back
    After a double PASS, a local reviewer capability and explicit human approval can publish one scoped DataHub Analysis document. The workflow is idempotent, concurrency-safe, auditable, and compensates safely after partial failure. It never mutates schemas, rows, lineage, or warehouse data.

The chat itself remains read-only. Requests that imply a change are routed to analysis or to a governed human-in-the-loop proposal. Prompt injection cannot bypass the write controls.

Live deployment

Try the public read-only demo here:

https://lineageguard.hackdev.tech

Source code:

https://github.com/omarfh111/LineageGuard-AI

The public deployment exposes catalog exploration, verified RAG, impact analysis, review results, observability, and audit history. The reviewer capability is intentionally not published because it is a security boundary for the controlled DataHub write-back path.

Why DataHub is the foundation

DataHub is not a decorative integration in this project. It is the system of record and source of truth for:

  • exact asset identity and platform disambiguation;
  • field names and types;
  • upstream and downstream lineage;
  • owners, domains, tags, descriptions, and criticality context;
  • the final governed Analysis document written back after approval.

LineageGuard uses DataHub OSS/Core and the official DataHub MCP Server.

Qdrant accelerates candidate discovery, but it does not override DataHub. A retrieved record becomes admissible evidence only after live MCP confirmation.

If an asset is ambiguous, such as orders across Snowflake, dbt, S3, and PostgreSQL, the agent asks the user to choose a platform. If an asset does not exist, retries remain locked to that target and the agent returns a safe limitation instead of substituting a convenient but unrelated asset.

How we built it

The backend is a FastAPI application using Pydantic contracts and explicit server-owned workflow states.

LangGraph provides planning, retrieval, MCP tool execution, reasoning, verification, repair, conversation routing, and safety nodes. The official DataHub MCP server supplies allow-listed read tools and the narrowly scoped governed save_document operation.

Qdrant stores searchable projections of DataHub metadata, never source-data rows. SQLite persists workflow checkpoints, approvals, idempotency records, audit events, and compensation state.

The frontend uses React, TypeScript, Vite, and Three.js. A 3D graph renders the shared server-side catalog cache. The cache refreshes in the background, uses catalog and lineage fingerprints to detect real changes, retains the last successful graph during a failure, and swaps completed graph generations atomically.

The model layer is separated by responsibility:

  • OpenAI powers the conversational planner/reasoner and one independent judge;
  • Groq provides the second independent judge;
  • NVIDIA NIM provides an optional advisory critique;
  • deterministic validators remain authoritative for routing, schema conflicts, target matching, evidence constraints, safety gates, and workflow state.

LangSmith tracing captures agent execution, tool calls, retries, latency, token usage, and evaluation runs. The user interface intentionally shows concise, auditable rationales and evidence IDs instead of hidden chain-of-thought.

Agentic RAG path

Question → LangGraph planner → Qdrant candidates → DataHub MCP confirmation → exact target lock → answer generation → claim-level verification → verified answer or safe limitation

Governed action path

Change request → exact DataHub target → schema and multi-hop lineage → deterministic validation → risk and remediation report → immutable snapshot → optional NVIDIA critique → OpenAI + Groq judges → local reviewer capability → explicit HITL decision → scoped DataHub Analysis document → audit and compensation

Safety and governance

The main security boundary is server-owned. The browser cannot submit a replacement report, select an arbitrary mutation, or bypass an approval state.

The backend hashes an immutable analysis snapshot before review and revalidates it at each transition.

Additional controls include:

  • exact URN locking across retries;
  • tool allow-lists and bounded MCP timeouts;
  • claim-to-evidence validation for schema and lineage answers;
  • deterministic rejection of duplicate ADD_COLUMN, rename collisions, unchanged types, and incompatible type changes;
  • double independent PASS before a proposal can advance;
  • explicit human rationale and a local reviewer capability;
  • write flags that fail closed when disabled;
  • idempotency and concurrency protection for competing approvals;
  • one narrowly scoped save_document operation;
  • durable audit events and safe compensation for partial failures.

Challenges we ran into and how we resolved them

Qdrant could retrieve relevant but incorrect assets

Early retrieval could find semantically similar Qdrant records without proving that they represented the user’s requested DataHub asset.

Resolution: Qdrant now only guides MCP confirmation. The exact DataHub URN is resolved and locked before schema or lineage reads. Unsupported or unrelated candidates are removed from the reasoning context.

Citation presence was not enough to prevent hallucinations

An answer could contain an evidence ID while still using evidence belonging to another platform or another dataset.

Resolution: verification is now claim-level and target-aware. A schema request for Snowflake orders cannot pass using order_details, dbt orders, or any unrelated platform evidence.

Multi-hop lineage is difficult in real metadata graphs

Real catalogs include duplicate names, cycles, disconnected assets, and high-degree nodes.

Resolution: we implemented bounded traversal, exact path reconstruction, deduplication, depth limits, asset limits, and explicit evidence records for every reported lineage path.

The 3D graph initially refreshed too aggressively

Rebuilding a large graph on every polling cycle caused visible resets, loss of relationships, and an unresponsive interface.

Resolution: the cache now separates cheap root-catalog change detection from expensive lineage enrichment, limits concurrency, uses timeout backoff, retains the last good graph, detects incomplete batches, recovers safely, and atomically swaps complete generations.

Providers are fallible

NVIDIA, OpenAI, and Groq can time out, return invalid structured output, or be temporarily unavailable.

Resolution: provider calls use bounded retries, strict structured-output normalization, explicit availability states, and fail-closed handling. A provider error never silently becomes a PASS. Judge disagreement is visibly preserved as NEEDS_REPAIR.

Write-back safety was harder than calling a write tool

The difficult problem was guaranteeing correct behavior around a write under duplicate approvals, concurrent actions, partial failure, or compensation failure.

Resolution: we added immutable server-owned snapshots, state-machine guards, local reviewer capabilities, idempotency keys, concurrency control, durable receipts, post-write MCP verification, and a compensation flow that remains visible and recoverable if it cannot finish immediately.

Browser confirmations were hard to test reliably

Native browser confirmation dialogs blocked automated validation and made the approval boundary difficult to demonstrate.

Resolution: native dialogs were replaced with explicit in-page HITL confirmation panels while preserving the server-side reviewer capability requirement.

Accomplishments we are proud of

  • DataHub is used for both evidence reads and a real governed write-back.
  • The same exact target follows the user from chat to impact analysis and review.
  • Every schema and lineage claim must map to target-matched MCP evidence.
  • Unsafe, ambiguous, unsupported, and nonexistent requests end as explicit limitations, not plausible guesses.
  • Four schema-change types share one deterministic and auditable workflow.
  • The catalog remains usable while 3D lineage enrichment runs in the background.
  • Independent judges can agree, disagree, or fail safely; no judge is treated as automatically correct.
  • Human approval is a real security boundary, not a decorative confirmation button.
  • Professional validation artifacts, evaluations, runbooks, diagrams, and troubleshooting notes are versioned in the public repository.

Validation and results

The repository includes automated tests, live evaluation evidence, browser validation, reproducible reports, and a governed write-back proof.

  • Backend quality: 167 tests passed; 4 DataHub integration tests are intentionally opt-in and skipped by default.
  • Frontend quality: 23 Vitest tests passed, plus TypeScript compilation and a Vite production build.
  • Professional live Agentic RAG evaluation: 30/30 queries completed; 22 queries had manually reviewed exact ranking labels.
  • Precision@6: 0.902
  • Recall@6: 0.909
  • MRR@6: 0.909
  • NDCG@6: 0.909
  • Routing accuracy: 1.000
  • Tool-selection accuracy: 1.000
  • Verification accuracy: 1.000
  • Target-resolution accuracy: 1.000
  • Unsupported-claim escape rate: 0.000
  • Observed p95 latency: 9,516.9 ms
  • Recorded OpenAI usage: 4,693 tokens, estimated at $0.0039472 for the benchmark.

We also completed a controlled end-to-end proof:

analyze → deterministic validation → independent judging → HITL approval → one DataHub Analysis document → compensation → ROLLED_BACK

No schema, lineage, or business-data mutation was attempted or permitted.

These metrics describe the versioned evidence run in the repository, not a universal production SLA. Provider latency and availability vary by environment.

What we learned

The main lesson was that “agentic” should mean controlled delegation, not reduced accountability.

Vector retrieval is valuable for discovery, but a metadata system of record must confirm identity. LLM review is useful for critique, but deterministic validation and explicit workflow state must control writes. Conversation memory is useful for references such as “its schema,” but remembered context must never become fresh evidence.

We also learned that observability is part of the product. Evidence ledgers, traces, immutable snapshots, costs, retries, reviewer decisions, and audit events make an agent understandable to data engineers and defensible to governance teams.

What's next

  • Add organization-specific policy packs and approval roles.
  • Support contracts, assertions, data products, and more compatibility rules.
  • Add event-driven incremental synchronization from DataHub to Qdrant.
  • Expand reviewed evaluation datasets and platform-specific schema rules.
  • Provide a resettable public sample environment for safer hands-on demonstrations.
  • Contribute reusable reliability and governance patterns back to the DataHub ecosystem.

Open source

LineageGuard AI is published under the Apache License 2.0.

The public repository contains the application, Docker setup, architecture, API reference, test plans, evaluation datasets, versioned evidence, runbooks, troubleshooting notes, media assets, and the governed write-back proof.

Demo access: the public deployment provides full read-only catalog exploration, verified Agentic RAG, impact analysis, and independent review. The final DataHub write-back is intentionally protected by a private local reviewer capability. Its complete analyze → judge → HITL → write → compensation proof is available in the public video, evaluation evidence, and repository documentation.

Built With

Share this project:

Updates