DataHub LineageGuard: Self-Healing Data Agent

Autonomous AI agent using DataHub MCP & LangGraph to detect stale data, compute blast radius, write back metadata, and auto-generate dbt models & Airflow DAGs to heal data pipelines in real time.

Comment

Inspiration

Modern data teams struggle with invisible pipeline failures: stale tables breaking downstream dashboards, silent schema drift corrupting ML models, and undocumented datasets accumulating technical debt.

We built LineageGuard to turn DataHub context graph into an autonomous defense system that continuously detects anomalies, traces downstream blast radius, auto-generates production dbt & Airflow code, and heals pipelines in real time.

What it does

LineageGuard runs a continuous Detect -> Diagnose -> Act -> Report loop:

  1. Detects: Scans DataHub for stale data, missing ownership/descriptions, schema drift, and failing assertions.
  2. Diagnoses: Computes full downstream blast radius across datasets, Looker dashboards, and production ML models (e.g. flagging nyc_taxi.trip_data staleness before it corrupts demand_predictor_v2).
  3. Acts: Reads DataHub metadata to auto-generate production-ready dbt staging models, schema.yml test suites, Airflow DAGs with freshness validation, and Great Expectations JSON quality checks.
  4. Writes Back to DataHub: Auto-tags entities (lineageguard:stale), populates documentation, raises operational incidents, and emits execution metrics back to DataHub GMS.
  5. Open-Source Contribution: Includes datahub-skill/pipeline-health, a reusable DataHub Skill for pipeline health check & assertion suggestions designed for upstream submission to datahub-project/datahub-skills.

How we built it

  • Orchestration: LangGraph stateful multi-agent system.
  • DataHub Integration: Bi-directional integration using DataHub MCP Server and DataHub Python SDK.
  • Code Generation: Metadata-aware templates producing clean dbt SQL and Airflow Python DAGs.
  • Frontend & API: FastAPI backend + glassmorphism React-style dark-mode dashboard with interactive lineage visualization.

Accomplishments that we are proud of

  • Built a complete, end-to-end working system with 7/7 passing unit/integration tests.
  • Covered 3 hackathon challenge tracks simultaneously plus the bonus open-source skill contribution.
  • Implemented bi-directional DataHub graph integration (reads context AND writes back tags, docs, and incidents).

What we learned

Building metadata-aware AI agents requires deep graph context traversal. DataHub MCP Server makes entity querying seamless, allowing agents to act with high context precision.

What is next for DataHub LineageGuard

Deploy LineageGuard as a continuous background job via Cloud Composer / Airflow to provide 24/7 self-healing data pipeline protection for enterprise data lakes.

Built With

Share this project:

Updates