LineageGate: AI-Powered Data Lineage PR Interceptor

LineageGate intercepts GitHub PRs, analyzes SQL schema changes against DataHub lineage using Gemini AI, and alerts developers to breaking downstream impacts before merging to production.

Comment

💡 Inspiration

In modern data engineering workflows, breaking schema changes—such as dropping or renaming a column in an upstream SQL model—silently pass standard CI/CD unit tests as long as the SQL syntax is valid. However, in production, these changes cause catastrophic silent failures across downstream models, BI dashboards, and data contracts.

We built LineageGate to shift data governance left: intercepting breaking changes inside the pull request workflow before they ever hit production.

🛡️ What It Does

LineageGate is an autonomous AI governance agent that sits inside your CI/CD pipeline:

  1. PR Interception: Automatically triggers when a developer opens a Pull Request containing schema or SQL model changes.
  2. Contextual Lineage Retrieval: Queries a live DataHub catalog (hosted on AWS EC2) via GraphQL to extract the real-time upstream and downstream dependency tree.
  3. AI Blast-Radius Analysis: Feeds the exact code diff and lineage metadata into Gemini 3.6 Flash, which reasons about potential schema breaks and impacted primary/foreign key relationships.
  4. Automated PR Remediation: Posts a rich, structured Markdown comment directly on the PR, highlighting broken downstream assets (with status indicators like ❌ Broken) and providing concrete remediation steps for the developer.

🏗️ How We Built It

  • Data Catalog & Lineage: Deployed DataHub via Docker Compose on an AWS EC2 instance, exposing GMS GraphQL endpoints.
  • AI Intelligence: Integrated the official google-genai Python SDK using Gemini 3.6 Flash for rapid reasoning over complex lineage graphs and code diffs.
  • CI/CD Orchestration: Configured custom GitHub Actions workflows leveraging PyGithub and custom Python interceptor scripts to read PR patches and post dynamic issue comments.
  • Open Source Standards: Licensed under Apache 2.0 with complete documentation, reproducible environment configurations, and exported dataset schemas.

🛠️ Tech Stack

  • AI Model: Google Gemini 3.6 Flash
  • Data Governance: DataHub GMS & GraphQL API
  • Cloud Infrastructure: AWS EC2
  • Automation / DevOps: GitHub Actions, Python 3.11+, REST APIs
  • Version Control & Docs: Git, Markdown, Apache 2.0 License

🏃 Challenges We Overcame

  • GraphQL Schema Resolution: Mapping complex GraphQL queries across DataHub entities (searchAcrossLineage vs entity-level lineage objects) to fetch multi-hop downstream dependencies efficiently.
  • Stateful Cloud Network Configurations: Managing real-time ingress rules on AWS Security Groups to allow secure, low-latency API access during CI/CD execution without restarting instances.
  • Prompt Engineering for Developers: Tuning system instructions for Gemini to ensure responses remain authoritative, structured, and developer-focused without hallucinating non-existent dependencies.

📈 What's Next for LineageGate

  • Automated PR Blocking: Adding configurable policy gates to automatically block PR merges if critical production dashboards are in the blast radius.
  • Multi-Platform Support: Expanding native lineage connectors beyond DataHub to support Atlan, OpenLineage, and dbt Cloud artifacts.
  • Column-Level Lineage Parsing: Enhancing Gemini's prompt context with AST-parsed column-level lineage for pinpoint accuracy on deeply nested SQL transformations.

Built With

Share this project:

Updates