Inspiration
Every data team has experienced this: someone renames a column in a staging model, merges it on Friday afternoon, and by Monday morning the executive dashboard is broken, the ML feature pipeline is failing silently, and nobody knows why.
The lineage data already exists in DataHub — but it's sitting in a catalog that nobody checks during PR review. Engineers review SQL diffs in a vacuum, blind to the 15 downstream dashboards, 3 ML features, and 2 SLA-reports that depend on that column.
We asked: what if the PR itself could tell you what it breaks — before you merge?
LineageLens connects the dots between GitHub pull requests and DataHub's lineage graph, using AI to assess blast radius and risk, then posting a clear verdict directly on the PR. No new tool to learn, no workflow change — just confidence before you merge.
What it does
LineageLens is a full-stack web application with a 6-stage AI pipeline:
Automatic trigger — A GitHub webhook fires on every PR open/sync. No manual action required. File extraction — Parses the diff to identify changed SQL/dbt models and column-level modifications. DataHub entity resolution — Searches DataHub's catalog via REST API + GraphQL to find affected datasets, schemas, and owners. Lineage tracing — Traverses full upstream and downstream lineage in DataHub to map the complete blast radius — every dashboard, feature, and report that depends on changed columns. AI risk assessment — Sends the schema diff, lineage graph, and change context to NVIDIA NIM (Llama 3.1) which produces a quantified risk score (0–1), identifies breaking changes, and recommends actions. PR verdict — Posts a structured comment directly on the GitHub PR with a clear verdict: SAFE TO MERGE, CHANGES REQUESTED, or BLOCK — including risk breakdown, affected entities, schema diffs, and action items. The dashboard provides real-time monitoring with live SSE updates, incident tracking with expandable evidence, aggregate analytics (verdict distribution, risk heatmaps, top repos), and a full audit log for compliance.
How we built it
Backend: Node.js + Express, MongoDB (Atlas) for persistence, JWT + GitHub OAuth for auth, Server-Sent Events for real-time dashboard updates.
Frontend: Pure HTML5 + CSS3 + ES6 — zero frameworks, zero build steps. Dark-themed dashboard with sidebar navigation, 5 views (All Incidents, Failed, Warnings, Stats, Audit Log), inline-expandable rows, and CSS-only charts.
DataHub integration: Used the DataHub REST API (/aspects, /entities, GraphQL queries) for entity search, schema inspection, and lineage traversal. Built a GraphQL fallback layer so the system works even when the MCP server is unavailable. Ingested Snowflake datasets with DatahubRestEmitter and MetadataChangeProposalWrapper from the Python SDK.
AI pipeline: NVIDIA NIM REST API with meta/llama-3.1-8b-instruct for schema analysis and risk assessment. Structured prompts produce JSON-scored risk assessments with per-factor contributions.
Deployment: Dockerized backend on Render, MongoDB Atlas for production data, GitHub OAuth with CSRF-protected state tokens, bcrypt password hashing, and rate limiting on all auth endpoints.
Challenges we ran into
DataHub MCP server complexity — We initially tried to use the DataHub MCP Server for lineage queries, but couldn't get the container running reliably. Pivoted to a REST API + GraphQL fallback architecture, which ended up being more resilient. The MCP documentation was sparse for self-hosted setups. Lineage graph traversal — DataHub's lineage API returns upstream/downstream entities, but mapping from column-level changes to dataset-level blast radius required custom traversal logic. We had to build a recursive walker that follows the lineage DAG and correlates column names across connected datasets. Docker deployment with monorepo — The repo has frontend files at the root and backend in a subdirectory. Getting Render's Docker build context to include both required restructuring the project layout three times before settling on a public/ directory inside the backend. GitHub OAuth redirect URI mismatch — The redirect URI sent during token exchange must exactly match what's registered in the GitHub OAuth app. Small differences (trailing slash, http vs https, localhost vs production URL) caused silent failures. Added detailed logging to trace the exact URI being sent. MongoDB module loading — Having both yarn.lock and package-lock.json in the repo caused require('mongoose') to hang indefinitely. Removing the npm lockfile and doing a clean yarn install fixed it, but it took hours to diagnose.
Accomplishments that we're proud of
End-to-end lineage-aware PR review — From webhook to verdict in under 5 seconds. The system actually works: push a PR, get a comment with lineage-backed risk assessment. Column-level precision — Not just "this table changed" — LineageLens knows that renaming order_total in stg_orders breaks the revenue_dashboard SQL template and the daily_revenue ML feature. Real-time dashboard — SSE-powered live updates with toast notifications and progress bars. The dashboard shows incidents as they happen, with expandable evidence blocks. Zero-framework frontend — 700+ lines of hand-written CSS with glassmorphism, scroll reveals, animated counters, and a premium landing page. No React, no Tailwind, no build step. Production-ready auth — Email/password + GitHub OAuth, JWT with refresh tokens, CSRF state verification, rate limiting, audit logging, and role-based access control. All endpoints protected. DataHub Python SDK ingestion — Successfully ingested Snowflake datasets with full schemas and lineage chains into DataHub using DatahubRestEmitter, proving the integration works end-to-end.
What we learned
DataHub's lineage graph is incredibly powerful but underutilized — The data is there, the APIs work, but most teams never look at it during development. Making it actionable (not just browsable) is the real unlock. Column-level lineage is the hard part — Dataset-level lineage is straightforward. But the real value is knowing that column_x in model A feeds column_y in model B which powers dashboard C. Getting that column-level mapping right required deep understanding of DataHub's schema and aspect models. AI needs structured context, not just diffs — Throwing a git diff at an LLM gives shallow results. The magic is in the preparation: correlating the diff with DataHub lineage, schema history, and ownership data before sending it to the model. Monorepo Docker builds are tricky — Build context boundaries matter. A .gitignore at the repo root silently becomes a .dockerignore fallback, blocking directories you didn't intend to exclude. SSE > WebSockets for this use case — Server-Sent Events gave us unidirectional real-time updates with auto-reconnect, zero extra dependencies, and trivial implementation. Perfect for dashboard notifications.
What's next for LineageLens
DataHub MCP Server integration — Once the MCP container is stable, we can use structured tool calls for lineage queries instead of raw GraphQL, enabling more complex multi-hop traversals. Batch PR analysis — Analyze all open PRs on a schedule and surface conflicts before merge — proactive, not reactive. Slack/Teams notifications — Alert data owners when a PR touches their datasets, with a one-click "review this PR" link. Lineage-aware CI checks — Block CI if the blast radius exceeds a threshold. Integrate with GitHub Branch Protection rules. DataHub quality assertion integration — After merge, automatically verify that downstream assertions still pass. Close the loop: pre-merge risk assessment → post-merge quality validation. Multi-repo support — Analyze cross-repository impacts when one team's dbt models depend on another team's data contracts.
Log in or sign up for Devpost to join the conversation.