Landing Page
Architecure Diagram
Dashboard
Risk Scoring
Live Feeds

OVERSIGHT: The Change-Intelligence Layer for DataHub

Inspiration

In modern data ecosystems, the real magic of AI occurs when agents possess complete organizational context. Without reliable knowledge of schemas, lineage, ownership, and governance, both human engineers and AI agents make blind changes, resulting in silent data corruption, broken downstream dashboards, and hallucinated metrics.

We realized that while traditional tools monitor data quality after data reaches production, there is a massive gap in proactive, preventive guardrails.

We were inspired to build OVERSIGHT as an autonomous, operational change-intelligence layer. Our goal was to "shift data quality left" by intercepting code and schema changes during the pull request (PR) process, checking them against DataHub's rich metadata context graph, and remediating issues before they ever hit production.

What It Does

OVERSIGHT acts as an active, preventative sentinel between developer/database workflows and production data environments.

1. Ingest & Parse

It intercepts schema and pipeline changes via GitHub PR webhooks and DDL migration files (e.g., Flyway/Liquibase SQL scripts).

2. Enrich with DataHub

It resolves affected table/view schemas and fetches N-degree lineage graphs, ownership, glossary terms, and governance tags directly from DataHub.

3. Compute Blast Radius & Risk

It calculates an operational risk score using an in-memory graph representing the affected data pipelines.

The risk score is determined mathematically based on the severity of the code change, the downstream blast radius, and the criticality of affected nodes:

Risk Score

Risk Score = 100 × Severity × ∑ (γᵈ × Criticality(d)), for d = 1 … N

Where:

Severity Score ∈ [0, 1.0]
Node Criticality ∈ [0, 1.0]
γ ∈ (0, 1] is the distance decay factor.
d is the downstream distance from the changed node.
N is the maximum downstream lineage depth considered.

4. Generate Remediation & Visualizations

It automatically spawns AI-agent runs to output code patches (e.g., dbt updates, SQL backfill scripts) and renders raw Mermaid lineage diagrams of the impacted paths.

5. Closed-Loop Write-Back

OVERSIGHT posts risk reports as comments on GitHub PRs and alerts owners on Slack.

Crucially, it writes deprecation statuses and impact documentation directly back into DataHub, ensuring that the central catalog is automatically kept up-to-date.

How We Built It

OVERSIGHT is architected as a modular, enterprise-ready full-stack application:

Backend Engine: FastAPI & Python

Orchestrated using the uv package manager.
The core pipeline handles ingestion, metadata enrichment, risk assessment, and third-party integrations (GitHub App, Slack).

Lineage & Graph Core

Built using NetworkX to parse lineage lists fetched from DataHub's APIs.
Constructs in-memory dependency DAGs for efficient traversal.

DataHub Client Layer

Integrated deeply with the DataHub Python SDK, GraphQL API, and custom Model Context Protocol (MCP) server endpoints.
Performs schema queries and commits metadata write-backs, including deprecations and institutional documents.

AI & Evaluation Framework

Leverages LLM agents (reviewer, validation, heuristics) to interpret code diffs.
We built a complete offline test framework (backend/app/evals/) featuring YAML case definitions and Markdown rubrics to systematically test and grade agent outputs.

Frontend Dashboard

Created with:

TypeScript
Tailwind CSS
shadcn/ui
ReactFlow

It provides a clean, visual portal to:

Track the change-risk feed
Drill into specific PR diagnoses
View interactive lineage blast radii
Access direct navigation links back to DataHub

Database & Infrastructure

Powered by PostgreSQL 16
SQLAlchemy 2.x
Alembic for schema migrations
Packaged with Docker Compose for local deployment alongside a self-hosted DataHub instance.

Challenges We Ran Into

Mapping Code Diffs to DataHub URNs

Extracting abstract schema definitions from raw SQL strings or dbt files and correctly identifying their corresponding DataHub Entity URNs required robust parsing, especially when dealing with nested structures or inconsistent database schema naming conventions.

Scale-Resilient Graph Analysis

Real-world enterprise lineages are massive. Traversing deep dependencies and calculating multi-variable risk scores dynamically during a webhook lifecycle required optimizing NetworkX queries and designing efficient caching layers.

Designing the Closed-Loop Write-Back

Ensuring that OVERSIGHT could cleanly update DataHub without causing infinite event loops or corrupting metadata required strict synchronization.

We overcame this by isolating the mutations to specific deprecation aspect updates and structured institutional memory document attachments.

Accomplishments That We're Proud Of

End-to-End Functional Integration

We successfully built a fully functioning, closed-loop pipeline where a Git PR triggers an automated risk calculation, alerts developers, and writes documentation back to DataHub without manual human intervention.

Robust Evaluation Suite

Rather than relying on simple prompt engineering, we implemented a structured evaluation framework (evals/) utilizing customized YAML holdout/adversarial cases.

This ensures our AI code reviewer and DDL diagnostic engines are verifiable and statistically accurate.

Polished, Developer-Centric UX

The Next.js 15 dashboard, coupled with interactive ReactFlow charts, provides an intuitive interface that makes abstract "blast radius" concepts immediately clear to data platform leads.

What We Learned

Metadata Is the Lifeline of AI

Building this project highlighted that DataHub's metadata graph is not just a search tool for data discovery - it is a critical contextual foundation that enables AI agents to make reliable, grounded decisions about software workflows.

Shift-Left Works for Data

Moving data governance and validation steps directly into the CI/CD pull-request lifecycle is infinitely more cost-effective than attempting to remediate broken metrics and pipeline outages in production.

Strict Modeling Over Raw Text

Utilizing structured schema mappings and defined severity tables ensures that risk scores remain predictable, consistent, and defensible for enterprise compliance.

What's Next for OverSightAI

Orchestrator DAG Diffing

Extend our ingestion layer to parse and evaluate Airflow, Prefect, and Dagster DAG changes before deployment.

Advanced Data Contract Enforcement

Integrate DataHub's data contracts and assertions directly into the PR review block, failing builds automatically if a schema modification violates an active consumer agreement.