Sentinel AI

Sentinel AI stops breaking data schema changes before merge by using DataHub MCP context to assess blast radius, block risky PRs, generate safe fixes, and preserve decisions.

Comment

Inspiration

Data teams often discover breaking schema changes only after they reach production. A removed column or incompatible type change can silently break dashboards, transformation jobs, and machine-learning features several hours later.

The problem is not usually a lack of metadata. Platforms such as DataHub already know about schemas, ownership, lineage, and historical queries. The missing piece is an agent that can use that context before code is merged and turn it into an enforceable engineering decision.

That inspired Sentinel AI: an autonomous pre-merge data reliability engineer that investigates schema changes, measures their blast radius, recommends a merge decision, generates carefully constrained remediation, and preserves the result for future engineers and agents.

What it does

Sentinel AI analyzes schema changes from a GitHub pull request or a submitted before-and-after schema.

For every investigation, it:

  1. Extracts the actual schema change deterministically.
  2. Detects removed columns, additions, nullability changes, renames, and incompatible type changes.
  3. Queries DataHub through the official Model Context Protocol server for schemas, ownership, lineage paths, downstream consumers, and query usage.
  4. Separates confirmed impact from assets that are merely connected.
  5. Produces an evidence graph with provenance for every claim.
  6. Calculates severity, evidence completeness, and a transparent recommendation such as SAFE_TO_MERGE, BLOCK, or INSUFFICIENT_EVIDENCE.
  7. Generates candidate SQL remediation with SQLGlot AST validation.
  8. Requires human review when a change affects predicates, joins, grouping, or other business semantics.
  9. Enforces the decision through GitHub Actions.
  10. After authenticated human approval, it can optionally post the result to GitHub and write investigation memory back into DataHub.

Sentinel fails closed when live evidence is unavailable. It visibly distinguishes LIVE_DATAHUB, DEMO_FIXTURE — NOT LIVE VERIFIED, and DATAHUB_UNAVAILABLE so fixture evidence cannot be mistaken for a live integration.

How we built it

The backend is implemented with Python and FastAPI. Pydantic validates API boundaries, SQLAlchemy provides persistence, and PostgreSQL is used by the production Docker Compose profile.

The investigation engine combines several focused stages:

  • Deterministic schema parsing and compatibility analysis
  • DataHub MCP JSON-RPC tool discovery and invocation
  • Multi-hop lineage and query-context collection
  • Evidence provenance and completeness scoring
  • Rule-based risk classification
  • SQLGlot AST remediation validation
  • Human approval and authenticated mutation gates
  • DataHub document/tag writeback
  • GitHub review automation

The frontend is a responsive Next.js and React application. It provides an analysis form, investigation dashboard, evidence and impact views, remediation review, integration status, and approval-controlled actions.

GitHub Actions runs backend tests, frontend regression tests, linting, production builds, and the schema-change firewall. Docker Compose packages the API, web application, PostgreSQL database, and pinned DataHub MCP sidecar.

The repository also includes reproducible breaking, safe, and insufficient-evidence scenarios so judges can inspect the generated investigation reports, evidence graphs, GitHub comments, validation results, and remediation patches without running the complete environment.

Challenges we faced

Preventing evidence hallucination

A connected downstream asset is not automatically a confirmed consumer. We had to preserve provenance for every piece of evidence and distinguish confirmed query or column-level usage from general graph connectivity.

Failing safely

A disconnected DataHub service must not silently fall back to convincing fixture data. Sentinel therefore exposes its evidence mode and blocks risky changes when the available evidence is insufficient.

Generating safe remediation

Removing every reference to a deleted column can alter business logic. We used SQLGlot AST analysis and require human review when a column appears in WHERE, JOIN, HAVING, GROUP BY, window, or qualification logic.

Reproducible MCP integration

A healthy DataHub GMS endpoint does not necessarily mean that MCP negotiation, required tools, authentication, and mutation permissions are working. Sentinel verifies the MCP protocol and required tools separately.

Safe external actions

Analysis is side-effect-free. GitHub comments and DataHub writeback require persisted approval and an authenticated operator. The request payload cannot approve itself.

What we learned

We learned that trustworthy agent behavior depends more on evidence quality and explicit boundaries than on producing a confident explanation.

Metadata provenance must remain visible throughout the investigation. Confirmed impact, inferred connectivity, missing evidence, and fixture evidence must never be blended together.

We also learned that autonomous remediation needs semantic limits. AST validation can prove that generated SQL is syntactically structured, but human review remains necessary when business predicates or aggregation semantics could change.

Finally, DataHub becomes especially valuable when its context is used operationally. Sentinel does not only display metadata—it turns DataHub context into a pre-merge policy decision and can preserve the investigation back in the metadata graph.

Accomplishments

  • Deterministic schema compatibility and risk analysis
  • Official DataHub MCP integration with tool negotiation
  • Evidence provenance and completeness scoring
  • Multi-hop lineage and query-context analysis
  • SQLGlot AST remediation safety checks
  • GitHub pre-merge enforcement
  • Authenticated human approval and mutation controls
  • Reproducible breaking, safe, and insufficient-evidence examples
  • 57 backend tests and 2 frontend component regressions
  • Passing lint, production build, Docker builds, and GitHub CI

What's next

Next steps include a managed hosted deployment, additional schema and transformation formats, reusable organization-specific policy packs, richer column-level lineage visualization, enterprise OIDC integrations, and historical analysis that learns from previous DataHub investigation documents.

Built With

Share this project:

Updates