Pareis

Mission control for production ML on DataHub lineage

Comment

Inspiration

Most production ML failures aren't model failures — they're silent data failures. A bad 3 a.m. load leaves prices NULL, nothing crashes, and models keep serving confident garbage. The failing dbt test fires in the data team's world; the damage lands in the ML team's world days later. The knowledge that connects them — table → feature → model — exists in exactly one place: DataHub's lineage graph. Walking it during an incident is nobody's job. We made it an agent's job.

What it does

Pareis closes the full incident lifecycle for production ML, on real signals:

  • Detect — scans the catalog for failing data-quality assertions (real dbt test results ingested through the official dbt connector). No mocked lineage, assertions, or metadata anywhere — the demo deliberately corrupts a real warehouse and lets the real test catch it.
  • Contain — column-precision blast radius: reads which column failed, re-roots to origin columns, closes transitively over fine-grained lineage into features and models. Our demo incident flags exactly the poisoned features per model (2/6 and 2/5, MEDIUM) instead of crying HIGH on everything — with the failed-row count attached.
  • Time-travel — blind-window contamination analysis: bounds when corruption began from assertion run history, cross-references every MLflow registry version's training timestamp, verdicts CONTAMINATED/CLEAN, and names the newest pre-window version as the safe rollback. A model retrained inside the blind window stays poisoned after the data heals — Pareis directly answers "which version do we roll back to," by name.
  • Enforce — a 10-check retrain gate with CI exit codes and --json; NO-GO physically blocks a retrain pipeline.
  • Act where teams work — native DataHub incidents (raised and auto-resolved), at-risk tags, incident notes, stateful Slack/Jira/Linear tickets (one per incident, resolution comments on heal).
  • Recover — when assertions go green, Pareis stands down on its own: flags removed, incidents resolved, and a self-written postmortem (timeline, time-to-heal, blast, classification) published as a searchable DataHub Document.
  • Prevent — pareis protect: transparent 0–100 reliability scores, untested critical columns ranked by model exposure, gaps matching past incidents' blast patterns flagged first, and paste-ready dbt tests generated to close them.
  • Prove — pareis evidence: an EU AI Act Article 10 data-governance evidence pack regenerated on demand from live metadata (obligations for high-risk systems generally apply from 2 August 2026).

Pareis uses a deterministic safety core — same incident, same answer, auditable — exposed to AI agents through DataHub's official MCP server, a DataHub Skill, and six specialized read-only Pareis tools. No paid model API is required for the core demo; fully reproducible on a laptop.

How we built it (DataHub-native by construction)

  • Metadata in: three official ingestion connectors (postgres, dbt with column-level lineage and test results, mlflow) over a real stack — 1.07M real UCI Online Retail II transactions in Postgres, dbt staging/marts with real tests, sklearn models registered in MLflow.
  • Reads: direct aspect and relationship reads via the DataHub SDK — avoiding search-index lag — plus assertion run-event timeseries for corruption-window bounding.
  • Writes: tags, institutional-memory notes, native Incidents, Documents, and typed structured properties (io.pareis.reliability_score, io.pareis.retrain_status, …) so DataHub itself can sort assets by risk.
  • Agent surfaces: six read-only MCP tools (stdio + stateless streamable HTTP per the 2026-07-28 spec) designed to run alongside DataHub's official MCP server (config shipped in .mcp.json.example); a repo-native DataHub Skill chaining both; and Pareis registered in DataHub's Agent Registry (urn:li:aiAgent:pareis) with lineage to the models it protects — the guardian appears inside the graph it guards.
  • Ops: Mission Control dashboard, watch daemon, CI gate workflow, automated test suite + GitHub Actions.

Challenges we ran into

  • dbt's artifacts overwrite each other (docs generate vs test), which silently destroys either column lineage or test results — we built a snapshot pipeline and fixed the silent part upstream (datahub#18823).
  • Column-level lineage attributes derived columns to raw tables, so asking "what's downstream of the failing column" naively returns nothing — our walk re-roots to origin columns first (and we verified first-hand where point-to-point MCP path lookup can and can't answer this).
  • Index lag: DataHub acknowledges writes before search catches up — we poll-until-consistent instead of sleeping.
  • Structured properties replace as a unit — read-merge-write or you delete your own metadata (caught by review, fixed with the test that would have caught it).

Accomplishments we're proud of

  • The blind-window contamination analyzer — the "was anything trained while we were blind, and what do we roll back to" question, answered from metadata alone.
  • A 100% real demo: real data, real failing tests, real registry versions, real incidents in DataHub's own UI. AUC drops from 0.7798 to 0.7734 when the model trains on the corrupted window — even the poison is real.
  • An upstream DataHub fix submitted from this work, labeled community-contribution (datahub#18823).

What we learned

Metadata is enough. Every hard question we started with — what's poisoned, is it safe to retrain, what do we roll back to, what should we test next — turned out to be answerable deterministically from the graph DataHub already has, if you read it carefully enough.

What's next

Gating continuous learners (pause the loop, not the retrain job), serving endpoint lineage, and upstreaming an ml-incident-response Skill.

Built With

Share this project:

Updates