Inspiration
Most data incident systems answer one question: should the pipeline stop? In a real data platform, that question is too broad. A corrupted source field may affect one downstream data product while another branch remains completely safe. Stopping everything creates unnecessary downtime, while continuing everything spreads bad data.
LineageProof was inspired by Axiomatic Intelligence: treat every claim as a hypothesis that must survive evidence, contradiction, and a declared falsification test. Instead of asking an LLM to produce a confident answer, LineageProof creates named, versioned, and challengeable claims grounded in deterministic SQL and DataHub's context graph.
The goal was to build an agent that can say:
This incident is real. This exact branch is affected. This sibling branch is safe. Here is the evidence that would disprove the decision.
What it does
LineageProof is a multi-agent data incident circuit breaker powered by DataHub.
When a quality issue is selected, it:
- Runs a deterministic, read-only SQL check against the source data.
- Uses the official DataHub MCP Server to retrieve the affected entity, metadata, ownership, tags, glossary context, upstream lineage, and schemas.
- Compares the affected and protected branch schemas to prove the impact boundary.
- Sends the evidence to three adversarial agents:
- Investigator: builds the strongest evidence-backed incident claim.
- Skeptic: attacks the claim, scope, and proposed action.
- Arbiter: chooses the narrowest safe decision.
- Calculates confidence deterministically instead of trusting model confidence alone.
- Requires explicit human approval before any metadata mutation.
- Writes the complete report back to DataHub as a durable document linked to the affected asset.
The current MVP persists a verified halt decision in DataHub. It does not directly pause an external Airflow or Dagster job. A production orchestrator could consume this durable decision and enforce the halt.
The verification protocol
LineageProof follows four stages:
1. Diverge
Collect independent evidence from two systems:
- Live, read-only SQLite assertions establish whether the source violation exists.
- DataHub MCP provides organizational context, lineage, ownership, governance metadata, and schemas.
2. Collide
The Investigator and Skeptic evaluate the same evidence from opposing positions. Agent responses must include the exact row count and affected assets. Generic or unsupported model output is rejected and replaced with a deterministic grounded fallback.
3. Converge
The Arbiter selects either:
HALT_AFFECTED_BRANCHNO_ACTION
A selective halt is allowed only when DataHub schema evidence proves that the incident fields exist in the affected branch and do not exist in the protected branch.
4. Validate
The system calculates evidence confidence using:
$$ C = 0.35S_{SQL} + 0.20S_{lineage} + 0.10S_{metadata} + 0.15S_{scope} + 0.20S_{agents} $$
The LLM does not calculate or control the final score. A human must then review the generated report and explicitly approve the DataHub writeback.
Why DataHub is essential
DataHub is not used as a simple metadata search box. LineageProof depends on DataHub to determine whether selective action is justified.
Through the official MCP Server, the application uses:
searchto discover healthcare assets.get_entitiesto retrieve ownership, tags, glossary terms, and asset metadata.get_lineageto trace upstream dependencies across three hops.list_schema_fieldsto compare affected and protected branches.save_documentto persist the approved report and link it to the affected asset.
Without DataHub, the SQL query could prove that bad rows exist, but it could not safely determine which downstream product must stop.
The writeback also creates compounding organizational knowledge. The next engineer or agent can discover the incident, evidence, decision, provenance, and falsification condition directly inside DataHub.
Demo dataset and results
LineageProof uses DataHub's official synthetic healthcare dataset with 55,500 patient records and a forking lineage graph:
raw_patients → staging_patients → mart_billing
raw_patients → staging_patients → mart_demographics
Four planted incidents were verified:
| Incident | Live result | Decision |
|---|---|---|
| Negative billing amounts | 1,215 rows, 2.19% | Halt mart_billing; protect mart_demographics |
| Impossible patient ages | 832 rows, 1.50% | Halt mart_demographics; protect mart_billing |
| Missing patient names | 555 rows, 1.00% | Inconclusive; no automatic halt |
| Swapped admission and discharge dates | 277 rows, 0.50% | Halt mart_billing; protect mart_demographics |
The missing-name case demonstrates the fail-safe behavior. The name field exists in both marts, so DataHub cannot prove that only one branch is affected. LineageProof verifies the defect but refuses to automate a selective halt.
How we built it
The application is written in Python and uses:
- DataHub Core running locally through Docker.
- DataHub MCP Server over MCP stdio.
- Gemma 3 running locally through Ollama.
- SQLite for deterministic read-only evidence.
- Streamlit for the interactive investigation and approval interface.
- Pydantic for structured agent outputs and validation.
- pytest for evidence, grounding, confidence, and schema-boundary tests.
- GitHub Actions for automated tests on every push and pull request.
Secrets remain in a gitignored .env file. Source data is opened using SQLite read-only mode. The MCP server may expose mutation capabilities, but LineageProof uses only save_document, and only after a checkbox and explicit approval click.
Challenges we faced
DataHub OSS setup on Windows
Running DataHub locally required resolving Docker port conflicts, an interrupted GitHub compose-file download, authentication configuration, and personal access-token setup. We created a diagnostic script that checks SQLite evidence, Ollama, MCP connectivity, available tools, schemas, and selective scope before starting the demo.
SQLite type affinity
The healthcare dataset stores age values as text. An initial numeric comparison incorrectly marked all 55,500 rows as invalid because of SQLite's type-affinity rules. We fixed the query with an explicit integer cast and added a regression test. The correct result is 832 invalid-age rows.
Preventing plausible hallucinations
Local models can produce fluent explanations that are not tied to the supplied evidence. We added output schemas, exact grounding requirements, permitted verdicts, deterministic fallbacks, and independent confidence scoring.
Proving that a sibling branch is safe
Lineage alone proves connectivity, but it does not prove field-level impact. We added DataHub schema comparison as a required condition for selective halting. This is why missing names produce INCONCLUSIVE instead of an unsafe automated decision.
What we learned
The most important lesson was that lineage is necessary but not sufficient. Safe automation requires both dependency paths and schema-level impact boundaries.
We also learned that confidence should come from evidence coverage, not how convincing model prose sounds. Finally, metadata writeback changes an agent from a temporary assistant into part of the organization's knowledge system.
Accomplishments
- Completed the full DataHub read, reason, approve, and writeback loop.
- Produced reproducible row-level evidence for all four planted incidents.
- Demonstrated selective halting and a fail-safe inconclusive path.
- Persisted a verified report in DataHub linked to
mart_billing. - Added nine automated tests and a passing GitHub Actions workflow.
- Used only local open-source infrastructure and a local language model.
What's next
Future versions can integrate the verified DataHub decision with Airflow, Dagster, or Prefect to enforce selective pipeline pauses. Other planned improvements include column-level lineage, DataHub assertions, incident remediation verification, confidence calibration across historical incidents, service-account authentication, and automatic reopening of a branch when its kill test returns zero.
LineageProof demonstrates what becomes possible when AI agents do not merely read metadata—they use context to make bounded decisions and contribute verified knowledge back to the graph.
Log in or sign up for Devpost to join the conversation.