Hydra - Verified Data Recall for DataHub

Find every consumer of contaminated data, contain the blast radius, execute a real recovery, and prove what recovered.

Comment

Inspiration

Repairing a corrupt source table does not repair the systems that already consumed it. The same bad field may have reached dbt models, executive dashboards, ML features, models, and production deployments—each with a different owner and recovery requirement. Incident response becomes a manual lineage investigation, a coordination problem, and an evidence problem.

Hydra applies the familiar idea of a product recall to data. It asks not only “where is the bad value?” but “which systems are now untrustworthy, what can we safely recover, what must remain quarantined, and what proof should the next person inherit?”

What it does

Hydra is a verified data-recall agent. It accepts a failed quality signal for a source dataset field and uses DataHub at runtime to confirm the field, traverse downstream column and entity lineage, and enrich every affected asset with owners, domains, subtypes, criticality, and quality context.

Hydra then performs a governed six-stage protocol:

  1. Detect the real failing condition.
  2. Map the live downstream blast radius from DataHub.
  3. Contain every affected asset with RECALL_ACTIVE and an initial recall document.
  4. Recover through a fixed, allowlisted snapshot restore and dbt build.
  5. Verify by rerunning the same checks that originally failed.
  6. Close only the assets supported by proof and write the final incident memory back to DataHub.

In the demo, one retail order amount is multiplied by 100, changing $142.80 to $14,280.00. Hydra discovers seven affected assets across four owner groups: the source table, two transformed datasets, an executive dashboard, a feature table, an ML model, and its production deployment. It restores a trusted DuckDB snapshot, runs a real dbt build, and proves both the accepted-range and modeled-revenue checks now pass.

Five data and reporting assets are cleared with evidence. The ML model and production deployment remain under RECALL_ACTIVE, because rebuilding data does not prove that a model was rolled back or retrained. Hydra makes that unresolved owner action explicit instead of claiming a fictional recovery.

How we built it

  • DataHub OSS/Core is the metadata graph and system of record for schemas, column lineage, entity lineage, owners, domains, tags, quality state, ML metadata, and recall documents.
  • DataHub's official MCP Server provides constrained runtime tools over stdio for schema lookup, lineage traversal, entity context, tag mutations, and document write-back.
  • DataHub Agent Context Kit and Python SDK provide a compatibility transport, deterministic graph seeding, and quality-result reporting.
  • A Python state machine implements DETECTED → MAPPED → CONTAINED → RECOVERING → VERIFIED → CLOSED, with safe failure behavior and compensating containment if closeout cannot be proven.
  • DuckDB provides a repeatable local warehouse, and dbt-duckdb builds the downstream models and runs the validation suite.
  • FastAPI, Pydantic, vanilla JavaScript, HTML, and CSS power a responsive interface that acts as a visual audit trail rather than a generic chat box.
  • An optional schema-constrained OpenAI reasoning layer may explain priorities and draft owner actions, but it cannot discover assets, execute commands, or change verification results. The recorded demo uses the complete deterministic fallback and needs no API key.

The recall engine contains no hard-coded consumer list. It first asks DataHub for field-level downstream lineage, then expands entity lineage from only the affected datasets. Empty or unavailable lineage is an explicit failure. The only executable recovery is a fixed allowlist; model-generated text is never executed.

Meaningful use of DataHub

Hydra cannot complete its central workflow with a static JSON graph. DataHub must confirm the source field and return the live impact graph. Column lineage keeps the selection precise, while entity lineage carries the incident into dashboard and ML consumers. Ownership and criticality drive coordination and priority. Hydra also contributes back to the graph by applying containment and clearance tags and saving both active and verified-recovery documents. The catalog becomes durable organizational memory for humans and future agents.

Challenges we ran into

The hardest challenge was making the automation honest at every boundary. Column lineage is not row-level lineage. Rebuilding dbt does not prove a model was rolled back. A metadata mutation is not successful until DataHub confirms it. An LLM explanation must not be allowed to invent asset URNs or execute recovery commands.

Those constraints shaped the architecture: bounded runtime graph discovery, explicit mutation acknowledgements, an allowlisted subprocess, before-and-after validation, compensating containment when closeout fails, and permanent quarantine for ML assets without system-specific recovery evidence.

We also found that the local DataHub quickstart can have healthy GMS and UI services without exposing the guessed /mcp HTTP endpoint. We switched the default integration to DataHub's official standalone MCP server over stdio, which made the local setup reproducible and kept credentials in the subprocess environment rather than command arguments.

Accomplishments that we're proud of

  • A repeatable one-command end-to-end check against a real local DataHub OSS instance.
  • Seven runtime-discovered assets spanning data, BI, feature, model, and deployment metadata.
  • A genuine corrupt → recover → rebuild → verify transition.
  • DataHub containment tags plus active and verified-recovery documents.
  • Safe partial clearance: five assets cleared with proof, two visibly left for an owner.
  • A focused interface that tells the incident story without requiring a chat transcript.
  • Tests for empty lineage, unavailable DataHub, recovery failure, failed tag clearance, compensating containment, deterministic reasoning fallback, and the judge-facing API.

What we learned

Metadata becomes much more valuable when an agent can both read context and leave structured memory behind. We also learned that trustworthy automation is less about giving a model unlimited authority and more about separating authority cleanly: DataHub determines impact, policy determines allowed action, dbt and DuckDB produce recovery evidence, and only confirmed evidence permits clearance.

The practical lesson was equally important: integration details matter. The MCP tool surface, the transport mode, mutation acknowledgements, idempotent seeding, and honest handling of unsupported recovery paths determined whether the product was a trustworthy agent or merely an impressive-looking demo.

What's next

Production Hydra would add event-driven assertion ingestion, durable workflow orchestration, authenticated approvals, incident and notification integrations, policy-specific recovery adapters, retries and idempotency keys, and real model registry rollback or retraining workflows. It would retain the same invariant: no asset is cleared without system-specific proof.

Built with

DataHub OSS/Core, DataHub MCP Server, DataHub Agent Context Kit, DataHub Python SDK, Python, FastAPI, DuckDB, dbt-duckdb, Pydantic, MCP Python SDK, JavaScript, HTML, CSS, pytest, Ruff, uv, TestReel/Playwright, FFmpeg, and Kokoro ONNX.

Built With

Share this project:

Updates