Lineage Lifeboat

A DataHub-powered recovery compiler that turns live lineage into an approved, dependency-correct, executable recovery program.

Comment

Inspiration

Disaster-recovery runbooks are usually organized by platform: restore storage, restart transformations, rebuild features, redeploy models, and refresh dashboards. That organization misses the dependency order across those platforms. A restored table does not prove that its features, model, or dashboard are ready to trust, and static runbooks drift as lineage and ownership change.

I built Lineage Lifeboat around a different question: if DataHub already knows how the estate is connected, can that graph become safe, executable recovery control flow?

What it does

Lineage Lifeboat turns current DataHub lineage and metadata into an approval-gated recovery program. In the hosted synthetic scenario it:

  1. initializes a disposable commerce estate and triggers an explicitly confirmed outage;
  2. binds the incident to current DataHub MCP entity and lineage evidence;
  3. computes the affected subgraph while retaining a healthy prerequisite and excluding an unrelated inventory branch;
  4. compiles six recovery targets into five dependency-correct waves;
  5. requires a human to approve the exact persisted plan ID;
  6. executes real idempotent Parquet, DuckDB, Python, and report adapters;
  7. validates every target before a downstream consumer may proceed; and
  8. writes a recovery marker to DataHub, immediately rereads it through MCP, and retains a verifiable run record.

If a required validation fails, execution stops before downstream consumers run. A resumed run skips already verified work and reuses stable idempotency keys.

How I use DataHub

DataHub is the dependency and governance authority, not a decorative catalog page. The project uses the DataHub MCP Server to read exact entities and complete direct lineage for the allocated lifeboat.* graph. Planning is bound to a fingerprint of that verified graph and fails closed if an expected entity or edge is missing.

After successful recovery, the DataHub Python SDK emits a supported globalTags update. The workflow immediately rereads the target through MCP and requires the marker to be present before reporting a verified DataHub outcome. Reset and writeback are restricted to exact project fixtures.

How I built it

The service is a Python and FastAPI workflow controller with Pydantic contracts, NetworkX graph planning, a persisted approval and run ledger, a registry of recovery adapters, and deterministic validation. The judge console is a small same-origin HTML, CSS, and JavaScript interface.

The demonstration estate uses Parquet snapshots, DuckDB transformations, deterministic Python feature and model builds, and a report refresh. The planning and safety path is deterministic; no LLM output is trusted as execution authority, and no paid model is required to reproduce the workflow.

Challenges I faced

The first hard problem was proving complete MCP lineage rather than accepting a successful but bounded response. I inspect the advertised tool schema, request a safe explicit limit, and refuse to plan unless every expected edge is present.

Reset behavior also differed by DataHub entity type. Dataset status could be reset safely, while the project Domain and Tag controls needed to remain. The reset now touches only the exact dataset allowlist, invalidates readiness before mutation, and supports idempotent reseeding.

Finally, recovery evidence had to survive later runs. Stable filenames allowed a new run to replace an older report, so evidence is now written atomically beneath validated per-run paths and cannot be replaced with different bytes.

What I learned

Recovery readiness is a graph property, not a service-health property. I also learned that a successful write is not proof until the read path observes it, and that evidence storage is part of the product contract rather than an implementation detail. The most important fail-closed behavior appears during partial resets and transient indexing, not only during obvious outages.

What I am proud of

  • Six real recovery targets execute in a dependency-correct order across four adapter families.
  • Exact-plan approval, validation gates, idempotency, and resume are visible in one workflow.
  • DataHub MCP reads and supported writeback are verified by immediate reread.
  • The unrelated branch remains excluded throughout the incident.
  • The public demo stays reproducible, synthetic, and below the three-minute limit.

Safety and limitations

The outage and recovery actions operate only on a disposable local estate. DataHub writes are namespace- and fixture-guarded, reset is allowlisted, and approval is required before execution. The MVP does not perform cloud failover, claim autonomous production recovery, or guarantee RPO or RTO. Adopting it for production would require organization-specific adapters, authentication, role-aware approval, and operational review.

What's next

I would add adapters for orchestrators, warehouses, feature stores, and BI systems; notify owners discovered from DataHub when manual work is required; and export approved plans to tools such as Airflow, Dagster, or Prefect.

Built With

Share this project:

Updates