Ripple

DATA INCIDENT TRIAGE

Comment

Inspiration

A production table breaks at 3am. Before anyone can fix it, someone has to answer, by hand: what downstream dashboards and tables are now serving wrong data? Who owns them? Which are customer-facing right now? And where do I write this down so nobody re-investigates it tomorrow? That's 30–60 minutes of frantic clicking through lineage graphs — every single incident.

DataHub already knows the entire dependency graph. I wanted an agent that turns that knowledge into instant, actionable incident response — and, crucially, writes the result back into the catalog so the knowledge compounds instead of evaporating.

What it does

Ripple is a data-incident triage agent for DataHub. Given a broken asset's URN, it:

  • Reads context — traverses downstream lineage across every hop (Snowflake → dbt → Looker / PowerBI / Tableau), pulling ownership and platform, through the DataHub MCP Server / SDK.
  • Reasons — ranks the blast radius by criticality (customer-facing dashboards and owned production tables first) and assigns a severity (SEV1–3).
  • Acts + writes back — applies an incident tag, saves a runbook into the asset's documentation, and raises a native DataHub Incident entity, so the next engineer or agent inherits full context.

On a seeded demo it lights up 8 affected assets across 5 platforms, 3 of them customer-facing, in one command.

It also does root-cause (upstream) analysis, column-level blast radius (which column traces to the break), and an auto-trigger watch loop. It ships two front-ends: a rich terminal UI and a read-only web dashboard with an interactive lineage graph (light/dark themes, no external JS).

How I built it

  • DataHub Core quickstart + the showcase-ecommerce data pack for a realistic catalog. A seeder script emits a purpose-built incident scenario — a source table fanning out to 5 dbt/warehouse tables and 3 BI dashboards, with owners and column-level lineage — so the demo is dramatic and reproducible.
  • The agent talks to DataHub through the Python SDK (DataHubGraph, GraphQL searchAcrossLineage, MetadataChangeProposalWrapper, mce_builder) and the self-hosted MCP Server (wired into Claude Code during development to validate query shapes against the live graph).
  • Key design choice: facts are deterministic, prose is LLM-generated. Lineage traversal, owner resolution, and criticality ranking are plain code — the LLM only writes the human-readable narrative. This keeps the impact analysis trustworthy and the demo reproducible.
  • The whole read → reason → write workflow is packaged as a reusable DataHub Agent Skill (skills/incident-triage/) for any MCP-compatible client.
  • UIs: Rich for the terminal; FastAPI + a dependency-free custom SVG renderer for the web graph.

Challenges I ran into

  • Docker's default 16 GB disk filled up, pushing OpenSearch past its flood watermark. The only symptom was searchAcrossLineage returning all shards failed — nothing pointed at disk.
  • datahub docker quickstart --restore-indices finished cleanly but rebuilt only the search index, not the graph relationship edges — so lineage stayed empty until a full datahub docker nuke.
  • The GraphQL addTag mutation refuses to apply a tag whose entity doesn't exist yet (Urn does not exist); I had to create the tag entity first and emit GlobalTags via the SDK.
  • SchemaMetadata.platform requires a full dataPlatform URN, with a cryptic "Provided urn urn" is invalid error.
  • Drawing an accurate lineage DAG meant reading multiple aspect types — dashboards express their inputs via DashboardInfo.datasets, not UpstreamLineage.

Accomplishments I'm proud of

  • A complete read → reason → write loop against live DataHub — an agent that acts on the catalog, not just a read-only query tool.
  • Four capabilities beyond the core (root-cause, column-level, auto-trigger, native incidents) plus two polished UIs — all working end-to-end.
  • A reusable Agent Skill designed to be contributed back to DataHub.

What I learned

  • DataHub's metadata model and aspect system — lineage (table and column), tags, incidents, schema, documentation — and how to emit each via the SDK.
  • How DataHub's search index and graph index differ, and how they fail (and recover).
  • That the most trustworthy agent design keeps the LLM out of the fact-gathering path entirely.

What's next for Ripple

  • Assertion-failure event subscriptions to replace the polling auto-trigger, so triage fires the moment a data-quality check fails.
  • Auto-remediation PRs — when a schema change breaks a downstream model, generate the fix and open a pull request.
  • Notifications & routing (Slack / PagerDuty), impact quantification from usage stats, and ML-asset support via DataHub's end-to-end ML lineage.

Built With

Share this project:

Updates