Faultline

Inspiration

Every data engineer knows this moment: a pipeline breaks, a dashboard looks wrong, and someone spends hours manually tracing lineage to figure out what's affected and who to tell. Worse, that knowledge disappears the second the incident is closed, the next failure starts the investigation from zero. DataHub already has the map: lineage, ownership, schemas, all connected. It just needed an agent that actually used it, instead of just displaying it.

What it does

Faultline watches for a data pipeline failure and runs the full incident response loop on its own:

1. Reads DataHub : walks the lineage graph from the broken asset to find every downstream dataset, dashboard, or model affected, and who owns each one.
2. Reasons : asks Claude to turn a raw error message into a clear, plain-English root cause summary.
3. Acts : posts targeted Slack alerts (not a generic broadcast, the real owner of each affected asset) and opens a Jira ticket pre-filled with full lineage context.
4. Writes back : this is the part most agents skip. Faultline tags every affected asset [INCIDENT: ACTIVE] in DataHub and logs a structured incident note on the origin asset.

That last step matters most: the next engineer or the next run of the agent and inherits the history instead of re-investigating the same failure from scratch.

How I built it

Faultline is a Python agent built on the acryl-datahub SDK, talking to a self-hosted DataHub instance via DataHubGraph and its GraphQL API for lineage traversal (searchAcrossLineage), ownership lookups, and metadata write-back (GlobalTagsClass, InstitutionalMemoryClass).

• Reasoning: Claude (Anthropic API) converts raw error text + lineage context into a human-readable root cause.
• Slack: slack-sdk, posting to a dedicated #data-incidents channel via a real bot app.
• Jira: REST API v3, creating tickets with the full incident description, affected assets, and owners.
• Orchestration: a single entrypoint (src/triggers/simulate_failure.py → handle_incident()) runs the entire read → reason → act → write-back loop.

The whole thing is designed so that in production, the manual trigger script would simply be replaced by a real signal, a dbt test failure, an Airflow callback, or a DataHub Action.

Challenges I ran into

Honestly? Infrastructure, not code.

Running DataHub's full local stack, Kafka, OpenSearch, MySQL, and GMS all at once on an 8GB laptop turned into days of memory starvation, Docker corruption, and crash loops. I tuned JVM heap sizes, adjusted WSL2 memory limits, restarted Docker Desktop more times than we'd like to admit, and eventually accepted the hardware just wasn't enough. Moving the whole environment to GitHub Codespaces fixed it in minutes and GMS went from repeatedly crashing on 2GB of available RAM to starting cleanly in under a some minutes. Lesson learned the hard way: know your resource ceiling before you spend days fighting it.

Accomplishments that I am proud of

Getting a genuinely complete loop working not stubs, not simulated output. Every piece (DataHub lineage read, Claude reasoning, real Slack alerts, real Jira tickets, DataHub write-back) runs against live services and was verified end-to-end, visually confirmed in each tool, not just trusted from terminal logs.

What I learned

That "contribute back to the graph" is a much higher bar than "read from the graph" and it's the difference between a bot that's mildly convenient and something a data team would actually trust to run unattended. Writing durable, structured incident history into DataHub, rather than just firing an alert and forgetting, is what makes the next incident faster to resolve too.

What's next for Faultline

• Real triggers : replace the manual simulation script with an actual dbt test webhook or a DataHub Actions listener, so Faultline runs automatically instead of on command.
• Impact-weighted blast radius : prioritize notifications by business criticality (an exec dashboard vs. an internal scratch table), not just lineage distance.
• Resolution tracking : automatically clear [INCIDENT: ACTIVE] tags once the upstream fix lands, closing the loop fully.
• Multi-agent handoff : let Faultline's incident notes seed a second agent that proposes the actual fix (e.g., a dbt PR), not just the alert.

Built With

• claudeapi
• datahub
• docker
• github-codespaces
• graphql
• jiraapi
• python
• rest-api
• slackapi