DataHub Guard AI: Autonomous Incident Triage Engine

An autonomous observability agent that uses DataHub lineage to isolate pipeline failures, halt bad data, and auto-generate code fixes.

Comment

Inspiration

Modern data engineering teams are constantly fighting fires. When a pipeline fails in Airflow or dbt, the immediate questions are always: "What dashboards are broken? Which machine learning models are using bad data? Who do we need to alert?"

We realized that while DataHub holds the ultimate source of truth (the lineage metadata graph), it is often treated as a passive catalog. We asked ourselves: What if we gave DataHub a brain? What if it could proactively act as an autonomous agent that stops bad data from spreading and writes the code to fix the root cause?

What it does

DataHub Guard AI is an autonomous data observability and remediation agent. When a pipeline fails, DataHub Guard AI springs into action:

  • Blast Radius Analysis: Instantly traverses the real-time DataHub lineage graph to determine exactly which downstream datasets, ML features, and executive dashboards are impacted.
  • Circuit Breaker: If the risk score is too high, the agent acts autonomously to pause downstream Airflow/dbt jobs, preventing data contamination from spreading to business users.
  • Self-Healing Loop: Using advanced LLMs, it analyzes the failed query and schema metadata, writes a code fix (e.g., a dbt SQL patch), and simulates compilation until the code is validated.
  • ChatOps & One-Click Resolution: Surfaces a rich alert to Slack where a human data engineer can click "Approve & Merge PR" to instantly apply the fix to GitHub and resolve the incident.
  • Agentic Copilot: Engineers can chat directly with the AI Agent in the dashboard to ask questions like "Who owns the impacted dashboards?" and receive immediate answers based on DataHub metadata.

How we built it

We architected a fully decoupled, cloud-native application:

  • Backend (FastAPI & Python): A high-performance REST API acting as the agentic orchestrator. It integrates directly with DataHub's GraphQL and OpenAPI endpoints to fetch lineage, datasets, and tags.
  • Frontend (Next.js & React): A modern, dark-mode dashboard built using Next.js, Tailwind CSS, and shadcn/ui, featuring interactive lineage visualization (@xyflow/react) and a floating Chatbot overlay.
  • Deployment: Frontend deployed on Vercel's Edge Network; Backend API containerized via Docker on Render.

Challenges we ran into

Integrating DataHub metadata, LLM pipelines, and GitHub source control into a cohesive, sub-second autonomous loop presented several challenges:

  • Lineage Traversal: Navigating DataHub's GraphQL API to recursively fetch multi-level upstream and downstream dependencies.
  • LLM Hallucinations: Implemented a self-healing CI loop that feeds compilation errors back into the LLM until valid, production-ready SQL is produced.
  • React State Management: Coordinating real-time UI state between the floating Chat Agent, Circuit Breaker alerts, and asynchronous backend diagnostic jobs.

Accomplishments that we're proud of

  • The Circuit Breaker: Successfully implementing an automated guardrail that actively protects downstream data assets based on DataHub lineage metadata.
  • Production-Grade UI/UX: A highly polished web application that feels responsive, intuitive, and enterprise-ready.
  • Deep Integration Vision: Building an end-to-end bi-directional pipeline executing real API calls across DataHub, GitHub, and OpenAI.

What we learned

Active metadata is immensely powerful. By treating DataHub not just as a search engine, but as an operational graph database for an AI agent, the possibilities for automating data engineering workflows are virtually limitless.

What's next for DataHub Guard AI: Autonomous Incident Triage Engine

  • Automated Rollback Deployments: Automatic rollbacks for corrupted schema migrations.
  • Cost Anomaly Detection: Deep integration with Snowflake query history for real-time cost anomaly detection.
  • Voice-Activated Ops: Voice-command capabilities for on-call engineers resolving pipeline outages.

Built With

Share this project:

Updates