DataGuard AI: Autonomous Data Reliability Operations Center

An autonomous data operations center using LangGraph multi-agent AI and DataHub to trace root cause lineage, auto-heal broken pipeline schema configs, and alert dataset owners in real-time.

Comment

Inspiration

Modern enterprise operations run on data. However, as data stacks expand across thousands of tables, pipelines, and machine learning models, they become fragile. A silent schema change upstream or a broken pipeline transformation can instantly corrupt downstream metrics, misleading decision-makers and breaking AI models. Tracking down these failures manually is a painstaking process of mapping lineage, examining code, and pinging teammates on Slack.

We built DataGuard AI to solve this. We wanted to move data teams away from reactive "firefighting" and toward autonomous, self-healing data operations. By combining the unified metadata graph of DataHub with a multi-agent AI system, we created a tool that can not only identify where things break, but proactively diagnose why they broke, generate the code fixes, and execute them automatically.

What it does

DataGuard AI acts as a central, autonomous operations center for your entire data stack:

Interactive Lineage & Blast Radius Explorer: A beautiful, real-time visual representation of your data flow. You can see how tables in Snowflake flow into machine learning models in SageMaker and dashboards in Looker. Autonomous Self-Healing Loops: When a quality assertion fails on DataHub, DataGuard AI intercepts the event. The specialized agent network traces the error upstream, locates the root cause schema modification, writes the corrected SQL/dbt code to fix the pipeline, and deploys it. Proactive Change Simulator: Allows developers to test "what-if" scenarios (e.g., “What happens if I drop or rename this column?”) and preview the exact downstream blast radius before pushing changes to production. AI Agent Desk Chat: A conversational interface that allows any team member to ask natural language questions about schemas, data ownership, or pipeline statuses, which the agent resolves by querying DataHub's metadata graph.

How we built it

The system is built as a production-grade monorepo split into two components:

The Backend Engine (FastAPI + LangGraph): Orchestrated using a LangGraph State Graph consisting of 5 specialized agents: Orchestrator, Detective, Remediation, Documentation, and Communication. Integration with Fireworks AI (utilizing deepseek-v4-flash) for lightning-fast, highly accurate reasoning and code generation. Leverages the DataHub SDK and API layers to query lineage, parse schemas, inspect owners, and write back tags, descriptions, and operational incidents. The Frontend Operations Center (Next.js 14): Built as a premium, glassmorphic dark-mode dashboard. Utilizes React Flow for dynamic, interactive lineage graph rendering. Connects to the backend via REST APIs and WebSockets for real-time streaming of multi-agent execution steps.

Challenges we ran into

Python 3.14 Compatibility: Developing under a Python 3.14 pre-release environment presented significant compilation challenges for packages containing native C-extensions (like older versions of the DataHub SDK). We bypassed this by building a custom abstraction layer that interacts cleanly via REST APIs and GraphQL interfaces. LLM Tool Grounding: Ensuring that LLM agents stay grounded in metadata realities without hallucinating schema structures. We solved this by implementing strict input/output Pydantic schemas and feeding the agents contextual lineage data fetched directly from DataHub prior to code generation. Port Conflict & Tunneling: Exposing local Next.js and FastAPI servers securely to external devices for remote demo testing, which we resolved by implementing standard Serveo SSH tunnels with customized CORS middleware.

Accomplishments that we're proud of

Fully Functioning Self-Healing Loop: Successfully getting the Remediation Agent to parse a broken SQL query, consult DataHub's schema catalog, and return a working, syntactically correct fix. Visual Lineage Graph: Building a premium, interactive dependency map that visually flows data status changes (green, yellow, red) down the lineage nodes in real-time. Zero-Touch Setup: Constructing a dual-mode service layer that seamlessly falls back to realistic e-commerce mock metadata schemas, allowing judges to test the app instantly on Render without spinning up a heavy Docker cluster.

What we learned

Metadata is the ultimate context: AI agents are incredibly powerful, but they are helpless without structured, real-time context. Grounding agents in DataHub’s metadata graph turns standard chat models into highly precise system operators. Multi-agent design patterns: Coordinating multiple specialized, narrow agents yields far better code quality, lower token consumption, and fewer loops than relying on a single large prompt.

What's next for DataGuard AI: Autonomous Data Reliability Operations Center

Continuous Integration Integration (CI/CD): Embedding the Impact Simulator directly into GitHub actions, so pull requests are automatically commented on with their metadata blast radius before merge. Multi-Platform Execution Connectors: Adding connectors for Airflow, Prefect, and Fivetran to allow the Remediation Agent to automatically restart corrected pipeline tasks directly in target schedulers. LLM-Based Metadata Tag Suggestion: Scanning search queries and query logs to automatically suggest glossary terms and tags on DataHub for undocumented tables.

Built With

  • datahub
  • deepseek
  • fastapi
  • fireworks-ai
  • git
  • github
  • graphql
  • javascript
  • langchain
  • langgraph
  • next.js
  • openai-api
  • openssh
  • python
  • react
  • react-flow
  • render
  • rest-api
  • sdk
  • serveo
  • ssh
  • tailwindcss
  • uvicorn
  • web-sockets
Share this project:

Updates