DataGuardian AI

What Inspired Us Modern data ecosystems are incredibly complex. Data engineers, ML engineers, and governance teams spend countless hours answering variations of the same painful questions: "Why is this dashboard broken?", "Who changed the schema upstream?", or "What happens if I drop this column?"

Current tools are great at throwing alerts, but terrible at investigating them. We were inspired by the concept of Site Reliability Engineering (SRE) in software development and realized the data world desperately needed the same. We set out to build DataGuardian AI—not just another alerting tool, but an autonomous, AI-powered Data Reliability Engineer that proactively investigates incidents, determines root causes, and recommends remediations, all grounded in enterprise metadata.

How We Built It DataGuardian AI is built on a modular, event-driven architecture designed around a core principle: Deterministic Before AI. We wanted the reasoning power of Large Language Models, but the reliability of deterministic code.

Here is the tech stack and architecture we used:

The Brain (Orchestration): We used LangGraph to build a multi-agent orchestration layer. Instead of one massive prompt, we designed specialized agents with single responsibilities (e.g., Metadata Agent, Schema Drift Agent, Root Cause Agent, Blast Radius Agent). The Engine (Backend): Built with FastAPI (Python) for high-performance async processing, backed by PostgreSQL for workflow persistence. The Ground Truth (Metadata): We deeply integrated with DataHub via MCP (Model Context Protocol). Every AI decision originates from verified metadata retrieved from DataHub, eliminating hallucinations. The LLM Gateway: We built an internal Enterprise LLM Gateway to interface with Gemini. It handles prompt construction, context assembly, tool registries, and strict structured output validation. The Interface (Frontend): A responsive Next.js dashboard where data teams can observe agent timelines, explore lineage, and approve high-risk remediations (Human-in-the-Loop). The Challenges We Faced Taming LLM Hallucinations: Early on, the AI would occasionally invent upstream dependencies that didn't exist. We solved this by enforcing a strict "Metadata First" rule. The LLM is never allowed to guess; it must query the DataHub API first. Multi-Agent Coordination: Managing state across a dozen autonomous agents is chaotic. An agent investigating a schema drift could easily get stuck in a loop. Implementing LangGraph allowed us to define strict state transitions, cyclic graphs with exit conditions, and deterministic routing. Balancing Confidence and Automation: We had to figure out how to quantify the AI's confidence in its root-cause analysis before triggering automated alerts in Slack or Jira. To solve this, we implemented a weighted confidence scoring model within our Validation Agent. The final confidence score $\mathcal{C}$ for a proposed root cause is calculated as a weighted sum of metadata evidence, lineage proximity, and LLM probability:

$$ \mathcal{C} = \alpha \sum_{i=1}^{n} M_i + \beta \left( \frac{1}{d_{lineage}} \right) + \gamma P_{LLM} $$

Where:

$M_i$ represents deterministic metadata validation checks. $d_{lineage}$ is the graph distance between the incident and the suspected root cause. $P_{LLM}$ is the structured output probability from the LLM. $\alpha, \beta, \gamma$ are tunable weights (where $\alpha + \beta + \gamma = 1$). Actions are only taken autonomously if $\mathcal{C} > \tau$ (our safety threshold).

What We Learned Context is Everything: LLMs are incredibly capable at data engineering tasks, provided you give them the right context. Assembling the context (System Prompt + Agent Prompt + Workflow Context + DataHub Metadata + Lineage) is more important than the model itself. Observability is Non-Negotiable: When an AI is making decisions about your data infrastructure, you need to know exactly why. We learned that logging every LLM call, token count, and metadata query is critical for trust. AI Should Assist, Not Replace: The most powerful workflow isn't fully autonomous; it's the "Human-in-the-loop" pattern. DataGuardian does the heavy lifting of investigation, but presents the engineer with a clearly explained root cause and a generated SQL rollback script for final approval.

What Next in DataGuardian Version 1.5 — Intelligence Expansion The immediate next step is focused on making the AI smarter, more efficient, and easier for teams to collaborate with.

Smarter AI Workflows: Implementation of workflow templates, intelligent retry strategies (when an agent gets stuck), and better context compression to save on token costs. Enhanced Team Collaboration: Adding custom notification rules and better ways for teams to interact with incident histories. Advanced Analytics: Building workflow analytics and advanced dashboard metrics to track how much time the AI is saving the engineering team. Performance: Enhanced caching for faster execution and improved scalability for the backend. Version 2.0 — Autonomous Operations This is the major leap from AI-assisted investigation (suggesting fixes) to AI-assisted operations (executing complex fixes safely).

True Autonomy: The system will move towards autonomous workflow planning and multi-step remediation recommendations (e.g., automatically rolling back a bad pipeline deployment, restoring the data, and notifying stakeholders). Predictive Capabilities: Moving from reactive to proactive with intelligent anomaly detection and predictive incident analysis (catching issues before downstream users notice). Semantic Search: Implementing semantic metadata search so users can chat naturally with their data catalog. Executive Reporting: High-level AI-generated reporting for engineering leadership. 🔌 Expanded Enterprise Integrations As DataGuardian grows, it plans to deeply integrate beyond DataHub, GitHub, and Slack to include:

Data Platforms: Snowflake, BigQuery, Databricks, and Apache Kafka. Orchestrators: Apache Airflow and dbt. Operations & Ticketing: ServiceNow, PagerDuty, and Microsoft Teams. Enterprise Features To support large-scale organizations, the roadmap includes robust security and management features:

Single Sign-On (SSO) and Multi-Factor Authentication (MFA). Fine-grained Role-Based Access Control (RBAC). Team workspaces and Tenant isolation. Compliance reporting and data retention policies. Long-Term Vision & Research Further down the line, the team is exploring advanced AI techniques such as:

Graph Neural Networks (GNNs) for highly advanced lineage analysis. Reinforcement Learning to optimize agent workflows based on past successes and failures. Vector-based Semantic Memory so the AI retains long-term context about your organization's specific data quirks and past incidents.