DataGuardian AI - Your Autonomous Data Incident Commander

An autonomous AI incident commander that detects, investigates, and fixes data pipeline failures end-to-end — opening the PR, notifying Slack, and logging every step, without waiting for a human.

Comment

Inspiration Every data team has the same Friday-night story: a critical ETL pipeline fails, the pager goes off, and an on-call engineer spends hours reading logs, tracing lineage, and squinting at stack traces — only to discover the fix was a one-line schema change. We realized most incident time isn't spent fixing; it's spent on grunt work an AI could handle: gathering metadata, checking recent migrations, testing hypotheses. So we set out to build the incident response squad every data engineer has dreamed of at 2 AM — one that never sleeps, never pings "anyone seen this before?" in Slack, and fixes the pipeline before the coffee kicks in.

What it does DataGuardian AI is an autonomous data incident commander. You upload a "broken table" — a simulated schema corruption or pipeline failure — and it dispatches a team of 11 specialized AI agents that detect, diagnose, fix, and verify the issue end to end:

Planner scopes the incident and builds a response plan Metadata scans catalog tables and recent migrations for the failure context Root Cause runs an LLM-driven hypothesis analysis against incident history Repair generates the candidate fix (typically a column_rename patch) Validator applies and tests the fix, counting pass/fail against our fix database Docs & GitHub agents write the documentation and open a real pull request with the diff, description, and test report Slack, Discord, Email, DataHub, Notifier keep humans in the loop in real time Every incident gets a full audit trail: agent timeline, code diff, notifications, and measured MTTR — all visible on a live dashboard that streams progress as the agents work.

How we built it The core is a LangGraph orchestrator — a linear state machine with a strict handoff protocol:

$$planner \rightarrow metadata \rightarrow root_cause \rightarrow repair \rightarrow validator \rightarrow docs \rightarrow github \rightarrow slack \rightarrow email \rightarrow datahub \rightarrow notifier$$

Each agent writes structured output to shared state; the next agent consumes and builds on it. Around that core: FastAPI on Vercel, Prisma Postgres for incident/run/step storage, a Next.js dashboard with Clerk auth (per-user incident isolation), and real integrations — GitHub PRs via API, Slack/Discord webhooks, Resend email, and DataHub metadata.

Challenges we ran into LangGraph's namespace collisions. Node names and state keys share a namespace, and our metadata and root_cause identifiers collided with framework keywords, producing cryptic validation errors. We fixed it with aggressive naming: metadata_data, root_cause_data, root_cause_agent. The invisible newline bug. Our first "successful" run through all 11 agents was marked failed with "Invalid non-printable ASCII character in URL" — a trailing \n in SLACK_CHANNEL_ID interpolated into a Slack URL. We now strip whitespace from every env var at load time. Production integration reality. GitHub tokens with read-only scopes returning silent 403s, Resend test mode refusing to email anyone but the owner's verified address, and asyncpg rejecting Prisma's sslmode=require — all fixed, one unglamorous bug at a time. Live status confusion. The timeline initially showed agents as "awaiting" even after completion — solved with per-step status mapping and live auto-refresh polling. Accomplishments that we're proud of From broken table to merged-ready PR in ~10 seconds. The GitHub agent opened a real PR (guardian-ai-fix/pull/1) with a branch named after the incident, four files, and a test report — fully automated. 11/11 agents completing in production on Vercel serverless, with MTTR: $$MTTR = \frac{1}{N}\sum_{i=1}^{N}(t_{fix} - t_{detect})_i$$

A production-grade dashboard with live agent timelines, code diffs, per-user incidents, and real time-ago timestamps. What we learned Orchestration is the hard 80%. The LLM agents are the easy 20%; the state machine that coordinates, failures, and streams progress is the real engineering. Observability wins. Streaming agent state and measuring MTTR turned a demo into a credible product. Environment hygiene is a debugging superpower. Strict env validation would have caught the newline bug in minutes, not hours. Works in dev ≠ works in production. Every "small" integration detail — token scopes, TLS flags, email verification modes — became a real incident. What's next for DataGuardian AI - Your Autonomous Data Incident Commander Real-time streaming of agent reasoning (not just status) to the dashboard Multi-incident parallelism and prioritization — an actual commander that triages multiple failures at once Auto-merge policies with rollback safety, letting teams configure how autonomous the "autonomous" gets Deep database integrations (Snowflake, BigQuery, Databricks) so repairs apply to real data warehouses Human-in-the-loop approval gates per agent, for teams that want the AI to propose — and humans to dispose Post-incident reviews generated automatically, turning every MTTR number into a learning signal

Built With

Share this project:

Updates