DataGuardian AI

Inspiration

What it does

Inspiration

Modern data catalogs like DataHub contain critical enterprise metadata, but querying schemas, updating column documentation, or evaluating downstream risk often requires writing complex GraphQL queries, navigating dense web interfaces, or digging through manual mapping tables.

We noticed two major pain points in everyday data governance workflows:

High friction for non-engineers: Stakeholders speak natural language (or mixed phrasing like Hinglish/Hindi), whereas metadata tools expect rigid identifier names and uppercase database schemas.
Unsafe mutations: Modifying catalog descriptions or altering column definitions without checking blast radius leads to broken Looker dashboards, failing dbt models, and broken production pipelines.

We built DataGuardian AI to act as a governance-safe copilot — allowing data teams to talk to DataHub the way they speak to a colleague, with strict safety controls, semantic matching, and blast-radius safeguards.

What it does

DataGuardian AI is an LLM-driven, multi-agent data governance assistant built on top of DataHub GMS. Key features include:

🌐 True Multilingual & Mixed-Language Support: Understands complex queries in English, Hinglish ("dim_customers table ke customer_email field ka description update karo"), and Devanagari Hindi using native LLM reasoning instead of translation rules.
🎯 Semantic Entity Resolution: Resolves shorthand names, typos, and casing variations (customer_email $\rightarrow$ CUSTOMER_EMAIL) against live DataHub schemas using vector embeddings rather than static alias tables.
🛡️ Confidence-Gated Mutations: If an intent or target entity is ambiguous, the system asks a clarifying question rather than executing a blind update.
💥 Blast-Radius & Lineage Impact Analysis: Calculates risk scores and identifies downstream dependencies (Looker dashboards, ML models, dbt models) before a column is dropped or changed.
⚡ Grounded Code Generation: Generates production-ready dbt staging models strictly grounded in verified live schema data.
🔁 Self-Healing & Verification: Retries transient DataHub network failures automatically and re-queries DataHub after every mutation to verify persistence before reporting success.

How we built it

DataGuardian AI is built as a layered, multi-agent state machine rather than a monolithic prompt:

Agent Orchestration: LangGraph coordinates dedicated single-responsibility nodes (Supervisor Router, Intent Agent, Entity Resolution Agent, Action Agent, Lineage/Impact Agent, Codegen Agent, and Verification Agent).
LLM & Semantic Search: Google Gemini (gemini-2.5-flash / gemini-1.5-flash) for structured intent extraction and Gemini Embeddings (text-embedding-004) for vector similarity matching against schema candidate fields.
Backend Framework: FastAPI (Python 3.11) with a custom repository layer for DataHub GMS GraphQL calls.
Catalog Core: DataHub GMS (Docker containerized instance).
Frontend: React (Vite) + Tailwind CSS for a real-time governance chat experience.

Challenges we ran into

Table vs. Column Entity Collisions: When users asked to "update description of dim_customers", early iterations extracted dim_customers as both the table and the column, causing GraphQL mutation failures. We solved this by building strict sanity guardrails in our entity resolution layer to enforce dataset-level updates when target names collide.
LLM Quota & Rate Limits: Hitting Google Gemini free-tier rate limits (429 RESOURCE_EXHAUSTED) previously crashed backend HTTP requests. We introduced a deterministic regex-based fallback inside intent_agent so the app degrades gracefully without throwing unhandled exceptions.
DataHub GraphQL Casing & Schema Alignment: DataHub stores warehouse column names in uppercase (CUSTOMER_EMAIL), whereas users enter lowercase shorthand. Embedding similarity combined with live schema inspection solved exact casing mapping before mutations land.

Accomplishments that we're proud of

Zero Unhandled Exceptions: Mapped all GraphQL, network, and validation errors into a classified, actionable error taxonomy instead of generic "Unknown Error" messages.
End-to-End Post-Verification Loop: Successfully implemented post-mutation re-reading so the UI only confirms an update once DataHub GMS confirms the write persisted.
True Multilingual Extraction: Achieved reliable structured entity extraction across English, Hinglish, and Devanagari Hindi without losing technical field accuracy.

What we learned

How to design modular LangGraph workflows with typed Pydantic state contracts between agents.
Deep integration with DataHub GMS GraphQL APIs, schema aspect structures (schemaMetadata), and metadata aspects.
How to combine vector similarity search with LLM extraction to eliminate hardcoded lookup tables in AI agents.

What's next for DataGuardian AI

Slack & Microsoft Teams Integration: Allow data stewards to approve or deny proposed metadata updates directly from Slack notifications.
Automated Data Quality Policy Checks: Integrate with Great Expectations / Soda Core to flag schema drift before executing mutations.
Interactive Lineage Graphs: Render interactive visual lineage graphs directly inside the chat UI during impact analysis queries. ## How we built it