Inspiration
Modern data stacks are growing faster than human operations teams can manage. While platforms like DataHub provide exceptional metadata cataloging, metadata remains largely passive—a static repository that engineers consult after a pipeline breaks or during an audit.
At the same time, the rise of LLMs and Text-to-SQL tools has introduced serious operational risks: models generated queries without context regarding data privacy, column-level security, or upstream pipeline health. We were inspired to invert this paradigm: What if metadata became an active, runtime operational primitive for AI agents?
We built Blueprint: Agentic Governance & Observability on DataHub to bridge the gap between AI autonomy and enterprise governance. By transforming DataHub from a static catalog into a dynamic toolset, we empowered autonomous agents to trace lineage, respect PII policies, perform zero-hallucination query generation, and triage pipeline incidents in real time.
What it does
- Context-Aware Text-to-SQL GenerationBefore an AI agent or engineer executes a query against a data warehouse, the system queries DataHub’s GraphQL API in real time to fetch schema definitions, column descriptions, and primary/foreign key relationships. This eliminates hallucinated column names and invalid joins.2. Deterministic PII Masking & Policy EnforcementWhen an LLM requests or constructs queries involving sensitive data, the blueprint inspects DataHub Business Glossary tags (such as PII, Confidential, or Restricted). If a protected column is detected, an interceptor automatically injects cryptographic hashing into the generated SQL:$$\text{MaskedColumn} = \text{SHA256}(\text{RawColumn} \mathbin{\Vert} \text{Salt})$$This guarantees regulatory compliance at generation time without relying on non-deterministic system prompts. ## How we built it The system architecture combines event-driven streaming, graph-based lineage traversal, and LLM tool calling into a closed-loop governance engine.Core Architecture ComponentsAgentic Tool Wrapping over GraphQL:We created asynchronous Python wrappers around DataHub’s GraphQL API (searchAcrossEntities, scrollLineage). This allows the LLM reasoning loop to dynamically query schema definitions, column descriptions, and upstream/downstream dependencies before generating code or diagnosing pipeline health.Deterministic PII Interception & Hashing:To prevent data leaks, the agent inspects DataHub Business Glossary tags (e.g., PII, Confidential). When a protected column is detected, the agent applies dynamic hashing directly in generated SQL outputs:$$\text{MaskedColumn} = \text{SHA256}(\text{RawColumn} \mathbin{\Vert} \text{Salt})$$This guarantees policy enforcement at generation time without relying on non-deterministic LLM prompts.Event-Driven Incident Triage:We subscribed an event listener to DataHub's Kafka Metadata Change Event (MCE) stream. When a dbt job fails or a schema drift occurs, the event triggers an automated diagnostic loop. The agent traverses upstream dependencies using a topological scoring algorithm to rank candidate root-cause nodes:$$S(u) = \alpha \cdot \text{FailureSeverity}(u) + \beta \cdot \text{DepthDistance}(u, v)^{-1}$$ ## Challenges we ran into Cyclic Lineage Traversal & Context Inflation: Traversing deeply nested or cyclic dbt DAGs via GraphQL caused query timeouts or generated massive JSON payloads that blew past LLM context limits. We resolved this by building a server-side client interceptor with explicit depth limits ($maxDepth = 2$) and cyclic node detection (visited_urns).Synchronous SDK Blocking in Async Loops: Modern AI agent execution loops rely on asyncio. The standard acryl-datahub Python SDK emitter calls were synchronous, which initially blocked thread execution and delayed UI responses. Wrapping SDK calls in asyncio.to_thread restored background non-blocking execution.Payload Noise in Metadata Events: Raw Kafka MCE/MAE event payloads contain significant aspect overhead. Distilling these raw JSON streams into concise, high-signal event summaries for the LLM required writing custom middleware parsers. ## Accomplishments that we're proud of
- Inverting the Metadata Paradigm From Static Catalog to Active Primitive: Successfully transformed DataHub from a passive UI repository into an active, real-time runtime control plane for autonomous AI agents.
Context-Aware Reasoning: Proved that LLMs can query live platform metadata via GraphQL to inspect schema types, tags, and column descriptions before executing actions, preventing hallucinated or blind query generation.
- Zero-Trust & Deterministic PII Security Deterministic Masking Engine: Engineered a interception mechanism that reads DataHub Business Glossary tags (e.g., PII, Confidential) and automatically applies cryptographic SHA256 hashing to sensitive columns directly at generation time.
Eliminating Prompt Reliability Risks: Bypassed non-deterministic system prompts by using hard-coded governance interceptors to guarantee data protection compliance.
- Event-Driven & Autonomous Incident Triage Real-Time Kafka Event Listener: Subscribed an asynchronous agent worker directly to DataHub's MetadataChangeEvent (MCE) Kafka stream to detect pipeline failures and schema drifts instantly without database polling.
Topological Root-Cause Ranking: Built an automated diagnostic traversal algorithm using graph-distance and failure-severity scoring:
S(u)=α⋅FailureSeverity(u)+β⋅DepthDistance(u,v) −1
This generates structured root-cause reports identifying upstream dbt/Airflow breaking changes in seconds.
- Overcoming Technical Bottlenecks Non-Blocking Async Architecture: Solved thread-locking issues in the standard synchronous Python SDK by wrapping emitter socket operations in non-blocking thread execution routines (asyncio.to_thread), preserving low latency in streaming loops.
Server-Side Graph Pruning: Overcame recursive cyclic timeouts on complex ETL graphs by introducing in-memory cycle detection (visited_urns) and token-efficient depth limits (maxDepth=2).
What we learned
Metadata is the Ultimate Guardrail for Generative AI: Prompt engineering alone cannot guarantee regulatory compliance. Combining strongly typed metadata (glossary tags, access tiers) with hard-coded interceptors provides deterministic security for LLM output.
Event-Driven Architecture Unlocks Proactive AI: AI agents shouldn't wait for human prompts. By hooking directly into DataHub's real-time event topics, agents become proactive team members capable of detecting and triaging incidents before users notice downtime.
Token Efficiency Demands Graph Pruning: Raw graph dumps are expensive and slow for LLMs to process. Translating complex graph structures into compact, topological summaries is critical for maintaining fast and cost-effective agent reasoning loops.
What's next for Blueprint: Agentic Governance & Observability on DataHub
Blueprint: Agentic Governance & Observability on DataHub expands from a hackathon prototype into a enterprise-ready control plane that shifts data governance from reactive monitoring to autonomous, self-healing data operations.
The next phase of development focuses on three primary tracks: production hardening, deeper native platform integration, and multi-agent governance orchestration.
Log in or sign up for Devpost to join the conversation.