Inspiration
Modern data platforms contain enormous amounts of metadata, lineage, ownership, and governance information, but identifying what actually needs attention still requires significant manual investigation.
We wanted to build an agent that doesn't just observe a data catalog, but can understand it, investigate problems, explain their business impact, recommend remediation, and take action.
That led to DataGuardian AI — an AI-powered metadata reliability and governance agent built around DataHub.
What We Built
DataGuardian AI connects to DataHub's metadata graph and continuously evaluates datasets across multiple dimensions including reliability, governance, lineage, ownership, metadata completeness, and business impact.
Instead of presenting users with a list of disconnected metadata problems, DataGuardian turns them into actionable cases.
The system can:
- Discover datasets and metadata from DataHub
- Detect reliability and governance findings
- Assign severity based on evidence and business impact
- Trace lineage to understand downstream impact
- Investigate individual datasets and root causes
- Generate natural-language explanations
- Recommend remediation actions
- Present proposed changes for review
- Apply approved metadata changes back to DataHub
- Re-evaluate the catalog after remediation
- Calculate an organization-level health score
The core workflow is:
Discover → Detect → Investigate → Reason → Recommend → Review → Act → Re-evaluate
Why DataHub Matters
DataHub is the system of record for the metadata context DataGuardian reasons over.
The agent uses DataHub's datasets, metadata, ownership, tags, glossary terms, and lineage relationships as evidence rather than treating data quality as an isolated rules engine.
This allows DataGuardian to answer a more useful question:
"What is wrong, why does it matter, what will it affect, and what should we do about it?"
Multiple Data Environments
Our demonstration uses three datasets in DataHub:
- Healthcare
- Fiction Retail
- NYC Taxi
The health score dynamically changes as findings are introduced and remediation is applied, demonstrating that the system is evaluating the state of the metadata environment rather than displaying a static dashboard.
Agentic Workflow
DataGuardian is designed around multiple specialized reasoning responsibilities rather than a single chatbot.
The system combines:
- Reliability analysis
- Governance analysis
- Lineage and impact analysis
- Business-impact assessment
- Investigation and root-cause reasoning
- Remediation planning
- Metadata write-back
- Post-action re-evaluation
The result is an agent that can move from finding a problem to helping resolve it.
Technical Challenges
One of the biggest challenges was making the system useful with real DataHub metadata while keeping the application deterministic and reproducible for development and demonstration.
We implemented a DataHub integration layer that supports both live DataHub connectivity and a deterministic mock environment.
We also designed the remediation workflow around review-before-write behavior so that potentially impactful metadata changes are not silently applied without user visibility.
Another challenge was connecting lineage to business impact. A finding on one dataset can have very different consequences depending on which downstream assets depend on it. DataGuardian therefore considers lineage and asset context when investigating findings.
What We Learned
The biggest lesson was that an effective data agent needs more than an LLM.
The valuable part is the combination of:
metadata + lineage + deterministic evidence + reasoning + controlled actions.
DataHub provides the metadata graph that makes this possible.
DataGuardian AI demonstrates how an agent can use that graph to move beyond catalog search and toward autonomous data reliability and governance operations.
What's Next
Future versions can extend the system with deeper statistical data-quality profiling, additional DataHub integrations, stronger policy reasoning, richer business-impact models, scheduled monitoring, and more sophisticated remediation automation.
For this hackathon, our focus was on demonstrating the complete agent loop:
Observe → Reason → Explain → Act → Verify.
Log in or sign up for Devpost to join the conversation.