🚀 Inspiration
Data downtime is a silent killer in modern data stacks. As data engineers, we’ve felt the panic when a critical dashboard breaks because a source column went null, or when a dbt model fails in production due to unexpected schema changes. While DataHub excels at centralizing metadata and lineage, there is no out-of-the-box "pre-flight check" for datasets before they hit production. We built DataGuardian to turn DataHub from a passive catalog into an active, metadata-aware reliability agent.
⚙️ What it does
DataGuardian integrates directly with DataHub to automate production-readiness checks. It allows users to:
- Browse & Select any dataset indexed in DataHub.
- Scan for Risks using deterministic rules (nullability, volume anomalies, field drift, lineage depth).
- Analyze Impact by traversing DataHub’s lineage graph to calculate downstream dependencies.
- Generate Remediation Artifacts – outputting executable SQL cleanups, dbt model patches, markdown reports, and metadata change proposals.
- Manage Workflows via a built-in approval UI (Approve/Reject/Apply).
🛠️ How we built it
We adopted a modern, decoupled architecture:
- Frontend: Next.js 14 with TypeScript and Tailwind CSS for a reactive, clean UI.
- Backend: FastAPI (Python 3.12) with Pydantic v2 for strict data validation and async scanning pipelines.
- Metadata Source: Direct integration with DataHub GMS REST API and DataHub MCP server for fetching schemas, ownership, and full lineage.
- Storage: SQLite for rapid prototyping, with production-ready migration paths to PostgreSQL.
- AI Integration: We added an optional Anthropic Claude adapter that translates risk signals into human-readable explanations and actionable code snippets.
- Deployment: Designed for Vercel (frontend) and containerized backend deployment (Render/Fly.io).
🔥 Challenges we ran into
- Pagination Hell: Traversing DataHub’s lineage graph for deeply nested downstream dependencies required careful cursor management and caching to avoid API throttling.
- Demo Mode Fidelity: Simulating a realistic DataHub environment without an actual connection was tricky. We built a seed data generator that mimics real-world metadata, complete with nested schemas and cyclic lineage.
- LLM Output Structuring: Getting Claude to output syntactically correct SQL and dbt YAML on the first try required iterative prompt engineering and strict JSON schema constraints.
- State Synchronization: Keeping the frontend approval state consistent with the backend scan history in a stateless environment led us to implement a lightweight SQLite-based session store.
🧠 What we learned
- Deep dive into DataHub’s metadata model (aspects, snapshots, and relationships).
- Balancing real-time data fetching with precomputed scan results to maintain UI responsiveness.
- Designing deterministic risk rules that are generic enough to work across industries but configurable for specific business logic.
- How to structure an "Agent" workflow – where the system doesn't just report problems but actively proposes tangible fixes.
🏆 Accomplishments we're proud of
- A fully functional, end-to-end reliability agent that cuts manual data quality checks from days to minutes.
- Seamless integration with DataHub without requiring any modifications to the existing DataHub deployment.
- The ability to generate production-ready remediation scripts automatically.
🚧 What's next for DataGuardian
- Multi-Source Support: Expand to OpenMetadata and Amundsen.
- Time-Series Anomaly Detection: Leverage historical metadata trends to flag gradual quality degradation.
- CI/CD Gatekeeper: Integrate with GitHub Actions to block PRs if a critical dataset fails the readiness scan.
- Custom Rule Marketplace: Allow the community to share and import rule packs.
Log in or sign up for Devpost to join the conversation.