AIDataGate — DataHub-aware Operations Copilot

Inspiration

AIDataGate is a lightweight agent backend that queries DataHub metadata, builds lineage-aware recommendations, and renders concise, role-personalized guidance (ownership, impact, quality). The backend is Python + FastAPI with a caching DataHub client, recommendation engine, ownership service, rendering service, audit logging, and feedback capture.

What it does

• Retrieves dataset metadata and 2‑hop lineage from DataHub
• Analyzes ownership, staleness, quality and downstream impact
• Generates prioritized recommendations with confidence scores
• Renders HTML summaries personalized by role
• Supports feedback submission and audit logging

How we built it

• Backend: Python + FastAPI
• Data stores: PostgreSQL (SQLAlchemy), Redis cache
• DataHub integration: async DataHub client with retry, TTL cache, lineage traversal
• Core services: RecommendationEngine, OwnershipService, RenderingService, AuditService, Feedback endpoints
• Observability: OpenTelemetry + Prometheus, structured logging
• Delivery: Docker + docker-compose, tests with pytest

Challenges we ran into

• Resilient DataHub integration during outages → implemented stale-cache fallback
• Balancing concise human-readable recommendations with metadata completeness → iterative confidence scoring
• Building explainable lineage traversal while preventing cycles → depth-limited traversal (2 hops)

Accomplishments that we're proud of

• End-to-end agent workflow (DataHub → recommendation → render) with tracing
• Robust caching + stale fallback and retry logic
• Full feedback capture API linked to requests
• Comprehensive unit tests for core services (coverage reports available)

What we learned

• Metadata grounding produces far more actionable guidance than LLM-only answers
• Small, focused workflows (impact + ownership) make a stronger demo

What's next for AIDataGate — DataHub-aware Operations Copilot

• Add a minimal frontend demo and hosted example
• Add a small open-source contribution to DataHub Skills for easier agent-context reuse
• Short automated demo script and a 3-minute video

Built With

• datahub-oss
• docker
• fastapi
• jinja
• mcp-server-/-agent-context-kit
• opentelemetry
• postgresql
• prometheus
• python
• redis
• sqlalchemy