-
-
DataHub Copilot Live Interactive Chat Demo
Inspiration## Inspiration
Data analysts and engineers spend countless hours writing complex SQL queries and Airflow DAGs. They often face errors and hallucinations due to unknown schema definitions or outdated table lineages. We wanted to build an autonomous agent that reads metadata directly from DataHub to eliminate hallucinated queries.
What it does
DataHub Copilot is a metadata-aware AI agent. It connects to DataHub's context platform to extract table schemas, column data types, and dependencies. When a user asks a natural language question, the agent uses this context to automatically generate zero-hallucination SQL queries and production-ready data pipelines.
How we built it
- Metadata Context: Integrated with DataHub MCP Server and Python SDKs.
- AI Logic: Built with Python to translate prompts into context-informed SQL/DAG code.
- Documentation & Web Interface: Hosted interactive documentation via GitHub Pages.
Challenges we ran into
Integrating deep schema context into prompt structures without hitting context limits while keeping query generation fast and precise.
Accomplishments that we're proud of
Successfully generating 100% schema-valid SQL code using dynamic DataHub metadata lookups.
What's next for DataHub Copilot
Adding support for automated column-level lineage tracking and direct integration into Slack/Teams bots for enterprise data teams.
Log in or sign up for Devpost to join the conversation.