Inspiration
Data teams constantly lose institutional knowledge. Every time someone joins a team or needs to understand an unfamiliar table, they either dig through DataHub's UI manually, ask a teammate on Slack or just guess. I wanted an agent that could do that lookup-and-explain work automatically, reading DataHub the way a helpful senior teammate would, then handing back a clear answer instead of raw metadata.
What it does
DataHub Documentation Assistant lets anyone type in a table name and get back a plain-English explanation of what that table is, where its data comes from and what depends on it. It queries a live DataHub instance for schema, description and full upstream and downstream lineage, then passes that context to an LLM to generate a clear explanation which turns raw catalog metadata into something a new analyst can actually use in seconds.
How I built it
• DataHub: self-hosted via Docker (datahub docker quickstart), ingested with DataHub's official sample dataset, queried live through its GraphQL API.
• Groq API: generates the natural-language explanation from the retrieved metadata.
• Streamlit: simple web interface, deployed on Streamlit Community Cloud.
• ngrok: tunnels the self-hosted DataHub instance so the deployed app can reach it publicly.
Challenges I ran into
Running DataHub in a cloud dev environment came with real infrastructure hurdles. Docker containers running out of disk space and crashing, GitHub Codespaces' port-forwarding rejecting external requests, and ngrok tunnel/port mismatches that took careful debugging to trace through container logs, port mappings and tunnel configs. Getting a self-hosted, publicly reachable DataHub instance took real persistence, but the result is a fully working, end-to-end pipeline from raw metadata to plain-English explanation.
What's next for Datahub Document Assistant
• Support write-back to DataHub. e.g. auto-suggesting descriptions for undocumented tables. • Multi-table context, so the agent can explain a whole pipeline, not just one table at a time. • Persistent hosting so the DataHub backend doesn't require a live Codespace session.
Log in or sign up for Devpost to join the conversation.