DataHub Buddy

Inspiration

Data engineers spend too much time searching through catalogs to find the right datasets, understand schemas, or trace lineage. We wanted to build an AI assistant that could answer natural language questions directly from real metadata in DataHub, eliminating the manual lookup and making data discovery instant.

What it does

DataHub Buddy is a chat-based AI agent that answers questions about datasets, columns, descriptions, ownership, and lineage by querying a live DataHub instance. You can ask things like “Show columns for fct_orders”, “List all Snowflake datasets”, or “View lineage of fct_orders” – and it fetches the actual metadata from DataHub’s REST API, displaying it in formatted markdown.

How we built it

Backend agent: We wrote a custom system prompt that instructs the LLM (via the Lovable AI gateway) to output structured JSON decisions – either a REST call to DataHub or a plain text reply. The API route /api/chat executes those REST calls against a local DataHub instance exposed through ngrok.

Metadata parsing: We built a parser in TypeScript that consumes DataHub’s entity response (DatasetSnapshot) and extracts schema fields, descriptions, lineage, and ownership.

Frontend: The UI was scaffolded using Lovable (TanStack Start, React, Tailwind, shadcn/ui) and then heavily customized – we replaced the entire colour palette with DataHub’s official blue and green, added a connection health badge, rewrote the empty state and error messages, and swapped all AI‑generated logos for the real DataHub logo.

Health monitoring: A dedicated health endpoint (/api/health/datahub) and a ConnectionBadge component give real‑time feedback on the tunnel status.

Environment: DataHub runs locally via Docker Quickstart, with a few datasets ingested directly via the REST API. ngrok provides the public tunnel for the deployed app.

Challenges we ran into

The biggest hurdle was authenticating API calls to DataHub from outside the Docker network. The Quickstart disables token generation by default, and the GraphQL endpoint returned empty results even with correct credentials. After extensive debugging, we switched to the REST API and used Basic authentication (datahub:datahub) combined with the ngrok‑skip‑browser‑warning header to bypass ngrok’s interstitial page. Another challenge was parsing DataHub’s complex nested response (DatasetSnapshot with multiple aspects) and presenting it cleanly in a chat UI. We had to carefully map aspect types to readable markdown, especially for schema fields and lineage relationships.

Accomplishments that we're proud of

Fully functional agent that reads real metadata from a running DataHub instance – no mock data.

Polished, professional‑looking UI that doesn’t feel like a generic AI chat – custom branding, DataHub colours, and a live connection indicator.

Resilient error handling: the app gracefully shows “Could not reach DataHub” when the tunnel drops, instead of crashing.

Successfully navigating the complex local‑deployment stack (Docker, ngrok, auth) and documenting it clearly in the README.

What we learned

How DataHub’s REST and GraphQL APIs work under the hood, including the snapshot model and aspect‑based metadata.

The importance of testing with real infrastructure early – many issues (like auth and ngrok warnings) only surfaced with live endpoints.

How to design a system prompt that reliably outputs structured JSON for tool use, and how to gracefully fall back when the AI returns an unexpected response.

What's next for DataHub Buddy

Support dynamic dataset discovery via the /browse endpoint instead of hardcoded URNs.

Integrate DataHub’s MCP server for richer, real‑time context.

Add write capabilities – let users add tags, ownership, or documentation through natural language.

Package as a standalone Docker image that companies can deploy alongside their own DataHub instance.

Add voice input and multimodal support for dashboard screenshots or error logs.

Built With

• ai-agent
• chatbot
• data-catalog
• datahub
• docker
• lineage
• lovable
• metadata
• ngrok
• opensource
• react
• rest-api
• shadcn/ui
• tailwind-css
• tanstack-start
• typescript