DataHub Navigator

Ask anything about your data stack. Watch the AI agent think.

Comment

Inspiration

Data engineers spend hours answering the same questions: What feeds this model? What breaks if I rename this column? Which tables contain PII? The answers exist in DataHub but getting them requires manual graph traversal, tribal knowledge, and time nobody has.

I wanted to build something that made the invisible visible: an agent that doesn't just return an answer, but shows its work in real time.

What it does

DataHub Navigator lets you ask questions about your data stack in plain English. An AI agent reads your DataHub entity graph via the official MCP Server, plans a traversal, then animates through your lineage graph in real time, nodes light up, edges trace, the camera follows the reasoning step by step.

The animation IS the thinking made visible.

How I built it

  • Graph layer: Connects to DataHub via the Model Context Protocol (@modelcontextprotocol/sdk) over stdio. Three tools: datahub_get_lineage (BFS), datahub_search_entities, datahub_get_entity. Falls back to 15-node sample graph when DataHub isn't configured.
  • AI reasoning: Query + graph sent to Ollama (Llama 3). Returns structured JSON: ordered node IDs, per-step narrative, final answer.
  • Animated traversal: useTraversal hook steps through the plan at 600ms/node. Node status updates (idle → active → dimmed), edges glow and flow, camera auto-follows with React Flow fitView.
  • Live/demo badge: Shows ● DataHub MCP when connected to a real instance, ● Demo Mode when using sample data.

Challenges

The hardest problem was making the agent's output trustworthy. Early versions hallucinated node IDs that didn't exist in the graph. I fixed this by injecting the full node list into the prompt and adding strict output validation, if a returned node ID doesn't exist in GRAPH_NODES, the traversal silently skips it rather than breaking.

The second challenge was latency. A full graph traversal query + Ollama inference + streaming response had to feel responsive. Progressive streaming and optimistic UI updates (showing "Traversing your data stack..." immediately) solved the perceived latency problem.

What I learned

MCP is underrated for internal tooling. The stdio transport pattern, spawning a subprocess and communicating via JSON-RPC — is surprisingly clean and composable. Building a mock MCP server first let me develop and test the entire client layer without needing a real DataHub instance.

What's next

  • Replace Ollama with a cloud LLM so the deployed version is fully functional
  • Support real DataHub connections with authentication
  • Natural language filtering ("show me only ML models")
  • Slack/Teams integration for async lineage queries

Built With

Share this project:

Updates