Chat Widget and Response from Orchestrator
Architecture Diagram
Configure Claude/OpenAI Key and Cusotm Skills
Flow Diagram

DataHub Nexus

Inspiration

DataHub already holds the metadata teams need to make better calls on engineering, governance, and privacy: schemas, ownership, lineage, classifications, freshness, and usage signals. At Atlassian that footprint is huge. We have over 5 million datasets cataloged in DataHub across many platforms, each one carrying real context about sensitivity, org-level ownership, and classification. The problem is that most of this intelligence stays locked behind dashboards, APIs, and one-off scripts. To actually use it, engineers and privacy folks have to step out of their workflow, learn DataHub's data model, and stitch the context together by hand.

So we asked a simple question. What if DataHub wasn't just a catalog, but an AI-native copilot? Instead of clicking through metadata, people should be able to ask a question in plain English and let an agent that already understands DataHub do the legwork.

What it does

DataHub Nexus brings a conversational agent right into DataHub. You can ask questions, dig into metadata, check governance signals, and kick off agentic workflows without leaving the UI.

Conversational DataHub access Ask plain-English questions about datasets, ownership, lineage, schema, freshness, and downstream impact, and get streaming answers inside the DataHub UI.
Multi-provider reasoning Send the heavy reasoning to a remote LLM (Anthropic Claude or OpenAI) while DataHub stays the trusted source of metadata. API keys live encrypted in the backend and never touch the browser.
Custom skills Users write their own named skills (custom system prompts) and pick one per conversation, so they can shape the agent's behavior without writing code.
Integration with the DataHub Action Framework The agent hooks into DataHub's Action Framework, so it can respond to configured actions rather than just answering questions. When a metadata event fires (a new dataset lands, a tag changes, an owner is added), the agent can pick it up and act on it, running the right skill and calling the matching DataHub tools. This turns Nexus from a request-response chatbot into something that reacts to what's happening in the catalog, whether that's auto-tagging a new dataset, flagging a governance gap, or triggering a downstream follow-up.
MCP-based metadata access We use the Model Context Protocol to expose DataHub metadata to the agent runtime in a structured, extensible way.
Agent orchestration A dedicated orchestrator runs the agent loop. It tracks conversation state, picks the right prompt, and decides which DataHub tools to call.
Conversation memory Context carries across turns using a "recent plus summary" approach (recent messages kept as-is, older ones summarized) with an idle timeout, backed by MySQL.

How we built it

We built DataHub Nexus as a thin agentic layer on top of DataHub, pulling together a native UI entry point, an orchestration service, MCP tooling, and remote LLM calls.

Native DataHub UI We added a chat entry point straight into the DataHub React frontend as a floating assistant, with an in-header model and skill picker.
Agent orchestrator A FastAPI service powers the chat. It takes the user's prompt, runs the streaming agent loop, manages session state, resolves the active skill, and decides which tools or agents to invoke.
MCP integration The orchestrator talks to DataHub through MCP-compatible tools for metadata search, schema lookup, lineage exploration, ownership lookup, and governance-signal retrieval.
Encrypted key and config storage We reused DataHub's (GMS) secret store, so provider keys stay encrypted at rest and get resolved just in time at chat.
Skills registry Custom skills sit in their own store so they can be created, viewed, selected, and reused across conversations.
Dockerized setup We packaged DataHub, the orchestrator, and the MCP server into one Docker Compose setup for easy local and demo runs.

Challenges we ran into

No enterprise knowledge graph to lean on We grounded the agent in DataHub metadata, but we kept bumping into questions that need context(enterprise/org context) beyond the catalog, like who owns a service, which team runs a project, or what doc explains a decision. Without a Rovo-style hook into a broader enterprise knowledge graph, the agent still sends users back out to other tools for that context, which breaks the "do it all inside DataHub" experience we were aiming for.
No transactions for long, multi-step workflows This is still an open design problem. When the agent runs a longer workflow that touches several things in a row, like tagging a batch of columns, adding owners, and then writing a governance annotation, there's no transaction wrapping those operations. Each tool call commits on its own, so if the agent (or a downstream API) fails halfway through, we can end up in a partial state: some columns tagged, others not, with no clean rollback. For steps that really need to happen together as a unit, we don't yet have atomicity or a compensating "undo" path, which makes multi-operation actions harder to trust and to retry safely.

Accomplishments that we're proud of

Remote agent invocation We took this past a basic chatbot by securely handing deeper governance and privacy reasoning to a remote LLM while DataHub stays the trusted source.
End-to-end agentic flow We showed a prompt travel from the DataHub UI to the orchestrator, out to DataHub metadata tools over MCP, into the remote reasoning agent, and back as a grounded, useful answer.
Integration with the DataHub Action Framework We got the agent working inside DataHub's Action Framework, so it isn't just answering questions on demand, it can react to what's happening in the catalog. When a metadata event fires (a dataset lands, a tag changes, an owner gets added), the agent can pick it up, run the right skill, and call the matching DataHub tools without anyone prompting it. That turned Nexus from a chatbot into something that can quietly do work in the background, like auto-tagging a new dataset or flagging a governance gap the moment it appears.
User-configurable skills People can define and switch skills on the fly, which makes the same agent useful for very different tasks.
Extensible architecture The design leaves plenty of room for future skills beyond the first batch, like data-quality debugging, lineage impact analysis, and governance-aware automation.

What we learned

Tools beat prompts for reliability Grounding the agent in real MCP tool calls gave far more trustworthy answers than prompt-only tricks.
The contract between UI, orchestrator, and DataHub matters most Keeping a clean split (browser to orchestrator for chat, orchestrator to GMS for secrets and metadata) kept things secure and easy to evolve.
Secrets belong in the platform's secret store Reusing DataHub's encryption and secret APIs was simpler and safer than rolling our own key handling.
Small memory heuristics go a long way A plain "recent plus summary" policy gave us coherent multi-turn conversations without runaway token cost.

What's next for DataHub Nexus: Agent to Agent

Configurable agent runtime Let teams plug in Rovo or other enterprise agent runtimes.
More reusable skills Starting with privacy risk scoring, data-quality debugging, lineage impact analysis, and governance-aware pipeline generation.
Agent-to-agent collaboration Explore ACP compatibility so specialized enterprise agents can work together.
Permissions-aware actions Tighten authentication and authorization so agent actions respect user permissions and enterprise access controls.
Fold Nexus into the Agent Context Kit DataHub already ships the Agent Context Kit, a set of guides, SDKs, and an MCP server for building AI agents grounded in DataHub context (ownership, lineage, glossary terms, quality signals, sample queries, and more). What it doesn't have yet is a first-party conversational agent that lives inside DataHub, and that's exactly what Nexus is. We see Nexus becoming a core component of the Agent Context Kit: the native, in-product experience that sits on top of the same MCP tools and SDKs the Kit already exposes, so instead of every team wiring up their own LangChain or Google ADK agent from scratch, they get a ready-made chat surface, skills, and orchestration out of the box, with the Kit's SDKs still there for anyone who wants to go custom.