-
-
One broken table, three dashboards down across Looker, Tableau and Power BI, with the three catalog calls that found them.
-
The on-call question in plain English. The header line names the catalog and the model this session is wired to.
-
Start screen: what the agent does with a question, and three you can click.
-
Questions the catalog cannot answer are classified before it is touched, so they cost zero catalog calls.
Inspiration
Every data team has some version of the same bad morning. A table breaks, and before anyone can fix it somebody has to work out what else just broke with it. That means walking lineage across Snowflake, dbt and whatever BI tools the company bought, checking who owns each hop, and remembering which columns hold personal data. A data engineer who knows the estate can do it in twenty minutes. An agent without that context cannot do it at all, because it will invent a table name and confidently give you the wrong owner.
DataHub already holds the context. What I wanted was an agent that reads it, works the question through, and then puts the answer back into the catalog so the next person starts from something instead of nothing.
What it does
You ask it a question the way you would ask a colleague:
"If the raw orders table breaks tomorrow, which dashboards go down?"
It searches the catalog, walks lineage across platforms, reads schemas, owners and governance tags, and answers with the URNs it actually read. Under every answer there is an evidence panel listing the catalog calls it made, in order, with their arguments. That panel is the reason you can trust the answer, or catch it when it is thin.
If the answer is worth keeping, one click writes it back into DataHub as a document, titled and tagged, sitting there for whoever asks the same question next month.
It also declines things. Ask it what the weather is and it tells you in one line that it only knows what your catalog knows, without spending a search to find that out.
Everything runs on free infrastructure: DataHub Core in Docker on your own machine, Gemini's free tier as the model. No tenant, no trial clock, no bill. Worth being precise about what "local" means, though. The catalog stays on your machine, but the metadata the agent reads travels to Gemini inside the model conversation, the same as with any hosted model.
How I built it
The tool layer is DataHub's Agent Context Kit, and only that. One call to build_google_adk_tools(client) gives the agent eleven functions: search, get_entities, list_schema_fields, get_lineage, get_lineage_paths_between, get_dataset_queries, get_dataset_assertions, search_documents, grep_documents, get_me and save_document. I wrote none of them. Write access is then filtered down to save_document by an allowlist that names what is permitted rather than what is banned, so if a future release of the kit ships something destructive, it stays out by default.
The instructions come from DataHub Skills. skills_loader.py reads the SKILL.md files DataHub publishes in datahub-project/datahub-skills, installed with npx skills add and pinned by content hash in skills-lock.json, so what the agent follows is DataHub's text rather than my paraphrase of it. Five of them go into the system instruction: using-datahub, datahub-search, datahub-lineage, datahub-enrich and datahub-quality. One for each group of tools the agent actually holds.
Those skills were written for Claude Code, and this agent is a plain Python ADK process with no command palette, so two things get rewritten on the way in. Sections about tools it does not have, like the datahub CLI, are stripped out, because an agent told to run datahub search list-filters will try it and then have to recover. And references like Use `/datahub-enrich` get pointed at the workflow that is already in the instruction, since the slash command leads nowhere here.
The catalog itself is a local datahub docker quickstart loaded with the showcase-ecommerce datapack. The same code runs against a DataHub Cloud tenant if you change DATAHUB_GMS_URL.
Around all of that: Python, Google's Agent Development Kit, gemini-3.5-flash-lite, and Streamlit for the chat UI. There are 112 tests, run against a fake tool layer, so the answer and save loop is covered without a network or a model anywhere in the suite.
Challenges I ran into
The one I am least proud of is that I fixed the same bug three times. Somebody typed "who are you" into the running app, and the agent searched the catalog for that phrase, matched an unrelated GDPR glossary term, and reported back that the catalog held no entity describing who it was. I fixed it with a list of phrases. Then "which model are you using" missed the list, so the list became regular expressions. Then "what's the date" needed a second set of them. By the third patch it was obvious that the problem was never the phrasings. There was no layer deciding what kind of question had arrived, and there is no list of phrasings that ever gets finished.
The actual fix is a classifier that labels each message as a catalog question, a question about the app, or neither, before anything touches DataHub. It fails open: if it errors, or the free tier throttles it, the question goes to the catalog exactly as it would have before. The worst a broken classifier can do is waste a search. It can never refuse a real question.
Then there was the write-back, which took 61 seconds to make a single tool call. All of that time went into the model reading 83,000 characters of workflow instructions before it was allowed to touch save_document. Saving now runs on a second runner carrying a four line instruction and one tool, on its own session, which also keeps long answer bodies out of the chat history. Measured against the running instance: 60.8 seconds down to 7.4.
The free tier shaped more of the code than I expected. Bounded run configs, backoff that reads the retry delay out of the error message, greetings answered without calling the model at all, and repeated identical tool calls refused rather than run twice. A demo that dies on a 429 in front of a judge is worth nothing.
Accomplishments that I'm proud of
The evidence panel, mostly. When it says orders is fed by three tables, the lineage call that established it is sitting right there with its arguments. You can disagree with the answer on the evidence rather than on vibes.
The write-back checks itself, which took two attempts to get right. save_document reports failure by returning a message instead of raising, so a save that quietly did nothing still looked like a success. The app now confirms that the tool call actually happened and that its result reported success, rather than believing the model's summary of its own work.
And it has a boundary it respects. Three questions that used to cost catalog searches now cost none.
What I learned
Instructions change behaviour more than I assumed. Going from three skills to five moved the PII answer from stopping at get_entities in 43.6 seconds to reaching list_schema_fields in 86.8 seconds. Twice the wait, but a column level answer to a column level question, which is what was asked.
The other lesson is about my own debugging. When the same class of bug came back a second time, that was the moment to stop patching and look at the shape of the thing. I got there on the third try instead.
What's next for CatalogPilot
Saving the same question twice currently creates two documents rather than revising one, because save_document needs the URN of a document that already exists and every phrasing I tried for "look one up first" made the agent stop and ask for confirmation instead of saving at all. That is the next thing to solve.
After that, the rest of the enrich workflows: tags, glossary terms and ownership written back straight from an answer, so the agent can correct the catalog it just read instead of only adding notes to it.
Log in or sign up for Devpost to join the conversation.