Metadata Steward Agent Inspiration Data catalogs like DataHub are incredible single sources of truth, but only when they are populated. In reality, data teams constantly face blank descriptions, unmapped column metadata, and missing PII labels—leading to broken trust and repeated manual discovery work.
We were inspired to build an agent that does more than just analyze or report errors; we wanted an agent that closes the operational loop. By reading metadata, generating grounded fixes, and writing them directly back into DataHub, the next engineer or agent who inspects the dataset inherits instant, high-quality context instead of a blank canvas.
How We Built It We structured the project around a lightweight read-think-write pipeline designed for quick, end-to-end execution:
Read (DataHub API / MCP): The agent fetches dataset schemas, native column types, existing sparse documentation, and upstream/downstream lineage graphs via GraphQL queries.
Think (LLM Grounding & Classification): We feed the schema and lineage context into Gemini 2.5 Flash. Using strict JSON schema output constraints via Pydantic, Gemini drafts:
A high-level operational summary of the dataset.
Clear column-level descriptions.
Automated PII identification (flagging sensitive fields like email, ssn, or IP addresses).
Write Back (Mutations & Audit Trail): The agent executes GraphQL mutations directly against DataHub to update editableProperties descriptions and attach classification tags (Has-PII, Auto-Documented).
To ensure reliability during evaluation, we configured fallback handling for GraphQL endpoints alongside MCP interfaces.
Challenges We Faced API Mutation Handshakes: Writing metadata back reliably meant making sure GraphQL mutations targeted the correct editableProperties and tag URN structures (urn:li:tag:...) without overwriting existing critical metadata.
Deterministic Structured Output: LLM hallucination in metadata tagging is dangerous. Enforcing Pydantic schemas on Gemini's output was critical to ensure column field mapping matched the target schema precisely before executing write-backs.
What We Learned Building this project reinforced that AI agents are most valuable when they mutate state to save human labor. Reading and summarizing data is helpful, but persisting that context back into enterprise infrastructure like DataHub converts a temporary AI response into permanent, shared organizational knowledge.
Log in or sign up for Devpost to join the conversation.