-
-
Document list
-
Document parse (1)
-
Document parse (2)
Inspiration
Business documents often contain the context needed to understand data—metric definitions, ownership, policies, operational rules, and caveats—but that knowledge is usually disconnected from the data catalog. We wanted to make documentation a first-class part of data governance without allowing an AI system to silently change metadata. That led us to build a review-first workflow that helps people connect documents to trusted DataHub assets safely.
What it does
DataHub Document Automation analyzes Markdown or plain-text business documents and recommends relevant DataHub metadata, including datasets, schema fields, domains, tags, and owners.
It retrieves candidates from the existing DataHub catalog, ranks them with deterministic rules and an optional LLM, and presents the result for human review. Only approved recommendations can be published as a native DataHub Document. The project also tracks related dataset metadata and flags published documents for review when their context becomes stale.
How we built it
We built a full local demo environment with Docker Compose, including DataHub 1.6.0 and its supporting Kafka, MySQL, Elasticsearch, Neo4j, and Schema Registry services.
The backend uses Python, FastAPI, Pydantic, SQLite, and versioned dbmate migrations. It provides document upload, extraction, catalog retrieval, recommendation, review, publishing, conflict handling, and freshness-check APIs.
The frontend uses React, TypeScript, Vite, and Mantine to give reviewers a clear interface for assessing recommendations before publication. We seeded DataHub with a Jaffle Shop retail catalog containing datasets, schemas, domains, owners, and tags, then used realistic business documents such as revenue-recognition policies, payment-failure runbooks, and customer-PII guidance to test the workflow.
Challenges we ran into
The hardest problem was balancing useful automation with safe metadata governance. LLM output can be persuasive but incorrect, so we designed the system to ground every recommendation in a bounded set of DataHub catalog candidates and validate the output before it reaches the reviewer.
We also had to make publishing reliable. A failed or repeated request should not create duplicate documents or partially apply metadata, so we introduced stable URNs, read-back verification, and idempotent retries.
Finally, reproducing a complete DataHub environment locally required coordinating several dependent services and ensuring that the demo data was deterministic and isolated from unrelated catalog assets.
Accomplishments that we're proud of
- Built an end-to-end, review-first workflow from document upload to safe DataHub publishing.
- Made the demo usable without an LLM API key through deterministic rule-based recommendations.
- Added safeguards including read-only retrieval, schema-validated LLM output, conflict review, stable publishing identifiers, and freshness monitoring.
- Created a realistic Jaffle Shop demo catalog and annotated document scenarios for evaluation and manual validation.
- Packaged the entire experience into a one-command Docker demo for judges.
What we learned
We learned that good AI-assisted data governance is less about generating metadata automatically and more about making trustworthy decisions easier for people. Grounding recommendations in the existing catalog, showing evidence, and preserving a human approval step are essential.
We also learned that metadata changes over time. Publishing a document is not the end of the workflow; systems need a way to detect when related datasets, schemas, or ownership information have changed.
What's next for datahub-go
Next, we want to support more document formats, including PDFs and collaborative knowledge-base pages, and improve semantic retrieval for larger catalogs. We also plan to add richer reviewer explanations, feedback-driven recommendation quality metrics, role-based approval workflows, and broader DataHub entity support such as glossary terms, dashboards, and data products.
Longer term, we want DataHub Document Automation to become a practical bridge between human business knowledge and governed data assets: automated enough to save time, but always transparent, traceable, and human-controlled.
