datahub

Inspiration LinkedIn’s shift from warehouse to lake left teams asking “Where is my data and how did it get here?”—we set out to give every dataset a living, searchable résumé . What it does DataHub is an open-source metadata platform that streams lineage, quality and ownership info into one searchable catalog, so humans and machines can discover, trust and govern data in real time . How we built it We rewrote the first-gen tool (WhereHows) as a stream-first architecture on Kafka; every SQL job, Airflow DAG or dbt run now pushes metadata to a central graph store that updates sub-second . Challenges we ran into Scaling a metadata graph that grows 10× faster than the data itself; keeping a 13 k-member community in sync while shipping enterprise-grade security and multi-tenant cloud . Accomplishments that we're proud of 13 k+ community, 3 k+ orgs (Netflix, Apple, Visa) in production, column-level lineage across 50+ sources, and a brand-new 1.0 design system built with community PRs . What we learned Metadata must be treated as a product—versioned, reviewed and automated—otherwise data lakes become swamps again . What's next for DataHub 2025 roadmap: Metrics catalog, end-to-end dashboard lineage, policy push-back to source systems, incident management, Python SDK v2, and live data explorer inside the UI .

Built With

• java
• mysql
• vue