DataHub Semantic Search Agent

Inspiration

Data catalogs like DataHub are only as useful as they are searchable. Traditional keyword search often misses the right dataset simply because the query doesn't share exact wording with a table's name or description — even when the data is exactly what someone needs. I wanted to build something that searches by meaning, not just text overlap.

What it does

DataHub Semantic Search Agent lets you search a DataHub catalog using natural language. Instead of matching literal keywords, it understands the intent behind a query and ranks datasets by semantic relevance. For example, searching "warehouse inventory levels" correctly surfaces the inventories dataset — whose description explicitly covers inventory levels — ahead of several datasets that are simply named "warehouses" but aren't actually about inventory tracking.

How I built it

1. Fetch — Connects to a self-hosted DataHub instance via its GraphQL API and pulls dataset metadata (names, descriptions, platform info).
2. Embed — Encodes each entity's name + description using sentence-transformers (all-MiniLM-L6-v2).
3. Search — Encodes the user's natural language query the same way and ranks all catalog entities by cosine similarity.

The agent was tested against DataHub's showcase-ecommerce sample data pack, covering 67 entities across dbt, Snowflake, S3, and Postgres platforms.

Challenges I ran into

Getting a full local DataHub stack running reliably was the biggest challenge — including a Windows-specific bug in the DataHub CLI's file-ingestion path handling that required moving the whole workflow into WSL2, plus resolving Docker Desktop's WSL integration, stale container path labels, and CPU resource contention during ingestion. Once the environment was stable, building the actual search logic was comparatively fast.

What I learned

A lot about DataHub's architecture (GMS, the metadata graph, GraphQL API) and about how much infrastructure reliability matters before you can even start building on top of a platform. Also reinforced how much value a small, well-chosen embedding model can add over naive keyword search, even with a lightweight setup.

What's next

Adding a write-back feature that lets the agent suggest and apply tags to datasets directly via the DataHub API, so search insights can feed back into improving the catalog itself.

Built With

• datahub
• docker
• graphql
• machine-learning
• numpy
• python
• sentence-transformers