Sherlock — The Metadata Detective for DataHub

An autonomous agent that investigates your catalog's "cold cases" — undocumented, unowned tables — by cross-referencing schemas, lineage & real SQL queries, then writes the knowledge into the graph.

Comment

Inspiration

Every data catalog rots the same way: tables nobody documented, nobody owns, and nobody dares to delete. Documentation sprints don't scale — humans hate writing metadata. But the evidence to document these tables already exists: in lineage, in schemas, and in the SQL queries people actually run. It just takes a detective to piece it together.

What it does

Sherlock patrols your DataHub graph like a detective works cold cases:

  1. 🧊 Finds cold cases — searches the graph via the DataHub MCP Server for datasets with no owner, no description, or mostly-undocumented columns.
  2. 🔍 Gathers evidence — schema fields, upstream/downstream lineage, sibling table naming conventions, and crucially the real SQL queries referencing the table.
  3. 🧠 Deduces — an LLM reasons strictly over the evidence bundle: table purpose, per-column descriptions, likely owner (inferred from who actually queries it), and undeclared PII columns. Every deduction cites its evidence and carries a confidence score.
  4. ✍️ Writes it back — table & column descriptions via update_description (clearly marked AI-generated), PII-Suspect tags via add_tags, and a full Case Report document saved into the graph via save_document.
  5. 📈 Remembers — future patrols recognize Sherlock's own writes and skip solved cases. Run it twice: the second patrol reports "the graph is healthy." Knowledge compounds instead of evaporating.

How we built it

  • Pure MCP: every read AND write flows through the official mcp-server-datahub (18+ tools, mutations enabled) — the same interface any other agent would use.
  • A single long-lived MCP session on a dedicated worker task (solving the anyio cancel-scope pitfall), wrapped in a sync facade.
  • Deduction engine works with any OpenAI-compatible LLM endpoint; grounded prompting with mandatory evidence citations.
  • Safety by design: confidence-gated writes (below threshold = report only), ownership is suggested for human confirmation rather than silently assigned, every write is attributed and reversible.
  • Demo estate: a seed script creates a realistically messy e-commerce catalog (cryptic LTV table, orphaned staging export full of PII, real query history) against DataHub quickstart.

Challenges we ran into

  • MCP tool schemas in the wild differ from docs — we built against the real server's introspected schemas (SQL-like filter syntax, replace/append operations, document types).
  • asyncio/anyio cancel scopes require MCP sessions to live and die in one task — fixed with a queue-fed worker thread.
  • Detecting "already documented" needed merging native schema docs with DataHub's editable layer.

Accomplishments we're proud of

  • True end-to-end closure: patrol → investigate → deduce → write back → next patrol inherits the knowledge and finds nothing to redo.
  • The owner inference actually works: Sherlock correctly deduced bob.finance owns the LTV table purely from query authorship evidence.
  • All 7 seeded PII columns caught (emails, phone, DOB, home address) — flagged for human review, not silently acted on.

What's next

  • Read prior case reports at patrol start for richer memory (search_documents is auto-enabled once reports exist).
  • Owner proposals via DataHub's governed proposal workflow.
  • Coverage metrics over time — a "graph health" dashboard written back as a document.

Built With

Share this project:

Updates