Schema Whisperer — Deep MongoDB Profiling Agent for DataHub

Inspiration

DataHub's MongoDB ingestion samples only 1,000 documents per collection to infer schema (datahub#9287). For schemaless collections that evolve over time, this misses rare fields, type conflicts, and nested structure. Schema Whisperer was built to close this gap autonomously.

What it does

Schema Whisperer is an AI agent that:

1. Reads DataHub to find MongoDB datasets and their shallow current schema
2. Scans MongoDB directly via aggregation pipeline ($objectToArray, $group, $facet) — finding field frequency, type conflicts, rare fields, and schema evolution variants
3. Reasons with an LLM to generate rich descriptions, detect PII, and infer business glossary terms
4. Writes back enriched metadata to DataHub — descriptions, tags, glossary terms, and full profile reports

How we built it

• MongoDB aggregation pipeline — $objectToArray for field discovery, $group for frequency counts, $facet for parallel multi-query profiling
• DataHub GraphQL API — read datasets, write descriptions/tags/terms/documents
• OpenRouter LLM (DeepSeek V4 Flash) — generates human-readable descriptions, detects PII, infers business meaning
• Express API + single-page demo UI — interactive dark-theme interface for running profiles and viewing results

Challenges we ran into

DataHub's GraphQL write-back required careful handling of null content fields and proper REST sink SYNC mode configuration. Getting the MongoDB aggregation pipeline to correctly detect nested document schemas across 23,000 documents required iterative refinement of the $objectToArray + $group pattern.

What we're proud of

• End-to-end autonomous loop: read → scan → reason → write-back
• Found fields DataHub completely missed (e.g., legacyUuid on 0.5% of docs, type conflict on createdAt)
• 100% open source with Apache 2.0 license
• Works with any MongoDB collection — zero configuration needed

What's next

• Support for PostgreSQL and MySQL schema profiling
• Scheduled drift detection (diff snapshots over time, alert on schema changes)
• Integration with DataHub MCP Server for agent-to-agent workflows

Built With

• datahub
• deepseek
• express.js
• mongodb
• node.js
• openrouter