cert-sync

Inspiration

Every org has a moment like this: someone in Slack says a metric means one thing, the data model says another, and nobody ever actually settles it. Today that's annoying but survivable — a human eyeballs both sources and picks one. But DataHub is increasingly the place AI agents go to answer questions about your data. The moment an agent starts answering business-critical questions straight from the metadata layer, an unresolved definition conflict stops being annoying and starts being a silent source of confidently wrong answers — and nobody would ever know, because nothing in DataHub reflects that a disagreement even exists.

That's the gap Cert Sync is built for: catch the disagreement, force an actual human owner to decide, write the decision back into DataHub as the system of record, and — critically — prove the fix reaches every agent downstream, not just assume it does.

What it does

Cert Sync watches real sources (Slack threads, docs) for a glossary term, uses an LLM to extract what each source actually claims the term means, and compares that against what's currently registered in DataHub. When it finds a genuine conflict:

It resolves the real owner of the term from DataHub's own ownership metadata — it never auto-decides, only routes to a human.
Once certified, the correct definition is written back into DataHub with a full audit trail of where it came from.
A completely separate consumer agent — no shared code, no shared session — answers questions about the term before and after certification, just by reading DataHub again. Wrong before, right after. That's the actual proof the fix propagated, not a log line claiming it did.

It also runs as a standing service, not a one-off script: a worker proactively scans for drift on a schedule, every run is persisted to a queryable audit log, and a dashboard shows the whole thing live, including a real Slack API pull rather than a canned fixture.

How I built it

I started with the riskiest assumption first: can a certified change in DataHub actually reach a second, independent agent with zero shared state? I built the smallest possible version of that loop before anything else — the whole project only made sense if that held up.

From there: the detection → owner-resolution → certification pipeline, a small eval harness against five hand-authored scenarios (including one with no registered owner at all, to force the "route to a human, don't guess" path), and tuning until it hit 100% precision/recall across all five.

Once the core logic worked as a script, I rebuilt it as an actual service: a FastAPI backend holding one long-lived, mutation-enabled MCP session (each open_session() call spawns a real subprocess, so pooling mattered), a deliberately dumb worker that just polls /scan over HTTP, and a Streamlit dashboard that talks to the API instead of importing agent code directly. Every detection and certification lands in a persistent SQLite audit log. I swapped a static Slack-thread fixture for a live Slack Web API integration for one scenario, so the demo shows a real fetch, not a mock. Everything ships in Docker Compose.

Challenges I ran into

A session that died quietly. The long-lived read-only MCP session would go unresponsive after sitting idle, hanging requests indefinitely with no error — nasty because it looked identical to a slow LLM call. Fixed by giving that agent a fresh short-lived session per request instead, while keeping the mutation-enabled session long-lived and lock-guarded.
Free-tier LLM latency and quota. OpenRouter's free tier has real latency variance (tens of seconds per call) and a hard daily request cap, which shaped real design decisions — like how often the worker is allowed to scan unattended — not just demo-recording timeouts.
Keeping the eval honest. The live Slack integration had to be wired in without ever touching the offline eval pipeline or its static ground truth — otherwise a passing eval score would stop meaning anything.

What I learned

That "the fix propagates" is a claim worth actually proving, not asserting — and that proof is only meaningful if the two agents involved genuinely don't share anything. Also: long-lived stdio sessions to a subprocess are a lot less "fire and forget" than they look, and free-tier LLM constraints belong in the architecture, not just in a rate-limit try/except.

What's next

Extending live ingestion beyond Slack (Confluence, dbt docs), broadening beyond single-term two-source conflicts, and proposing the underlying pattern upstream to DataHub itself.