Syndicate

Smart Memory & Trust Ledger for DataHub: AI agents learn from past decisions, skip redundant scans, and build proven reliability.

Comment

Inspiration It started with a bold idea: build an intelligent Data Onboarder Agent that could autonomously discover, classify, and register new datasets directly into DataHub using the MCP (Model Context Protocol) server. We were excited — a fully autonomous AI agent that could replace hours of manual metadata onboarding with a single command.

Then we hit the wall.

As we dug deeper into the DataHub MCP server, we discovered it was read-only. The MCP tools available only allowed agents to observe the catalog — not write to it. Our entire Data Onboarder concept depended on mutation capabilities that simply weren't there. We had a working architecture, a clean design, and zero way to execute it.

We sat back and asked ourselves: what's the real problem with AI agents operating on data catalogs?

The answer came fast — it wasn't just about onboarding. It was about trust and memory. Who checks whether an AI agent is actually making the right calls? What happens when one agent tags sensitive customer data as public? What happens when 10 agents waste compute re-analyzing the same table someone already reviewed last week?

That question became our inspiration. We scrapped the onboarder and built something we believe is far more impactful: a native Trust and Memory Ledger for AI agents operating on DataHub.

What it does Syndicate is an Agent Provenance Ledger that brings memory, accountability, and trust directly into DataHub — using only standard APIs, with zero external databases.

It orchestrates 4 specialized sub-agents in a sequential pipeline:

Memory Checker — Before running any expensive investigation, it scans past Decision Records stored in DataHub for a similar prior case. If the algorithmic similarity check misses (due to column naming variations or schema differences), an AI Semantic Scan Fallback kicks in, using semantic PII concept matching to find relevant precedents that pure token algorithms would overlook.

Evidence Gatherer — If no memory hit is found, this sub-agent investigates the target dataset: scanning schema columns for PII signals, tracing upstream lineage dependencies, and querying recent dataset operations. It computes a final confidence score (0–1).

Proposer — Based on the agent's current trust level and confidence score, it decides the action: auto-apply tags for high-trust agents or propose-only for lower-trust agents. It then attaches provenance tags (proposed-by:agent-id) to datasets and persists a Decision Record — a permanent, tamper-evident log of what the agent did and why — directly into DataHub as a Glossary Term with a full reasoning chain embedded in its definition.

Trust Scorer — After human Data Stewards review pending proposals in the DataHub UI, their feedback is fed back as approvals or rejections. The Trust Scorer uses Laplace-smoothed scoring to update each agent's live trust profile, stored natively in DataHub. Agents that consistently make correct proposals earn higher autonomy. Reckless agents get progressively restricted.

How we built it We built the entire system in Python, wiring it to a live DataHub instance running on Docker Desktop with WSL2 on a local machine.

DataHub Backend: Deployed the full DataHub quickstart stack (GMS, Frontend, Kafka, OpenSearch, MySQL, Actions) using datahub docker quickstart. All agent interactions happen via the DataHub REST/RestLi GMS API at http://localhost:8080.

MCP-Compatible Client: We engineered a DataHubClient that mirrors the exact tool surface of the official mcp-server-datahub — so every call (get_entities, list_schema_fields, get_lineage, search_documents, save_document, add_tags, add_structured_properties) matches an MCP tool 1-to-1. This means the entire system can swap its transport to a real MCP server with zero code changes.

Decision Records as Glossary Terms: We had no external database. Instead, we serialized each Decision Record as a JSON blob embedded in a DataHub Glossary Term's definition field. This gives us a fully queryable, UI-visible audit log with zero infrastructure overhead.

AI Semantic Fallback: When rule-based Jaccard similarity fails, the Memory Checker escalates to an AI scan. It supports OpenAI GPT-4o-mini, Gemini 1.5 Flash, and a built-in zero-dependency Local Semantic Reasoner — so it works even without API keys.

Trust Profiles via Structured Properties: Agent trust scores are stored and updated as Glossary Term definitions in DataHub, making them directly inspectable and filterable in the DataHub UI.

Challenges we ran into

  1. The MCP Pivot The biggest challenge wasn't technical — it was psychological. Discovering that our original Data Onboarder idea was blocked by a read-only MCP server two days in was a gut punch. Scrapping work is never easy. But pivoting quickly and decisively to the Provenance Ledger concept was the right call.

  2. DataHub GMS RestLi Quirks The DataHub REST API has subtle quirks that cost us hours of debugging. For example, GET /entitiesV2 expects repeated ids=urn:li:... query params — not a JSON-encoded array. Passing the wrong format returns a silent java.net.URISyntaxException from the Java backend. We methodically debugged the raw HTTP requests until every endpoint behaved correctly.

  3. Windows Encoding Hell Running Python on Windows with Unicode output characters (✔, →) caused UnicodeEncodeError crashes on the CP1252 codepage. The fix was simple ($env:PYTHONIOENCODING="utf-8") but finding it cost us time during a live demo run.

  4. Docker on WSL2 on Acer ALG Hardware virtualization was disabled in BIOS by default on the Acer ALG laptop. The DataHub Docker stack silently failed to start with a cryptic HCS/ERROR_NOT_SUPPORTED error. We traced it all the way to a BIOS F2 setting.

  5. Search Aspect Mismatch search_documents initially returned zero results because the DataHub /entitiesV2 batch endpoint returned empty responses for Glossary Term URNs. We had to pivot to individual /aspects/{encoded_urn} calls per URN — a much more reliable pattern that now works perfectly.

Accomplishments that we're proud of Full live demo working on Docker — The entire pipeline runs against a real, live DataHub GMS instance. No mocks. No fakes. Real API calls.

AI + Algorithmic Memory Hybrid — We built a two-pass memory system that first tries rule-based Jaccard similarity and falls back to semantic AI concept matching. This hybrid approach catches cases that pure tokenization would miss.

Trust Scores that actually diverge — Agent A's trust climbs to 0.67 while Agent B's drops to 0.33 in a single demo run. The system demonstrably rewards good agents and penalizes careless ones.

Zero External Dependencies for Governance — Every Decision Record, Trust Profile, and Provenance Tag lives natively inside DataHub. Auditors don't need a separate system — they just use the DataHub UI.

MCP-Ready Architecture — Our DataHubClient is a 1-to-1 mirror of the official MCP tool surface. Swapping to a live MCP server requires changing exactly one line.

What we learned Constraints breed creativity. A read-only MCP server felt like the end of the road but became the forcing function that led us to a more meaningful and novel solution.

DataHub is a governance platform, not just a catalog. Glossary Terms, Structured Properties, Tags, and Lineage are powerful primitives that can encode complex agent behavior without a single line of server-side modification.

Trust is not binary. Building Laplace-smoothed trust scoring taught us that good governance systems need probabilistic thinking — not just pass/fail rules.

AI fallbacks require graceful degradation. Engineering the Semantic Scan to work with OpenAI, Gemini, and a built-in local reasoner (zero dependencies) made the system genuinely robust in any environment.

What's next for Syndicate MCP Live Transport: Connect the DataHubClient directly to a live mcp-server-datahub instance so agents communicate via the Model Context Protocol natively.

Multi-Agent Conflict Resolution: When Agent A and Agent B disagree on a classification, trigger a structured debate round — majority-rules or trust-weighted voting — before escalating to a human.

Explainability Dashboard: Build a DataHub custom front-end dashboard (using the DataHub Metadata Tests / Assertions framework) that visualizes agent trust trajectories and decision record timelines.

Automated Escalation Policies: When an agent's trust score falls below a configurable threshold, automatically escalate all its pending proposals to a senior Data Steward — closing the governance loop end-to-end.

Cross-Agent Memory Sharing: Today, Memory Checker only searches decision records from the same agent. Next, we want agents to search each other's prior decisions — enabling true institutional knowledge sharing across the entire agent fleet.

Built With

Share this project:

Updates