Cairn

• 

  Cairn agents pipeline

Inspiration

Every agent (and every human) that touches a dataset in DataHub learns something useful along the way — "this column looks derived, not authoritative," "this description no longer matches how the table is actually queried." That knowledge normally evaporates the moment the session ends. The next agent or teammate starts from zero, or worse, overwrites a partial understanding with an equally partial one.

I wanted to build something that treats that as the actual problem — not documentation-generation, not a chatbot, but a deliberate way to leave a trail marker before context disappears. A cairn is a stack of stones travelers leave along a trail: not a full map, just enough signal for the next traveler to find their way. That's exactly what we wanted our agent to do for DataHub.

What it does

Cairn is a context relay agent with three cooperating parts:

• Sentinel inspects a dataset's schema, lineage, description, and query history, and produces findings — gaps between what's documented and what the data actually shows. It implements two strategies: query_drift (heavily queried but undocumented columns) and documentation_gap (an existing description that may be stale relative to the current schema/lineage).
• Governance gate decides which findings are worth acting on. Writes are confidence-gated, rate-limited, and cooled down per entity — Cairn behaves like a careful colleague, not a bot that floods your catalog with guesses.
• Capsule writer turns an approved finding into a small, structured handoff capsule written via DataHub's structured_properties — machine-readable fields any downstream agent can parse programmatically. Alongside it, Cairn also saves a short, human-readable reflection document linked to the dataset, so a person skimming the DataHub UI sees the contribution too.

Cairn doesn't answer questions the way DataHub's Analytics Agent does — it watches for context that's about to be lost and leaves a marker before it disappears.

How we built it

The pipeline is a straight line: dataset URN → Sentinel → Governance gate → Capsule writer → DataHub (via MCP Server). Writes land as io.cairn.* structured properties (agent ID, confidence, finding type, unresolved questions, assumptions made, a human-review flag) plus a linked Context document. Everything talks to DataHub through the official MCP Server.

Challenges we ran into

Building against a real, self-hosted DataHub quickstart instance surfaced concrete integration bugs that wouldn't have shown up from reading docs alone:

• add_structured_properties initially sent the wrong payload shape — a flat {urn, structured_properties} object instead of the real tool's {property_values, entity_urns} signature.
• io.cairn.sessionTimestamp is registered as a date-type property, which DataHub validates strictly as YYYY-MM-DD — a full ISO 8601 datetime was rejected server-side until we fixed it.
• save_document initially used an invented parent_folder parameter that doesn't exist on the real tool.
• Our MCP client's __aenter__() originally wrapped connection setup in asyncio.wait_for(), which breaks anyio's cancel-scope-to-Task binding and raised a TypeError on connect.

Each of these was found by reading the actual mcp-server-datahub tool source and testing live writes against the instance, then fixed and covered by regression tests.

Accomplishments that we're proud of

Getting the full write path verified end-to-end against a live instance: a high-confidence finding gets written as both structured properties and a reflection document, while a low-confidence finding in the same run is skipped with a logged reason. That restraint — Cairn choosing not to write — is the point, not a limitation.

What we learned

Governance has to be a first-class citizen in an agent that writes back to a shared system of record, not an afterthought bolted on later. Confidence thresholds, cooldowns, and rate limits changed how we designed the whole pipeline, not just the last step.

What's next for Cairn

• A third finding strategy for lineage breaks
• Multi-dataset batch runs with a single governance budget
• Surfacing Cairn's capsules directly inside DataHub's Analytics Agent so a person can just ask "what has Cairn noticed about this table?"

Built With

• anthropic-claude
• asyncio
• click
• datahub
• docker
• mcp
• pytest
• python