majestic

Inspiration

An alert fires: a dashboard is empty, a report looks wrong. The alert tells you what broke, never why. Today, finding the actual cause means manually jumping between the lineage graph, the ETL scheduler, and Slack — hoping someone remembers what changed three hops upstream. DataHub already has the lineage graph and the metadata to answer that question. Nobody was using it as a causal-reasoning surface — just as a place to browse and search.

What it does

• diagnose — walks the lineage graph upstream from a broken entity, hop by hop, cross-referencing real evidence at each hop (incident tags, missing owners, stale data, recent schema changes) into a single causal chain. No LLM speculation: every link is backed by a fact read from the graph, or the chain stops right there.
• Lag-aware ranking — when two candidates look equally suspicious, Majestic breaks the tie by when each anomaly happened (exponential recency decay) and discounts evidence that's probably just inherited from further upstream, instead of an independent signal. It returns a ranked top-K, not just one guess.
• Memory — writes the diagnosis back to DataHub as structured properties (pattern signature, diagnosis, confidence, timestamp). The next time it sees the same structural pattern on another entity, it reuses the diagnosis instead of reasoning from scratch.
• impact — same lineage traversal, inverted: before touching a dataset, see what breaks downstream and who owns it.
• check-change — a CI/CD gate that blocks a change automatically when downstream blast radius combined with ownership orphanhood crosses a threshold.
• MCP server — the same core exposed as a majestic_impact tool so other agents can invoke it directly, not just the CLI.

How we built it

A Python agent on top of DataHub's Python SDK (DataHubGraph). src/graph handles the client and a deduplicated, paginated BFS lineage traversal. src/core holds the root-cause diagnoser (evidence collection + causal chain, lag-aware weighting) and the agent that writes and reads back structured properties, with a two-plan search (structured filter first, free-text fallback) for finding previous diagnoses. src/impact mirrors the traversal downstream for blast radius assessment. Everything was verified against a real, running DataHub instance — not just mocks.

Challenges we ran into

• Found a real bug in DataHub's own UI: it breaks when a structured property's text value contains a URN-shaped substring anywhere inside it — and our evidence text does exactly that (e.g. "... (hop 1): ..." embedding another entity's URN). Fixed it on our side and documented it as a ready-to-file issue.
• Our memory mechanism (matching a structural signature across entities) risked false positives — two unrelated incidents with the same shape could get treated as the same one. We anchored the signature to the causal node's platform to cut that down, and made the reuse message explicit that it's a structural match, not a confirmed identical incident.
• The local DataHub stack can get overwhelmed if left running unattended for many hours — we hit this mid-project and had to debug it while finishing up.

Accomplishments that we're proud of

79 unit tests plus 4 integration tests, all run against the real SDK's actual classes and methods (confirmed by introspecting the installed SDK, not assumed). Every core flow — doctor check, seeding, diagnose with write-back, impact simulation, and memory reuse on a second entity — was validated live, end to end, against a real DataHub instance. We also wrote an unfiltered self-audit scoring our own project against this hackathon's criteria and published it in the repo, because we'd rather find the gaps ourselves than have a judge find them first.

What we learned

Real graph evidence beats a confident-sounding guess. Keeping root-cause evidence 100% deterministic — facts from the graph only, with narration strictly downstream of evidence — was the right call for a project built for an Agent Hackathon. We also learned that "passes in CI with mocks" and "works against a real instance" are genuinely different claims: we found more than one bug we only caught because we ran against a live DataHub.

What's next

Registering the MCP server as a proper DataHub Skill / Agent Context Kit surface. Calibrating evidence weights against real historical incident data instead of reasoned defaults. Anchoring memory reuse to domain/tag overlap, not just platform, to further cut false-positive pattern matches.

Built With

• api
• cli
• compose
• datahub
• docker
• elasticsearch
• git
• github
• graphql
• kafka
• mcp
• mysql
• pytest
• python
• rest