Sentinel: An Autonomous Incident Agent for DataHub

Sentinel is an autonomous incident response agent for DataHub. It investigates, coordinates safe response, and writes verified knowledge back to the DataHub catalog. Every incident learns.

Comment

Sentinel: An Autonomous Incident Agent for DataHub

Inspiration

It's 2 AM. A freshness breach just fired on the NYC taxi pipeline: six hours stale against a one-hour SLA. Priya is on-call. She opens the war room.

What happens next is the same on every data team: thirty minutes of searching for context that someone, somewhere, already solved. Who owns this asset? When did it last break? Wasn't there a post-mortem from the last incident? The answers exist, in archived Slack threads, in a Confluence page nobody updated, in the head of the engineer who left last quarter. The knowledge is real. The catalog doesn't have it.

DataHub is a brilliant catalog. It documents your system: assets, lineage, ownership, governance tags. But a catalog documents. It doesn't remember.

That gap, between what your organization knows and what your catalog contains, is where every incident wastes its most expensive resource: the work someone already did last time.

Sentinel started as one question: what if the catalog could remember?

What if every incident, instead of just paging a human and dying in a thread, wrote its own post-mortem back into DataHub, so the next time the same asset broke, the agent started where the last one ended, not from scratch?

That question reframed the whole project. Sentinel isn't an alerting tool. It's the mechanism by which a catalog becomes a memory.

What It Does

When a freshness, schema, or quality signal trips in DataHub, Sentinel runs the full incident-response loop, autonomously, governed, and in the open.

Investigates. Reads the failing asset from DataHub. Traverses the lineage graph to size the blast radius. Pulls ownership, glossary terms, governance tags, and, critically, any post-mortem from the last time this asset broke. A visible ReAct loop: observe → think → act. Every reasoning step shown, every tool call grounded in the live context graph.

Acts. Tightens the freshness SLA. Files a GitHub issue with the root cause and the fix. Posts a Block Kit triage card to the on-call Slack channel. Never merges a pull request. Stops and asks for approval the moment an action touches PII-tagged data.

Writes back. Composes a structured post-mortem (root cause, blast radius, recommended action) and persists it to DataHub via the Agent Context Kit. Also writes a glossary proposal, an ownership proposal, and the new SLA assertion.

That third step is the whole point. The post-mortem Sentinel writes in Run 1 is the context it reads in Run 2. The next time this asset breaks, Sentinel doesn't start from scratch. It reads yesterday's knowledge first, produces a shorter reasoning trace, and resolves faster.

We measured it:

Run 1 Run 2 (same incident, post-mortem now in catalog)
Reasoning steps 24 23
Tool calls 7 6
Resolution time 66.4s 2.5s

That's not a feature. That's the catalog learning. Every incident leaves it smarter.

How We Built It

Architecture: a ReAct loop over DataHub's MCP tools, with a write-back path through the Agent Context Kit, under a code-level guardrail.

  • Read layer. Sentinel calls DataHub's MCP Server directly: map_get_entities, map_get_lineage, ownership, glossary, governance tags, prior post-mortem search. Every read is a real MCP tool call, no synthetic wrappers.
  • Reasoning layer. A ReAct (observe → think → act) loop, streamed live via Server-Sent Events so the dashboard shows the agent thinking in real time. Multi-provider LLM with failover: Groq Llama 3.3 70B as primary,Gemini 2.0 Flash as fallback, behind a code-level circuit breaker and rate limiter. No single provider's outage can stop the agent.
  • Action layer. Connectors to the GitHub REST API (issues plus draft PRs, never merges) and the Slack Web API (Block Kit triage cards). Idempotent by design: if an open issue with the same title exists, Sentinel appends context as a comment instead of opening a duplicate.
  • Write-back layer. Structured post-mortem, glossary proposal, ownership proposal, and SLA assertion ingested into DataHub via the Agent Context Kit. This is the closed loop: the write that makes the next read smarter.
  • Guardrail. Code-level, not prompt-level. PII refusal, no-merge rule, approval gate. The LLM cannot bypass it by rephrasing; the check runs in pre-exec.ts before the tool executes, not in the prompt.

Stack: Next.js 16 (App Router, Turbopack), TypeScript 5 strict, Tailwind CSS 4 + shadcn/ui, Prisma + SQLite (Turso libSQL in production), streaming SSE, z-ai-web-dev-sdk. Runs from a fresh clone in under a minute: seeded SQLite catalog, no Docker, no live DataHub required.

The demo runs live. Real GitHub issues in sodiq-code/sentinel-demo-pipeline. Real Slack posts in #sentinel-bot. Real DataHub write-backs. Every artefact is externally verifiable: click the links in the README's "Verified end-to-end" section.

Challenges We Ran Into

The PII guardrail. Early on, the agent wrote a post-mortem about a PII-tagged asset (customer_pii_dataset, tagged PII and Restricted) directly into DataHub. That's a leak: a shared context document describing a PII asset exposes PII-adjacent details to anyone who reads the catalog. The obvious fix was to skip the write. But skipping it silently was worse: no audit trail, no way for an operator to know what was refused and why.

The solution was a graded response. The guardrail refuses the PII write, but instead of an empty refusal, Sentinel writes a governance refusal record (URN, tags, reason, timestamp) to DataHub. The decision is persisted; the content never is. Safe actions still fire (GitHub issue, Slack triage). The run resolves WRITE-BACKS=1 instead of 2. A PendingApproval row lets an operator approve a redacted post-mortem later. Status "degraded" means "safe subset ran, unsafe write refused," the correct graded response, not a failure dressed up as one.

This had to be code-level. A prompt-level guardrail ("don't write PII") fails the moment the LLM rephrases or the model changes. The check runs in pre-exec.ts, before the tool executes. The LLM cannot bypass it.

The closed-loop proof. We could narrate the compounding thesis. Proving it was harder. We needed the same incident, run twice, with the post-mortem from Run 1 visible in Run 2's context, and a measurable difference in reasoning steps, tool calls, and resolution time. Building the replay harness that makes Run 1's write-back visible to Run 2's read, and measuring both, took longer than the agent itself. The 66.4s → 2.5s number is the output of that harness, not a claim.

Multi-provider failover. Gemini's free tier rate-limits aggressively under a 24-step reasoning load. Groq's Llama 3.3 70B is faster but has different tool-calling conventions. The circuit breaker and failover had to handle both without dropping the reasoning trace mid-loop. Solved with a server-side retry queue that preserves the ReAct state across provider switches.

Accomplishments That We're Proud Of

  • The 66.4s → 2.5s compounding result. Run 1 investigates from scratch. Run 2 reads Run 1's post-mortem first. The catalog didn't just record the failure, it learned from it. That number is the thesis, measured.
  • Everything in the demo is real. Two GitHub issues (#12, #13) in an open repo, two Slack triage cards in #sentinel-bot, one post-mortem persisted to DataHub. A judge can click any link and verify. No mocks, no dry-run, no theatre.
  • The PII guardrail as a graded response. Not a binary allow/deny, but a refusal that persists the decision without exposing the content. An autonomous agent that refuses unsafe action and leaves an audit trail is rarer than it sounds.
  • The bookended film. The three-minute demo opens on the motion graphic ("Run 1 writes. Run 2 reads.") as a promise and closes on the same visual as proof. The film itself is shaped like the loop it's selling. (Watch it.)
  • The closed-loop RFC. The write-back pattern generalizes beyond incidents; any metadata agent that reads context, acts, and writes back compounds. The RFC ships in the repo at rfc/closed-loop-metadata-agents.md.
  • The packaged DataHub Skill. incident-triage ships as a standalone Skill compatible with Cursor, Claude Code, Copilot, Codex, and Gemini CLI. Not just a demo, a reusable artifact.

What We Learned

The value of an autonomous agent isn't in the first run. It's in the second.

A first-run agent that investigates and resolves an incident is useful. A second-run agent that starts with the first run's post-mortem, using a shorter trace, fewer tool calls, and faster resolution, is a different category of thing. The first run is a tool. The second run is proof the system learned.

That reframed how we think about agent design. Most agent benchmarks measure single-run performance: accuracy, latency, tool-call count. Almost none measure compounding: whether the agent gets better the second time it sees the same problem. But compounding is where the actual leverage is. A flat-performance agent is a utility. A compounding agent is a memory.

The second thing we learned: a guardrail that refuses is not a weakness. It's a feature, if it refuses well. The PII refusal felt like a limitation in early demos. Once we made it write a governance refusal record, persisting the decision without the content, it became one of the most cited parts of the demo. Engineers trust an agent that knows what it doesn't know.

The third: the closed loop is the product, not the agent. Sentinel-the-agent is the visible part. But the thing that compounds is the loop: incident → context → reasoning → action → write-back → smarter-next-time. The agent is the worker. The loop is the asset. Designing for the loop, not just for the agent, is what makes the catalog get smarter instead of just busier.

What's Next for Sentinel

Generalize the closed loop beyond incidents. The pattern Sentinel proves (read context from DataHub, act under a guardrail, write the outcome back so the next agent starts smarter) is not specific to incidents. It applies to any metadata agent: schema drift, data quality, cost anomalies, access reviews, lineage proposals. The closed-loop-metadata-agents RFC in the repo generalizes the pattern. The next step is to ship it as a reusable primitive, not a Sentinel-specific feature, but a DataHub capability.

The post-mortem as a first-class DataHub entity. Right now, Sentinel's post-mortems are written via the Agent Context Kit and visible in the catalog. The next iteration makes them first-class: queryable, versioned, linked to the incidents they resolved and the assets they reference. A post-mortem shouldn't be a document. It should be a node in the graph, reachable from the asset, the incident, and the next run that reads it.

Cross-asset compounding. Today, Run 2 reads Run 1's post-mortem when the same asset breaks again. The richer version: Run 2 reads post-mortems from related assets, sharing the same upstream source, the same owner, or the same glossary term. The catalog doesn't just remember this incident. It remembers the shape of incidents like this one.

The catalog as a memory the whole platform learns from. That's the line we opened with. That's the line we're building toward. Every incident leaves the catalog smarter, until the catalog isn't a catalog anymore. It's the memory the whole data platform learns from.

Built With

Share this project:

Updates