Blast Radius

Inspiration

Every SRE has lived this: a P1 page fires on a user-facing service, but that service isn't actually broken — something several hops down the dependency graph is. The first 30–45 minutes of the incident go to mechanical detective work: which dependency regressed, what deployed, who's affected, what do we do? It's repeatable, high-stakes, and it happens at 2 AM. That's the ideal job for an agent — so we built one.

What it does

Blast Radius is an autonomous agent that triggers on a Splunk alert and produces a complete incident brief in seconds:

• Finds the true root cause, distinguishing the service that paged from the service that's actually broken by walking the service dependency graph.
• Computes the blast radius — every impacted downstream service and an estimate of user-facing traffic affected.
• Correlates the onset with recent changes to name the probable cause.
• Classifies operational vs security so the right team is engaged.
• Writes a ranked, plain-English runbook with the recommended remediation.

In our demo incident, the page fires on edge-gateway, but Blast Radius correctly identifies redis-cache as the root cause — a cost-savings config change that dropped the connection-pool limit one minute before the cascade — lists the three impacted services and ~850 req/s of affected traffic, and recommends rolling back the change.

How we built it

A deterministic 6-step reasoning loop orchestrates Splunk data and AI models:

1. Ingest alert + metrics + logs + change events from Splunk via the MCP Server.
2. Confirm anomalies with the Cisco Deep Time Series hosted model (real? worsening?).
3. Localize the root cause as the sink of the anomaly subgraph (depends on no other anomalous service).
4. Blast radius by walking the graph upward to all impacted services + user impact.
5. Correlate anomaly onset with the nearest prior change event.
6. Classify with Foundation-Sec-1.1-8B, then gpt-oss writes the narrative + runbook.

The agent is backend-agnostic: a mock mode runs entirely offline, and flipping two env flags points it at a live Splunk MCP Server and the hosted model endpoints — no code change.

Splunk capabilities used

• Splunk MCP Server — every read of Splunk data is an MCP tool call; the agent operates agent-style over the platform.
• Splunk Hosted Models — Cisco Deep Time Series, Foundation-Sec-1.1-8B-Instruct, and gpt-oss, each doing what it's best at.

Challenges we ran into

The core insight took iteration: naively, "the deepest anomalous node" can pick the symptom instead of the cause. The fix was to define the root as the sink of the anomaly subgraph — the anomalous service that depends on no other anomalous service — and to compute its upward reach to break ties. Keeping the reasoning deterministic and traceable (so every conclusion is auditable) while still using AI where it adds the most value was the central design tension.

Accomplishments that we're proud of

• An agent that acts on data rather than just summarizing it.
• A fully explainable trace — every step is visible and justifiable, which is what it takes for an SRE team to trust automation in their incident path.
• Runs end-to-end with zero setup for judges, yet has real MCP and hosted-model integration paths.

What we learned

The hardest part of incident response isn't reading data — it's reasoning across topology + time + change. Combining a deterministic graph layer with Splunk's hosted models gives you both accuracy and an explanation, which is what production trust requires.

What's next for Blast Radius

Real-time Splunk alert-action trigger, topology auto-discovery from traces, closed-loop remediation with approval gates and post-action verification, and a feedback loop where engineers confirm or correct the root cause to tune the agent.

Built With

• ai-agents
• mcp
• python
• splunk
• splunk-hosted-models