BlastMap: AI Driven Failure Propagation & Impact Analyzer

Inspiration

In modern microservice environments, a single failure rarely stays isolated. A minor latency spike, container crash, or configuration error can silently propagate across dependencies, eventually manifesting as a major outage.

While observability platforms like Splunk excel at collecting logs, metrics, and traces, answering higher‑level questions often remains manual:

Where did the failure originate?
Which services are impacted downstream?
What is the true blast radius of this incident?

BlastMap was inspired by the need for impact‑aware observability—moving beyond detection toward understanding and explanation. We wanted to explore how Agentic Ops and LLMs embedded in Splunk could help engineers reason about failure propagation rather than just react to alerts.

What we built

We built an LLM‑powered failure propagation and impact analysis app for Splunk that:

Ingests logs, container events, and infrastructure signals into Splunk
Correlates failure signals across services and time
Uses Splunk‑hosted LLM capabilities to:
- Identify likely failure origins
- Reason about dependency‑driven propagation paths
- Summarize affected services and users
- Explain why an outage expanded the way it did
Writes AI‑generated impact analysis back into Splunk as enriched fields
Visualizes both raw telemetry and AI‑derived blast radius insights in Splunk dashboards

The result is a system that transforms fragmented operational data into a coherent, impact‑focused incident narrative.

How we built it

BlastMap is implemented as a Splunk‑native app using an agentic, LLM‑driven architecture:

Logs / Metrics / Container Events ↓ Splunk Index ↓ AI Agent (Splunk‑hosted LLM) ↓ Failure Origin + Propagation Reasoning ↓ Blast Radius & Impact Insights ↓ Dashboards & Investigations

Key Components

Splunk App Backend (Python)
Coordinates searches, extracts cross‑service context, and manages enrichment workflows.
Splunk REST APIs
Used to retrieve telemetry and persist AI‑generated impact analysis.
Splunk‑Hosted LLM Capabilities
Perform reasoning, summarization, and causal explanation without relying on external LLM APIs or paid subscriptions.
Agentic Reasoning Workflow
The AI agent:
1. Observes failure signals across services
2. Infers dependency‑driven propagation paths
3. Explains downstream impact and blast radius

All intelligence is delivered within Splunk, aligning directly with the Agentic Ops vision.

Why This Is Agentic Ops

Traditional observability answers “What broke?”
BlastMap focuses on “How did this failure spread, and what does it impact?”

The AI agent reasons over relationships between services, infrastructure, and time, producing explanations and impact assessments, not just alerts.

We frame incident response efficiency as:
Time to Recovery = Time to Detection + Time to Understanding + Time to Impact Assessment + Time to Remediation

By using LLMs for causal reasoning and blast‑radius analysis, BlastMap directly reduces Time to Understanding and Time to Impact Assessment, accelerating overall recovery.

Challenges we ran into

Correlating noisy, heterogeneous telemetry across services required careful context selection.
Designing LLM prompts that produce consistent and defensible causal explanations was challenging.
Working with evolving Splunk AI and agentic features required experimentation and iteration.
Balancing architectural ambition with hackathon constraints meant focusing on clear, high‑value impact insights rather than full automation.

Accomplishments that we're proud of

Built a true Agentic Ops workflow on Splunk
We designed an LLM‑powered agent that reasons over logs, containers, and infrastructure data to explain how failures propagate across microservices—moving beyond alerting into impact‑aware operations.
Achieved blast‑radius analysis without external AI services
All reasoning and summarization is performed using Splunk‑hosted LLM capabilities, with no reliance on external APIs or paid LLM subscriptions.
Turned fragmented telemetry into a coherent incident narrative
BlastMap automatically transforms raw signals into clear explanations of failure origin, downstream impact, and affected services, significantly reducing manual correlation effort.
Demonstrated causal reasoning, not just detection
The system focuses on why an outage spread the way it did, producing human‑readable causal explanations rather than simple classifications or scores.
Delivered Splunk‑native enrichment and visualization
AI‑generated insights are written back into Splunk and visualized alongside raw data, preserving analyst trust and existing workflows.
Validated Agentic Ops as a practical observability pattern
The project shows how LLMs embedded in Splunk can act as operational copilots, helping engineers reason under pressure during real incidents.

What we learned

Failure propagation is as much a reasoning problem as a data problem.
LLMs are particularly effective for causal explanation and impact summarization in observability workflows.
Agentic systems provide the most value when tightly coupled with the operational data platform.
Clear, explainable AI output is critical for engineer trust during incidents.

What's next for BlastMap: AI Driven Failure Propagation & Impact Analyzer

BlastMap demonstrates how agentic AI can transform observability into impact‑aware operations by helping teams understand not just what failed, but why it mattered.

Future work includes: