Splunk AttackGraph AI Agent: Autonomous SOC Investigations

Inspiration

In modern Security Operations Centers (SOCs), analysts suffer from severe alert fatigue. A single malicious campaign can trigger dozens of disparate alerts across firewalls, endpoint detection (EDR), and cloud infrastructure. Human analysts must manually stitch these disjointed logs together to form an attack narrative, a process that is slow, error-prone, and overwhelming.

We were inspired by the question: What if the SIEM could autonomously investigate its own alerts using a knowledge graph? We wanted to build an AI agent that doesn't just read logs, but mathematically maps relationships, traverses attack paths, and closes investigations without human intervention.

How We Built It

We built the Splunk AttackGraph AI Agent natively within Splunk Enterprise, heavily leveraging the new Model Context Protocol (MCP) to grant an LLM secure, decoupled access to our internal graph data.

  1. The Telemetry Engine: We configured savedsearches.conf to continuously hunt for malicious activity (DDoS, Brute Force, Malware, SSRF). When triggered, Splunk routes the JSON payload to our custom seed_graph.py alert action.
  2. The Graph Engine: The alert payload is dynamically parsed and injected into an in-memory knowledge graph powered by NetworkX. Entities (Users, IPs, Hosts) become nodes, and the alerts become the relational edges.
  3. The MCP Bridge: We exposed the graph data to the AI agent by building a custom Splunk REST API handler (mcp_handler.py) mapped in restmap.conf. The MCP server registers 14 dedicated tools (e.g., graph_generate_attack_path, graph_get_patient_zero) that the LLM can seamlessly invoke.
  4. The Agent Brain: Following a strict 7-step Standard Operating Procedure (SOP), the LLM queries the graph via MCP, identifies Patient Zero, hunts for missing SPL evidence, maps the attack to MITRE ATT&CK, and publishes a final Markdown/JSON incident report natively inside a Splunk Dashboard.

Mathematical Hypothesis Scoring

To prevent hallucinated verdicts, the AI Agent doesn't just guess the attack type, it scores hypotheses based on evidence weights embedded in the graph.

When calculating the probability of a specific attack Hypothesis ($H$) given the observed Evidence ($E$), the system conceptually utilizes a Bayesian-inspired summation model:

$$ P(H | E) = \frac{P(E | H) \cdot P(H)}{P(E)} $$

In practice, our MCP tool (graph_score_hypotheses) evaluates the confidence score $S$ for a given attack classification $C$ by aggregating the normalized weights $w_i$ of all evidence edges $e_i$ connected to Patient Zero:

$$ S_C = \sum_{i=1}^{n} \left( w_i \cdot \delta(e_i \in C) \right) $$

Where $\delta(e_i \in C)$ is an indicator function that equals $1$ if the evidence supports classification $C$, and $0$ otherwise. If $S_C$ exceeds the operational threshold, the agent confirms the hypothesis.

Challenges We Faced

Integrating deeply into Splunk's proprietary ecosystem presented several major hurdles:

  • Environment Isolation: Splunk runs background scheduled searches in a strictly sandboxed Python environment. We experienced hard crashes (ModuleNotFoundError) because external packages like python-dotenv were blocked. We solved this by using pip to statically bundle third-party dependencies natively inside the app's bin/ directory.
  • Platform Architecture & Tool Registration: Initially, grasping the proprietary internal routing of Splunk and figuring out how to register the MCP tools securely as custom REST endpoints was exceptionally tough. It required deep dives into configurations to ensure the LLM could seamlessly invoke tools.
  • State Contamination: Our LLM agent kept evaluating past threats because the knowledge graph was accumulating already-ingested data from older alerts. We solved this by implementing a graph_get_historical_investigations tool to archive past context, and enforcing a mandatory graph_reset wipe at the beginning of the system prompt to guarantee a clean slate for each new incident.

What We Learned

  • MCP is a Game Changer: Decoupling the AI logic from the Splunk backend using the Model Context Protocol allowed us to write clean, modular Python tools without worrying about complex LLM integration inside the Splunk core.
  • Graph Theory in Security: Representing security alerts as a graph (rather than tabular logs) makes it exponentially easier for an AI to identify lateral movement and root causes.
  • Splunk Internals: We gained deep, battle-tested knowledge of Splunk's internal routing, custom alert actions, savedsearches.conf, and its embedded Python environment quirks.

What's Next

The foundation is built. Next, we plan to implement bi-directional mitigation capabilities, allowing the AI Agent to not only generate incident reports but also execute MCP tools that automatically quarantine infected hosts and suspend compromised Active Directory accounts.

Built With

Share this project:

Updates