Splunk IR Triage Agent

Inspiration

SOC analysts drown in alerts. Traditional SOAR runs static playbooks that either escalate everything or suppress everything. We wanted an agent that thinks like a tier-1 analyst: pull context, weigh evidence, and say "I don't know" when data is missing.

What it does

On alert fire, the agent receives the Splunk alert payload, autonomously queries Splunk for surrounding events, entity history, and historical alert patterns via the Splunk MCP Server, then emits a structured JSON triage card with classification, severity, a discrete recommended action (escalate / contain / investigate / suppress), calibrated confidence score, and explicit uncertainty flags.

How we built it

• Splunk MCP Server (Splunkbase app #7931, v1.1.3) exposes 10 read-mostly tools at the Splunk management port
• Gemini 2.5 Flash via Vertex AI handles ambiguity reasoning through function-calling loops
• Python orchestrates the agent loop: alert in, tool calls, triage card out
• Agent autonomously decides which Splunk queries to run (typically 2-5 tool calls per alert)

Challenges we ran into

Getting the agent to be honest about missing data instead of hallucinating findings. Most LLM-on-SOC demos invent a story when Splunk returns empty results. We tuned the system prompt to treat "no data found" as a valid and important signal.

What we learned

The Splunk MCP Server makes it trivial to give an LLM agent read access to Splunk. The harder problem is trust: structured output with calibrated uncertainty is what makes the difference between a demo and something an analyst would actually use.

Built With

• context
• mcp
• model
• splunk
• vertex