mike_splunk_native_agent

Splunk-native agent that captures SOC analyst reasoning, not just events, and serves it back as SPL commands, a CLI, and MCP tools for AI agents.

Comment

Inspiration

When a senior SOC analyst with six years of tenure leaves, they take their mental model with them. Which alerts are the Monday finance batch job. Which subnets are sanctioned pentest ranges. Which correlation rules were never updated after the 2019 network re-segmentation. None of this lives in a system. It lives in human heads. SIEMs capture events. Nobody captures why analysts decide what they decide. IMA fixes that.

Existing SIEM tooling captures events. Nobody captures analyst reasoning at scale and turns it into queryable organizational memory. The closest thing in the market is expensive security data lake consulting projects. We wanted to build it natively inside Splunk, as a first-class data surface, so that when someone like Mike walks out the door, the knowledge doesn't leave with them.

What it does

IMA watches alert dispositions, prompts analysts for a 10-second "why?" on each closure, clusters those annotations by (event_type, disposition) through Foundation-Sec-1.1-8B, and writes structured knowledge entries into Splunk KV Store. That institutional memory then surfaces anywhere an analyst works:

  • | imaannotate records a disposition and reason from any SPL search
  • | imabuild clusters annotations and synthesizes knowledge via LLM
  • | imaquery question="finance Monday" retrieves relevant patterns in plain English
  • | imaaboutasset asset="acct-prod-01" pulls per-asset behavioral memory
  • ima mcp serve exposes the entire knowledge graph as MCP tools for Claude Desktop, SAIA Agent Mode, or any MCP-compatible AI agent

A modular input (ima_autobuild) keeps the graph current autonomously on a configurable interval without any manual intervention.

How we built it

  • Python CLI (Typer + Rich) as the dev harness, with full command coverage for annotate, watch, build, query, and MCP serve
  • Three Splunk KV Store collections (ima_annotations, ima_knowledge, ima_assets) as the persistence layer, declared in collections.conf and initialized via ima kv init
  • Custom Search Commands using the Python Splunk SDK 3.0 chunked protocol, making IMA callable from any saved search, dashboard, or Splunk alert action
  • Foundation-Sec-1.1-8B for structured JSON extraction from informal analyst text; Ollama-hosted Llama-3.1-8B as a local dev stand-in (one .env flag switches to the Splunk-hosted endpoint)
  • nomic-embed-text for vector embeddings and cosine similarity search backing | imaquery
  • FastMCP server (ima/mcp_server.py) exposing five tools over stdio for Claude Desktop or streamable-HTTP for remote agents
  • Simple XML dashboard (ima_overview.xml) giving the SOC a single pane: contributor stats, disposition mix, knowledge table, and an interactive "ask the agent" input

The central architectural decision: annotation is instant and cheap, persisted directly to KV Store with zero added latency. LLM synthesis is a batch step, run on demand or on a schedule.

Challenges we ran into

SPL tokenizes underscores as internal field prefixes, so | ima_query silently fails as "unknown command 'ima'". Custom search command stanza names must be concatenated (imaquery, not ima_query). This produced maximally unhelpful HTTP 400 errors with no traceback and took a full debug cycle to isolate.

Splunk 10.x does not bundle splunklib for custom search commands, and a failed import produces error code 1 with no Python traceback in any indexable log. The fix is vendoring the full SDK into bin/lib/, including the splunk_sdk-3.0.0.dist-info folder that splunklib queries at runtime via importlib.metadata during the protocol handshake.

Microsoft Store-packaged Claude Desktop ignores the mcpServers block in claude_desktop_config.json entirely; it uses a DXT connector system gated by an account allowlist. MCP integration testing required the standalone installer from claude.ai/download.

Accomplishments that we're proud of

The confidence scoring from cluster size is elegant: one annotation is an observation (confidence ~0.1-0.35), three or more is an institutional pattern (confidence ~0.9-1.0). The batch synthesis architecture keeps analyst annotation latency at zero. And full MCP exposure means any AI agent, not just Splunk users, can query and update institutional memory through a standard protocol with no Splunkbase install required.

If even 20% of analyst expertise is captured before someone leaves, it reduces mean time to investigate (MTTI) measurably and cuts false positive closure time for new analysts who inherit the environment cold.

What we learned

Foundation-Sec is well-suited to structured extraction from informal analyst shorthand. KV Store as a knowledge graph substrate works better than expected at this scale. And the hardest design problem was not technical: it was keeping the annotation prompt short enough that analysts actually fill it in while an alert is still fresh.

What's next for Mike

  • Full population of the ima_assets graph from aggregated annotations, enabling richer per-asset behavioral exception tracking
  • Slack and Teams integration so annotation happens where analysts already work, without a context switch to Splunk Web
  • Multi-analyst conflict detection: surfacing alert types where analysts historically gave different dispositions as a first-class knowledge signal

Built With

  • fastmcp
  • foundation-sec-1.1-8b
  • llama-3.1-8b-instruct
  • model-context-protocol-(mcp)
  • nomic-embed-text
  • ollama
  • python
  • python-splunk-sdk-3.0
  • simple
  • splunk-enterprise-10.4
  • splunk-kv-store
  • typer
  • xml
Share this project:

Updates