SplunkGuard

🏆 Splunk Agentic Ops Hackathon — Observability track + Best Use of Splunk MCP Server.

Inspiration

Splunk indexes everything. The hard part is asking the right question. You know the answer is in there. You just don't know the SPL.

What it does

SplunkGuard is a Gemini-driven agent that investigates operational questions against your Splunk data. Ask "What CI pipelines failed last night and why?" or "Are there auth anomalies in the last 24h?" — Gemini queries Splunk (via the official MCP Server or REST), returns a structured SplunkInvestigationReport with root cause, failure category, time range, affected components, and recommended actions (each with paste-ready SPL).

Both paths verified end-to-end against a real Splunk Enterprise instance with ~294 events (30 pipelines + ~264 jobs) ingested from the public gitlab-org/cli project:

How we built it

Agent core (pipelineguard/splunk_agent.py). Gemini 2.5 Flash drives Splunk investigations. Two backends:

• SplunkDirectBackend — POST /services/search/jobs → poll → GET results. Bearer token auth. One Gemini call per investigation. Fast loop.
• SplunkMCPBackend — talks to Splunkbase App #7931 ("Splunk MCP Server", official, beta) over the Streamable HTTP transport (MCP 2025-06-18 spec). Fully dynamic tool discovery via list_tools() — the agent never hardcodes Splunk MCP tool names; whatever the server advertises is what Gemini calls. Auth: Authorization: Splunk <encrypted-token>. Iterative tool-call loop capped at 15 iterations.

Structured output: the system prompt constrains Gemini to produce a typed SplunkInvestigationReport (root_cause, investigation_category, affected_components, time_range, recommended_actions[]). Each recommended action carries an optional spl_query field so the output is wire-ready for Slack/PagerDuty automation.

Ingester (pipelineguard/ingesters/gitlab_to_splunk.py, 257 LOC, 31 tests). Reads GitLab pipelines + jobs via python-gitlab, attaches log tails (last 50 lines) for failed/canceled jobs, posts batches of 100 events to Splunk HEC. Verified at 294 events / 37s against real data.

Pre-submission audit caught 3 HIGH-severity bugs in the MCP backend before judges would have hit them:

1. verify_ssl=False was silently dropped — built an ssl.SSLContext but never passed it to the MCP SDK's transport. Fixed by wiring a custom httpx_client_factory.
2. Prompt tool-name drift — old names like run_splunk_query didn't match Splunkbase App #7931's actual advertised surface (splunk_run_query, splunk_run_saved_search, …). Fixed + added "use exact advertised names — do not guess" instruction.
3. Wrong MCP transport — App #7931 uses Streamable HTTP, but we were using mcp.client.sse.sse_client which does GET → got HTTP 405. Swapped to mcp.client.streamable_http.streamablehttp_client. This is what unlocked the MCP-mode demo (commit 7edeb95).

Challenges we ran into

• The "App #7931 won't boot on Splunk 10.4" scare was a misdiagnosis. Install reported success, then splunkd appeared to refuse to restart on Splunk Enterprise 10.4.0. The real cause wasn't a version conflict — it was the in-container splunk restart command (see next bullet). With docker restart splunk, App #7931 boots cleanly on both 10.4.0 and 9.4.11. We recorded the MCP-mode demo on 9.4.11 and the REST-mode demo on 10.4.0; SPLUNK.md documents both.
• splunk restart doesn't work in Docker — it kills splunkd but never re-spawns it (no log output after the Interrupt). Use docker restart splunk instead — Docker's entrypoint script handles boot correctly. This was the actual blocker, and it cost us ~30 minutes (and a false "10.4 incompatibility" conclusion) before we tried the container-level restart.
• MCP transport mismatch — initial code used the SSE transport and got HTTP 405 from Splunk's POST-only endpoint. Real fix: swap to the Streamable HTTP transport (the modern MCP spec).
• Splunk container permission model — docker exec defaults to root but Splunk runs as the splunk user; running install as root once corrupts /opt/splunk/var ownership and bricks the container. Documented in SPLUNK.md.
• Distinguishing user-canceled jobs from real failures — solved with the investigation_category enum (anomaly / threshold_breach / pipeline_failure / security_event / performance_degradation / data_gap). The MCP-mode demo correctly identifies a script_failure failure_reason that --direct mode missed.

Accomplishments that we're proud of

• Both paths verified end-to-end — --direct REST and MCP. Same agent, same structured output, same SPL recommendations format. The user picks the trade-off (fast vs. deep).
• MCP path is real, not vaporware. 37.84s, 2 real MCP tool calls, real Splunkbase App #7931 v1.1.0. The agentic loop autonomously identifies specific failing job names that --direct's single-call mode missed.
• 100% dynamic MCP tool discovery — the agent never hardcodes a Splunk MCP tool name. Forward-compatible with whatever tools the Splunk team ships next.
• Bundled GitLab ingester (257 LOC, 31 tests) — judges can reproduce the whole pipeline-failure scenario in ~5 minutes from a blank Splunk Docker container.
• Pre-submission 10-pass code audit caught and fixed 3 HIGH-severity bugs in the MCP backend (SSL, prompt drift, wrong transport) before judges would have hit them.
• Honesty about reproduction quirks. We document the Docker splunk restart foot-gun (which initially masqueraded as a Splunk 10.4 incompatibility), the root-permission trap, and the MCP transport switch — so a judge reproducing the demo doesn't waste time on the traps we hit.

What we learned

• The MCP transport matters. Splunkbase App #7931 uses the modern Streamable HTTP transport (MCP 2025-06-18 spec), not the older SSE transport. Picking the right mcp.client.* module is the difference between "works" and "HTTP 405".
• splunk restart in Docker is a foot-gun. It kills splunkd but never re-spawns it; you have to docker restart the container instead.
• Verify your restart method before blaming versions. What looked like an "App #7931 won't boot on Splunk 10.4" incompatibility was actually the splunk restart foot-gun. Once we used docker restart, the app booted on both 10.4.0 and 9.4.11 — confirming the integration boots (via docker restart) is step 0.
• Audit your code before judges do. Three HIGH-severity bugs (SSL silently dropped, prompt tool-name drift, wrong transport) would have caused silent failures. A systematic 10-pass audit caught all three.
• REST fallback is a real safety net, not a hedge. The same agent + same structured output + same prompt run unchanged in --direct mode — zero code changes to switch paths.

What's next for SplunkGuard

• Upstream a docs note / Splunkbase issue clarifying that App #7931 needs a container-level docker restart (not splunk restart) after install — the trap that nearly sent us downgrading.
• Auto-create Splunk saved searches for recurring patterns the agent finds.
• Slack / PagerDuty webhook output of investigation reports.
• Multi-index correlation (correlate CI failures with infrastructure metrics in a different index).
• Submit pipelineguard/backends/splunk_mcp.py upstream as a community reference for how to consume Splunk MCP Server with Gemini over Streamable HTTP.

Built With

• docker
• fastmcp
• gemini-2.5-flash
• google-genai
• httpx
• mcp
• mcp-server
• model-context-protocol
• observability
• python
• python-gitlab
• rich
• splunk
• splunk-enterprise
• splunk-hec
• splunk-mcp-server
• streamable-http