SPL Guard

Inspiration

The inspiration for SPL Guard came from a massive structural shift happening right now in enterprise operations. With the release of the Splunk Model Context Protocol (MCP) Server v1.2, organizations can finally connect autonomous AI agents directly to their operational data streams. However, during development and testing, we observed a critical operational bottleneck: autonomous agents often generate unoptimized, non-deterministic SPL—firing wildcard index searches ($index=*$) or completely unbounded historical time ranges ($earliest=-30d$ or $alltime$).

In a production enterprise environment, this isn't just inefficient—it introduces severe search head concurrency risks and drives unintended Splunk Virtual Compute (SVC) consumption across the environment. While native platform rate limiting exists, it acts as a blunt instrument that drops requests and breaks the agent's workflow. We realized that the ecosystem didn't need another passive monitoring dashboard; it needed an inline, active runtime optimization and governance layer. That is why we built SPL Guard.

What it does

SPL Guard intercepts every SPL query an AI agent sends to Splunk, rewrites dangerous ones before they execute, and records every decision back into Splunk natively — keeping agents productive and search heads stable without manual intervention.

How we built it

We engineered SPL Guard as a transparent, high-performance MCP-to-MCP reverse proxy that sits directly between the LangGraph agent framework and the Splunk MCP Server.

The architecture is entirely modular, split across five highly optimized layers:

1. Layer 1 (The Interceptor Proxy): A high-performance, asynchronous FastAPI application that exposes a single /mcp endpoint. It intercepts native JSON-RPC tools/call envelopes designated for Splunk search tools while letting lifecycle traffic pass through completely transparently.

2. Layer 2 (SPL Intelligence Engine): Built entirely with high-speed, deterministic regex inspection. By deliberately avoiding secondary LLM calls in the critical path, we keep execution latency under $3\text{ms}$ while processing a three-tier verdict system: SAFE, REWRITTEN, or BLOCKED.

3. Layer 3 (Query Memory Cache): A local SQLite caching layer keyed by a cryptographic hash of the raw query, $H = \text{SHA-256}(\text{raw_spl})$. If an agent repeats a dangerous query structure, it bypasses re-inspection and instantly serves the approved rewrite from cache.

4. Layer 4 (Splunk-Native Audit Trail): To remain completely secure, SPL Guard operates over a single channel using Splunk's own encrypted authentication token—no side channels, no second credentials. It utilizes the native collect command to log all proxy decisions right back into an audit index.

5. Layer 5 (FinOps Dashboard): A Streamlit-based operational UI that merges local SQLite metrics with live Splunk _internal telemetry to display real-time SVC units saved and active query mitigation counts.

Challenges we ran into

primary challenge was solving the core trade-off between governance and latency. An LLM-based rewriter would add 1–3 seconds of latency to every query in the critical path — unacceptable in a live multi-agent workflow. I made a deliberate architectural decision to use a deterministic regex rule engine instead, keeping inspection latency under $3\text{ms}$ with fully auditable, reproducible decisions on every call.

A second hurdle was ensuring total transparency. I had to carefully unpack and rebuild the JSON-RPC envelope on the fly without changing the underlying message IDs, ensuring that neither the LangGraph agent nor the Splunk MCP Server was aware an intermediary had modified the payload.

The correct solution was routing all Splunk communication — query forwarding, telemetry reads, and audit writes — through a single MCP channel with one encrypted token. This is not a limitation — it is a deliberate security boundary. One token means one audit surface, one revocation point, and zero credential sprawl across the proxy layer.

Accomplishments that we're proud of

🏆 Accomplishments I'm Proud Of

Solving a problem Splunk documented but didn't fix. SPL Guard directly addresses the resource contention issue that drove Splunk to ship rate limiting as a GA feature in MCP Server v1.2. The gap between observing bad queries (MCP Watch) and preventing them (SPL Guard) was real and unoccupied. Filling it with a working, tested proxy felt significant.

Sub-3ms inspection latency with zero LLM dependency. Keeping the entire SPL inspection and rewrite pipeline under $3\text{ms}$ — using a deterministic regex rule engine with no secondary model call in the critical path — proved that governance does not have to cost performance. Every decision is auditable, reproducible, and fast enough to be invisible to the agent.

A genuinely single-channel architecture. Routing all Splunk communication — query forwarding, telemetry reads, and audit writes — through one encrypted MCP token and one channel was an architectural discipline, not a shortcut. One token means one audit surface and one revocation point. No credential sprawl anywhere in the proxy layer.

18/18 tests passing before submission. The SPL inspector covers the full verdict and risk classification matrix — safe queries, five categories of rewriteable patterns, three blocked patterns including the CVE-2025-20381 subsearch injection signature, and hash stability for the memory cache. Every test passing before a line of the submission was written mattered.

The learning layer. Most governance tools are stateless — they inspect the same bad query a thousand times. SPL Guard's SQLite cache keyed by $H = \text{SHA-256}(\text{raw_spl})$ means the proxy gets smarter with every session. The same unsafe pattern is never inspected twice. That compound improvement over time is the detail I am most proud of technically.

Splunk as the audit store, not an afterthought. Writing every proxy decision back into Splunk via a collect SPL command through the MCP channel — rather than logging to a file or an external system — means platform teams get a native, queryable audit trail in the system they already trust. Splunk governs the agents. The agents do not govern themselves.

What we learned

Determinism is a feature, not a fallback. The instinct when building AI infrastructure is to solve every problem with another model call. SPL Guard taught me that in the critical path — where latency is measured in milliseconds and consistency is a compliance requirement — a deterministic rule engine outperforms an LLM on every dimension that matters: speed, auditability, reproducibility, and cost.

The protocol layer is where governance belongs. Observability tools watch what agents do after execution. Rate limiters react when thresholds are already breached. Neither changes the outcome. Sitting at the MCP protocol layer — intercepting the tools/call envelope before it reaches Splunk — is the only position in the stack where you can correct behaviour in real time without breaking the agent's workflow.

Constraints produce better architecture. The single-channel constraint — Splunk's encrypted MCP tokens are exclusively for MCP authentication and cannot be used for direct REST calls — forced every operation through one channel. What felt like a limitation early in the design produced an architecture with a smaller security surface, zero credential sprawl, and a cleaner audit trail than the original multi- connection design would have.

Splunk's own data is the right observability source. Routing telemetry reads against index=_internal through the MCP channel rather than opening a side REST connection meant the FinOps dashboard is sourced from the same data plane platform teams already monitor. No custom metrics pipeline. No external dependency. The system observes itself through Splunk — which is exactly what Splunk is for.

Enterprise AI governance is an infrastructure problem. The assumption going in was that making agents smarter would reduce unsafe queries. The reality is that non-deterministic SPL is a structural output of how LLMs generate queries — not a reasoning failure that better prompting fixes. Governance has to be enforced at the infrastructure layer, not requested at the model layer. That distinction shaped every design decision in SPL Guard and is the principle I will carry into every agentic system I build from here.

What's next for SPL Guard

HITL Intercept Mode — v1.1 The governance toggle is already in the dashboard UI. The next step is wiring it fully into the LangGraph interrupt() pattern so high-risk queries pause at the proxy and surface an Approve / Deny decision to the platform team before execution. This closes the loop between automated optimization and human oversight — the two modes every enterprise security team needs.

Semantic Query Memory — v1.2 The current SQLite cache matches on exact query hashes — $H = \text{SHA-256}(\text{raw_spl})$. The next evolution replaces exact matching with semantic similarity using Qdrant and Gemini embeddings. Queries that are structurally similar but not identical — index=* sourcetype=access_combined and index=* sourcetype=web_logs — would map to the same approved rewrite pattern. The proxy learns families of bad queries, not just individual instances.

Splunkbase App Packaging — v2.0 SPL Guard's architecture is designed to install alongside the Splunk MCP Server as a native Splunk app. A configuration UI in Splunk Web would let platform administrators set index allowlists, time range budgets, per-role SVC limits, and HITL thresholds — without touching a config file. This is the path to distribution across Splunk's enterprise customer base.

Multi-Platform Query Governance — v3.0 The SPL Intelligence Engine is the domain-specific layer. The proxy core — intercept, inspect, rewrite, cache, audit — is platform-agnostic. Expanding the rule engine to cover Elastic Query DSL, Datadog Log Query Language, and PromQL would make SPL Guard the governance layer for AI agents across any observability platform, not just Splunk. Same architecture, swappable rule engines.

The Broader Principle As agentic AI moves from experimentation to production, every enterprise will face the same structural problem SPL Guard solves for Splunk today: autonomous agents interacting with operational systems without explicit boundaries. The inline governance layer pattern — deterministic, protocol-level, learning across sessions — is the architecture that makes that transition safe. SPL Guard is the first implementation. It will not be the last.

Built With

• cryptography
• docker
• fastapi
• gemini
• google-gemini-api
• httpx
• json-rpc
• langchain
• langchain-google-genai
• langgraph
• langsmith
• mcp
• opentelemetry
• opentelemetry-exporter-otlp-proto-grpc
• otlp-grpc
• pydantic
• pydantic-settings
• python
• python-dotenv
• splunk-ai-assistant
• splunk-internal-index
• splunk-mcp-server
• splunk-observability-cloud
• splunk-otel-collector
• splunk-python-sdk
• sqlite
• streamlit
• uvicorn
• wsl2