SentryOps Copilot

Inspiration

The phrase "human in the loop" has become a compliance checkbox in enterprise AI deployments. Teams add it to architecture diagrams and feature lists, and then the gate gets bypassed the first time it slows down an incident response. The problem is that most implementations enforce the human gate at the UI layer — a button that can be skipped, an approval that can be configured away. The Splunk Agentic Ops Hackathon asked builders to change that. If AI agents are going to operate autonomously in enterprise security and observability stacks, the safety constraints need to be architectural, not behavioural.

What it does

SentryOps Copilot is an autonomous Splunk security-ops agent that drives a real Splunk MCP Server at runtime, with a human-approval gate enforced at the MCP boundary. When a Splunk alert fires, the copilot triages it live: it generates SPL from the alert, runs it against the Splunk instance through the MCP Server's run_splunk_query tool (returning the real indexed events), resolves the affected service's downstream dependencies through an inputlookup, and computes an anomaly confidence with an SPL z-score over the events. It then surfaces the finding and a proposed remediation to an operator. The gate does not allow execution until the operator explicitly confirms — and that confirmation is a signed warrant the agent cannot forge.

After approval, the action is applied through the MCP surface and the full decision trace is recorded in a tamper-evident audit chain. The gate is not implemented in the UI — it is a hard constraint at the MCP boundary. The tool that executes remediation requires a signed approval token generated only by the operator's confirmation UI. The orchestrator cannot generate this token itself.

How we built it

The orchestrator talks to a Splunk MCP Server over JSON-RPC. For the demo we run a stdlib MCP server (server/splunk_mcp_server.py) that exposes a live Splunk Enterprise 10.4 instance — every run_splunk_query executes real SPL against Splunk and returns the real indexed events, so the submitted demo genuinely uses Splunk at runtime with no simulated output. The agent's entire read surface resolves to real SPL on the live instance: search returns the correlated events, the service-dependency map is an inputlookup, and the anomaly confidence is an SPL z-score computed by Splunk (peak per-source failure rate vs. baseline) — not a number the agent invents. NL→SPL is a deterministic rule in the MCP server; a production deployment can swap it for the Splunk AI Assistant, and the z-score for a Splunk hosted model, without touching the gate.

The remediation execution tool requires a signed operator warrant — an HMAC-SHA256 of the proposed action payload, generated client-side in the approval UI and verified server-side at the MCP boundary before execution. A warrant minted for one action cannot authorize a different one, because the signature is bound to the exact action payload. Every boundary action is also linked into a tamper-evident HMAC audit chain: altering any historical entry invalidates every entry after it, so a judge can replay the trace and confirm nothing was inserted or edited.

Runtime proof (no simulated Splunk output)

In the live run the agent's MCP search returns 4 real events from index=secops host=billing-api-07 on a live Splunk Enterprise 10.4 instance; the service map resolves 3 downstream dependencies via inputlookup; and the anomaly confidence (0.90) is computed by Splunk with an SPL z-score, not by the agent. All incident data is synthetic (no real hosts, customers, or schemas), but the Splunk integration is live — the demo makes genuine MCP calls to a genuine Splunk Server at runtime.

Challenges we ran into

The hardest property to get right was making the agent unable to fabricate a finding. Early versions could be coaxed, via prompts implying evidence existed, into emitting a finding from an empty result set. The fix was structural: every MCP tool response carries an explicit evidence_count, and the orchestrator is contractually forbidden from emitting a finding when evidence_count == 0. We then hardened the approval gate against prompt injection — a 40-attempt self-approval suite, all denied, because the gate is a cryptographic check, not a prompt rule.

Accomplishments that we're proud of

The full flow — alert arrives, the agent triages against live Splunk, operator approves, action executes, trace exports — runs end to end, and the safety properties are proven by tests, not asserted: a 7-test suite covers warrant verification, action-binding, tamper rejection, the 40-attempt injection suite, evidence-gated findings, and audit-chain tamper detection. All green. The accomplishment isn't that the demo is fast — it's that the safety property is provable and the Splunk integration is real.

What we learned

Building on Splunk's MCP Server forced an architectural discipline that direct API access would have bypassed. Every action the agent can take must be expressed as an MCP tool call with a defined schema and documented preconditions, which makes the agent's capability surface explicit — there are no implicit paths through the system. The discipline the MCP protocol imposes turns out to be the same discipline that makes an agent trustworthy in production: the boundary design is the product; the orchestrator is just the interface.

What's next

A configurable autonomy dial: below a confidence threshold the copilot escalates to a human-review queue; high-confidence, low-risk actions can be fast-tracked; novel or high-risk scenarios always surface to a human. And the natural Splunk-native upgrades the boundary already anticipates: the Splunk AI Assistant for NL→SPL and a Splunk hosted model for anomaly scoring, slotted in behind the same gate.

Built With

• hmac-sha256
• model-context-protocol
• python
• spl
• splunk-enterprise
• splunk-mcp-server