sockeye

Autonomous, evidence-backed SOC triage on the official Splunk MCP Server.

Comment

Inspiration

Most "AI for security" demos start after the hard part is over: an alert already fired, and the model just summarizes it. But the real grind in a SOC is tier-1 triage; the human judgment of deciding whether an alert is even real, whether the access actually succeeded, what happened next, and whether you can defend that conclusion to an incident commander.

That work is mostly pattern, pivot, and proof. It looked like exactly the shape of problem an agent should own end to end; not summarize, but investigate. The catch is trust: an analyst can't act on a confident paragraph with no evidence behind it. So we set out to build an agent that does the investigation and leaves a verifiable trail.

The needle-in-a-haystack framing makes the stakes concrete. Our demo buries a full intrusion inside ~1,976 events over 36 hours, where exactly one malicious login succeeds:

$$ p_{\text{breach}} = \frac{1\ \text{successful compromise}}{1{,}976\ \text{events}} \approx 5.1 \times 10^{-4} $$

A useful agent has to find that signal, prove it, and ignore the louder-but-benign noise around it.

What it does

Sockeye is an autonomous SOC analyst. It connects Claude to a live Splunk Enterprise instance through the official Splunk MCP Server, then runs a real investigation:

  1. Sweeps the index by sourcetype and volume to get the lay of the land.
  2. Measures authentication failures by source IP and account.
  3. Pivots on outcome did any suspicious attempt actually succeed?
  4. Follows the compromised identity into privilege escalation, lateral movement, data staging, and outbound transfer.
  5. Emits a Markdown report: verdict + severity, timeline, IOCs, evidence, and ranked containment actions.
  6. Appends an Execution Audit that the runner generates from the actual MCP tool calls; the exact SPL behind every claim, not model prose.

In the demo it correctly separates a distributed password spray (220 failures, zero successes) from a focused brute force against svc-backup (180 failures, one success), then traces that account through sudo su -, a first-ever login on the file server, archival of finance/HR directories, and $570{,}904{,}170$ bytes ($\approx 570.9$ MB) exfiltrated to the original attacker IP. The prompt teaches the agent the method, never the planted answer.

How we built it

  • Splunk Enterprise 10.4 in a pinned Docker environment.
  • Splunk MCP Server 1.2.0, which exposes a read-only splunk_run_query tool over MCP streamable HTTP.
  • The Claude Agent SDK drives the investigation loop. This was a deliberate choice: it runs on the local Claude Code CLI, so the whole thing works on a Claude Pro/Max subscription with no API key — while still supporting an ANTHROPIC_API_KEY for containerized deployment.
  • scripts/seed_data.py generates a deterministic 36-hour scenario (fixed RNG seed, stable event_ids) so the demo is reproducible and safe to re-ingest.
  • scripts/setup_splunk.sh installs the app, creates index=security, provisions a dedicated least-privilege role and user, and mints a 30-day encrypted MCP token for that identity.
  • The runner fails closed: it validates a successful SDK result and the required report sections before atomically writing the artifact. ## Challenges we ran into
  • The token contract. The MCP Server doesn't accept a generic Splunk JWT; it requires an RSA-encrypted token with audience mcp, minted through the app's own /services/mcp_token endpoint. Discovering that meant reading the handler source.
  • An RBAC footgun. Setting role capabilities via the REST role endpoint replaces the entire capability list; we briefly stripped admin's own rights. The fix was to ship capabilities declaratively in a dedicated app's authorize.conf instead of mutating roles at runtime.
  • A subtle secret leak. The Agent SDK serializes MCP server config — including headers — into the CLI's --mcp-config argument, which means a literal bearer token would be visible in the process list. The correct fix was to put a ${SPLUNK_MCP_TOKEN} placeholder in the header and let the CLI expand it from the subprocess environment at runtime, keeping the token out of argv entirely. We verified end-to-end that the agent still authenticates this way.
  • TLS honesty. Splunk's default certificate fails Node's hostname check, so a process-wide TLS bypass was the wrong answer. Instead we bind every port to loopback and use plain HTTP only for the local management/MCP port in this disposable demo — and require HTTPS for any remote URL.
  • Failing closed. An agent run can return an error result that still carries partial text. Sockeye now refuses to write an error as if it were a finished report. ## Accomplishments that we're proud of

  • It investigates — it doesn't follow a script. The agent forms hypotheses, writes its own SPL, reads the results, and pivots. We give it the method, never the planted answer, and it still reconstructs the full attack chain.

  • Every claim is provable. The runner appends an Execution Audit built from the actual MCP tool calls — the exact SPL submitted to Splunk, independent of what the model says in prose. An analyst can re-run the evidence instead of trusting a paragraph.

  • It found the needle. Inside ~1,976 events it cleanly separated a noisy password spray (220 failures, 0 successes) from the one focused brute force that actually breached svc-backup, then traced escalation, lateral movement, staging, and ~570.9 MB of exfiltration to the original attacker IP.

  • MCP as a real security boundary. The agent authenticates as a dedicated, expiring identity — not the Splunk admin — scoped server-side to index=security with a single read-only tool, all ports loopback-bound. Least privilege end to end.

  • No API key required. By building on the Claude Agent SDK, the whole agent runs on a Claude Pro/Max subscription — while still supporting an API key for containerized deployment.

  • We closed a real secret leak. We discovered the SDK serializes MCP headers into the CLI's argv, then fixed it with runtime ${SPLUNK_MCP_TOKEN} expansion and verified end-to-end that the token never touches the process list.

  • It's actually reproducible. Deterministic seeded data, one-command setup, a pinned dependency lockfile, unit tests, linting, dependency auditing, Compose validation, CI, and a root-level architecture diagram — a judge can clone and run it.

    What we learned

    The biggest lesson is that MCP is most valuable when you treat it as a security boundary, not a convenience layer. The pattern that worked: an expiring identity, narrow RBAC, explicit client-side tool allowlisting, server-side query scoping (the agent can only ever touch index=security), and an execution trail a human can re-run. Agentic autonomy and auditability aren't in tension — the audit trail is what makes the autonomy safe to trust.

We also learned how much rigor the "boring" plumbing demands: deterministic data, reproducible setup, and verifying claims (like "the token stays out of argv") by actually running them rather than assuming.

What's next for sockeye

  • Scheduled runs that diff today's verdict against the previous one.
  • Writing reports back into Splunk for dashboards and analyst review.
  • Evaluation fixtures that score detection rate, evidence fidelity, and hallucinated claims.
  • Parallel sub-investigations per high-priority account or source IP.

Built With

  • anthropic
  • claude
  • claude-agent-sdk
  • claude-code
  • css
  • docker
  • docker-compose
  • fastapi
  • github-actions
  • hec
  • html
  • javascript
  • mcp
  • model-context-protocol
  • pydantic
  • pytest
  • python
  • server-sent-events
  • spl
  • splunk
  • splunk-enterprise
  • splunk-mcp-server
  • uvicorn
Share this project:

Updates