Veritrace - autonomous SOC analyst for Splunk

Inspiration

Security teams want to hand Tier-1 triage to AI agents, because alert volume is crushing and analysts burn out on repetitive investigations. The blocker is trust. In security, an agent that takes actions you cannot audit is a non-starter: if software says "this is a breach and I locked the account," an analyst has to see exactly why, check the evidence, and stand behind it in an incident review. I built Veritrace to solve that trust problem head on: it does the Tier-1 investigation autonomously, and it proves every step inside Splunk.

What it does

Veritrace investigates a Splunk alert the way a careful analyst does, and proves every step.

• It pulls evidence from Splunk over the Splunk MCP Server and discovers the attack from the data. Each stage is a real detection that finds the malicious entity by behaviour and correlation, not by assuming it: the brute-force source by failure-to-success ratio, the compromised host by the suspicious commands the account ran, the lateral-movement target by a subsearch that keeps only the SMB peer that also exfiltrates to the attacker, and the C2 domain by the DNS answer that resolves to the attacker. Each discovered entity feeds the next detection.
• It reasons with the Foundation-Sec cybersecurity model, which interprets each real result, maps it to MITRE ATT&CK, scores its own confidence, and renders the verdict.
• It writes its full chain of evidence, reasoning, confidence and actions back into Splunk as a tamper-evident, replayable ledger. Every step is hash-chained to the one before it, so the ledger is a verifiable chain of custody: you can re-read the steps straight out of Splunk and prove the investigation was not altered. The agent is a glass box, not a black box.
• It proposes reversible containment for a human to approve and never runs a destructive action on its own.
• It ships detection-as-code: a higher-fidelity correlation detection that would have caught the incident as one signal, backtested against history, deployable as a savedsearches.conf stanza.

It is not a one-scenario demo. The bundled data embeds two distinct intrusions with entirely different attackers, accounts, hosts and C2 domains, and the same engine solves each by discovering its attack from the data.

How I built it

• Agent core in Python, built around the Splunk Python SDK and splunklib.ai agentic patterns: a triage, reason-and-pivot, verdict, containment and detection loop.
• A purpose-built Splunk MCP Server with FastMCP that exposes safe, typed investigation tools, each behind an SPL guardrail that blocks write, execute and export commands. The agent reaches Splunk only through these MCP tool calls, which is what makes its access auditable.
• A detection-driven investigation engine that discovers the attack entities by correlation rather than assuming them.
• Pluggable Foundation-Sec reasoning: the model runs locally through Ollama or vLLM, the Splunk-hosted model through the AI Toolkit ai SPL command on Splunk Cloud, or a deterministic replay provider for tests and offline runs.
• A tamper-evident ledger written into Splunk over HTTP Event Collector, each step hash-chained to the previous, re-read and recomputed to prove integrity.
• A React console that streams the investigation live over Server-Sent Events: reasoning timeline, MITRE kill chain, live attack map, detection diff, containment approvals, and a ledger integrity check.
• A one-command Docker stack, and a Splunk app that passes App Inspect with zero failures.

Challenges I ran into

The hardest lesson was knowing what to trust a small local model with. An 8B model interprets evidence and maps MITRE well, but it is unreliable at authoring correct multi-stage SPL: left to write its own pivots it chased the wrong hosts and found nothing. The fix was to split the work: the agent runs detection-driven pivots that correlate on known anchors to beat the benign noise, and the model does the reasoning over the live results. Keeping the live demo, the offline run and the sample data consistent meant defining the reference incident in one place.

Accomplishments that I'm proud of

An autonomous agent that earns trust instead of asking for it. Every action is named, guarded and written back into Splunk, and the chain of custody is provable, not just present. It does not just close the ticket, it improves the detection that produced it, and it generalizes across incidents rather than being a single hard-coded scenario.

What I learned

Agentic operations in security live or die on auditability. The interesting work is not making an agent act, it is making every action provable.

What's next

Multi-alert correlation across concurrent investigations, learning which pivots were most decisive to tune the playbook, and pushing approved detections through a review and version-control workflow.

Built With

• docker
• fastapi
• fastmcp
• foundation-sec
• http-event-collector
• model-context-protocol
• ollama
• python
• react
• splunk
• splunk-app-inspect
• splunk-mcp-server
• splunk-python-sdk
• splunklib.ai
• tailwind-css
• typescript
• vite