• 

  Running inside Splunk: the splunklib.ai agent's verdict rendered as native @splunk/react-ui controls via A2UI.

• 

  Verdict at a glance — COMPROMISED: 4 critical, 7 high, 15 MITRE techniques, as A2UI VerdictBadge, KPI cards and severity bar.

• 

  The reconstructed MITRE ATT&CK kill-chain: every YARA detection with severity chips and T-codes, as a native A2UI table.

• 

  Prioritized remediation the agent recommends — isolate the DC, reset krbtgt, hunt lateral movement — as native cards.

• 

  Hands-off SOC workflow: detect, then | ai triage, then a notable incident — the AI analysis is generated inside SPL.

• 

  Living-off-the-land binaries carved from memory (PowerShell, cmd, ntdsutil, PsExec) — Volatility3 data in the forensics index.

Find Evil — Project Story

💡 Inspiration

Memory forensics is slow, manual, and reserved for experts. When a domain controller is breached, an analyst has to mount a multi‑gigabyte memory dump, run Volatility, scan it with YARA, and then manually correlate hundreds of processes and detections — all while the attacker is busy exfiltrating NTDS.dit, the crown jewels of Active Directory.

I wanted to find out: could an AI agent run that entire investigation — and deliver an incident verdict — in seconds, without ever leaving Splunk? Not a chatbot bolted on the side, but an agent that lives inside the search platform, queries it through real tools, and respects its security model.

🔎 What it does

Find Evil turns Splunk into a forensic database that an AI agent can investigate in natural language. Starting from a real memory image (base-dc-memory.img, a Windows Server 2016 DC from the public SRL‑2018 scenario), it:

1. Extracts artifacts with YARA‑X (15 custom APT rules) and Volatility3 (124 processes / 122 distinct PIDs).
2. Ingests them into a forensics index via the official splunk‑sdk‑python streaming API, with a CIM‑aligned add‑on.
3. Exposes 5 purpose‑built forensic tools through the official Splunk MCP Server.
4. Investigates with the official splunklib.ai Agentic SDK: the agent auto‑discovers those tools, reasons with Claude under Splunk RBAC, and — through the ai_triage tool — even has the LLM reason inside SPL via the AI Toolkit's | ai command.
5. Renders the result as A2UI, mapped to native @splunk/react-ui controls (verdict badge, KPI cards, severity bar, MITRE table with chips, remediation list) across all four dashboards.
6. Automates a SOC loop: a scheduled alert detects → triages with | ai → writes a notable incident.

On the demo image, the verdict is COMPROMISED — 4 critical / 7 high detections — reconstructing the full AD credential‑exfiltration kill‑chain (T1003.003 ntdsutil, T1003.001 mimikatz, T1074.001 staging, T1021.002 PsExec, T1047 WMI…).

🛠️ How I built it

• Forensic extraction: YARA‑X (Python bindings + 15 scenario‑tuned rules) and Volatility3 → JSON artifacts.
• Data plane: a dedicated forensics index fed by splunk-sdk-python (index.attached_socket), plus a versioned TA for the index and CIM‑aligned sourcetype props.conf.
• Control plane: the official Splunk MCP Server, where I registered 5 forensics_* tools as safe, frozen SPL templates via /services/mcp_tools.
• Agent: real splunklib.ai agents (SDK 3.x) running as app scripts — tool auto‑discovery, Claude reasoning, and Pydantic structured output.
• UI: an enriched A2UI v0.9 catalog + one reusable React renderer that drives every dashboard from JSONL snapshots — produced either by the LLM agent or by a fast deterministic generator.
• Delivery: a Terraform module (splunk/splunk provider) that packages the apps into .spl and installs them + the index — validated end‑to‑end and idempotent.

🧗 Challenges I ran into

• Getting the AI Toolkit's | ai command to actually run on local Splunk Enterprise (macOS Apple Silicon): the right Python‑for‑Scientific‑Computing build, removing macOS quarantine on the embedded Python (Gatekeeper was SIGKILL‑ing it), granting apply_ai_commander_command via the mltk_admin role, and wiring the Anthropic connection.
• Designing an A2UI catalog rich enough to produce dense, useful Splunk‑native UI instead of generic cards or raw Markdown.
• Loading the React bundle from a dashboard's script= under Splunk's RequireJS — a UMD dependency kept registering an anonymous define() and breaking the dashboard until I shadowed define/module/exports in the bundle.
• Keeping the agent strictly inside Splunk's RBAC (no sidecar that bypasses auth) while still getting full tool auto‑discovery.
• Small but sharp data issues: 2018‑era timestamps rejected by MAX_DAYS_AGO, search‑time JSON re‑extraction causing duplicate fields, and Anthropic strict mode rejecting the MCP row_limiter's integer bounds.

📚 What I learned

The combination of the MCP Server, the Agentic SDK (splunklib.ai), and the AI Toolkit (| ai) genuinely enables a production‑shaped agentic loop inside Splunk: the agent discovers purpose‑built tools, the model can reason either from the SDK or natively in a search, and RBAC is enforced throughout. A2UI turned out to be a clean bridge from agent reasoning to polished Splunk UI — no custom visualizations from scratch. And the Terraform delivery proved that even an AI + forensics app can be treated as plain infrastructure‑as‑code.

🚀 What's next

Run the | ai path behind a local open‑source security model (e.g. Foundation‑Sec‑8B) for a no‑egress / air‑gapped posture, extend the tools to disk forensics (E01), and make the A2UI catalog reusable across other Splunk AI use cases.

Built With

• a2ui
• anthropic
• claude
• esbuild
• javascript
• model-context-protocol
• node.js
• pydantic
• python
• react
• spl
• splunk
• splunk-ai-toolkit
• splunk-mcp-server
• splunk-react-ui
• splunk-sdk-python
• splunklib.ai
• styled-components
• terraform
• volatility3
• yara-x