Splunk MCP Playbook Validation Lab

• 

Inspiration

As AI adoption grows, security teams may increasingly use AI to generate detection searches and response workflows. However, an AI-generated SOAR playbook should not be trusted simply because it appears correct.

A playbook may contain missing fields, unsafe dependencies, inconsistent compiled logic, or destructive actions that lack human approval. I built the Splunk MCP Playbook Validation Lab to close the gap between generating a security workflow and deploying it safely. The lab validates the playbook's structure, dependencies, compiled logic, and approval requirements before a workflow is promoted to production use.

What it does

The project turns real BOTS v3 security evidence into validated, approval-gated SOAR workflows.

It guides an analyst through five stages:

1. Investigate BOTS v3 evidence through Splunk MCP.
2. Generate a human-readable YAML SOAR playbook.
3. Compile it into a machine-readable workflow JSON.
4. Validate its structure, fields, dependencies, conditions, and safety controls.
5. Dry-run the response and export the complete evidence package.

The lab includes four BOTS v3 investigation scenarios:

• Suspicious authentication
• Web and phishing investigation
• Endpoint malware investigation
• Suspicious network communication and possible C2

The validator checks that:

• Required fields match the Splunk result schema
• Optional enrichment branches cannot block the main workflow
• YAML steps match compiled JSON nodes
• Dependencies reference valid workflow nodes
• Conditions and input mappings are preserved
• High-risk containment actions require human approval

The dry-run simulator performs threat-intelligence enrichment, calculates risk, evaluates decision gates, and simulates ticketing and reporting. Destructive actions such as blocking an IP, blocking a domain, isolating an endpoint, or revoking sessions remain blocked until human approval.

How I built it

The backend uses Python and FastAPI. The frontend uses React, TypeScript, and Vite.

Splunk MCP is the primary investigation path. It executes SPL against a local Splunk Enterprise instance containing the BOTS v3 dataset and returns structured evidence to the application.

The platform then:

• Extracts the returned field schema
• Builds incident context and training hypotheses
• Generates a deterministic YAML playbook
• Compiles YAML into workflow JSON
• Runs semantic and safety validation
• Simulates execution without performing destructive actions
• Exports YAML, compiled JSON, validation reports, Splunk evidence, and dry-run results

MITRE ATT&CK mappings provide investigation context while clearly remaining hypotheses rather than automatic incident confirmation.

VirusTotal is integrated in lookup-only mode for live URL, domain, IP, and file-hash reputation. Private and internal indicators are filtered locally and are never sent to VirusTotal. No files or URLs are uploaded for scanning.

Challenges

One major challenge was distinguishing available fields from reliable fields. BOTS v3 events have different sourcetypes and field coverage, so a field visible in some rows should not automatically become mandatory for every alert.

Another challenge was workflow dependency semantics. An optional enrichment step may be skipped when its artifact is absent, but it must not prevent required investigation, ticketing, or reporting steps from continuing.

Threat-intelligence enrichment required similar care. The first search result could contain an internal hostname while later rows contained valid public indicators. The system now safely selects an eligible public IOC from the result set while continuing to filter private indicators.

Finally, YAML validation alone was not enough. I added compiled JSON parity validation so the machine-executable workflow cannot silently diverge from the human-readable playbook.

Accomplishments

• Connected Splunk MCP to real BOTS v3 evidence
• Built deterministic and explainable SOAR playbook generation
• Added YAML and compiled workflow validation
• Added non-blocking optional enrichment branches
• Implemented approval gates for high-risk actions
• Integrated live, read-only VirusTotal reputation
• Added MITRE ATT&CK investigation context
• Created a safe dry-run simulator and exportable evidence package
• Added automated backend and frontend test coverage

What I learned

I learned that trustworthy security automation requires more than generating plausible YAML. A useful SOAR workflow needs schema awareness, dependency analysis, compiled-artifact validation, explicit approval boundaries, and evidence provenance.

I also learned that enrichment results and ATT&CK mappings should support analyst judgment, not replace it. Confidence, provider mode, evidence source, and simulation status must remain visible to the user.

What's next

Future improvements could include:

• Native Splunk SOAR playbook export and execution
• Additional threat-intelligence providers
• Live MITRE ATT&CK knowledge synchronization
• Analyst approval and audit history
• More BOTS v3 investigation scenarios
• LLM-assisted playbook drafting constrained by deterministic validation
• CI validation for playbooks before deployment

The long-term goal is to make AI-assisted SOAR development safer, explainable, and useful for both analyst training and controlled production adoption.

Built With

• fastapi
• jason
• mitreatt&ck
• python
• react
• splunkbotsv3
• splunkenterprise
• splunkmcp
• splunkspl
• typescript
• virustotalapi
• vite
• yaml