• 

  TrustOps for Splunk

TrustOps for Splunk

Inspiration

Security operations teams are increasingly asked to move faster, investigate more alerts, and rely on AI-assisted recommendations. But speed alone is not enough. In high-stakes security environments, analysts still need to understand the evidence, challenge recommendations, and make accountable decisions.

TrustOps was inspired by a simple question:

How do we measure and improve the human decision in an AI-augmented SOC?

Splunk already gives analysts powerful visibility into security telemetry. Splunk AI Assistant and agentic workflows can accelerate investigation. TrustOps builds on that foundation by adding a human-in-the-loop decision layer that captures how analysts evaluate AI recommendations, how much they trust them, whether they challenge them, and how their final decisions are made.

The guiding idea behind the project is:

Splunk investigates the alert. TrustOps measures and improves the human decision.

What It Does

TrustOps for Splunk is a human-in-the-loop agentic security triage assistant.

It helps analysts investigate suspicious alerts using:

• Splunk security data
• Splunk AI Assistant
• MCP-enabled workflows
• A sequential agentic investigation orchestrator
• MITRE ATT&CK mapping
• Evidence review and trust calibration
• Analyst decision telemetry

The demo scenario focuses on alert TO-VPN-2026-514, where user jsmith experiences seven failed VPN/SAML authentication attempts followed by a successful VPN login from Romania.

TrustOps retrieves the related Splunk events, summarizes the evidence, runs a tool-backed agentic investigation, maps the activity to MITRE ATT&CK, and guides the analyst through a structured decision process.

For this alert, TrustOps maps the activity to:

• Credential Access — Brute Force — T1110
• Initial Access — Valid Accounts — T1078

The analyst must then review evidence, challenge the AI recommendation, document supporting and contradicting evidence, and submit a final decision. That decision is logged back into Splunk with confidence, trust, evidence reviewed, time-to-decision, AI recommendation status, and automation-bias risk.

How We Built It

TrustOps was built as a full-stack Splunk application prototype.

The core architecture includes:

• Splunk Enterprise for indexing and searching authentication and decision telemetry
• FastAPI backend for alert APIs, Splunk search integration, AI support, and decision logging
• React / Vite frontend for the analyst console
• Splunk AI Assistant integration for SPL explanation and generation
• Optional Splunk MCP integration for agentic workflows
• MITRE ATT&CK enrichment using local mappings and optional mitreattack-python
• Agent-run telemetry logged back into Splunk

The backend includes a sequential agent orchestrator with specialized SOC agent roles:

1. Evidence Agent — queries Splunk and summarizes real alert telemetry
2. Triage Agent — classifies severity using Splunk-grounded evidence
3. SPL Agent — generates follow-up searches and uses Splunk AI Assistant where available
4. MITRE ATT&CK Mapping Agent — maps evidence to ATT&CK tactics and techniques
5. Contradictory Evidence Agent — identifies benign explanations and evidence gaps
6. SOP Agent — maps the case to response actions
7. Trust Calibration Agent — provides human-AI oversight guidance

These are not uncontrolled autonomous bots. They are specialized, tool-backed workflow roles that execute in sequence and return a transparent execution trace showing tools used, evidence, recommendations, and timing.

The frontend presents this workflow in an analyst-friendly dashboard with an alert queue, investigation panel, agent execution trace, MITRE mapping, SPL explanation, evidence review checklist, challenge-the-AI fields, and decision submission.

What We Learned

This project reinforced that AI in security operations should not only be evaluated by whether it produces a recommendation. It should also be evaluated by how well it supports human judgment.

We learned that a strong SOC assistant needs three layers:

1. Investigation support — helping the analyst understand what happened
2. Decision support — helping the analyst evaluate severity and next steps
3. Decision telemetry — measuring how the human and AI worked together

The most important insight was that trust is not a simple “accept” or “reject” decision. Trust has to be calibrated. Analysts need to see evidence, review alternatives, consider benign explanations, and understand the limits of AI-generated recommendations.

TrustOps captures that process through structured telemetry. Over time, this could help SOC leaders understand patterns such as overreliance, excessive overrides, low-confidence escalation, or analyst training needs.

Challenges We Faced

One of the biggest challenges was balancing agentic AI with transparency. It would have been easy to present a black-box chatbot, but that would not align with accountable security operations. Instead, TrustOps uses a structured agentic workflow where each step is visible, explainable, and auditable.

Another challenge was reliability. Hackathon demos need to work consistently, even when cloud-connected AI services or external data sources are unavailable. To address this, TrustOps uses deterministic local fallback logic while still integrating with Splunk AI Assistant and optional MITRE ATT&CK enrichment when available.

We also had to carefully design the human-AI decision workflow. The goal was not just to let the analyst click “accept.” The analyst needed to review evidence, challenge the AI recommendation, document reasoning, and submit a decision that could be measured later.

Finally, we had to keep the user interface clear. TrustOps includes a lot of information: Splunk events, agent traces, MITRE mappings, SPL explanations, trust calibration, and decision telemetry. The challenge was to organize that information into a workflow that supports fast triage without overwhelming the analyst.

What Makes It Different

TrustOps is not just another alert dashboard or chatbot.

It focuses on the relationship between AI-assisted investigation and human decision quality.

The project shows how Splunk can be used not only to investigate security telemetry, but also to observe the human-AI process used to investigate that telemetry.

TrustOps logs:

• Security event evidence
• Agent workflow telemetry
• Analyst decision telemetry
• Trust and confidence scores
• Evidence review behavior
• Automation-bias risk
• AI recommendation acceptance, modification, or rejection

This creates a measurable loop for improving SOC decision-making.

What’s Next

Future enhancements could include:

• Additional alert scenarios across cloud identity, endpoint, email, and OT security
• Expanded MITRE ATT&CK coverage and ATT&CK Navigator export
• Deeper Splunk MCP tool usage
• Analyst skill profiles and adaptive training recommendations
• More advanced automation-bias detection
• Integration with SOAR platforms for human-approved response actions
• Longitudinal analysis of analyst trust, confidence, and decision accuracy

Closing

TrustOps demonstrates a practical model for agentic security operations where Splunk data, AI-assisted investigation, MITRE ATT&CK mapping, and human accountability work together.

The goal is not to replace SOC analysts.

The goal is to help them make faster, better, and more accountable decisions.

Built With

• agent-run-telemetry
• analyst-decision-telemetry
• analyst-decisions
• and-agent-run-telemetry-dev-tools:-cursor
• architecture
• automation-bias-risk-scoring-apis:-fastapi-rest-endpoints
• bash
• css
• curl
• custom-css-backend:-fastapi
• deterministic-local-fallback-logic-for-reliable-demos-cybersecurity-frameworks:-mitre-att&ck
• fastapi
• github
• html
• javascript
• mcp
• mermaid
• npm
• optional-mitreattack-python-for-att&ck-enrichment-data-/-telemetry:-synthetic-vpn/saml-authentication-logs
• optional-splunk-mcp-server-ai-/-agentic-workflow:-sequential-agent-orchestrator
• pydantic
• python
• python-virtual-environment
• react
• shell-scripts-visualization:-react-dashboard-ui
• spl-frontend:-react
• splunk
• splunk-ai-assistant
• splunk-ai-assistant-api-where-available-storage-/-databases:-splunk-indexes-for-authentication-events
• splunk-ai-assistant-for-spl-explanation-and-generation
• splunk-dashboard-xml
• splunk-dashboards
• splunk-indexes
• splunk-management-api
• splunk-rest-api
• splunk-sdk-for-python
• splunk-search-apis
• trust-and-confidence-scoring
• uvicorn-security-/-data-platform:-splunk-enterprise-splunk-technologies:-splunk-search-processing-language
• vite