Splunk Sentinel

Splunk Sentinel turns raw alerts into AI kill-chain reports, analyst-approved containment, and measurable saved-search coverage improvements in the Splunk workflow.

Comment

Inspiration

SOC analysts do not investigate incidents in one place. A single serious alert can send them across Splunk searches, event timelines, MITRE ATT&CK mappings, threat context, containment decisions, audit requirements, and detection engineering follow-up.

That manual workflow is slow, repetitive, and easy to fragment. An analyst may reconstruct the attack path, but the containment plan might live somewhere else. Detection gaps may be noticed, but never turned into a deployed saved search. Confidence scores may be produced, but not explained. In real SOC environments, this gap between investigation, response, and future detection coverage is where time is lost.

We built Splunk Sentinel to close that loop.

Instead of stopping at “AI summary,” Splunk Sentinel turns a raw Splunk alert into an explainable investigation, a reconstructed MITRE kill chain, an analyst-approved containment plan, and measurable saved-search coverage improvement inside the Splunk workflow.

What it does

Splunk Sentinel is an agentic SOC investigation platform built around Splunk Enterprise.

A security trigger starts a multi-agent investigation pipeline. The system classifies the threat, queries Splunk, reconstructs the attack path, maps observed behavior to MITRE ATT&CK, enriches the finding with threat and technique context, and produces a polished investigation report.

The live dashboard shows the investigation as it happens:

  • agent pipeline status
  • forensic event feed
  • confidence progression
  • live kill-chain reconstruction
  • final dashboard/report reconciliation after completion

The final report includes:

  • attack classification and severity
  • evidence confidence
  • reconstructed attack path
  • confirmed vs inferred stage labels
  • MITRE technique mapping
  • confidence breakdown
  • performance SLO status
  • SHA-256 SPL audit-chain verification
  • PDF export

Splunk Sentinel also includes analyst-in-the-loop response. It generates a phased containment plan, lets the analyst refine actions in natural language through a copilot, and requires confirmation before execution. Containment actions and rollbacks are written back into Splunk for auditability.

The strongest workflow is the closed-loop detection coverage feature. After an investigation, Splunk Sentinel compares the mapped MITRE techniques against existing Splunk saved searches. It identifies detection gaps, generates safe SPL for uncovered techniques, deploys the SPL as a Splunk saved search, then re-runs coverage with a fresh Splunk saved-search scan. The before/after panel shows coverage before deployment, coverage after deployment, delta points, newly covered techniques, gaps closed, remaining gaps, and saved searches checked.

The result is not just an investigation report. It is a workflow that helps the SOC improve future detection coverage.

How we built it

The backend is a FastAPI application orchestrating a LangGraph multi-agent pipeline. The agents share typed investigation state and coordinate the flow from triage to reconstruction, enrichment, synthesis, reporting, containment, and detection coverage analysis.

The core agents include:

  • Triage Agent for classification, severity, and routing
  • Reconstruction Agent for bounded ReAct-style kill-chain reconstruction from Splunk telemetry
  • Threat Intelligence Agent for external indicator context
  • TTP Agent for MITRE ATT&CK mapping and enrichment
  • Synthesis Agent for final reasoning, confidence, and narrative generation
  • Report Agent for persistence, PDF/report output, and Splunk write-back

Splunk integration is done through the Splunk Python SDK and SPL. The backend executes scoped searches, writes investigation summaries to sentinel_findings, writes containment actions to sentinel_actions, creates saved searches for generated detections, and exposes the workflow through a native Splunk app dashboard.

For durability, LangGraph checkpointing uses SQLite through AsyncSqliteSaver, with checkpoints stored at backend/checkpoints.db. This enables checkpoint status checks and resume support in the local demo environment.

The frontend is React 18 with Vite. It uses streaming responses for the live investigation dashboard and includes polished SOC-style views for the dashboard, report, containment workflow, detection gap analysis, history, and PDF export.

For persistence, the full demo workflow uses Supabase PostgreSQL with JSONB report storage. For retrieval and enrichment, the project uses Qdrant-backed RAG for MITRE and security context where configured. Prompt versioning uses Langfuse when configured, with fallback prompts available. LangSmith tracing is supported when configured.

The project includes GitHub Actions CI and 425 passing backend tests across guardrails, reconstruction, containment, detection gap analysis, confidence breakdown, API contracts, audit logging, parallel agents, schema validation, synthesis, and trigger categorization.

Challenges we ran into

The hardest challenge was making the AI output defensible for a security workflow. In cybersecurity, a confident sentence is not enough. Every claim needs to be tied back to evidence, SPL, telemetry, or a clearly labeled inference.

We had to be careful with wording throughout the product. Detection coverage improvement is described as saved-search coverage posture, not proof of real-world prevention. MLTK enrichment is shown honestly as MLTK Validated, MLTK Review, or NOT RUN depending on whether validation was available for that investigation. Confidence is explainable and factor-based instead of presented as a magic AI score.

Another challenge was streaming state. During a live investigation, the dashboard receives partial agent progress, while the final report is produced after synthesis. We fixed dashboard/report mismatches by reconciling the final kill chain, final confidence, and agent state when the investigation completes.

We also had to handle real Splunk integration issues. Saved-search deployment through the Splunk SDK can fail if the backend session expires, so we added a retry path with re-authentication. Generated SPL also needed safety controls, so the project includes SPL guardrails, index scope checks, and audit-chain logging.

Finally, reproducibility was a major focus. The system touches Splunk, FastAPI, React, Supabase, LangGraph checkpointing, optional prompt tooling, and optional RAG services. The README documents setup, environment variables, API endpoints, checkpointing, tests, and fallback behavior so the project can be reproduced after cloning.

Accomplishments that we're proud of

We are proud that Splunk Sentinel is a complete closed-loop SOC workflow, not just a chatbot or report generator.

The system can start from an alert, run an AI-assisted investigation, reconstruct the MITRE kill chain, produce an explainable report, propose containment actions, identify detection gaps, generate SPL, deploy a Splunk saved search, re-run coverage, and show measurable saved-search coverage improvement.

We are also proud of the proof artifacts inside Splunk:

  • generated saved searches appear in Splunk
  • investigation summaries are written to sentinel_findings
  • containment actions are written to sentinel_actions
  • SPL audit-chain verification is exposed in the report
  • the native Splunk dashboard shows recent investigations and security activity

On the engineering side, the project includes durable checkpointing, analyst-in-the-loop containment refinement, forced coverage refresh after deployment, dashboard/report state reconciliation, a polished report UI, and 425 passing backend tests with green CI.

What we learned

We learned that useful AI security tools need more than model output. They need evidence, guardrails, auditability, and human control.

We also learned that detection engineering should not stop at recommendations. The most valuable workflow is closing the loop: investigate the incident, find the gap, create the detection, deploy it, and measure whether saved-search coverage improved.

Streaming agent interfaces also require careful design. Users see the system think in real time, but they also expect the final dashboard and final report to agree. Final-state reconciliation became essential.

Most importantly, we learned that trust is built through transparency. Splunk Sentinel does not ask analysts to blindly accept an AI answer. It shows the SPL, the evidence, the confidence factors, the audit chain, the coverage delta, and the actions before execution.

What's next for Splunk Sentinel

Next, we would focus on production hardening and deeper Splunk packaging.

Planned improvements include:

  • multi-tenant deployment with role-based access control
  • deeper Splunk app packaging and Splunk Cloud readiness
  • broader MITRE ATT&CK coverage
  • expanded MLTK validation coverage
  • richer SOC playbook integrations
  • distributed checkpointing for multi-node deployments
  • more evaluation coverage for reconstruction and synthesis quality
  • deeper saved-search quality testing beyond coverage matching
  • enterprise deployment guides for SOC teams

The long-term vision is for Splunk Sentinel to become an agentic SOC teammate that investigates incidents, keeps analysts in control, and continuously improves detection coverage inside Splunk.

Built With

Share this project:

Updates