• 

• 

• 

• 

• 

• 

ThinkingSOC — Devpost Project Story

Inspiration

Security Operations Centers receive more alerts than analysts can investigate deeply. Even when an analyst completes a strong investigation, the useful knowledge often stays inside one ticket, one chat, or one person's memory. When the same detection fires again, another analyst repeats the same questions, writes similar SPL, collects the same evidence, and spends more time reaching the same decision.

We built ThinkingSOC to make every accepted investigation improve the next one. Our goal was not unrestricted security automation. We wanted a practical system that combines collaborating AI Agents with deterministic validation, live Splunk evidence, and human approval. The system should reduce repetitive work while keeping analysts in control of every important security decision.

What it does

ThinkingSOC receives Splunk alerts and routes them through security or observability analysis workflows. Specialized Agents collect context, inspect inventory and threat information, propose findings, and produce a structured result that is reviewed by a Judge stage. The investigation, evidence, Agent outputs, timing, and analyst actions are stored for later review.

When an analyst acknowledges an investigation, ThinkingSOC can convert it into a reusable Runbook. The configured LLM produces a small set of reusable investigation intents rather than trusted executable queries. The backend then generates fresh, alert-specific, read-only SPL, validates the syntax, runs it against Splunk, and records the returned evidence. Splunk MCP is preferred when available, with authenticated Splunk REST as the fallback.

A Runbook cannot approve itself. ThinkingSOC derives its status from parser and evidence results, then requires a separate human approval. Approved Runbooks can only be reused on another stored alert with the exact same Alert Name and a different alert identifier. Every reuse regenerates and validates fresh SPL for the target alert.

Analysts can:

• view all Runbooks grouped by exact Alert Name;
• inspect every immutable revision and its evidence state;
• search, filter, select and sort Alert Names;
• view, edit, import and export safe intent-only Runbook JSON;
• run read-only Shadow Evaluations before approval;
• review a visual Runbook execution graph;
• inspect Safe Response Previews that never execute containment;
• follow the complete Autopilot Agent trace; and
• ask Runbook questions or request execution for a specific alert through natural-language Chat.

How we built it

The backend is built with FastAPI, Python and Pydantic. PostgreSQL stores analysis results, Runbook revisions, approvals, replays, Agent traces and Chat history as auditable records. Qdrant supports Runbook-aware retrieval for Chat, while Neo4j supports relationship and correlation views. The analyst interface is built with Next.js, React and the project's shared dark visual system.

Splunk alerts enter through a webhook. Investigation and Runbook queries use the existing read-only SPL sanitizer, parser and execution pipeline. The configured LLM is accessed through LiteLLM, allowing the deployment to select a supported provider without changing the safety contract.

Agent collaboration

Runbook Autopilot uses five specialized Agents working together inside a bounded backend workflow:

1. Supervisor plans the workflow, delegates tasks and produces the final recommendation.
2. Evidence Scout loads the stored alert, checks the latest analyst action and gathers available evidence.
3. Runbook Engineer creates reusable investigation intent and requests fresh read-only SPL validation.
4. Policy Guard checks exact Alert Name matching, evidence status, approval requirements and safe-reuse rules.
5. Response Advisor prepares evidence-based next steps and non-executable Safe Response Previews.

The Agents do not receive unrestricted shell access. They use a defined set of storage, Runbook, LLM and read-only Splunk tools. Every Agent start, handoff, tool call, tool result, retry, duration and failure is recorded in an append-only trace. Autopilot can prepare and verify work, but it cannot acknowledge alerts, approve Runbooks, execute containment or bypass the human gate.

We also built an installation workflow, demo PostgreSQL bundle, JSON fallback, live health checks, Runbook logging, automatic SPL refinement, typed SDK methods, frontend and backend tests, and production smoke tests. The demo data includes a complete same-name/different-alert Runbook journey so reviewers can inspect the workflow immediately after installation.

Challenges we ran into

One major challenge was keeping AI output useful without treating it as trusted execution. LLMs can produce good investigation ideas while still returning invalid SPL or overly specific incident details. We solved this by separating reusable intent from executable SPL, enforcing read-only rules, validating every query, and adding a bounded repair-and-retry path for syntax failures.

Another challenge was representing asynchronous work honestly in the UI. A Runbook may have an older incomplete revision while a new build is still running. We added explicit in-progress and verification-incomplete states so active work is not incorrectly shown as a red failure.

We also had to handle alerts with the same name but different identifiers correctly. The Alert Name controls compatibility, while the unique alert identifier keeps records, runs and evidence isolated. Duplicate webhook identifiers are avoided automatically in the test tooling.

Finally, real environments are unreliable. MCP may be unavailable, Splunk may be offline, free LLM endpoints may respond slowly, and installation environments vary. We added REST fallback, retries, timeouts, lifecycle logs, connection diagnostics, deterministic demo data and end-to-end smoke tests instead of hiding these failures.

Accomplishments that we're proud of

We are proud that ThinkingSOC is an end-to-end operational workflow rather than a standalone AI demo. It connects ingestion, multi-Agent analysis, human acknowledgment, Runbook compilation, Splunk verification, approval, safe reuse, evaluation and Chat in one product experience.

We are especially proud that:

• AI-generated content cannot set its own verification or approval status;
• all executable investigation queries are fresh, read-only and parser-validated;
• Agent collaboration and tool use are visible instead of hidden behind one answer;
• every approval, edit, replay and target run is auditable;
• Safe Response Preview has no command or execution field;
• MCP failure does not stop the workflow when authenticated Splunk REST is available;
• the installer loads a complete judge-ready demo without deleting existing scenarios; and
• the value model separates measured evidence from assumptions.

In an illustrative six-analyst SOC scenario, reducing 30 eligible repeat investigations per business day from 25 minutes to 5 minutes returns about 2,600 analyst hours per year. That represents roughly 1.25 FTE of capacity and approximately $223,000 in annual gross capacity value using documented U.S. labor assumptions. Larger SOC programs can potentially save hundreds of thousands or millions of dollars, but ThinkingSOC keeps these projections separate from measured customer results.

What we learned

We learned that reliable AI for security is not created by using a larger prompt. It comes from combining specialized Agents, narrow tools, deterministic policy, visible evidence and human authority.

We also learned that failures are valuable product information. Parser errors, missing evidence, unavailable integrations and incomplete verification should remain visible and auditable. Clear failure states build more trust than an opaque confidence score.

Most importantly, reusable investigation intent is more durable than storing one generated query. Data, alert fields and infrastructure change, so the safe approach is to preserve the approved procedure and regenerate validated SPL for each new alert.

What's next for ThinkingSOC

Next, we want to evaluate approved Runbooks against larger labeled historical datasets and report precision, evidence coverage, drift and analyst time saved over time. We also plan to add scheduled revalidation so owners can see when a previously useful Runbook no longer matches current Splunk data.

We want to expand the Agent tool catalog with additional security platforms while preserving the same least-privilege and human-approval rules. A tiered memory layer could help Agents reuse approved organizational knowledge while keeping temporary cache, durable Runbook memory and sensitive evidence clearly separated.

Longer term, we plan to support policy-based approval roles, team review workflows, more response-system previews, and carefully evaluated semantic Runbook matching. Semantic reuse will only be enabled after enough approved evidence exists to measure its safety against the current exact-name policy.

Built With

• docker
• postgresql
• python
• typescript