SAO-CP (Splunk Agentic Ops for Critical Platforms)

Enhanced security for health data: AI analyzes, humans decide

Comment

Inspiration

In critical environments like university hospitals (CHU), medical record platforms are prime targets for cyberattacks. Security teams face a difficult dilemma: manual log analysis is too slow to react to modern threats, but giving full autonomy to an AI to block threats is too risky. A false positive from an autonomous AI could accidentally lock out a doctor from a life-saving patient file during an emergency. SAO-CP was born from this exact need: creating a security model where the speed of AI meets the wisdom of human judgment.

What it does

SAO-CP is a security operations platform that protects a healthcare document system by integrating an AI agent under strict human supervision. It operates through a clear 5-step workflow:

  1. Generation & Capture: Medical staff interact with the patient record platform. Every action generates logs that are streamed in real-time to Splunk.
  2. Agentic Analysis: An AI agent, connected to Splunk via the MCP protocol, continuously monitors these logs. If it detects an anomaly (e.g., unusual data exfiltration), it analyzes the context.
  3. Action Proposal: Instead of acting blindly, the agent formulates a remediation recommendation (e.g., "Block User IP" or "Revoke Session").
  4. Human Supervision: The alert and the agent's proposed action appear on a dedicated SOC Dashboard for security analysts. The analyst sees exactly why the AI wants to act.
  5. Validation & Execution: The human always has the final say. With a single click, the analyst gives the green light or cancels the action. The agent only executes the remediation task after this explicit validation.

How we built it

We are building a clean, highly structured MVP based on a 4-layer architecture:

  • Target Application: A web application built with Next.js that simulates a medical document platform. It emits event logs in JSON format.
  • Splunk Enterprise: The core engine. It ingests the target app's logs, stores them, and allows the AI to query them using SPL.
  • AI Agent & Splunk MCP: Developed with LangChain in Python, the agent leverages the Splunk MCP Server to natively interact with Splunk, search for anomalies, and formulate context-aware suggestions. Agent states and decisions are persisted in a PostgreSQL database.
  • SOC Dashboard & Backend: A Next.js frontend coupled with a FastAPI backend. FastAPI orchestrates the communication between the agent, the database, and the frontend via WebSockets to display alerts in real-time and capture the analyst's validation.

Challenges we ran into

  • Integrating the Official Splunk MCP Server: Initially, we struggled to find the correct way to deploy and connect to the official Splunk MCP Server. There was confusion between the MCP TA (for monitoring) and the actual MCP Server app. We had to carefully read the Splunkbase documentation to understand how to install it as an app inside Splunk Enterprise, generate an encrypted token, and expose the /services/mcp endpoint on port 8089.

  • Real-time Human-in-the-Loop Architecture: Designing the communication flow between the AI Agent and the SOC Dashboard was tricky. We didn't want the Agent to block its execution while waiting for a human to click "Approve", nor did we want it to spam Splunk by re-sending the same alert. We solved this by separating the Splunk polling (every 15s) from the approval checking (every 2s via FastAPI) and tracking pending_alerts in the agent's memory.

  • LLM JSON Output Consistency: Getting the LLM (Gemma) to consistently return strict JSON without markdown backticks or conversational text was challenging. We implemented a robust parsing function that cleans markdown formatting and extracts the JSON using regex as a fallback.

  • Local SSL/TLS Issues: Since Splunk Enterprise uses self-signed certificates by default, our Next.js app and Python Agent were throwing SSL errors when trying to send logs (HEC) or query the MCP. We had to implement custom SSL context patching in Python and configure the apps to bypass local certificate verification for the hackathon MVP.

Accomplishments that we're proud of

  • A Complete, Working Agentic Loop: We didn't just build a chatbot; we built a true "Agentic Ops" loop. The AI autonomously detects a threat, analyzes the context, proposes a specific remediation, waits for human validation, and executes the action. The human always has the final say.

  • Context-Aware AI Reasoning: We are incredibly proud of the AI's analytical capability. During our tests, when it saw an unauthorized access attempt from the IP 127.0.0.1, it didn't blindly propose "Block IP" (which would have taken down the server). Instead, it recognized it was a localhost address and proposed to disable the user account instead. That is next-level SOC analysis!

  • Official Splunk MCP Integration: We successfully integrated the official Splunk MCP Server, using the splunk_run_query tool to let our LangGraph agent natively query Splunk data via SPL. This makes the project highly relevant to the Splunk ecosystem.

  • Seamless Real-Time Dashboard: The WebSocket integration between FastAPI and the Next.js SOC Dashboard is fluid. The moment the agent detects a threat, it pops up on the screen instantly, making the demo feel like a real Security Operations Center.

What we learned

  • The Importance of Human-in-the-Loop in Critical Sectors: In healthcare (or any critical infrastructure), AI cannot be fully autonomous. A false positive could lock doctors out of life-saving patient records. We learned how to design UX and backend architectures that empower AI to assist, but keep humans accountable.

  • Splunk Architecture: We deepened our understanding of Splunk's architecture, specifically the difference between the Web UI port (8000), the HTTP Event Collector for ingestion (8088), and the Management/REST API port (8089).

What's next for SAO-CP (Splunk Agentic Ops for Critical Platforms)

  • Actual Remediation Execution: Currently, the "execution" step updates the database status. The next step is to integrate directly with the hospital's Identity Provider (e.g., Microsoft Entra ID / Active Directory) via API to truly disable user accounts or block IPs at the firewall level upon approval.

  • Multi-Agent System: Expand from a single agent to a team of specialized agents (e.g., a Network Agent, a Malware Analysis Agent, a Compliance Agent) orchestrated by a supervisor agent using LangGraph.

  • Integration with Splunk AI Assistant: Connect the SAO-CP agent to the Splunk AI Assistant via MCP to use tools like generate_spl, allowing the agent to write complex SPL queries on the fly based on natural language descriptions of emerging threats.

  • Broader Sector Application: Adapt the platform beyond healthcare to other critical sectors like Finance (detecting insider trading) or Energy (detecting unauthorized access to SCADA systems).

Built With

Share this project:

Updates