Enterprise Council AI

Splunk Live Dashboard
Architecture Diagram
Dashboard Layout
Authentication Gateway (RBAC)
Landing Page
Relationship Nodes (Scaled Twin Topology)
Focused view of Privilige Escalation (Filtered Graph)

Tagline

Multi-agent executive debate and decision intelligence powered by Splunk AI, Digital Twin graphs, the Model Context Protocol (MCP) and a Developer Python SDK.

Inspiration

When a critical security alert fires at 2:00 AM, the immediate instinct of a security analyst is to block the user account. But if that user is a Lead DevOps engineer currently pushing a hotfix to a payment gateway, blocking them immediately halts business transactions, costing the company thousands of dollars per minute.

Security wants containment. Infrastructure wants uptime. Compliance wants to preserve logs. Business wants revenue. Usually, this results in a high-pressure phone call between tired managers trying to balance competing risks with incomplete information.

I built Enterprise Council AI to automate this decision process entirely—letting domain-specific AI agents representing each business priority debate the risks, query live Splunk data for evidence and recommend balanced containment strategies in under five minutes, at any hour, for every incident.

Why it belongs to the Platform & Developer Experience Track

Enterprise Council AI was submitted under the Security track, but the Platform & Developer Experience angle is where it makes the strongest case for what Splunk's ecosystem can achieve.

Most enterprise security teams already use Splunk, but they still can't act on it fast enough. The bottleneck isn't data—it's decision-making. SPL is complex, incident context is siloed across teams and a single analyst can't simultaneously evaluate security risk, infrastructure blast radius, compliance obligations and business impact in minutes. Our platform solves this not just as a security tool, but as a developer-accessible intelligence layer on top of Splunk that anyone can consume, automating workflows through an agentic pipeline.

Specifically, here's how we address the Platform & Developer Experience track:

Splunk MCP Server as a True Developer Platform: Rather than building another static Splunk dashboard, I treated the MCP Server as an application development platform. I implemented 30 standard-compliant Splunk MCP tools (including splunk_run_query, saia_generate_spl and splunk_run_saved_search) exposing Splunk's full data indexes and AI capabilities as callable tools inside an agentic pipeline. This is a pattern any developer can follow to build intelligent applications on Splunk without managing raw REST calls or writing complex SPL from scratch.
FastAPI Developer SDK — One API Call to Multi-Agent Intelligence: The entire seven-stage pipeline—Digital Twin construction, agent council debate, impact simulation, consensus decision—is exposed as a single REST endpoint: POST /api/v1/incident/analyze. Any external SIEM, SOAR, or custom application can integrate Splunk-powered multi-agent incident response. This is the abstraction Splunk developers need—not another query interface, but a decision intelligence API that sits on top of their existing Splunk deployment with zero migration cost.
AI Assistant — SPL Democratization for Every Developer: Our AI Assistant tab lets any developer or analyst generate, explain and execute Splunk queries in plain English with auto-suggested investigation trails generated dynamically per active incident. A developer who has never written SPL can investigate a live incident in seconds. This directly lowers the barrier to building on Splunk data and addresses a real friction point in the developer experience.
Splunk App Packaging Standards: The application is packaged as a standard-compliant Splunk App directory structure containing default/app.conf and metadata/default.meta. This allows it to pass Splunk's App Inspect validation guidelines and deploy directly onto active Search Heads.

Splunk MCP Server Tools & Hosted AI Models

To connect our agents' reasoning directly to active enterprise data, we developed and exposed a robust array of Splunk integration tools at runtime:

1. Model Context Protocol (MCP) Tools

splunk_run_query: Executes custom SPL queries synchronously against active indexes.
splunk_get_info: Retrieves server configuration and runtime operational status.
splunk_get_indexes: Lists configured indexes (security, infrastructure, business, compliance).
splunk_get_index_info: Fetches storage sizing and total event count per index.
splunk_get_metadata: Inspects metadata (hosts, sources, sourcetypes) across logs.
splunk_get_user_list / splunk_get_user_info: Inspects active sessions and account permissions.
splunk_get_kv_store_collections: Accesses collections in Splunk's App KV store.
splunk_get_knowledge_objects: Lists pre-configured saved searches, lookups and dashboards.
splunk_run_saved_search: Dispatches configured Splunk alerts.
query_security_events / search_security_events: Fetches logs from the security index.
query_infrastructure_events: Queries server metrics and status from the infrastructure index.
query_business_context: Accesses department structures and permission levels.
query_compliance_events: Evaluates regulatory governance mappings.
get_user_activity: Constructs a unified user timeline of events across all indexes.
get_user_context: Queries the Digital Twin to get user criticality and anomalies.
get_related_alerts / get_system_dependencies: Performs multi-hop queries for blast radius maps.
saia_generate_spl: Formulates complete SPL queries from natural language questions.
saia_explain_spl: Explains complex SPL query syntax for junior developers.
saia_optimize_spl: Rewrites queries to improve execution performance.

2. Splunk Hosted AI Models (Runtime Integration)

Splunk Foundation-Sec (foundation-sec-1.1-8b-instruct): Called at runtime using the | ai pipeline command to zero-shot classify security descriptions, identify attack vectors and map alert behavior to active MITRE ATT&CK technique IDs (e.g., T1548 or T1048).
Cisco Deep Time Series (CiscoDeepTimeSeries): Accessed via the | fit command to perform infrastructure metrics forecasting and anomaly detection (e.g., predicting capacity load trajectories over a 24-hour horizon).

What It Does

Enterprise Council AI is a seven-stage decision intelligence console. It ingests raw Splunk security logs across four indexes (security, infrastructure, business, compliance) and builds a topological Security Digital Twin of the organization using NetworkX—linking users, systems, credentials, databases, departments and compliance mandates as a live relationship graph.

When an incident fires, the platform classifies it by type and severity, then dynamically assembles a council of four specialized AI agents:

Security Agent: Powered by Splunk's foundation-sec-1.1-8b-instruct hosted model, assesses threat severity and maps tactics to MITRE ATT&CK framework techniques.
Infrastructure Agent: Powered by the Cisco Deep Time Series model, forecasts peak load and calculates blast radius across downstream systems.
Compliance Agent: Evaluates regulatory obligations (GDPR, HIPAA, PCI-DSS, SOC2) and evidence preservation requirements.
Business Agent: Assesses revenue impact, deployment criticality and productivity loss of each containment option.

These agents run a structured three-round adversarial debate:

Opening Statements: Initial assessments based on raw alert data.
Cross-Examination: Agents challenge each other's positions with live SPL queries generated dynamically via saia_generate_spl and run via the MCP toolset.
Final Stances: Refined opinions incorporating simulated risk ratings.

Before reaching consensus, the Impact Simulation Engine traverses the Digital Twin graph to calculate the exact security, business, compliance and infrastructure risk percentages across three containment playbooks: Block User, Temporary Restriction and Monitor Only.

The Council Agent then aggregates all opinions, simulation results and weighted votes into a single recommended action with a confidence score, registers a cryptographic SHA-256 audit hash for compliance logging and allows operators to dispatch containment responses directly from the Operational Command Center dashboard.

How I Built It

I built the core pipeline in Python across seven modular stages. I chose NetworkX to model the Digital Twin topology because querying multi-hop node dependencies—such as finding which databases a compromised credential can access via peer devices—is significantly faster in a graph structure than in relational tables.

To bridge AI reasoning and live Splunk data, I leveraged the Model Context Protocol (MCP). I implemented all 14 native Splunk REST wrapper tools, plus 16 custom security and AI-powered (exposing a total of 30 tools to the agentic registry), allowing agents to write and execute their own SPL queries mid-debate to pull real-time evidence. This makes the agents resilient to schema changes—they adapt their queries dynamically rather than relying on hardcoded parsers.

To achieve balanced recommendations, the Council Agent runs a weighted aggregation of agent opinions and simulation outcomes. The consensus score for each playbook option is calculated as:

$$\text{Consensus Score} = (1 - \lambda) \frac{\sum_{i=1}^{N} w_i \cdot (100 - R_i)}{N} + \lambda \cdot (100 - I_{\text{sim}})$$

Where:

w_i is the confidence score of agent i (based on their domain relevance to the incident type).
R_i is the agent's risk rating (0-100%).
I_sim is the simulated impact of the containment playbook on the organization (0-100%).
λ (lambda) is a dynamic balancing coefficient (typically 0.4) that shifts weight toward simulation data when agent opinions diverge significantly.
The score indicates the overall viability/safety of the playbook (higher is better).

Production-Grade Security & Performance Hardening

To make the application enterprise-ready, we implemented several advanced security mechanisms:

SPL Injection Prevention: Developed regex input filters to sanitize user, event and severity fields, blocking malicious command injections at the API gateway.
Destructive Command Blocking: Restrained query execution by parsing and rejecting destructive SPL commands (delete, outputlookup, collect, run, outputcsv).
HMAC Integrity Hashing: Anchored audit logs with SHA-256 integrity hashes and simulated HSM HMAC signatures to prevent audit trail tampering.
SSL Verification & Auth Restrictions: Enforced SSL certificate verification against system CA bundles and blocked insecure basic credential fallback when BLOCK_BASIC_AUTH is toggled.
Parallel Query Engine: Implemented ThreadPoolExecutor parallel threads for Stage 4 MCP context queries and debate rebuttals, reducing consensus latency from 12+ seconds to under 3 seconds.

The developer REST API is built with FastAPI and SlowAPI rate limiting. I chose Streamlit for the Operational Command Center because it allowed rapid assembly of NetworkX topology plots, Plotly risk charts, and debate transcript timelines—with custom CSS injections to build a dark military SOC aesthetic. The landing page is a React application with Framer Motion animations deployed to Firebase Hosting.

Challenges I Ran Into

Streamlit Reload NameErrors: Restructuring the Streamlit dashboard layout was a major hurdle. I wanted to remove the default sidebar entirely and build a horizontal command center layout. Because Streamlit compiles top-to-bottom on every reload, checking the execution button state before it was declared in the layout columns threw repeated global NameError exceptions. I resolved this by refactoring the debate workflow scope to execute cleanly within the column rendering context.
Matplotlib & FPDF2 Thread-Locking: I hit a thread-locking conflict between Matplotlib (which renders the Digital Twin topology graph) and FPDF2 (which compiles the downloadable PDF incident report). Both rendering engines kept blocking each other during application reloads. I resolved this by enforcing a non-interactive Agg backend globally before any rendering calls.
LLM Fallback Decoupling: When the Splunk hosted model was unavailable, the fallback path attempted to access the Gemini .models attribute on the Splunk client object, throwing an AttributeError. I resolved this by decoupling both clients into separate global variables with independent error boundaries.
Offline CSV Fallback Connector: Built a robust offline CSV fallback connector so the application functions fully for judges even when a live local Splunk Enterprise management port is unavailable—with 30-50 realistic synthetic events per index matching real enterprise telemetry patterns.

Accomplishments That I am Proud Of

Live Querying During Debates: Managed to get four distinct LLM agents with structurally competing priorities to adjust their positions based on live Splunk query results and simulated risk percentages mid-debate. Watching the Infrastructure Agent successfully veto the Security Agent's block recommendation because the Cisco Deep Time Series model predicted an 89% peak load event on a downstream database was the moment I knew the adversarial council design was working.
Highly Responsive Performance: The entire pipeline, from log ingestion through Digital Twin construction, three-round debate, impact simulation, consensus aggregation and PDF compliance report generation with SHA-256 audit hashing, completes in under three seconds.
Clean SDK Abstraction: Exposing a multi-agent Splunk-powered incident response pipeline as a single endpoint and a lightweight client library reduces integration to three lines of Python for any developer.

What I Learned

LLM Tool Agency: Exposing system queries as LLM tools via MCP is far more scalable than writing hardcoded log parsing regexes. Giving agents the agency to write their own SPL queries makes the system resilient to telemetry schema changes.
Design by Conflicting Mandates: Multi-agent debate only works if the agents have structurally conflicting objectives by design. If agents share the same optimization goal, they converge immediately. Giving the Business Agent a mandate that directly conflicts with the Security Agent's mandate is what produces balanced decisions.
Streamlit Customization: Learned how to override default Streamlit stylesheets to build responsive dashboard cards that look and feel like a dedicated security operations center rather than a standard data science notebook.

What's Next for Enterprise Council AI

Active Containment Enforcement: Expand the SOAR capabilities from simulated mock dispatches to active directory domain policy enforcement—real Okta suspensions, real ServiceNow SIR tickets and real Cortex XSOAR playbook triggers.
Multi-LLM Benchmarking: Benchmark consensus quality by running the debate loops across different model backends (Claude, Gemini and local Llama instances) to measure how model choice affects decision quality and cross-domain balance.
Lateral Movement GNNs: Train graph neural networks directly on the Digital Twin topology to predict lateral movement attack paths before they appear in security logs—turning the system from reactive incident response into proactive threat anticipation.